How errors arrive
A tool never throws. It returns an error result whose text names the failure and the next step. The status code in that text comes from the Pome control plane, so the same codes mean the same thing on every tool:
Quota trips are always
402, never 429: a 429 clears on its own, a 402
does not.
Orientation
get_started
Returns the coach playbook. Call this before anything else. Takes no inputs. Returns — the playbook as markdown: the coach/examinee split, how to use twin URLs, and the error semantics above. Errors — none. This is the one tool that resolves no team, so it answers on an unauthenticated connection. Exampleget_platform_context for identity and
quota.
get_platform_context
Confirms identity, team, plan tier, and quota headroom.
Returns —
{ team, me, usage, entitlements, note }. usage is the live
concurrency snapshot (how many sandboxes are open now); entitlements is the
two monthly meters, agent evals and MCP calls. entitlements is best-effort
and is omitted if the meter read fails, so an orientation call still answers
“who am I”.
Errors — 401, 403.
Example
run_task, whose eval_cost prices one specific
run against the allowance this tool reports.
list_twins
Returns the hosted digital-twin catalog. Takes no inputs. Returns —{ twins: [{ id, name, description, criteria_vocabulary }] }.
This is the same allowlist that decides whether a sandbox can be provisioned:
a twin absent from it is not mounted at all. Today the catalog is GitHub,
Slack, Gmail, Linear, and Stripe.
criteria_vocabulary is "declared" when list_checks describes checks for
the twin and save_task can bind a [code] criterion to it. It is
"none_declared" when the twin runs but declares no checks yet — that twin is
still listed, and a [code] criterion naming it would not be graded.
Errors — none. The catalog is static and needs no team resolution.
Example
list_checks for what a given twin can grade.
Authoring
list_tasks
Returns the team’s task catalog, grading criteria included. Takes no inputs. Returns —{ tasks: [{ task_id, name, prompt, twins, criteria, discrimination }] }. An empty list is not an authorization error: the call is
already authenticated and team-scoped, so [] means the catalog is genuinely
empty. A row whose source no longer parses is still listed, with an error
field instead of the task view.
Errors — 401, 403.
Example
save_task to add one, run_task to
run one.
list_checks
Returns the closed set of typed[code] checks a twin declares.
Returns — for each check: its id, the English template it renders, what the
predicate actually compares, the substrate it reads, and every parameter with a
valid example. A twin with no declared checks returns
checks: [] and a note
saying so.
Pass a check id to save_task’s criteria and the system writes
the criterion text, so it cannot fail to bind. Do not hand-write [code]
sentences: one that binds nothing scores unmatched, leaves the score’s
denominator, and does not fall back to the judge.
Errors — 422 when the twin id is not mounted.
Example
validate_task to confirm what your criteria
bind to.
validate_task
Validates task markdown. Creates nothing.
Returns —
{ valid, title, twins, timeout_seconds, discrimination, criteria_count, criteria_kinds: { code, model }, code_criteria: { bound, unbound, not_checked }, notes[], criteria[] }, or
{ valid: false, issue }.
Read code_criteria, not criteria_kinds. criteria_kinds counts markers;
code_criteria counts what will actually be graded. A [code] criterion whose
text binds to no declared check comes back bound_to: null, scores
unmatched, and has nothing to fall back to. not_checked means the declared
vocabulary was unreachable, so binding is unknown — not that it failed.
A null binding that also carries corrupted_instance_of is not an unknown
sentence. It is that named check’s wording with a slot value the check rejects,
so re-render the check rather than rewording the criterion.
A
## Seed State section must contain a fenced json code block. Prose is
rejected with a named diagnostic — there is no .seed.json sidecar on this
surface.422 on a malformed request. A malformed task is not an error:
it comes back as a valid: false result.
Example
save_task to persist, and
evaluate_criteria to dry-run.
save_task
Validates, then upserts a task by name.
Returns —
{ task_id, name, source_hash, discrimination, … }.
discrimination reports whether any model has ever been shown to fail this
task. status: "unmeasured" is an absent result, not a clean one. It never
blocks a save — editing a task invalidates its measurement, and this call is
how a task is edited — but the corpus gate does block a shipped exam task on
it. Measure with run_trials, then record the returned
fingerprint in a ## Discrimination section.
save_task refuses to persist a task whose [code] criterion binds no
declared check. Build criteria from list_checks and that cannot happen.
Errors — 422 when the markdown fails validation or criteria collides
with a ## Success Criteria section, 413 when the source is over the limit.
Example
verify_seed before you spend a run on it.
evaluate_criteria
Dry-runs the deterministic[code] criteria against the seed’s initial state,
without running the examinee.
Returns —
{ twins, has_seed_state, deterministic, model }. The control
plane boots the twin in-process and evaluates each [code] criterion against
the twin’s real initial state. [model] criteria are advisory — the narrator
reads them at run time and they never enter the score — and are returned here
as informational.
The verdict you get here is the verdict the real run will give: a criterion is
scored the same way whether it is evaluated at authoring time or at finalize.
One case is worth knowing about — a criterion that quotes a string the platform
redacts (an API key, and with team PII redaction on, an email) comes back
skipped with reason subject_redacted. Redacted values never reach the
graded state, so no predicate can find one; it is reported unscoreable here
instead of quietly passing at run time. Assert on something the final state can
actually carry.
Errors — 422 when neither task_id nor task_source is supplied, 404
on an unknown task_id.
Example
verify_seed, which asks the opposite question:
does the seed already satisfy the criteria?
verify_seed
Flags any deterministic criterion that already passes on the seed’s initial state — a broken seed.
Returns —
{ no_criteria_prepass, has_seed_state, already_passing, deterministic, model, notes, verdict }. will_be_excluded_from_score names
the criteria scoring will drop from the denominator for that reason, including
the negative ones a good seed is supposed to pre-satisfy.
A criterion satisfied before the examinee acts means the seed already meets the
success condition, so the exam cannot measure anything.
Errors — 422 when neither task_id nor task_source is supplied, 404
on an unknown task_id.
Example
run_task, once the seed is fair.
register_agent
Registers a Managed-Agents-cloud agent under test. Idempotent on the derived slug.
Returns —
{ agent, next }. Use agent.slug (or agent.id) as
run_task’s agent_slug / agent_id. next.local_repo.run is the CLI verb
to run if a local repo turns out to be present.
A slug that near-misses an existing one comes back as needs_confirmation
rather than silently creating a second agent.
Errors — 422 on a rejected field, 409 on the near-miss guard (returned
as needs_confirmation, not an error result).
Example
intake_clone_scope when you also need
the clone scope captured.
intake_clone_scope
Registers the agent and captures its clone scope, sorun_task can assemble
the examinee.
Returns —
{ agent }, including the per-twin twins.pome.sh façade URLs.
Supplying a clone scope mints a durable façade token bound to the agent,
returned once. Re-intake updates the scope and rotates the token.
Errors — 422 on a rejected field, 409 on the near-miss guard.
Example
run_task, which consumes the captured scope.
list_agents
Lists the team’s registered agents. Takes no inputs. Returns — for each agent: id, slug, display name, framework, narrator model (the model that writes each run’s summary and handoff — it holds no score authority), the twins it may exercise, and any stored clone scope. Errors —401, 403.
Example
run_task, which takes agent_slug from here.
Run lifecycle
run_task
Provisions a sandbox with the task’s twins, each isolated and seeded, and returns the assembledexaminee_launch spec. It does not launch the clone —
the coach does.
Prefer
agent_slug. A near-miss of an existing slug comes back
needs_confirmation rather than silently creating a second agent; retry with
confirm_new_agent: true if you meant a new one.
Returns — examinee_launch (transport, instructions, mcp_servers[] with
per-sandbox twin URLs https://twins.pome.sh/<twin>/s/<sid>/mcp,
mcp_permission_policy, rest_urls, env, network allowlist), plus
session_id, expires_at, agent_token, examinee_task, and
how_to_finalize. The clone carries zero Pome tools.
eval_cost prices the run against the team’s monthly agent-eval allowance.
Show eval_cost.headline to the user before you launch: nothing is charged
until finalize_run, and a sandbox they stop — or one that fails — costs them
nothing. This is the last moment the choice is still theirs.
A response carrying stale_criteria means the task’s saved [code] sentences
no longer bind to the twin’s current declared vocabulary. The run proceeds,
those criteria come back “not evaluated”, the agent is not scored on them,
and the run reports INCOMPLETE. Re-render them from list_checks and call
save_task again — it updates in place.
Errors — 402 when the concurrency quota is full, 404 on an unknown
task_id or agent_id, 422 on a rejected field, 413 on an oversized task.
Example
finalize_run to grade the run, and
run_trials for N trials in one call.
run_trials
Provisions N trials of one task sharing a group — the batch form ofrun_task.
Not idempotent.
Returns —
{ group_id, requested, provisioned, trials[], errors[], note },
where each trial carries the same fields run_task returns. eval_cost and,
when applicable, stale_criteria behave exactly as under run_task. Each
trial is finalized separately and each finalize burns one agent eval, so n
trials cost n evals.
All N sandboxes are provisioned up front and share the team’s concurrency
quota, so launch and finalize promptly to free the slots. A per-trial failure
is collected in errors[] and does not abort the trials that already
succeeded.
Errors — 402 stops further provisioning but still returns what came up.
Otherwise as run_task. If a call is interrupted, finalize or stop the
sandboxes it returned rather than blindly retrying.
Example
save_task, whose discrimination field this is
how you measure.
list_sandboxes
Lists the team’s recent sandboxes with state and expiry.
Returns — recent sandboxes newest-first, each with its id, state, and
expires_at.
Errors — 401, 403.
Example
get_sandbox for one sandbox in detail.
get_sandbox
Probes one sandbox. Side-effect-free — use it when twin URLs start answering 404.
Returns —
{ session, twin_activity_detected, idle_seconds, activity, verdict }.
Errors — 404 on an unknown id. An expired sandbox is not an error: it
comes back with a verdict saying nothing is recoverable from it.
Example
finalize_run, once you have confirmed the
examinee ran.
stop_sandbox
Stops a sandbox early and frees its concurrency slot without evaluating — the run is discarded, not graded. Unrecoverable.
Pome creates the run row at finalize, so every open sandbox’s run is ungraded
by construction. An unconfirmed stop is therefore refused: the refusal
names the task, how long the sandbox has been open, and a
discard_token.
Re-call with confirm_discard set to that token only when you actually mean to
throw the run away.
To keep the run instead, call finalize_run — it grades the
run and tears the sandbox down, so this tool is not needed after it.
Returns — { stopped, graded: false }, or a refusal naming what would be
lost. Stopping an already-closed sandbox is idempotent success:
{ stopped, already_closed: true, state }.
Errors — 409 with reason ungraded_session is the refusal above, 404
on an unknown id.
Example
finalize_run, which is almost always what you
want instead.
finalize_run
Grades the run synchronously: captures the recorded twin tape, scores it, and tears the sandbox down. Call it withsession_id alone. The capture bearer is re-issued
server-side and the task was stamped at run_task, so both are inferred.
Never carry the sandbox JWT around just to finalize.
Returns —
{ run, note }. The score is in run directly; there is nothing
to poll. Call get_report with run.run_id for the full markdown.
The sandbox must still be live, and one evaluation is charged per run.
Errors — 409 with reason capture_incomplete when the tape could not be
captured, 409 with reason blob_modified on a racing upload (retry once),
402 when the monthly agent-eval allowance is spent. An expired sandbox is
refused by name before any capture is attempted — run the task again.
Example
get_report for the graded report.
get_report
Returns the scored evaluation report as markdown, plus a weblink for humans.
Returns — the report as markdown: the
[code] score with its criteria
results, each [model] criterion with its reading, and the narrator’s summary
and handoff — plus the app.pome.sh/runs/<id> weblink.
Errors — 404 on an unknown run_id, or one belonging to another team.
Example
run_trials to re-run the fix as a new version.
list_runs
Lists the team’s evaluation runs, newest first.
Returns —
{ runs, dashboard } — the matching runs and a link to the runs
dashboard.
Errors — 401, 403.
Example
get_report for one run in full.
Evaluate an agent over MCP
These tools in the order you call them.
Digital twins reference
What
run_task provisions against.