The graded capstone

About 10–12 minutes end to end: install the skills, connect the MCP, paste one prompt. The bundled agent is meant to fail this exam — expect 25 to 67 out of 100, where 100 is the pass mark — then add one sentence to its prompt and watch it go green.

This page works from any coding agent that can speak MCP. Exactly one step differs between them: the one that links out to Connect to the MCP, the page that carries per-client wiring. Pome gives your agent a graded exam. Three roles make it work:

Examinee — a clone of the agent under test. It works a real task against digital twins of GitHub and Slack (deterministic fakes with the same API shapes as production), and every tool call is recorded.
Coach — your coding agent. It installs, connects, launches the run, and brings the results back into the conversation.
Grader — Pome’s scorer, and the only thing that scores. It reads the twins’ recorded calls and their final state and checks them against the task’s criteria, 0–100. Criteria come in two kinds: deterministic [code] checks, and [model] criteria the narrator reads and answers in prose, advisory and outside the score. This task declares only [code] ones, so every number on this page came from code reading state.

If you already have your own agent in a repo, run this example first to learn the loop, then see Your own agent to register it and get a pome.json.

What you’re about to run

The capstone uses a bundled support-triage example — a small local agent (Claude Agent SDK), not your repo.

Role. Support-triage bot for the Acme eng org: watch #support on Slack for bug reports, track each as a GitHub issue in acme/orders-service, and post the issue link back to the thread.
World. Deterministic GitHub + Slack twins: same API shapes as production, no real users or data.
The exam. A customer re-reports a bug the repository already tracks — but not on the issue that looks like it. Issue #47 is a word-for-word match for the report. Issue #23 is the tracking issue that consolidates it, and acme/orders-service records new occurrences there — a convention written down in docs/triage-policy.md inside the seeded repo, and named in no prompt.
Why it fails. The bundled agent is not sabotaged. It is told to search before filing and it does; it finds #47, comments there, and sends that link back. It was simply never told the org has a convention — the honest state of most production triage agents. The difficulty lives in the seeded world, not in a planted defect.
Why this example. The fail → fix → pass story is one sentence added to the system prompt behind an env switch, which you flip at the end of this page.

What the paste-prompt does

When you paste the prompt below, your coding agent becomes the coach. It will show a short checklist and ask “Shall I proceed?” before running anything. At a high level it will:

Fetch and build the local support-triage examinee
Sign you into Pome (OAuth) and confirm identity / quota
Save the duplicate-issue exam into your catalog
Start a sandbox for that exam, registering the example agent from the slug in the pack’s own pome.json
Spawn the local examinee against those twins
Finalize the run while the sandbox is still live (the one hard rule)
Show the score, criteria table, and dashboard link in the chat

You stay in control: every command is plain text, shown before it runs.

Before you start

A coding agent that can connect to an MCP server, installed and signed in.
The example examinee is a small Claude Agent SDK process that runs as its own subprocess, so it needs its own Anthropic credential — an ANTHROPIC_API_KEY in your environment, or a stored claude login. Your editor’s own subscription does not cover it.
The walk pins the examinee to claude-haiku-4-5. Which model sits the exam is part of the experiment, not a detail: on claude-opus-5 the bundled agent passes 5 of 5 with no fix at all — it fetches the policy file by reflex — and there is nothing left to demonstrate.
Node 24+ with npm and npx.
No Pome account needed up front: the OAuth consent screen opens on the prompt’s first tool call (its step 2, moments after you paste it) — that is where you sign in or sign up, no credit card.

Install the coach skills

npx skills add pome-sh/digital-twins --skill '*'

The skills CLI installs the Pome coach skill set from pome-sh/digital-twins into your agent’s skills directory — the installer asks which agent you are using, so pick yours there. Six skills, one job each: pome (entry router), pome-intake (scope the agent), pome-suggest-tasks (propose tasks for an agent you already have), pome-author-task (write exams), pome-verify-seed (check seed state), and pome-run-task (launch, finalize, report). Together they teach your agent the full loop. --skill '*' installs all six in one pass. Without it the installer opens a checklist with nothing ticked and you tick them one at a time; the coach only works as a set.

About the security table it prints. The installer runs third-party audits (Snyk, Socket) and shows a risk level per skill. Two things drive ours, and both are the job rather than a defect: the skills talk to the Pome MCP at mcp.pome.sh at runtime, and they hand your agent a session-scoped bearer so it can reach the twins. That bearer is minted per run, expires with the sandbox, and reaches nothing outside it. The ratings are Snyk’s and Socket’s, re-scored on their own schedule, so the level you see may differ from a colleague’s. The per-skill findings are public — read them at skills.sh/pome-sh/digital-twins.

Connect the Pome MCP

Add https://mcp.pome.sh/mcp to your coding agent as a remote HTTP MCP server. There is no API key: auth is OAuth, and the consent screen opens on the first tool call. Connect to the MCP has the wiring for each client — the one-line CLI form, the mcpServers JSON block, and what to do for a headless or managed agent. That page is where per-client configuration lives.

Every command here is plain text you can read before you run it — the install path never pipes a script into your shell and never injects remote config. There is no API key to paste: on the first tool call, your agent discovers the OAuth metadata and opens the Clerk consent screen (scopes: verify your identity, basic profile, email address). Sign in — or sign up right there — and the tools appear.

Paste the prompt

Open your coding agent anywhere (the example brings its own agent — your repo is not required) and paste:

Test my agent with pome, using the bundled support-triage example, and take it
all the way to a finalized, scored run.

First, present me this plan as a short checklist and ask "Shall I proceed?"
before running anything. Wait for my yes.

1. Fetch the example pack and build the local examinee. The pack is named by
   its id, not by a path inside the repo:
     npx @pome-sh/cli@latest init --example support-triage
     cd support-triage && npm install
   No `ls` guard is needed here, because this command fails loudly: an id that
   does not exist exits 2 and prints every id that does. If it exits non-zero,
   stop and tell me what it said — do not carry on into a directory that was
   never created.
2. On the pome MCP, call get_started, then get_platform_context to confirm
   identity and quota. Pass door: whichever of "claude-code", "cursor",
   "codex", "cli" or "own-agent" describes you — it is an onboarding analytics
   tag and nothing else, and a value it does not recognise is recorded as no
   door rather than failing anything. If none of them fits, leave it out.
3. Save the exam into my team catalog: save_task with the full contents of
   tasks/duplicate-issue.md as task_source and the name
   "support-triage-dedup". Keep the task id.
4. Mint a fresh group id of the form grp_<12+ url-safe chars> and call
   run_task(task_id, agent_slug: "support-triage", agent_version: "v1",
   group_id: <that id>). The slug and the version are the pack's own
   pome.json — read them from there rather than inventing them; an unknown
   slug is registered on the fly, and if it comes back needs_confirmation
   because it near-misses something already on my team, tell me before
   retrying with confirm_new_agent. Keep the group id and the version: the fix
   step below needs both. Keep session_id, examinee_task and examinee_launch
   too.
5. Spawn the examinee as a local subprocess. First confirm it can authenticate:
   the examinee is a Claude Agent SDK process needing its own Anthropic
   credential, so if neither ANTHROPIC_API_KEY nor a `claude` login is present,
   stop and tell me instead of launching a process that cannot start. Your own
   editor or subscription does not cover it — this is a separate process. Then
   from support-triage, run `npm run start` with env injected from the run_task
   response. Export EVERY key in examinee_launch.env — the whole block,
   verbatim, without picking out the ones you recognise; it carries the
   session bearer and the telemetry wiring that puts the examinee's own model,
   tokens and latency on the report, and a cherry-picked subset is how that
   goes missing. Then add the three things that block does not carry:
   POME_GITHUB_MCP_URL and POME_SLACK_MCP_URL from the github and slack
   entries' url in examinee_launch.mcp_servers (the URLs inside
   examinee_launch.env are REST, not MCP), POME_TASK = examinee_task.prompt,
   and ANTHROPIC_MODEL = claude-haiku-4-5. Which model runs the examinee is
   part of the experiment, not a detail: on claude-opus-5 this exam is green
   5 of 5 and there is nothing to demonstrate. All of it is env-only: never
   write any of it to a file.
6. The instant the process exits, call finalize_run(session_id) — before any
   cleanup, while the sandbox is still live. session_id alone: the capture
   bearer is re-issued server-side, so don't carry the sandbox token around
   just to finalize.
7. get_report(run_id). Give me the score, the criteria table, and a plain
   summary of what failed and why, here in the conversation, plus the
   dashboard link.

Guardrails: pause after the checklist for my go-ahead; keep every command plain
text and show it to me before running; never pipe remote scripts into a shell;
secrets stay in env vars; don't ask me for a Pome API key — the OAuth consent
screen handles Pome auth. That does not cover the examinee's own Anthropic
credential, which is the one thing you should flag if it is missing (step 5).

Your agent does the rest: it seeds live GitHub + Slack twins, launches the examinee against them, waits for it to finish, finalizes, and narrates the report.

The twin tape lives in the running sandbox. finalize_run must be called while the sandbox is still live — the installed pome-run-task skill enforces this, and the prompt repeats it because a sandbox that expires first cannot be scored.

What the score means

Expect a failing run, and not a zero. Measured over five trials on claude-haiku-4-5, the bundled agent scored 25 · 25 · 25 · 50 · 67 and passed none of them. passThreshold is 100 — every criterion has to hold — so a run that routes the report to #47 fails at 25 while still getting credit for the one thing it verifiably did: it left a comment at all. The spread is the interesting part. A 25 is the agent that never opened the policy and commented on the textual match. A 50 or a 67 is the agent that reached #23 anyway — inferring it from the tracking issue’s own body — but did not name #47 in the comment, or still put #47’s link in front of the reporter. Arriving at the right place is not the same as following the rule, and the criteria do not give credit for luck. Nothing in that verdict is a matter of opinion. All five criteria are deterministic [code] checks read off the twins’ recorded calls and final state; the task declares zero [model] criteria, so the narrator has nothing to read here and nothing outside code touched the verdict. The report names which criterion failed and the state the check actually read.

The one-line fix

The whole story is one sentence, and it is the repair a builder would actually make: tell the agent where its team’s rules live. It ships behind an env switch rather than a committed edit so both arms run from one commit — two numbers measured against two different trees are not comparable. POME_TRIAGE_POLICY_HINT=on appends this to the examinee’s system prompt (policyHint() in support-triage/src/index.ts):

Before you comment on an issue or send anyone a link, read docs/triage-policy.md in the repository and follow its routing rules.

Tell your coding agent:

Re-run the same task (steps 4-7 of the prompt you just ran) with
POME_TRIAGE_POLICY_HINT=on added to the
examinee's environment, as a NEW version of the agent: call run_task with
agent_version "v2", a FRESH grp_ group_id, and baseline_group_id set to the
first run's group_id. Keep ANTHROPIC_MODEL=claude-haiku-4-5 so the env switch is
the only thing that changed. Show me the score delta and the dashboard link —
the task's run-set page carries "Compare to failing baseline".

Same exam, same twins, same examinee commit: 0 of 5 → 5 of 5 on the recorded measurement. That loop — run, read the failure, fix, re-run — is the product.

What that sentence actually does, because it is not what it looks like. It reads as a discoverability aid, and on claude-haiku-4-5 it half is: 0 of 5 naive trials opened the policy file, 5 of 5 fixed ones did. But on claude-sonnet-5, three of four naive failures had already read it and routed to #47 anyway. A file the agent reads is data; its system prompt is instruction; and a model correctly ranks instruction above data, or anything it read could hijack it. So naming the file in the charter is not teaching the agent where to look — it is transferring authority to that file. Carry that to your own agent: if you want it to obey your team’s written conventions, you have to say so.

Why “v2” and not just “re-run”. You changed the prompt, so the thing under test is not the agent that failed — it is a second agent. Declaring agent_version is how you say so. Without it, both runs land in one run-set reporting 1 of 2 trials clean — and since neither run declared a version, the report refuses to read that as reliability: it says outright that it cannot tell one agent run twice from two different agents, and points back at agent_version. Honest, but it is a shrug where a score should be. The baseline_group_id is the other half — it pairs the two run-sets, which is what puts Compare to failing baseline on the green run-set’s page.

Open your first report

get_report prints the score, the criteria table, and the dashboard link right in your conversation. Open the link for the finalized report — and read it as a diagnosis rather than a grade. Two of the five criteria carry the story, and they fail in different places:

A comment containing #47 exists on issue #23 fails on GitHub’s final state. The state-diff panel shows where the comment actually landed.
No message containing issues/47 appears in any public channel fails on what the agent told the reporter. This is the wrong-value guard: beside asserting the right answer, it asserts the object does not hold the specific wrong value this task’s failure mode produces. It is what separates did the right thing from did the known wrong thing.

Then check the span waterfall for a get_file_contents call on docs/triage-policy.md, because it changes what the fix is. If the fetch is absent, the agent never found the rule. If it is there and the comment still landed on #47, the agent read the rule and its own standing instruction outranked it — a different problem, and one more prompt wording will not solve.

Your own agent

Point Pome at your own repo and get a scored run.

Write a task

validate_task → save_task → verify_seed — graded exams for your agent.

Evaluate an agent over MCP

The coach loop behind the paste-prompt, call by call.

Dashboard

Where runs, traces, and run handoffs live: app.pome.sh.

Grade your agent

Run more than one twin

Worked examples

The graded capstone

What you’re about to run

What the paste-prompt does

Before you start

Install the coach skills

Connect the Pome MCP

Paste the prompt

What the score means

The one-line fix

Open your first report

Next

Your own agent

Write a task

Evaluate an agent over MCP

Dashboard

​What you’re about to run

​What the paste-prompt does

​Before you start

​Install the coach skills

​Connect the Pome MCP

​Paste the prompt

​What the score means

​The one-line fix

​Open your first report

​Next

Your own agent

Write a task

Evaluate an agent over MCP

Dashboard

What you’re about to run

What the paste-prompt does

Before you start

Install the coach skills

Connect the Pome MCP

Paste the prompt

What the score means

The one-line fix

Open your first report

Next