> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pome.sh/llms.txt
> Use this file to discover all available pages before exploring further.
# The graded capstone
> Point your coding agent at a bundled example, paste one prompt, and read a scored report of an agent failing a real task — then fix it in one sentence.
**About 10–12 minutes** end to end: install the skills,
connect the MCP, paste one prompt. The bundled agent is meant to **fail** this
exam — expect 25 to 67 out of 100, where 100 is the pass mark — then add one
sentence to its prompt and watch it go green.
This page works from **any coding agent** that can speak MCP. Exactly one step
differs between them: the one that links out to
[Connect to the MCP](/docs/mcp/connect), the page that carries per-client wiring.
Pome gives your agent a graded exam. Three roles make it work:
* **Examinee** — a clone of the agent under test. It works a real task against
**digital twins** of GitHub and Slack (deterministic fakes with the same API
shapes as production), and every tool call is recorded.
* **Coach** — your coding agent. It installs, connects, launches the run, and
brings the results back into the conversation.
* **Grader** — Pome's scorer, and the only thing that scores. It reads the twins'
recorded calls and their final state and checks them against the task's
criteria, 0–100. Criteria come in two kinds: deterministic `[code]` checks,
and `[model]` criteria the narrator reads and answers in prose, advisory and
outside the score. **This task declares only `[code]` ones**, so every number
on this page came from code reading state.
If you already have your own agent in a repo, run this example first to learn
the loop, then see [Your own agent](/existing-agent) to register it
and get a `pome.json`.
## What you're about to run
The capstone uses a bundled **support-triage** example — a small local agent
(Claude Agent SDK), not your repo.
* **Role.** Support-triage bot for the Acme eng org: watch `#support` on Slack
for bug reports, track each as a GitHub issue in `acme/orders-service`, and
post the issue link back to the thread.
* **World.** Deterministic **GitHub + Slack twins**: same API shapes as
production, no real users or data.
* **The exam.** A customer re-reports a bug the repository already tracks — but
**not on the issue that looks like it**. Issue **#47** is a word-for-word
match for the report. Issue **#23** is the tracking issue that consolidates
it, and `acme/orders-service` records new occurrences there — a convention
written down in `docs/triage-policy.md` inside the seeded repo, and named in
no prompt.
* **Why it fails.** The bundled agent is **not sabotaged**. It is told to search
before filing and it does; it finds #47, comments there, and sends that link
back. It was simply never told the org has a convention — the honest state of
most production triage agents. The difficulty lives in the seeded world, not
in a planted defect.
* **Why this example.** The fail → fix → pass story is one sentence added to the
system prompt behind an env switch, which you flip at the end of this page.
## What the paste-prompt does
When you paste the prompt below, your coding agent becomes the coach. It will
show a short checklist and ask **"Shall I proceed?"** before running anything.
At a high level it will:
1. Fetch and build the local support-triage examinee
2. Sign you into Pome (OAuth) and confirm identity / quota
3. Save the duplicate-issue exam into your catalog
4. Start a sandbox for that exam, registering the example agent from the slug in
the pack's own `pome.json`
5. Spawn the local examinee against those twins
6. Finalize the run while the sandbox is still live (the one hard rule)
7. Show the score, criteria table, and dashboard link in the chat
You stay in control: every command is plain text, shown before it runs.
## Before you start
* **A coding agent that can connect to an MCP server**, installed and signed in.
* The example examinee is a small
[Claude Agent SDK](https://docs.claude.com/en/agent-sdk/typescript) process
that runs as its **own subprocess**, so it needs its own Anthropic credential
— an `ANTHROPIC_API_KEY` in your environment, or a stored `claude` login. Your
editor's own subscription does not cover it.
* The walk pins the examinee to **`claude-haiku-4-5`**. Which model sits the exam
is part of the experiment, not a detail: on `claude-opus-5` the bundled agent
passes 5 of 5 with no fix at all — it fetches the policy file by reflex — and
there is nothing left to demonstrate.
* **Node 24+** with `npm` and `npx`.
* No Pome account needed up front: the OAuth consent screen opens on the
prompt's first tool call (its step 2, moments after you paste it) — that is
where you sign in or sign up, no credit card.
## Install the coach skills
```bash theme={"dark"}
npx skills add pome-sh/digital-twins --skill '*'
```
The [`skills` CLI](https://github.com/vercel-labs/skills) installs the Pome
coach skill set from
[`pome-sh/digital-twins`](https://github.com/pome-sh/digital-twins/tree/main/skills)
into your agent's skills directory — the installer asks which agent you are
using, so pick yours there. Six skills, one job each: `pome` (entry router),
`pome-intake` (scope the agent), `pome-suggest-tasks` (propose tasks for an
agent you already have), `pome-author-task` (write exams), `pome-verify-seed`
(check seed state), and `pome-run-task` (launch, finalize, report). Together
they teach your agent the full loop.
`--skill '*'` installs all six in one pass. Without it the installer opens a
checklist with nothing ticked and you tick them one at a time; the coach only
works as a set.
**About the security table it prints.** The installer runs third-party audits
(Snyk, Socket) and shows a risk level per skill. Two things drive ours, and
both are the job rather than a defect: the skills talk to the Pome MCP at
`mcp.pome.sh` at runtime, and they hand your agent a session-scoped bearer so
it can reach the twins. That bearer is minted per run, expires with the
sandbox, and reaches nothing outside it. The ratings are Snyk's and Socket's,
re-scored on their own schedule, so the level you see may differ from a
colleague's. The per-skill findings are public — read them at
[skills.sh/pome-sh/digital-twins](https://skills.sh/pome-sh/digital-twins).
## Connect the Pome MCP
Add `https://mcp.pome.sh/mcp` to your coding agent as a remote HTTP MCP server.
There is no API key: auth is OAuth, and the consent screen opens on the first
tool call.
**[Connect to the MCP](/docs/mcp/connect)** has the wiring for each client — the
one-line CLI form, the `mcpServers` JSON block, and what to do for a headless or
managed agent. That page is where per-client configuration lives.
Every command here is plain text you can read before you run it — the install
path never pipes a script into your shell and never injects remote config.
There is no API key to paste: on the first tool call, your agent discovers the
OAuth metadata and opens the Clerk consent screen (scopes: verify your
identity, basic profile, email address). Sign in — or sign up right there — and
the tools appear.
## Paste the prompt
Open your coding agent anywhere (the example brings its own agent — your repo is
not required) and paste:
```text theme={"dark"}
Test my agent with pome, using the bundled support-triage example, and take it
all the way to a finalized, scored run.
First, present me this plan as a short checklist and ask "Shall I proceed?"
before running anything. Wait for my yes.
1. Fetch the example pack and build the local examinee. The pack is named by
its id, not by a path inside the repo:
npx @pome-sh/cli@latest init --example support-triage
cd support-triage && npm install
No `ls` guard is needed here, because this command fails loudly: an id that
does not exist exits 2 and prints every id that does. If it exits non-zero,
stop and tell me what it said — do not carry on into a directory that was
never created.
2. On the pome MCP, call get_started, then get_platform_context to confirm
identity and quota. Pass door: whichever of "claude-code", "cursor",
"codex", "cli" or "own-agent" describes you — it is an onboarding analytics
tag and nothing else, and a value it does not recognise is recorded as no
door rather than failing anything. If none of them fits, leave it out.
3. Save the exam into my team catalog: save_task with the full contents of
tasks/duplicate-issue.md as task_source and the name
"support-triage-dedup". Keep the task id.
4. Mint a fresh group id of the form grp_<12+ url-safe chars> and call
run_task(task_id, agent_slug: "support-triage", agent_version: "v1",
group_id: ). The slug and the version are the pack's own
pome.json — read them from there rather than inventing them; an unknown
slug is registered on the fly, and if it comes back needs_confirmation
because it near-misses something already on my team, tell me before
retrying with confirm_new_agent. Keep the group id and the version: the fix
step below needs both. Keep session_id, examinee_task and examinee_launch
too.
5. Spawn the examinee as a local subprocess. First confirm it can authenticate:
the examinee is a Claude Agent SDK process needing its own Anthropic
credential, so if neither ANTHROPIC_API_KEY nor a `claude` login is present,
stop and tell me instead of launching a process that cannot start. Your own
editor or subscription does not cover it — this is a separate process. Then
from support-triage, run `npm run start` with env injected from the run_task
response. Export EVERY key in examinee_launch.env — the whole block,
verbatim, without picking out the ones you recognise; it carries the
session bearer and the telemetry wiring that puts the examinee's own model,
tokens and latency on the report, and a cherry-picked subset is how that
goes missing. Then add the three things that block does not carry:
POME_GITHUB_MCP_URL and POME_SLACK_MCP_URL from the github and slack
entries' url in examinee_launch.mcp_servers (the URLs inside
examinee_launch.env are REST, not MCP), POME_TASK = examinee_task.prompt,
and ANTHROPIC_MODEL = claude-haiku-4-5. Which model runs the examinee is
part of the experiment, not a detail: on claude-opus-5 this exam is green
5 of 5 and there is nothing to demonstrate. All of it is env-only: never
write any of it to a file.
6. The instant the process exits, call finalize_run(session_id) — before any
cleanup, while the sandbox is still live. session_id alone: the capture
bearer is re-issued server-side, so don't carry the sandbox token around
just to finalize.
7. get_report(run_id). Give me the score, the criteria table, and a plain
summary of what failed and why, here in the conversation, plus the
dashboard link.
Guardrails: pause after the checklist for my go-ahead; keep every command plain
text and show it to me before running; never pipe remote scripts into a shell;
secrets stay in env vars; don't ask me for a Pome API key — the OAuth consent
screen handles Pome auth. That does not cover the examinee's own Anthropic
credential, which is the one thing you should flag if it is missing (step 5).
```
Your agent does the rest: it seeds live GitHub + Slack twins, launches the
examinee against them, waits for it to finish, finalizes, and narrates the report.
The twin tape lives in the running sandbox. `finalize_run` must be called while
the sandbox is still live — the installed `pome-run-task` skill enforces this,
and the prompt repeats it because a sandbox that expires first cannot be scored.
## What the score means
Expect a **failing run, and not a zero**. Measured over five trials on
`claude-haiku-4-5`, the bundled agent scored **25 · 25 · 25 · 50 · 67** and
passed none of them. `passThreshold` is 100 — every criterion has to hold — so
a run that routes the report to #47 fails at 25 while still getting credit for
the one thing it verifiably did: it left a comment at all.
The spread is the interesting part. A 25 is the agent that never opened the
policy and commented on the textual match. A 50 or a 67 is the agent that
reached #23 anyway — inferring it from the tracking issue's own body — but did
not name #47 in the comment, or still put #47's link in front of the reporter.
Arriving at the right place is not the same as following the rule, and the
criteria do not give credit for luck.
Nothing in that verdict is a matter of opinion. All five criteria are
deterministic `[code]` checks read off the twins' recorded calls and final
state; the task declares **zero `[model]` criteria**, so the narrator has
nothing to read here and nothing outside code touched the verdict. The report
names which criterion failed and the state the check actually read.
## The one-line fix
The whole story is one sentence, and it is the repair a builder would actually
make: tell the agent where its team's rules live. It ships behind an env switch
rather than a committed edit so both arms run from **one commit** — two numbers
measured against two different trees are not comparable.
`POME_TRIAGE_POLICY_HINT=on` appends this to the examinee's system prompt
(`policyHint()` in `support-triage/src/index.ts`):
> Before you comment on an issue or send anyone a link, read
> `docs/triage-policy.md` in the repository and follow its routing rules.
Tell your coding agent:
```text theme={"dark"}
Re-run the same task (steps 4-7 of the prompt you just ran) with
POME_TRIAGE_POLICY_HINT=on added to the
examinee's environment, as a NEW version of the agent: call run_task with
agent_version "v2", a FRESH grp_ group_id, and baseline_group_id set to the
first run's group_id. Keep ANTHROPIC_MODEL=claude-haiku-4-5 so the env switch is
the only thing that changed. Show me the score delta and the dashboard link —
the task's run-set page carries "Compare to failing baseline".
```
Same exam, same twins, same examinee commit: **0 of 5 → 5 of 5** on the recorded
measurement. That loop — run, read the failure, fix, re-run — is the product.
**What that sentence actually does**, because it is not what it looks like. It
reads as a discoverability aid, and on `claude-haiku-4-5` it half is: 0 of 5
naive trials opened the policy file, 5 of 5 fixed ones did. But on
`claude-sonnet-5`, three of four naive failures had **already read it** and
routed to #47 anyway. A file the agent reads is *data*; its system prompt is
*instruction*; and a model correctly ranks instruction above data, or anything
it read could hijack it. So naming the file in the charter is not teaching the
agent where to look — it is **transferring authority** to that file. Carry that
to your own agent: if you want it to obey your team's written conventions, you
have to say so.
**Why "v2" and not just "re-run".** You changed the prompt, so the thing under
test is not the agent that failed — it is a second agent. Declaring
`agent_version` is how you say so. Without it, both runs land in one run-set
reporting 1 of 2 trials clean — and since neither run declared a version, the
report refuses to read that as reliability: it says outright that it cannot
tell one agent run twice from two different agents, and points back at
`agent_version`. Honest, but it is a shrug where a score should be. The
`baseline_group_id` is the other half — it pairs the two run-sets, which is
what puts **Compare to failing baseline** on the green run-set's page.
## Open your first report
`get_report` prints the score, the criteria table, and the dashboard link right
in your conversation. Open the link for the finalized report — and read it as a
diagnosis rather than a grade. Two of the five criteria carry the story, and
they fail in different places:
* **A comment containing `#47` exists on issue #23** fails on GitHub's final
state. The state-diff panel shows where the comment actually landed.
* **No message containing `issues/47` appears in any public channel** fails on
what the agent *told the reporter*. This is the wrong-value guard: beside
asserting the right answer, it asserts the object does not hold the specific
wrong value this task's failure mode produces. It is what separates *did the
right thing* from *did the known wrong thing*.
Then check the span waterfall for a `get_file_contents` call on
`docs/triage-policy.md`, because it changes what the fix is. If the fetch is
absent, the agent never found the rule. If it is there and the comment still
landed on #47, the agent read the rule and its own standing instruction
outranked it — a different problem, and one more prompt wording will not solve.
## Next
Point Pome at your own repo and get a scored run.
`validate_task` → `save_task` → `verify_seed` — graded exams for *your* agent.
The coach loop behind the paste-prompt, call by call.
Where runs, traces, and run handoffs live: app.pome.sh.