Skip to main content
Most agent failures are loud. This one is quiet, and quiet is worse. A merge bot reviews the open pull requests in a repository. It merges the safe ones, blocks the unsafe ones, and reports every outcome to a Slack channel. Give it a pull request with failing CI and it does the hard part perfectly: it reads the red ci/test status, refuses the merge, and leaves a CHANGES_REQUESTED review explaining why. Then it tells nobody. GitHub is in exactly the right state. Slack is silent. Nobody is watching the pull request page; everybody is in the channel. The half that is missing is the half a human would have acted on. This is the example that grades that. It runs on LangGraph, not the Claude Agent SDK, which is the second thing it demonstrates: the exam does not care what your agent is built with.
The example’s id is minimal-viktor-langgraph. Fetch it by that id:
Everything below is measured against that directory, and VERIFICATION.md inside it carries the run ids.

​
What breaks

The graph is five nodes: intake → gather → decide → act → report. The model makes one call, in decide, and returns a MERGE / BLOCK / FLAG decision per pull request. act carries the decision out in GitHub. report mirrors it to Slack. The committed defect is one constant in src/graph.ts:
Only merges get announced. Blocks and flags happen in GitHub and vanish. Two properties make this teachable rather than just broken: It is invisible from either system alone. Open GitHub: correct. Open Slack: quiet — which looks like nothing happened, not like something happened and you were not told. Only an exam that reads both final states at once catches it. It cannot rot green. The flaw is committed control flow, not a prompt. The model still decides BLOCK correctly and still writes a good reason — for a message that is never sent. A stronger model writes a better string for the same silence.

​
Run the failing baseline

The defect ships as the default, so the failing run is the plain one:
33, not 0. The GitHub check that is graded passes; both Slack checks fail. That is deliberate, and it is the whole lesson: an example where everything fails teaches the agent is broken. This one teaches the systems disagree.
verified red: claude-sonnet-5, 0/3 trials — 2026-08-04, re-measured 2026-08-30 from a provisioned throwaway account. A committed control-flow defect has the lowest rot risk of any baseline shape — no model capability routes around a branch that returns before the write — so this stamp does not need re-measuring per model generation, only after a twin-snapshot rebuild.

​
Read the report

You should not have to open the repository to know what went wrong. Open the run and the diagnosis is the top half of the page. The score names its own denominator. 33/100 — 1 of 3 evaluated criteria passed; 1 excluded as already true in the seed; 1 advisory, not scored. Not a percentage floating free of what was checked. The criteria split by kind, and the split is the signature. GitHub holds — one row graded green, one excluded because it never stopped being true. Every Slack check red. 0 message(s) scanned is the evidence, not an opinion — the Slack twin’s final state either carries that message or it does not. Read the last row again. The agent’s reasoning holds. It understood the situation, drew the right conclusion, and wrote it down. Only the mirror is missing. If the agent were simply bad at this task, that row would say so too. The state panel has two tabs. github shows a CHANGES_REQUESTED review from pome-agent and merged: 0. slack shows the eng-alerts channel present and correct — right members, right topic, "messages": []. Two systems, one action, one of them empty. The trace shows which node did nothing. Find the act and report rows:
act has a tool call under it. report is a leaf that took a millisecond. That row is the defect, visible without reading a line of the example’s source. Across three trials it is not flaky. The run-set page lays the trials out as a grid — rows are criteria, columns are trials — and the two Slack rows read 0/3 while every other row reads 3/3. Strict · pass^3: not met.
On a LangGraph or Vercel AI SDK agent, twin HTTP rows nest under the enclosing LLM turn rather than under the specific tool call that made them. The composer joins on a per-tool-call correlation id, and today only the Claude adapter injects one — with no join key it falls back to the enclosing turn instead of inventing an edge. Grading is unaffected: [code] criteria read the twins’ final state, not the trace.

​
The fix

One line in src/graph.ts:
You can find it from the report alone: the failing criterion names the eng-alerts channel, and that is the only place in the graph that writes there.

​
Re-run green

The run-set page keeps both sets, so the fix is a delta you can point at: 0/3 before, 3/3 after, same agent, same task, same twins. Strict · pass^3: met. And STEP report now has a TOOL slack_post_message child. The GitHub half never moved. Measured across both run sets, the graded code:github criterion holds 3/3 in the baseline and 3/3 in the fix — and the not-merged row stays excluded in both, because it never stopped being true. That invariance is the point: you can tell exactly which system broke.

​
Customize

Move the branch. Gate on FLAG instead of MERGE and re-run tasks 05 and 06. Now only the malicious pull requests go unannounced — same class, worse blast radius. Make the mirror fail instead of never firing. Put a fault seed on the Slack twin so the notify call errors partway through the batch. That composes this class with retry/partial-failure and teaches cross-system partial writes rather than a skipped mirror. Swap the framework. The same agent on the Vercel AI SDK is one id away:
Same task, same seed, same criteria, same report — which turns “any framework can take the exam” from a claim into something you can check. Point it at your own agent. The task file is ordinary task markdown. If your agent acts in one system and reports in another — a deploy bot that updates a status page, an on-call bot that files a ticket and posts a summary — the same paired-criteria shape grades it: every assertion on system A gets a mirror assertion on system B.

​
If your baseline passes, or your fix fails

The baseline came back green. Most likely the run never reached the report node — check the trace for act and report steps. If decide returned MERGE for a pull request with failing CI, the run is red for a different reason entirely and this is not the lesson you measured; re-read the two code:github criteria first. The defect only binds on non-MERGE outcomes, by design, so tasks 01 and 02 are green either way. The fix stayed red. Check which criterion. If a code:slack one is still failing, the message went somewhere else — the channel defaults to eng-alerts and the criteria name that channel literally. If a criterion reads NOT EVALUATED rather than failed, the run is INCOMPLETE: the grader could not read that state at all, which is a wiring problem rather than an agent problem. pome run exits non-zero for that too, and the score says so in its denominator. Your numbers differ from the ones above. They were measured on 2026-08-30 against claude-sonnet-5 with @pome-sh/cli@0.42.2 (an earlier 2026-08-04 measurement, before [model] rows went advisory and seed-true rows were excluded, printed 60 for the same red). Scores are date-stamped and model-stamped on purpose — VERIFICATION.md ships inside the directory you scaffolded and carries run ids and how the model was routed. What should not drift is the shape: GitHub criteria hold, Slack criteria flip, and one line moves them.