ci/test status, refuses the merge, and leaves a
CHANGES_REQUESTED review explaining why.
Then it tells nobody.
GitHub is in exactly the right state. Slack is silent. Nobody is watching the
pull request page; everybody is in the channel. The half that is missing is the
half a human would have acted on.
This is the example that grades that. It runs on LangGraph, not the Claude
Agent SDK, which is the second thing it demonstrates: the exam does not care what
your agent is built with.
The example lives at
examples/minimal-viktor-langgraph
in pome-sh/digital-twins. Everything below is measured against it —
VERIFICATION.md in that directory carries the run ids.What breaks
The graph is five nodes:intake → gather → decide → act → report. The model
makes one call, in decide, and returns a MERGE / BLOCK / FLAG decision per pull
request. act carries the decision out in GitHub. report mirrors it to Slack.
The committed defect is one constant in src/graph.ts:
Run the failing baseline
The defect ships as the default, so the failing run is the plain one:verified red:
claude-sonnet-5, 0/3 trials, 2026-08-04. A committed
control-flow defect has the lowest rot risk of any baseline shape — no model
capability routes around a branch that returns before the write — so this stamp
does not need re-measuring per model generation. It does need re-measuring after
a twin-snapshot rebuild.Read the report
You should not have to open the repository to know what went wrong. Open the run and the diagnosis is the top half of the page. The score names its own denominator.60/100 — 3 of 5 criteria passed. Not a
percentage floating free of what was checked.
The criteria split by kind, and the split is the signature.
Every GitHub check green. Every Slack check red.
0 message(s) scanned is the
evidence, not an opinion — the Slack twin’s final state either carries that
message or it does not.
Read the last row again. The judge passes the agent’s reasoning. The agent
understood the situation, drew the right conclusion, and wrote it down. Only the
mirror is missing. If the agent were simply bad at this task, that row would be
red too.
The state panel has two tabs. github shows a CHANGES_REQUESTED review
from pome-agent and merged: 0. slack shows the eng-alerts channel present
and correct — right members, right topic, "messages": []. Two systems, one
action, one of them empty.
The trace shows which node did nothing. Find the act and report rows:
act has a tool call under it. report is a leaf that took a millisecond. That
row is the defect, visible without reading a line of the example’s source.
Across three trials it is not flaky. The run-set page lays the trials out as
a grid — rows are criteria, columns are trials — and the two Slack rows read
0/3 while every other row reads 3/3. Strict · pass^3: not met.
On a LangGraph or Vercel AI SDK agent, twin HTTP rows nest under the enclosing
LLM turn rather than under the specific tool call that made them. The composer
joins on a per-tool-call correlation id, and today only the Claude adapter
injects one — with no join key it falls back to the enclosing turn instead of
inventing an edge. Grading is unaffected:
[code] criteria read the twins’
final state, not the trace.The fix
One line insrc/graph.ts:
eng-alerts channel, and that is the only place in the graph that writes there.
Re-run green
0/3 before, 3/3 after, same agent, same task, same twins. Strict · pass^3: met. And STEP report now has a TOOL slack_post_message child.
The GitHub half never moved. Measured across both run sets, the two
[code:github] criteria hold 3/3 in the baseline and 3/3 in the fix.
That invariance is the point: you can tell exactly which system broke.
Customize
Move the branch. Gate onFLAG instead of MERGE and re-run tasks 05 and
06. Now only the malicious pull requests go unannounced — same class, worse blast
radius.
Make the mirror fail instead of never firing. Put a fault seed on the Slack
twin so the notify call errors partway through the batch. That composes this
class with retry/partial-failure and teaches cross-system partial writes rather
than a skipped mirror.
Swap the framework.
examples/minimal-viktor
is the same agent on the Vercel AI SDK. Same task, same seed, same criteria, same
report — which turns “any framework can take the exam” from a claim into
something you can check.
Point it at your own agent. The task file is ordinary
task markdown. If your agent acts in one system and
reports in another — a deploy bot that updates a status page, an on-call bot that
files a ticket and posts a summary — the same paired-criteria shape grades it:
every assertion on system A gets a mirror assertion on system B.
If your baseline passes, or your fix fails
The baseline came back green. Most likely the run never reached thereport
node — check the trace for act and report steps. If decide returned MERGE
for a pull request with failing CI, the run is red for a different reason
entirely and this is not the lesson you measured; re-read the two code:github
criteria first. The defect only binds on non-MERGE outcomes, by design, so tasks
01 and 02 are green either way.
The fix stayed red. Check which criterion. If a code:slack one is still
failing, the message went somewhere else — the channel defaults to eng-alerts
and the criteria name that channel literally. If a criterion reads
NOT EVALUATED rather than failed, the run is INCOMPLETE: the grader could not
read that state at all, which is a wiring problem rather than an agent problem.
pome run exits non-zero for that too, and the score says so in its denominator.
Your numbers differ from the ones above. They were measured on 2026-08-04
against claude-sonnet-5 with @pome-sh/cli@0.18.0. Scores are date-stamped and
model-stamped on purpose — see
VERIFICATION.md
for the exact run ids and how the model was routed. What should not drift is the
shape: GitHub criteria hold, Slack criteria flip, and one line moves them.