Skip to main content
Braintrust and LangSmith run your eval. Pome is what your agent calls during it. Both of them sell something called a sandbox, and neither one is Pome’s. Theirs runs your code — the dataset loop, the scorers, and in LangSmith’s case containers for code an agent wrote. Pome’s is a digital twin of the SaaS APIs your agent talks to: a stateful emulation of Stripe, GitHub, Slack, Gmail or Linear that answers the same REST calls, boots from a starting state you declare, and remembers every request it received. That distinction is the whole recipe. Your dataset has rows; this gives each row its own world, and grades what your agent did to it. Three answers to one question, and the middle of the three is the same in both:
  • Braintrust — one score column per criterion, out of Eval().
  • LangSmith — one feedback key per criterion, out of evaluate().
  • Your agent’s spans — the framework-agnostic seam. Two standard OpenTelemetry variables, and the spans land wherever you point them.
The first two are one recipe with two renderers, so this page carries them together: pick a framework below and the tabs follow it through. Everything untabbed is true of both.

​
The shape

One dataset row becomes one isolated world:
Six rows, six sandboxes, six graded runs. Each row carries its own charge, its own amount, and its own injected fault — not one world reused six times.

​
Run one of the two examples

Both examples are bundled with the CLI and fetched by id, not by a typed GitHub path that can rot into a 404. pome init --example <id> writes the whole tree into ./<id> and nothing else: it does not touch a manifest you already have. It needs @pome-sh/cli 0.35.0 or later, which is what @latest below gets you.
You need three credentials. Two are yours already if you are reading this:
npm start runs Eval() in-process through tsx. It does not read a .env file — export the three variables, or the run reaches 401 invalid_auth. Braintrust’s own bt eval CLI does load .env; this example does not go through it, because Braintrust’s Sandbox evals are a Pro feature and the recipe must stay runnable on Starter.
On npm 11 the install prints allow-scripts warnings for braintrust, esbuild and fsevents. They are expected and nothing here needs those scripts — the measured run below was produced in exactly that state.npm test is fully offline. It needs no account and no credentials, and it includes a real Eval() run in Braintrust’s local noSendLogs mode.
You do not need either directory to use Pome from your eval. They exist so you can watch the thing work before you touch the eval you care about. If you already have an Eval() or an evaluate(), skip to how it fits together — the integration is two functions, and they drop into what you already have.
What has been measured, and what has not. The Braintrust half of this page was walked end to end on 2026-08-28 on a $0 Braintrust Starter account with no card on file; every number under it is from that run. The Pome half is the same code in both examples — src/pome.ts, copied, with both test suites pinning it case for case — and it was verified against api.pome.sh on 2026-08-27, three times, same split each run. The LangSmith half has not been run against a live LangSmith account. Its seam is verified against the real SDK by test/langsmith-seam.test.ts, which drives an actual evaluate() against a stub client, and its plan and OTLP facts are read from LangSmith’s own documentation on 2026-08-27. Where a LangSmith number below is expected rather than observed, it says so.

​
The failure both examples demonstrate

Neither framework needs Pome to check a trajectory. Braintrust already ships agentAssertionScorer — declarative assertions over tool calls, their ordering, and a call budget, read off its own spans — and a LangSmith evaluator is handed the run, child runs included, to assert the same way. This dataset is deliberately built around the one failure that is invisible to a check written that way, whichever framework writes it. A charge for $100.00 needs a partial refund of $50.00. A failure-injection rule loses the first refund’s response after the write lands:
The refund row is written and the money moves. The caller is told the call failed. Nothing in the response distinguishes that from a request that never arrived.
  • An agent that retries the 500 lands a second refund row. The customer gets $100.00 back instead of $50.00.
  • An agent that reads the charge back first sees amount_refunded: 5000 and stops.
Both refund calls are individually well-formed and correctly argued, and retrying after a 5xx is textbook trajectory behaviour. The trajectory is clean; the money is wrong. Only the twin’s aggregate state tells the two runs apart. That is trace versus tape. A span, or a LangSmith run, is the client’s record of what the agent meant to do. The tape is the twin’s record of what it actually received. An agent can emit a perfect span for a call that never happened; it cannot produce a refund row.

​
The dataset

Six rows: three worlds × two retry policies. Both policies are things a real team writes down, and both sound reasonable. Everything else — the job, the tools, the world — is identical, and test/agent.test.ts fails if that stops being true. Neither arm is ever told that a refund can land on a 500: an agent told that would be following an instruction, and the red would be authored rather than earned.
The refund must be partial. The twin computes refundable = amount - amount_refunded and refuses anything larger, so a second full refund is rejected with charge_already_refunded: one row is ever written, the over-refund assertion passes, and the demo shows all green while demonstrating nothing. test/dataset.test.ts pins it in both examples.

​
Why not just send an Idempotency-Key?

It is the first thing anyone who knows Stripe asks, and the answer is that it works. The Stripe twin implements the real idempotency semantics, including under the injected lost response. Measured 2026-08-28: one seeded world, two sandboxes, the only difference being the header on the retry. This is what makes the dataset an exam rather than a trap. There are two correct ways out of that world — send an idempotency key on the write, or read the charge back before retrying — and only an agent that does neither lands the second row.

​
What comes back

Four criteria, four columns, one per criterion — not one aggregate. The criterion ids you send at finalize become the column names.
pome/refund-count-is-one at 66.67% is two rows out of six: the retry-on-5xx arm in the two worlds that inject a fault. Open either cell and its metadata carries the criterion’s own sentence and the reason it reached that verdict.
[code] verdicts are numbers; [model] readings are categorical. A [code] criterion is a fact about the twin’s final state reached by code, so 1 and 0 mean what a number should mean, and a criterion that could not be evaluated scores null rather than 0 — both frameworks leave a null out of that column’s average, which is the honest arithmetic for “we did not find out”. A [model] criterion is read by Pome’s narrator, which has no score authority over it: the row comes back advisory (it read the tape) or abstained (the criterion names something this run never did), and the framework carries it as a classifier. Flattening it to a number would put an opinion back on your dashboard as a score.

​
How it fits together

Two moving parts, the same two in both frameworks: the function that runs a row, and the function that renders its verdicts. That is the entire integration.

​
The task function mints the world

Braintrust calls task once per row and waits; LangSmith calls the target the same way. What happens inside is yours, and it is the same four calls either side:
source: "twin-pull" is what makes this reachable over plain HTTP: the control plane reads the tape and the final state off the live twin, so there is nothing for you to capture, gzip or upload. Two conditions — the sandbox must still be live, and the agent must actually have called the twin. Neither example takes a @pome-sh/* dependency at all; both are plain fetch against api.pome.sh/v1, so what you read is the HTTP contract rather than an SDK wrapping it. The full reference is the REST API. Three credentials reach three different places, and they are not interchangeable: The third is the key the twin expects to see inside the sandbox, the shape a real Stripe SDK would send. It does not authenticate you to twins.pome.sh: a call bearing it comes back 404 No twin pod for this session, because the proxy resolves which sandbox you mean from the bearer and only the agent_token says. Measured 2026-08-27.

​
The rendering step

Whatever your framework calls the thing that turns a verdict into a column, it sees only what the task returned — not the sandbox, not the finalize response. So the Pome evidence has to ride inside the return value: the one shape constraint the recipe is built around, the same in both.
A scorer receives input, output, expected, metadata and trace — and nothing else. A scorer that returns an array emits one column per item:
The renderer decides nothing: every verdict was already reached against the twin’s own tape and final state, and this only reshapes them for the framework. It is pure code. No model reads anything at this step, which also keeps the recipe runnable on a free account: Braintrust’s built-in models want a work email or a card on file, and a model-driven scorer would break that for anyone who signed up with a personal address.

​
Three mechanical differences, if you port one to the other

All against langsmith@0.9.0. The first two fail silently rather than throwing, and are pinned by test/langsmith-seam.test.ts, which drives a real evaluate(); the third is read off the SDK’s own source. 1. The score key field is key, not name. A copy-paste port is not rejected: coerceEvaluationResult carries an entry with no key straight through, _logEvaluationFeedback reads res.key — undefined — and hands that to createFeedback as the feedback key. The criterion’s identity is gone before the request is built and nothing throws. 2. Multiple scores: Python returns a bare list; TypeScript returns {results: [...]}. An empty envelope is also silent — _selectEvalResults reads results: [], iterates it zero times, and calls createFeedback never. No throw, no log, no feedback. The example refuses a finalize response with no criteria_breakdown for exactly that reason, and requires every row to carry at least one pome/ key rather than trusting it does. 3. maxConcurrency is the field the cap rides on. Nothing in 0.9.0 runs rows unbounded: leave every concurrency field unset and evaluate() falls back to a queue of one — sequential, “matching Python behavior” per the SDK’s own comment — so the miss costs you time, not a stampede of sandboxes. One number on maxConcurrency bounds the target and the evaluators together, and POME_EVAL_CONCURRENCY (default 2) is what the example passes through. Read off langsmith@0.9.0’s _runner.js, 2026-08-30.

​
Where the LangSmith network restriction does and does not apply

LangSmith’s “Network Access: You cannot access the internet from a code evaluator” binds their online / UI-defined code evaluators — the ones that run in LangSmith’s cloud, limited to stdlib plus numpy, pandas, jsonschema, scipy and scikit-learn, written inline in the UI. SDK evaluators passed to evaluate() run in your own process and are unrestricted. So this recipe needs no workaround: it could call the Pome API directly from the evaluator if it wanted to. The evidence travels through the target’s return value because that is cleaner and costs no second round trip, not because it has to. If you want Pome verdicts on production traces — inside LangSmith’s online evaluators rather than an offline evaluate() run — then the constraint does bite, and the answer is the same shape: put the finished Pome report into the run’s outputs at trace time so the cloud-side evaluator can read it without a network call. That is a narrower use case, and neither example is built around it. (Read from LangSmith’s own documentation on 2026-08-27.)

​
The trace half: spans, in any OTLP collector

Everything above produces columns and no trace. Run the Braintrust example as written and the experiment summary reads:
for a run that really did call a model and three tools. Eval() traces the task function’s input and output, not what happens inside it — and on this recipe everything happens inside it. LangSmith’s evaluate() has the same boundary: it auto-traces the target function and nothing within it. Nothing about Pome changes to fix that, and none of this is Pome instrumentation. Pome’s OTLP is inbound: the twin receives calls, it does not export spans. What ships the spans is the exporter your agent already has, and where they land is a URL — two standard OpenTelemetry variables, the ones every OTel SDK reads:
That is the whole seam, and it is why this is a section of its own rather than a Braintrust feature. Point those two at a vendor, at your own collector, or at nothing — leave OTEL_EXPORTER_OTLP_ENDPOINT unset and the Braintrust example builds no tracer at all; the columns never depended on it. OTEL_SERVICE_NAME overrides the name the spans arrive under, which defaults to pome-braintrust-refund-agent.

​
Where to point them

Both Braintrust endpoints answer on /otel/v1/traces, verified 2026-08-29. The LangSmith row is read from LangSmith’s own documentation on 2026-08-27 and is not measured here; Langsmith-Project is the structural analogue of Braintrust’s x-bt-parent, and a self-hosted instance appends /api/v1/otel to the instance URL. The bundled LangSmith example ships no exporter of its own — if you want the agent’s model calls in LangSmith, langsmith/experimental/vercel’s wrapAISDK(ai) adds a run per LLM call and per tool call, which is a much richer trace and a much larger share of the free tier’s 5,000. Measured 2026-08-28 on the Braintrust example, one six-row run landed 40 spans on api.braintrust.dev: 17 ai.generateText.doGenerate (type llm), 6 ai.generateText, and 17 tool spans across get_charge, list_refunds and create_refund. npm start prints Exporting agent spans via OTLP to … when an endpoint is configured.

​
x-bt-parent decides where they land, and the obvious value is the wrong one

The header takes three forms. They are not three spellings of one destination: Only experiment_id: reaches the experiment — which is the thing you came to fix. Point the exporter at a project and every part of it succeeds: the export is green, the spans are real, they are browsable in the project’s logs, and the experiment’s llm_calls / tool_calls / total_tokens still read zero. That looks exactly like the exporter not working. And the id does not exist until a run has created it. Eval() creates the experiment when it starts; the example reads the two variables once, before Eval() runs. So the order is:
  1. Run once with project_name: or project_id:. The spans go to the project’s logs and the experiment summary still reads zero. Expected.
  2. Take the id of the experiment that run created, from Braintrust.
  3. Set x-bt-parent=experiment_id:<id> and run again. Now the summary fills in.
A fourth value is accepted and not used here: a span slug from span.export(), which nests the trace under a span you already have.

​
The endpoint is a base, except when it is not

OTLP’s convention is that OTEL_EXPORTER_OTLP_ENDPOINT is a base and the SDK appends the signal path. The signal-specific variable is the full path instead:
Mixing the two up is not a subtle failure. POST https://api.braintrust.dev/otel answers 404 Cannot POST /otel — checked again 2026-08-29, and it is a plain Express 404, not an OTLP error. Only /otel/v1/traces is a route.

​
How this fails quietly

Not a Pro feature. Braintrust’s OTLP ingest is on no row of its plan table — what that table gates behind Pro is custom dashboards, Environments, custom retention policies, S3 export and SSO/RBAC. That is read off the pricing comparison on 2026-08-29 rather than measured, unlike the span counts above. Spans create no score column either, so the arithmetic in What it costs does not move.
The Braintrust example emits its spans through the Vercel AI SDK’s experimental_telemetry, wired to a tracer that only exists when an endpoint was configured. Swap src/agent.ts for your own agent and whatever it already uses to emit spans is what ships them. The two variables are the whole configuration surface, on either agent.

​
What it costs

On the Pome side, one sandbox per row — and a sandbox is the billing unit. POME_EVAL_CONCURRENCY (default 2) caps how many are open at once, in both examples. The metered unit on the other side is not the same one, so the arithmetic is worth doing before you point a 500-row dataset at either:
Scores are what a per-criterion column design consumes: rows × numeric columns. A 20-row dataset with 5 criteria spends 100. Starter includes 10,000 scores a month.Measured after the six-row run above, Braintrust’s own meter read 24 of 10,000 — six rows times the four numeric columns. There were five Pome columns per row, so the categorical [model] column is not metered as a score. Model credits stayed at $0 of $10, because nothing here calls a model on Braintrust’s side.
Braintrust’s Starter plan also includes 1 GB of processed data and 14-day retention, and gates Sandbox evals, Environments and extended retention behind Pro. The recipe stays inside Starter on purpose — it runs Eval() locally and scores with code.

​
If something goes wrong

401 invalid_auth on the first call. POME_API_KEY is not set, or you gave it the sandbox’s agent_token. They are different credentials for different hosts: the pme_ key reaches api.pome.sh/v1, and the agent_token reaches the twins on twins.pome.sh. A call to the control plane bearing an agent_token does not work, and neither does the reverse. 402 quota_exceeded. Too many sandboxes open at once. Lower POME_EVAL_CONCURRENCY. Every criterion came back skipped, and the row looks blank. The world did not arrive. The hosted door’s pinned Stripe seed parser is not strict yet, so a mistyped top-level key is dropped in silence and POST /v1/seeds/validate answers valid: true for a seed that will boot an empty world. (The twins themselves now refuse an unknown key — F-1689, and the local CLI’s twin start --seed already does — the hosted refusal arrives when the parser pin moves, F-1775.) The Stripe twin’s default world is empty, so there is no fallback state — every row must seed. Both examples read the charge back before starting the agent and refuse the row if it is not there; do the same in yours. Your harness rejects the mint. POST /v1/sandboxes answers 201, not 200. A check written as status === 200 fails on a sandbox that was created correctly. 409 capture_incomplete at finalize. The tape was empty — the agent never called the twin — or the sandbox was already torn down. The tape lives in the sandbox and does not survive teardown, so finalize while it is still live. The run is there but the task column reads unknown. task_source is optional on the mint, but both examples send the real task on purpose: it is what finalize grades against, and it is what names the run in the dashboard. A mint with a seed and no task is legal and starts the same world — it just produces a run nothing can grade.

​
Put it in your own eval

Nothing above needs either example directory. To add Pome columns to an eval you already have:
  1. Give each row a world. Add the four calls to your existing task or target function. Your agent needs two values out of the mint — per_twin.<twin>.api_url and agent_token — and nothing else changes about how you run it.
  2. Return the evidence in the output. A scorer or evaluator cannot see anything else the task produced, so the verdicts have to ride along in the return value.
  3. Add one renderer. Braintrust: a scorer that returns the array. LangSmith: an evaluator that returns {results: [...]}. One column, or one feedback key, appears per criterion.
src/pome.ts is the only Pome-specific file in either example — mint, assert, drive, finalize, stop, in about 450 lines of plain fetch with no @pome-sh/* dependency, and the same job on both sides: the two copies differ only in their comments and in the two renderer-facing field names. Copy it, or write your own against the REST API; it is a small enough surface that either is reasonable. If instead you want to start from an example and reshape it: swap src/agent.ts for your own agent — it takes a base URL and a bearer token, and Pome grades what the agent did to the twin, not how it was built. src/dataset.ts carries the worlds and src/task.ts the criteria. To write criteria that bind, start from the closed set each twin declares: GET /v1/checks serves the vocabulary and POST /v1/checks/render turns a check plus arguments into the exact sentence the grader binds against. A sentence that does not bind is neither a pass nor a fail.

REST API

The control plane both recipes call, endpoint by endpoint.

Write a task

[code] versus [model], and the checks each twin declares.

Stripe twin

What the Stripe twin serves, and the shape of its world.

Cross-twin consistency

The other worked example — one action, two systems, one of them silent.