> ## Documentation Index > Fetch the complete documentation index at: https://docs.pome.sh/llms.txt > Use this file to discover all available pages before exploring further. # Use your existing eval stack > Braintrust, LangSmith and OTLP over one recipe: every dataset row gets its own seeded digital twin, and every Pome criterion comes back as its own column. A refund agent whose trajectory is clean and whose money is wrong. Braintrust and LangSmith run your eval. Pome is what your agent **calls** during it. Both of them sell something called a *sandbox*, and neither one is Pome's. Theirs runs your code — the dataset loop, the scorers, and in LangSmith's case containers for code an agent wrote. Pome's is a **digital twin** of the SaaS APIs your agent talks to: a stateful emulation of Stripe, GitHub, Slack, Gmail or Linear that answers the same REST calls, boots from a starting state you declare, and remembers every request it received. That distinction is the whole recipe. Your dataset has rows; this gives each row its own **world**, and grades what your agent did to it. Three answers to one question, and the middle of the three is the same in both: * **Braintrust** — one score column per criterion, out of `Eval()`. * **LangSmith** — one feedback key per criterion, out of `evaluate()`. * **[Your agent's spans](#the-trace-half-spans-in-any-otlp-collector)** — the framework-agnostic seam. Two standard OpenTelemetry variables, and the spans land wherever you point them. The first two are one recipe with two renderers, so this page carries them together: [pick a framework below](#run-one-of-the-two-examples) and the tabs follow it through. Everything untabbed is true of both. ## The shape One dataset row becomes one isolated world: ```text theme={"dark"} row → POST /v1/sandboxes (twins + your seed) → your agent drives the twin → POST /v1/sandboxes/:id/finalize → one column per criterion ``` Six rows, six sandboxes, six graded runs. Each row carries its own charge, its own amount, and its own injected fault — not one world reused six times. ## Run one of the two examples Both examples are bundled with the CLI and fetched by **id**, not by a typed GitHub path that can rot into a 404. [`pome init --example `](/docs/cli/init#start-from-an-example) writes the whole tree into `./` and nothing else: it does not touch a manifest you already have. It needs `@pome-sh/cli` **0.35.0 or later**, which is what `@latest` below gets you. ```bash theme={"dark"} npx @pome-sh/cli@latest init --example braintrust cd braintrust npm install ``` You need three credentials. Two are yours already if you are reading this: | | What it is | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `BRAINTRUST_API_KEY` | Your Braintrust key. **Optional** — without it the eval still runs and prints a local summary instead of creating an experiment. | | `POME_API_KEY` | A Pome **team** key (`pme_…`), from the [dashboard](https://app.pome.sh) or `pome login`. | | `ANTHROPIC_API_KEY` | The bundled agent runs on the Vercel AI SDK. Override the model with `POME_AGENT_MODEL`. | ```bash theme={"dark"} export BRAINTRUST_API_KEY=… export POME_API_KEY=pme_… export ANTHROPIC_API_KEY=… npm start ``` `npm start` runs `Eval()` in-process through `tsx`. It does **not** read a `.env` file — export the three variables, or the run reaches `401 invalid_auth`. Braintrust's own `bt eval` CLI does load `.env`; this example does not go through it, because Braintrust's Sandbox evals are a Pro feature and the recipe must stay runnable on Starter. On npm 11 the install prints `allow-scripts` warnings for `braintrust`, `esbuild` and `fsevents`. They are expected and nothing here needs those scripts — the measured run below was produced in exactly that state. `npm test` is fully offline. It needs no account and no credentials, and it includes a real `Eval()` run in Braintrust's local `noSendLogs` mode. ```bash theme={"dark"} npx @pome-sh/cli@latest init --example langsmith cd langsmith npm install ``` Three credentials, and unlike the Braintrust variant none of them is optional: | | What it is | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `LANGSMITH_API_KEY` | From [smith.langchain.com/settings](https://smith.langchain.com/settings). The legacy `LANGCHAIN_API_KEY` works too; the SDK reads `LANGSMITH_* \|\| LANGCHAIN_*`. | | `POME_API_KEY` | A Pome **team** key (`pme_…`), from the [dashboard](https://app.pome.sh) or `pome login`. | | `ANTHROPIC_API_KEY` | The bundled agent runs on the Vercel AI SDK. Override the model with `POME_AGENT_MODEL`. | ```bash theme={"dark"} export LANGSMITH_API_KEY=… export POME_API_KEY=pme_… export ANTHROPIC_API_KEY=… npm start ``` **There is no local-only mode.** `evaluate()` calls `client.createProject()` inside its own `start()`, *before the first prediction*, so a LangSmith key is required where Braintrust's is not. The upside of that ordering is that a missing key costs nothing — it fails before any sandbox is minted, and the example says so in a sentence rather than letting a bare `401` do it. The example uploads its dataset before `evaluate()` runs, because `evaluate()` serves examples out of LangSmith's own store rather than out of an array. **The dataset name carries a digest of the row set**, so a reader who adds a world and re-runs is not quietly served the old rows under the same name. `npm test` is hermetic — no network, no credentials. **You do not need either directory to use Pome from your eval.** They exist so you can watch the thing work before you touch the eval you care about. If you already have an `Eval()` or an `evaluate()`, skip to [how it fits together](#how-it-fits-together) — the integration is two functions, and they drop into what you already have. **What has been measured, and what has not.** The Braintrust half of this page was walked end to end on 2026-08-28 on a **\$0 Braintrust Starter account with no card on file**; every number under it is from that run. The Pome half is the same code in both examples — `src/pome.ts`, copied, with both test suites pinning it case for case — and it was verified against `api.pome.sh` on 2026-08-27, three times, same split each run. **The LangSmith half has not been run against a live LangSmith account.** Its seam is verified against the real SDK by `test/langsmith-seam.test.ts`, which drives an actual `evaluate()` against a stub client, and its plan and OTLP facts are read from LangSmith's own documentation on 2026-08-27. Where a LangSmith number below is expected rather than observed, it says so. ## The failure both examples demonstrate Neither framework needs Pome to check a trajectory. Braintrust already ships [`agentAssertionScorer`](https://www.braintrust.dev/docs/evaluate/custom-code) — declarative assertions over tool calls, their ordering, and a call budget, read off its own spans — and a LangSmith evaluator is handed the run, child runs included, to assert the same way. This dataset is deliberately built around the one failure that is **invisible** to a check written that way, whichever framework writes it. A charge for \$100.00 needs a **partial** refund of \$50.00. A failure-injection rule loses the first refund's response *after* the write lands: ```json theme={"dark"} { "method": "POST", "path": "/v1/refunds", "attempt": 1, "mode": "after_handler", "status": 500 } ``` The refund row is written and the money moves. The caller is told the call failed. Nothing in the response distinguishes that from a request that never arrived. * An agent that retries the 500 lands a **second** refund row. The customer gets \$100.00 back instead of \$50.00. * An agent that reads the charge back first sees `amount_refunded: 5000` and stops. Both refund calls are individually well-formed and correctly argued, and retrying after a 5xx is textbook trajectory behaviour. **The trajectory is clean; the money is wrong.** Only the twin's aggregate state tells the two runs apart. That is *trace* versus *tape*. A span, or a LangSmith run, is the client's record of what the agent meant to do. The tape is the twin's record of what it actually received. An agent can emit a perfect span for a call that never happened; it cannot produce a refund row. ### The dataset Six rows: three worlds × two retry policies. Both policies are things a real team writes down, and both sound reasonable. | Arm | The one sentence that differs | | ------------------- | ------------------------------------------------------------------------------------------------------ | | `retry-on-5xx` | "If a call comes back 5xx, retry it once — a 5xx means the request did not go through." | | `verify-then-retry` | "…but a 5xx on a **write** does not tell you whether the write landed, so read the object back first." | Everything else — the job, the tools, the world — is identical, and `test/agent.test.ts` fails if that stops being true. Neither arm is ever told that a refund can land on a 500: an agent told that would be following an instruction, and the red would be authored rather than earned. | World | Charge | Refund | First response lost? | | ------------------ | ----------------------- | ------- | -------------------- | | `duplicate-charge` | `ch_test_200`, \$100.00 | \$50.00 | yes | | `cancelled-add-on` | `ch_test_318`, \$75.00 | \$25.00 | yes | | `goodwill-credit` | `ch_test_442`, \$42.00 | \$10.00 | no — the control | **The refund must be partial.** The twin computes `refundable = amount - amount_refunded` and refuses anything larger, so a second **full** refund is rejected with `charge_already_refunded`: one row is ever written, the over-refund assertion passes, and the demo shows all green while demonstrating nothing. `test/dataset.test.ts` pins it in both examples. ### Why not just send an Idempotency-Key? It is the first thing anyone who knows Stripe asks, and the answer is that it **works**. The Stripe twin implements the real idempotency semantics, including under the injected lost response. Measured 2026-08-28: one seeded world, two sandboxes, the only difference being the header on the retry. | Retry | Refund rows | `amount_refunded` | | ------------------------------- | ----------- | ----------------- | | with the same `Idempotency-Key` | 1 | 5000 | | without it | 2 | 10000 | This is what makes the dataset an exam rather than a trap. There are **two** correct ways out of that world — send an idempotency key on the write, or read the charge back before retrying — and only an agent that does neither lands the second row. ## What comes back Four criteria, four columns, one per criterion — not one aggregate. The criterion ids you send at finalize become the column names. ```text theme={"dark"} 6 rows → 6 Pome sandboxes (group bteval-mtcnj9u7), 2 at a time. ── duplicate-charge · verify-then-retry — Pome scored it 100/100 PASS pome/refund-exists charge "ch_test_200" has 1 refund row(s) PASS pome/refund-count-is-one charge "ch_test_200" has 1 refund row(s), wanted 1 PASS pome/charge-succeeded 1 of 1 charge(s) have status "succeeded" advisory pome/checked-before-retrying The agent attempted to create a refund … ── duplicate-charge · retry-on-5xx — Pome scored it 67/100 PASS pome/refund-exists charge "ch_test_200" has 2 refund row(s) FAIL pome/refund-count-is-one charge "ch_test_200" has 2 refund row(s), wanted 1 PASS pome/charge-succeeded 1 of 1 charge(s) have status "succeeded" advisory pome/checked-before-retrying The agent made a POST request to create a refund … Experiment summary ================== pome/charge-succeeded 100.00% pome/refund-count-is-one 66.67% pome/refund-exists 100.00% pome/run-score 89.00% duration 32.82s ``` | Column | Kind | What it says | | ------------------------------ | --------------- | --------------------------------------------------------------------------------------------------------- | | `pome/refund-exists` | numeric | A refund exists on the charge. **Passes for the careless agent too** — two rows are still "at least one". | | `pome/refund-count-is-one` | numeric | **The red.** Exactly one refund row. The only column a double refund fails. | | `pome/charge-succeeded` | numeric | A charge exists with status `succeeded`. | | `pome/checked-before-retrying` | **categorical** | What Pome's narrator read in the tape about the agent's method. | | `pome/run-score` | numeric | Pome's own 0–100 for the run, ÷ 100. A convenience for sorting. | `pome/refund-count-is-one` at 66.67% is two rows out of six: the `retry-on-5xx` arm in the two worlds that inject a fault. Open either cell and its `metadata` carries the criterion's own sentence and the reason it reached that verdict. `evaluate()` prints the experiment name and a compare URL and nothing else — no scores, no categoricals. So this example prints its own per-row report and summary, or the one thing it is about would be visible only in a browser. Expected shape: ```text theme={"dark"} created LangSmith dataset "pome-lost-response-double-refund-", uploaded 6 row(s). 6 rows → 6 Pome sandboxes (group lseval-), 2 at a time. Starting evaluation of experiment: pome-refund-agent- View results at https://smith.langchain.com/o/…/datasets/…/compare?selectedSessions=… ── duplicate-charge · retry-on-5xx — Pome scored it 67/100 PASS pome/refund-exists charge "ch_test_200" has 2 refund row(s) FAIL pome/refund-count-is-one charge "ch_test_200" has 2 refund row(s), wanted 1 PASS pome/charge-succeeded 1 of 1 charge(s) have status "succeeded" advisory pome/checked-before-retrying 1. The agent made a POST request to '/v1/refunds' … https://app.pome.sh/runs/run_… Experiment summary ================== pome/refund-exists 100.00% n=6 pome/refund-count-is-one 66.67% n=6 pome/charge-succeeded 100.00% n=6 pome/checked-before-retrying advisory 4, abstained 2 pome/run-score 89.00% n=6 ``` | Feedback key | Kind | What it says | | ------------------------------ | --------------- | --------------------------------------------------------------------------------------------------------- | | `pome/refund-exists` | numeric | A refund exists on the charge. **Passes for the careless agent too** — two rows are still "at least one". | | `pome/refund-count-is-one` | numeric | **The red.** Exactly one refund row. The only key a double refund fails. | | `pome/charge-succeeded` | numeric | A charge exists with status `succeeded`. | | `pome/checked-before-retrying` | **categorical** | What Pome's narrator read in the tape about the agent's method. | | `pome/run-score` | numeric | Pome's own 0–100 for the run, ÷ 100. A convenience for sorting. | `pome/refund-count-is-one` at 66.67% would be the two `retry-on-5xx` rows in the two injected worlds. The control world comes back green on both arms, and its `[model]` reading `abstained` — no refund call failed there, so there is nothing for the narrator to read. **`[code]` verdicts are numbers; `[model]` readings are categorical.** A `[code]` criterion is a fact about the twin's final state reached by code, so `1` and `0` mean what a number should mean, and a criterion that could not be evaluated scores `null` rather than `0` — both frameworks leave a null out of that column's average, which is the honest arithmetic for "we did not find out". A `[model]` criterion is *read* by Pome's narrator, which has no score authority over it: the row comes back `advisory` (it read the tape) or `abstained` (the criterion names something this run never did), and the framework carries it as a classifier. Flattening it to a number would put an opinion back on your dashboard as a score. ## How it fits together Two moving parts, the same two in both frameworks: the function that runs a row, and the function that renders its verdicts. That is the entire integration. ### The task function mints the world Braintrust calls `task` once per row and waits; LangSmith calls the target the same way. What happens inside is yours, and it is the same four calls either side: ```text theme={"dark"} POST /v1/seeds/validate does this world parse? free, nothing provisioned POST /v1/sandboxes twins + seed + task_source → session_id, agent_token, per_twin GET /v1/charges/:id did the world actually ARRIVE? POST /v1/sandboxes/:id/finalize source: "twin-pull" → run_id, score, criteria_breakdown ``` `source: "twin-pull"` is what makes this reachable over plain HTTP: the control plane reads the tape and the final state off the live twin, so there is nothing for you to capture, gzip or upload. Two conditions — the sandbox must still be live, and the agent must actually have called the twin. Neither example takes a `@pome-sh/*` dependency at all; both are plain `fetch` against `api.pome.sh/v1`, so what you read is the HTTP contract rather than an SDK wrapping it. The full reference is [the REST API](/docs/api). Three credentials reach three different places, and they are not interchangeable: | | Reaches | Give it to | | ------------------------------------- | ---------------------------- | ------------ | | `POME_API_KEY` (`pme_…`) | `api.pome.sh/v1` | your harness | | `agent_token` | the twins on `twins.pome.sh` | your agent | | `provider_credentials.stripe.api_key` | nothing, on its own | — | The third is the key the twin expects to *see* inside the sandbox, the shape a real Stripe SDK would send. It does not authenticate you to `twins.pome.sh`: a call bearing it comes back `404 No twin pod for this session`, because the proxy resolves which sandbox you mean from the bearer and only the `agent_token` says. Measured 2026-08-27. ### The rendering step Whatever your framework calls the thing that turns a verdict into a column, it sees only what the task **returned** — not the sandbox, not the finalize response. So the Pome evidence has to ride inside the return value: the one shape constraint the recipe is built around, the same in both. A scorer receives `input`, `output`, `expected`, `metadata` and `trace` — and nothing else. A scorer that returns an **array** emits one column per item: ```ts theme={"dark"} // task return { summary, pome }; // every verdict rides along in `output` // one column per [code] criterion — each entry is {name, score} export function pomeCriteria({ output }) { return output.pome.scores; } // Eval() takes the two kinds of verdict on two channels scores: [pomeCriteria, pomeRunScore], classifiers: [pomeNarratorReadings], ``` An evaluator receives the run's `outputs`, so the verdicts ride in the target's return value the same way. A TypeScript evaluator returning several results returns the **envelope**, not a bare array: ```ts theme={"dark"} // target return { answer, pome }; // every verdict rides along in the outputs // one evaluator, one feedback key per entry — each carries `key`, not `name` export function pomeVerdicts({ outputs }) { return { results: [...outputs.pome.scores, ...outputs.pome.readings] }; } ``` **One evaluator rather than two, deliberately.** LangSmith has no separate classifier channel: a categorical is a feedback entry carrying `value` where a score carries `score`, so there is nothing structural to split along. And `_runEvaluators` wraps *each* evaluator in its own `traceable` — a second one is a second traced run per row, which on a 5,000-trace tier costs twice as much for no extra information. The renderer decides nothing: every verdict was already reached against the twin's own tape and final state, and this only reshapes them for the framework. **It is pure code.** No model reads anything at this step, which also keeps the recipe runnable on a free account: Braintrust's built-in models want a work email or a card on file, and a model-driven scorer would break that for anyone who signed up with a personal address. ### Three mechanical differences, if you port one to the other All against `langsmith@0.9.0`. The first two fail **silently** rather than throwing, and are pinned by `test/langsmith-seam.test.ts`, which drives a real `evaluate()`; the third is read off the SDK's own source. **1. The score key field is `key`, not `name`.** A copy-paste port is not rejected: `coerceEvaluationResult` carries an entry with no `key` straight through, `_logEvaluationFeedback` reads `res.key` — `undefined` — and hands that to `createFeedback` as the feedback key. The criterion's identity is gone before the request is built and nothing throws. **2. Multiple scores: Python returns a bare list; TypeScript returns `{results: [...]}`.** An empty envelope is also silent — `_selectEvalResults` reads `results: []`, iterates it zero times, and calls `createFeedback` never. No throw, no log, no feedback. The example refuses a finalize response with no `criteria_breakdown` for exactly that reason, and requires every row to carry at least one `pome/` key rather than trusting it does. **3. `maxConcurrency` is the field the cap rides on.** Nothing in 0.9.0 runs rows unbounded: leave every concurrency field unset and `evaluate()` falls back to a queue of one — sequential, "matching Python behavior" per the SDK's own comment — so the miss costs you time, not a stampede of sandboxes. One number on `maxConcurrency` bounds the target and the evaluators together, and `POME_EVAL_CONCURRENCY` (default 2) is what the example passes through. Read off `langsmith@0.9.0`'s `_runner.js`, 2026-08-30. ### Where the LangSmith network restriction does and does not apply LangSmith's *"Network Access: You cannot access the internet from a code evaluator"* binds their **online / UI-defined** code evaluators — the ones that run in LangSmith's cloud, limited to stdlib plus numpy, pandas, jsonschema, scipy and scikit-learn, written inline in the UI. **SDK evaluators passed to `evaluate()` run in your own process and are unrestricted.** So this recipe needs no workaround: it could call the Pome API directly from the evaluator if it wanted to. The evidence travels through the target's return value because that is cleaner and costs no second round trip, not because it has to. If you want Pome verdicts on production traces — inside LangSmith's online evaluators rather than an offline `evaluate()` run — then the constraint does bite, and the answer is the same shape: put the finished Pome report into the run's outputs at trace time so the cloud-side evaluator can read it without a network call. That is a narrower use case, and neither example is built around it. (Read from LangSmith's own documentation on 2026-08-27.) ## The trace half: spans, in any OTLP collector Everything above produces columns and no trace. Run the Braintrust example as written and the experiment summary reads: ```text theme={"dark"} llm_calls 0 · tool_calls 0 · total_tokens 0tok ``` for a run that really did call a model and three tools. `Eval()` traces the task function's **input and output**, not what happens inside it — and on this recipe everything happens inside it. LangSmith's `evaluate()` has the same boundary: it auto-traces the target function and nothing within it. **Nothing about Pome changes to fix that, and none of this is Pome instrumentation.** Pome's OTLP is *inbound*: the twin receives calls, it does not export spans. What ships the spans is the exporter your agent already has, and where they land is a URL — two standard OpenTelemetry variables, the ones every OTel SDK reads: ```bash theme={"dark"} export OTEL_EXPORTER_OTLP_ENDPOINT=… export OTEL_EXPORTER_OTLP_HEADERS="…" ``` That is the whole seam, and it is why this is a section of its own rather than a Braintrust feature. Point those two at a vendor, at your own collector, or at nothing — leave `OTEL_EXPORTER_OTLP_ENDPOINT` unset and the Braintrust example builds no tracer at all; the columns never depended on it. `OTEL_SERVICE_NAME` overrides the name the spans arrive under, which defaults to `pome-braintrust-refund-agent`. ### Where to point them | Destination | `OTEL_EXPORTER_OTLP_ENDPOINT` | The header that routes it | | ------------------------ | -------------------------------------------- | ------------------------------------------------------------ | | Braintrust (US) | `https://api.braintrust.dev/otel` | `Authorization=Bearer , x-bt-parent=experiment_id:` | | Braintrust (EU) | `https://api-eu.braintrust.dev/otel` | same | | Braintrust (self-hosted) | your stack's Universal API URL, plus `/otel` | same | | LangSmith | `https://api.smith.langchain.com/otel/` | `x-api-key=, Langsmith-Project=` | | Anything else | your collector's OTLP endpoint | whatever it authenticates with | Both Braintrust endpoints answer on `/otel/v1/traces`, verified 2026-08-29. The LangSmith row is read from LangSmith's own documentation on 2026-08-27 and is **not** measured here; `Langsmith-Project` is the structural analogue of Braintrust's `x-bt-parent`, and a self-hosted instance appends `/api/v1/otel` to the instance URL. The bundled LangSmith example ships no exporter of its own — if you want the agent's model calls in LangSmith, `langsmith/experimental/vercel`'s `wrapAISDK(ai)` adds a run per LLM call and per tool call, which is a much richer trace and a much larger share of the free tier's 5,000. Measured 2026-08-28 on the Braintrust example, one six-row run landed **40 spans** on `api.braintrust.dev`: 17 `ai.generateText.doGenerate` (type `llm`), 6 `ai.generateText`, and 17 tool spans across `get_charge`, `list_refunds` and `create_refund`. `npm start` prints `Exporting agent spans via OTLP to …` when an endpoint is configured. ### x-bt-parent decides where they land, and the obvious value is the wrong one The header takes three forms. They are not three spellings of one destination: | Value | Where the spans land | | --------------------- | ---------------------- | | `project_id:` | the project's **logs** | | `project_name:` | the project's **logs** | | `experiment_id:` | **the experiment** | Only `experiment_id:` reaches the experiment — which is the thing you came to fix. Point the exporter at a project and every part of it succeeds: the export is green, the spans are real, they are browsable in the project's logs, and the experiment's `llm_calls` / `tool_calls` / `total_tokens` still read zero. That looks exactly like the exporter not working. **And the id does not exist until a run has created it.** `Eval()` creates the experiment when it starts; the example reads the two variables once, before `Eval()` runs. So the order is: 1. Run once with `project_name:` or `project_id:`. The spans go to the project's logs and the experiment summary still reads zero. Expected. 2. Take the id of the experiment that run created, from Braintrust. 3. Set `x-bt-parent=experiment_id:` and run again. Now the summary fills in. A fourth value is accepted and not used here: a span slug from `span.export()`, which nests the trace under a span you already have. ### The endpoint is a base, except when it is not OTLP's convention is that `OTEL_EXPORTER_OTLP_ENDPOINT` is a **base** and the SDK appends the signal path. The signal-specific variable is the full path instead: ```bash theme={"dark"} OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.braintrust.dev/otel/v1/traces ``` Mixing the two up is not a subtle failure. `POST https://api.braintrust.dev/otel` answers **`404 Cannot POST /otel`** — checked again 2026-08-29, and it is a plain Express 404, not an OTLP error. Only `/otel/v1/traces` is a route. ### How this fails quietly | What you see | What it actually is | | -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `404 Cannot POST /otel` | A traces exporter aimed at the base. Give it `/otel/v1/traces`, or use an SDK that appends the signal path itself. | | `400 No valid spans in this request: 1 rejected, 0 valid` | **A missing `x-bt-parent`.** Braintrust requires it, and the error names *spans*, so it reads like a malformed payload. It is an absent header. | | A `400` that names no header, with headers that look correct | The header string was split on every `=` instead of the **first**. `x-bt-parent=project_id:` carries a colon in its value and `Authorization=Bearer ` can carry base64 padding; splitting more than once truncates both. | | The run finishes, the spans never arrive | The batch processor was never flushed. A `BatchSpanProcessor` holds spans until its timer fires, and the last row finishes well inside that window — `forceFlush()` then `shutdown()`, both awaited, before the process exits. | | The project's logs fill up, the experiment still reads `llm_calls 0` | `x-bt-parent` names a project. See above. | **Not a Pro feature.** Braintrust's OTLP ingest is on no row of its plan table — what that table gates behind Pro is custom dashboards, Environments, custom retention policies, S3 export and SSO/RBAC. That is read off the pricing comparison on 2026-08-29 rather than measured, unlike the span counts above. Spans create no score column either, so the arithmetic in [What it costs](#what-it-costs) does not move. The Braintrust example emits its spans through the Vercel AI SDK's `experimental_telemetry`, wired to a tracer that only exists when an endpoint was configured. Swap `src/agent.ts` for your own agent and whatever it already uses to emit spans is what ships them. The two variables are the whole configuration surface, on either agent. ## What it costs On the Pome side, one sandbox per row — and a sandbox is the billing unit. `POME_EVAL_CONCURRENCY` (default 2) caps how many are open at once, in both examples. The metered unit on the other side is not the same one, so the arithmetic is worth doing before you point a 500-row dataset at either: **Scores** are what a per-criterion column design consumes: `rows × numeric columns`. A 20-row dataset with 5 criteria spends 100. Starter includes **10,000 scores a month**. Measured after the six-row run above, Braintrust's own meter read **24 of 10,000** — six rows times the four numeric columns. There were five Pome columns per row, so the categorical `[model]` column is not metered as a score. Model credits stayed at \$0 of \$10, because nothing here calls a model on Braintrust's side. Braintrust's Starter plan also includes 1 GB of processed data and 14-day retention, and gates Sandbox evals, Environments and extended retention behind Pro. The recipe stays inside Starter on purpose — it runs `Eval()` locally and scores with code. **Traces** are the binding limit. The **Developer** plan is \$0 and 1 seat, and includes **5,000 base traces a month**; with no payment method on file that is a hard stop rather than pay-as-you-go. Tracing, datasets, and both offline and online evals are all on it; Deployment, Engine and Tuned Evaluators are not. One traced run for the target plus one for the evaluator is **about two per row**, so six rows is roughly a dozen. The agent's own model calls are not traced by default — `evaluate()` auto-traces the target function and nothing inside it. **Experiment runs are created at extended retention (400 days) by default**, which is the more expensive tier — eval traffic is not billed like ordinary base traces. Fine at recipe scale; worth knowing before a 500-row dataset. Plan facts read from LangSmith's own documentation on 2026-08-27, not measured. ## If something goes wrong **`401 invalid_auth` on the first call.** `POME_API_KEY` is not set, or you gave it the sandbox's `agent_token`. They are different credentials for different hosts: the `pme_` key reaches `api.pome.sh/v1`, and the `agent_token` reaches the twins on `twins.pome.sh`. A call to the control plane bearing an `agent_token` does not work, and neither does the reverse. **`402 quota_exceeded`.** Too many sandboxes open at once. Lower `POME_EVAL_CONCURRENCY`. **Every criterion came back skipped, and the row looks blank.** The world did not arrive. The hosted door's pinned Stripe seed parser is not strict yet, so a mistyped top-level key is dropped in silence and `POST /v1/seeds/validate` answers `valid: true` for a seed that will boot an empty world. (The twins themselves now refuse an unknown key — [F-1689](https://linear.app/pome-sh/issue/F-1689), and the local CLI's `twin start --seed` already does — the hosted refusal arrives when the parser pin moves, [F-1775](https://linear.app/pome-sh/issue/F-1775).) The Stripe twin's default world is *empty*, so there is no fallback state — every row must seed. Both examples read the charge back before starting the agent and refuse the row if it is not there; do the same in yours. **Your harness rejects the mint.** `POST /v1/sandboxes` answers **`201`**, not `200`. A check written as `status === 200` fails on a sandbox that was created correctly. **`409 capture_incomplete` at finalize.** The tape was empty — the agent never called the twin — or the sandbox was already torn down. The tape lives in the sandbox and does not survive teardown, so finalize while it is still live. **The run is there but the task column reads `unknown`.** `task_source` is optional on the mint, but both examples send the real task on purpose: it is what finalize grades against, and it is what names the run in the dashboard. A mint with a seed and no task is legal and starts the same world — it just produces a run nothing can grade. ## Put it in your own eval Nothing above needs either example directory. To add Pome columns to an eval you already have: 1. **Give each row a world.** Add the four calls to your existing task or target function. Your agent needs two values out of the mint — `per_twin..api_url` and `agent_token` — and nothing else changes about how you run it. 2. **Return the evidence in the output.** A scorer or evaluator cannot see anything else the task produced, so the verdicts have to ride along in the return value. 3. **Add one renderer.** Braintrust: a scorer that returns the array. LangSmith: an evaluator that returns `{results: [...]}`. One column, or one feedback key, appears per criterion. `src/pome.ts` is the only Pome-specific file in either example — mint, assert, drive, finalize, stop, in about 450 lines of plain `fetch` with no `@pome-sh/*` dependency, and the same job on both sides: the two copies differ only in their comments and in the two renderer-facing field names. Copy it, or write your own against [the REST API](/docs/api); it is a small enough surface that either is reasonable. If instead you want to start from an example and reshape it: swap `src/agent.ts` for your own agent — it takes a base URL and a bearer token, and Pome grades what the agent *did to the twin*, not how it was built. `src/dataset.ts` carries the worlds and `src/task.ts` the criteria. To write criteria that bind, start from the closed set each twin declares: [`GET /v1/checks`](/docs/api/endpoints#get-v1checks) serves the vocabulary and [`POST /v1/checks/render`](/docs/api/endpoints#post-v1checksrender) turns a check plus arguments into the exact sentence the grader binds against. A sentence that does not bind is neither a pass nor a fail. The control plane both recipes call, endpoint by endpoint. `[code]` versus `[model]`, and the checks each twin declares. What the Stripe twin serves, and the shape of its world. The other worked example — one action, two systems, one of them silent.