What counts as an agent eval

Pome meters two things, and they are easy to confuse because both are counted per calendar month and both reset on the 1st at 00:00 UTC.

Meter	What consumes one	Free plan
Agent evals	one finalized, graded simulation run	100 / month
MCP / API calls	one request your agent makes through a twin	5,000 / month

A single eval typically spends many MCP / API calls — every tool call the agent under test makes against a twin is one of the latter. This page is about the first meter.

The rule

One agent eval is charged when a simulation run is graded at finalize, and at no other moment.

Nothing before finalize costs an eval. Starting a session, provisioning twins, launching your agent, watching it work, reading its trace — all free. The charge lands when the run is frozen and scored, because that is the moment Pome produces the thing you came for: a verdict backed by evidence.

What is free

The short version: a run you get no verdict from is never charged. Concretely —

Situation	Charged?	Why
You stop a session without grading it	No	No run is recorded. `stop_session` discards the run rather than scoring it.
The session expires unused	No	Same — nothing was graded.
Finalize fails on our side	No	The tape could not be captured, the trace was unreadable, or the judge errored. No run row is written, so nothing is counted.
You retry the same finalize after a timeout	No	One evaluation per run. A retry short-circuits onto the run that already exists; it never grades or bills twice.
The run tripped the per-run judge cost ceiling	No	You get a 402 and no verdict, so you are not charged. The run is still stored, marked unscored, so a retry does not re-invoke the judge.
Every criterion came back unscored	No	Whether our judge was unavailable or your `[code]` criteria matched no declared check, the run told you nothing about your agent. One criterion coming back passed or failed is enough to make the run a graded one — a partial grade is still a grade.
Replaying a production run against a twin	No	A replay launches no agent and calls no judge — it re-drives a recorded call sequence to see whether the failure reproduces.
Production telemetry (OTLP) arriving in Pome	No	Ingesting your production traces is not a simulation run. It is on a separate path and never touches this meter.

What is charged

Situation	Charged	Why
A run you finalize and get a report for	1 eval	This is the unit.
Running the same task again after a prompt fix	1 eval	A rerun is a new run: a new sandbox, a new agent attempt, a new judge call, and — the point of running it — a new independent verdict.
`run_trials n`	n evals	Each trial is its own session and is finalized separately, so `n` trials cost `n` evals. Seeing how much your agent’s behaviour varies between identical attempts is what you are buying.

On the free plan run_trials is pinned to one trial per task, so a run there always costs exactly one eval. The response tells you when it clamped a larger request.

What happens at the limit

On the free plan there is no overage. When the month’s 100 evals are used, the next finalize is refused with HTTP 402 — it does not silently bill, and it does not degrade to a cheaper grade. The allowance resets on the 1st. On paid plans, evals beyond the included allowance depend on overage billing being enabled on your subscription. Because a refused finalize happens after your agent has already done the work, the number is quoted up front instead:

Coach / MCP — run_task and run_trials return an eval_cost block with what the run will consume and what is left. get_platform_context reports both meters.
Dashboard — the running-sessions view and Usage show the same reading.

Both read the same counter the control plane enforces on, so the number you are quoted before a run is the number you are charged against.

Checking your usage

Dashboard → Usage shows used / quota / remaining for both meters and the reset date.
Over MCP, get_platform_context returns the same snapshot as entitlements.

Start

Guides

Examples

CLI

MCP

What counts as an agent eval

The rule

What is free

What is charged

What happens at the limit

Checking your usage

​The rule

​What is free

​What is charged

​What happens at the limit

​Checking your usage

The rule

What is free

What is charged

What happens at the limit

Checking your usage