> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pome.sh/llms.txt
> Use this file to discover all available pages before exploring further.
# What counts as an agent eval
> The exact rule for what consumes your monthly agent-eval allowance, when it is charged, and what is free.
Pome meters two things, and they are easy to confuse because both are counted per
calendar month and both reset on the 1st at 00:00 UTC.
| Meter | What consumes one | Free plan |
| ------------------- | ------------------------------------------- | ------------- |
| **Agent evals** | one finalized, graded simulation run | 100 / month |
| **MCP / API calls** | one request your agent makes through a twin | 5,000 / month |
A single eval typically spends many MCP / API calls — every tool call the agent under
test makes against a twin is one of the latter. This page is about the first meter.
## The rule
One agent eval is charged when a simulation run is **graded at finalize**, and at
no other moment.
Nothing before finalize costs an eval. Starting a session, provisioning twins,
launching your agent, watching it work, reading its trace — all free. The charge
lands when the run is frozen and scored, because that is the moment Pome produces
the thing you came for: a verdict backed by evidence.
## What is free
The short version: **a run you get no verdict from is never charged.** Concretely —
| Situation | Charged? | Why |
| ---------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| You stop a session without grading it | **No** | No run is recorded. `stop_session` discards the run rather than scoring it. |
| The session expires unused | **No** | Same — nothing was graded. |
| Finalize fails on our side | **No** | The tape could not be captured, the trace was unreadable, or the judge errored. No run row is written, so nothing is counted. |
| You retry the same finalize after a timeout | **No** | One evaluation per run. A retry short-circuits onto the run that already exists; it never grades or bills twice. |
| The run tripped the per-run judge cost ceiling | **No** | You get a 402 and no verdict, so you are not charged. The run is still stored, marked unscored, so a retry does not re-invoke the judge. |
| Every criterion came back unscored | **No** | Whether our judge was unavailable or your `[code]` criteria matched no declared check, the run told you nothing about your agent. One criterion coming back passed or failed is enough to make the run a graded one — a partial grade is still a grade. |
| Replaying a production run against a twin | **No** | A replay launches no agent and calls no judge — it re-drives a recorded call sequence to see whether the failure reproduces. |
| Production telemetry (OTLP) arriving in Pome | **No** | Ingesting your production traces is not a simulation run. It is on a separate path and never touches this meter. |
## What is charged
| Situation | Charged | Why |
| ---------------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A run you finalize and get a report for | **1 eval** | This is the unit. |
| Running the same task again after a prompt fix | **1 eval** | A rerun is a new run: a new sandbox, a new agent attempt, a new judge call, and — the point of running it — a new independent verdict. |
| `run_trials n` | **n evals** | Each trial is its own session and is finalized separately, so `n` trials cost `n` evals. Seeing how much your agent's behaviour varies between identical attempts is what you are buying. |
On the free plan `run_trials` is pinned to one trial per task, so a run there always
costs exactly one eval. The response tells you when it clamped a larger request.
## What happens at the limit
On the **free plan there is no overage**. When the month's 100 evals are used, the
next finalize is refused with HTTP 402 — it does not silently bill, and it does not
degrade to a cheaper grade. The allowance resets on the 1st.
On paid plans, evals beyond the included allowance depend on overage billing being
enabled on your subscription.
Because a refused finalize happens *after* your agent has already done the work, the
number is quoted up front instead:
* **Coach / MCP** — `run_task` and `run_trials` return an `eval_cost` block with what
the run will consume and what is left. `get_platform_context` reports both meters.
* **Dashboard** — the running-sessions view and **Usage** show the same reading.
Both read the same counter the control plane enforces on, so the number you are quoted
before a run is the number you are charged against.
## Checking your usage
* Dashboard → **Usage** shows used / quota / remaining for both meters and the reset
date.
* Over MCP, `get_platform_context` returns the same snapshot as `entitlements`.