> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate a model

> Measure whether a model or a checkpoint does your task, using rules, a code check, a validated judge, and paired comparisons.

An evaluation measures how well a model does your task, whether it's a model you could use as is or a checkpoint you trained. Build it before you train: the same definition of a good answer sets the bar a trained model must beat, selects training data, and decides whether a checkpoint ships.

A payments API company's support tool reads each incoming ticket and writes a JSON summary for the on-call engineer: a short `summary` and a `severity` from `sev1` to `sev4`. The tool runs on GLM 5.3 today. The team wants to know whether GLM 5.3 Flash, which costs less per token, or a smaller model they train can do the job as well.

## Prerequisites

* A [workspace API key](https://docs.baseten.co/organization/api-keys.md).
* [uv](https://docs.astral.sh/uv/) and Python 3.12 or later.
* Access to GLM 5.3 and Kimi K3 on [Model APIs](https://docs.baseten.co/inference/model-apis/overview.md).

Create the project and add the OpenAI and Pydantic packages:

```bash theme={"system"}
uv init --python 3.12 ticket-eval
cd ticket-eval
uv add openai pydantic
```

Export the key as `BASETEN_API_KEY`:

```bash macOS/Linux theme={"system"}
export BASETEN_API_KEY="YOUR_API_KEY"
```

```powershell Windows theme={"system"}
$env:BASETEN_API_KEY = "YOUR_API_KEY"
```

Every script calls Model APIs, which bill per token: each run sends every ticket to the model under test, and each summary that parses to the judge once per axis.

Download the example data, 134 synthetic tickets from 40 fictional merchant accounts, and write it as one ticket per line:

```bash theme={"system"}
curl -fL https://docs.baseten.co/assets/post-training/tickets.json -o tickets.json
uv run python -c 'import json; print("\n".join(map(json.dumps, json.load(open("tickets.json")))))' > tickets.jsonl
```

Each line of `tickets.jsonl` is one ticket, with the `account` that sent it and the `severity` it should get under the tool's severity policy:

```jsonl tickets.jsonl wrap theme={"system"}
{"id": "t-0045", "account": "granite-tickets", "severity": "sev3", "ticket": "Subject: Re: Webhooks not being sent for successful payments - partial resolution\n\nHi,\n\nJust an update on the webhook issue. We're seeing most `payment_intent.succeeded` events come through now, thanks for the quick fix. However, we're still not receiving `payment_intent.payment_failed` events for a small number of transactions that were declined. We've confirmed this by looking at the dashboard \u2013 the events show as \"skipped\" with no retry. For example, order ord_zyx98765 was declined at 16:20 UTC, and we only found out when we manually checked the dashboard. This is not a big deal since we have a nightly reconciliation script that pulls the status via the API, so we can work around it. But it would be nice to have those webhooks consistent.\n\nCould you look into why only that event type is missing? I can provide request IDs (req_a1b2c3d4e5) and the exact timestamps. Let me know if you need anything else.\n\nThanks,\nJordan from Granite Tickets"}
```

## Write down what a good answer is

Turn quality into rules a reviewer can apply to one output. Write each rule as a trigger and a required behavior, not an adjective:

| Avoid | Prefer |
| - | - |
| Summaries should be accurate. | If the summary names an order or request ID, the ID appears in the ticket. |
| Capture the key details. | The summary names the affected service, the error, and the customer impact. |
| Don't overstate the cause. | If the merchant guesses at a cause, the summary reports it as the merchant's guess. |

Collect rules from material that already encodes them, most trusted first:

* **System prompts and docs:** every must or must-not instruction is a rule someone wrote down.
* **Templates and schemas:** required fields, formats, and allowed values are stated there.
* **Production outputs:** the choices the outputs make consistently describe what production does, which isn't necessarily correct.
* **Escalations and corrections:** a complaint or a "don't do this again" note can outrank a pattern seen in hundreds of rows.

Write down what's acceptable as well as what isn't, so the evaluation doesn't punish it. For the support tool, paraphrasing the ticket is fine, and so is leaving out background the engineer doesn't need.

The support tool's system prompt holds most of its rules. Save it as `prompt.py`. The evaluation sends it with every ticket:

```python prompt.py theme={"system"}
SYSTEM_PROMPT = """You summarize support tickets for the on-call engineer at a payments API company.
Reply with only a JSON object with two keys:
- "summary": at most 120 words. Name the affected service, the error, and the customer impact.
- "severity": "sev1", "sev2", "sev3", or "sev4", following this policy:
  - sev1: payments fail, or money moves incorrectly, for many of the merchant's customers right now.
  - sev2: a core flow (checkout, payouts, or webhooks) is broken or badly degraded, with no workaround.
  - sev3: a problem with a workaround, or with limited impact.
  - sev4: a question, a feature request, or a cosmetic issue.
Never include an order, request, or account ID that isn't in the ticket."""
```

Two escalations add rules the prompt doesn't cover. An engineer spent an hour looking for an order ID that a summary invented, and another summary reported a merchant's guess about the cause as the diagnosis. Sort the rules by what can score them:

| Rule | Source | Scored by |
| - | - | - |
| The reply is a JSON object with a `summary` and an allowed `severity`. | System prompt | Code: `valid_json` |
| The summary is at most 120 words. | System prompt | Code: `length` |
| Every ID in the summary appears in the ticket. | System prompt and an escalation | Code: `ids_from_ticket` |
| The severity matches the ticket's labeled severity. | Severity policy and triage records | Code: `severity` |
| Every fact in the summary is stated in the ticket, and a merchant's guess stays a guess. | Escalation | Judge: `grounded` |
| The summary names the affected service, the error, and the customer impact. | System prompt | Judge: `coverage` |

Then decide with the task's owners which failures are worst. An invented ID or fact sends an engineer the wrong way during an incident, so for this tool `ids_from_ticket` and `grounded` outrank `length`. Where production outputs disagree on the same trigger, keep both versions of the rule and have the owners choose one. Record these calls, because they set how you weigh the scores later.

Find out where your reference outputs came from. In production, a ticket's `severity` label comes from triage; in this synthetic data, it's the severity each ticket was written to have. Either way, the `severity` check compares against that label. The tool has no verified reference summaries, only production outputs nobody checked, and scoring against those rewards matching production rather than being correct. So the judge scores each summary against the ticket and the rules, without a reference. If you need reference outputs, have a person verify a sample of them first.

## Split the data

Hold back tickets that never enter training data or prompt examples, and score every model on the same ones. Split the data three ways:

* **Training pool:** the only rows that training data comes from.
* **Dev set:** rows you read while you write rules, tune judge prompts, and choose checkpoints.
* **Holdout:** rows you score only to compare finished candidates. A score on rows you tuned against overstates how well the winner does.

Split by group, not by row. A group is whatever makes rows near-duplicates. For the support tool it's the merchant account, because an account's tickets follow up on each other and repeat the same IDs. If one account lands on both sides, the evaluation measures recall of training data instead of the task.

Save the following as `split.py`. It assigns each account by a hash of the account name and a fixed salt, so the split doesn't change between runs, and adding tickets later never moves an account to another split:

```python split.py theme={"system"}
import hashlib
import json

SALT = 20261106
HOLDOUT_FRACTION = 0.3
DEV_FRACTION = 0.2


def position(account):
    """A number in [0, 1) that depends only on the salt and the account."""
    digest = hashlib.sha256(f"{SALT}:{account}".encode()).hexdigest()
    return int(digest[:15], 16) / 16**15


splits = {"holdout": [], "dev": [], "train": []}
with open("tickets.jsonl", encoding="utf-8") as source:
    for line in source:
        row = json.loads(line)
        value = position(row["account"])
        if value < HOLDOUT_FRACTION:
            splits["holdout"].append(row)
        elif value < HOLDOUT_FRACTION + DEV_FRACTION:
            splits["dev"].append(row)
        else:
            splits["train"].append(row)

for name, rows in splits.items():
    with open(f"{name}.jsonl", "w", encoding="utf-8") as output:
        for row in rows:
            output.write(json.dumps(row) + "\n")
    accounts = len({row["account"] for row in rows})
    print(f"{name}.jsonl: {len(rows)} tickets from {accounts} accounts")

with open("holdout.jsonl", "rb") as holdout:
    print(f"holdout.jsonl sha256: {hashlib.sha256(holdout.read()).hexdigest()}")
```

To split the tickets, run the script from the directory containing `tickets.jsonl`:

```bash theme={"system"}
uv run python split.py
```

```text Output theme={"system"}
holdout.jsonl: 37 tickets from 11 accounts
dev.jsonl: 32 tickets from 11 accounts
train.jsonl: 65 tickets from 18 accounts
holdout.jsonl sha256: e6e7be49af735befdbed3fbdf253edeede627fac57775974b25c31874aa97b0e
```

The fractions apply to accounts, and accounts send different numbers of tickets, so each split's share of tickets varies. Record the salt and the holdout's hash with every score. Two scores are comparable only when both came from the same holdout file.

Size the holdout to the smallest change you need to see. A pass rate measured on `n` rows is uncertain by about ±1/√n at 95% confidence:

| Holdout rows | Uncertainty on a pass rate near 50% |
| - | - |
| 100 | ±10 points |
| 300 | ±5.7 points |
| 1,000 | ±3.1 points |

That estimate assumes every row is independent. The holdout's 37 tickets come from 11 accounts, and one account's tickets follow up on each other, so the real uncertainty is larger than ±1/√n. To measure it, resample whole accounts rather than single tickets.

Comparing two models on the same rows is tighter than this, because only the rows where they disagree carry signal. Read a slice, such as `sev1` tickets, on its own only when it has at least 30 rows. The 37-ticket holdout here is enough to run the scripts and see a large difference, not to measure a few points. Use 300 or more rows for a release decision.

## Score with code first

Use code for every rule a person wouldn't need to reason about: valid JSON, a required field, a length limit, an exact label, an ID copied from the input. Code runs locally and gives the same answer every time.

Save the following as `checks.py`. Its `check()` function scores one reply against the four code rules:

```python checks.py theme={"system"}
import json
import re

SEVERITIES = {"sev1", "sev2", "sev3", "sev4"}
# Ticket IDs: a lowercase prefix such as req, ord, payout, or pay_out, then a
# suffix of letters and digits.
ID_PATTERN = re.compile(r"\b[a-z][a-z_]*_[A-Za-z0-9]{5,}\b")


def ids_in(text):
    """The ID tokens in text, without words that only look like IDs."""
    return {
        token for token in ID_PATTERN.findall(text) if any(c.isdigit() for c in token)
    }


def parse(output):
    """Return the reply as a dict, or None if the tool couldn't read it."""
    try:
        reply = json.loads(output)
    except (TypeError, json.JSONDecodeError):
        return None
    if not isinstance(reply, dict) or not isinstance(reply.get("summary"), str):
        return None
    severity = reply.get("severity")
    if not isinstance(severity, str) or severity not in SEVERITIES:
        return None
    return reply


def check(row, output):
    reply = parse(output)
    if reply is None:
        return {
            "valid_json": False,
            "length": False,
            "ids_from_ticket": False,
            "severity": False,
        }
    summary = reply["summary"]
    ticket_ids = ids_in(row["ticket"])
    return {
        "valid_json": True,
        "length": len(summary.split()) <= 120,
        "ids_from_ticket": all(token in ticket_ids for token in ids_in(summary)),
        "severity": reply["severity"] == row["severity"],
    }
```

A reply that doesn't parse fails every check, because the tool can't show it to the engineer. Score format and task correctness separately: a reply can be valid JSON and still name the wrong severity. [Structured outputs](https://docs.baseten.co/inference/structured-outputs.md) can constrain the JSON at generation time; keep the check anyway, because it also catches a reply cut off at the token limit.

Report each check separately. A single combined score hides which direction a change moved: a model that rates every ticket `sev3` gains on the most common severity and loses on every other one.

## Judge what code can't decide

Use a large language model (LLM) judge only for rules that need reading and reasoning, such as whether every fact in a summary is stated in the ticket. Design each judged rule as its own axis:

* **One prompt per axis.** A prompt that scores several rules at once lets one obvious failure bleed into unrelated scores.
* **Pass or fail by default.** A binary verdict varies less between repeated judgments than a 1 to 10 scale. If you use a scale, set the passing threshold yourself rather than asking the judge whether an output passes.
* **A boundary for each axis.** State in words what passes and what fails at the edge. Judges and people disagree most on boundary cases.
* **Reasoning before the verdict.** Ask for the judge's reasoning and quoted evidence first, then the verdict, so the verdict follows from the evidence.
* **As few axes as the rules need.** Two axes that never disagree on the same rows measure one thing. Each extra axis is another judge call per row.

Choose the judge model deliberately:

* **A different model family from the one under test:** judges tend to [favor their own outputs](https://arxiv.org/abs/2404.13076), and to [favor models trained on data from their own model or family](https://arxiv.org/abs/2502.01534).
* **A reasoning model for correctness:** an axis like `grounded` needs the judge to check each claim against the ticket.
* **Sampling settings from its model card:** send them explicitly, and [measure how much the verdicts vary](#check-the-judge-before-you-trust-it) at those settings.
* **Cost at your data size:** forecast the judge calls so you know what each evaluation costs to run.

Save the following as `judge.py`. It judges each axis with its own call to Kimi K3 on Model APIs, a different family from the GLM models under test. Kimi K3's model card reports its results at a temperature of 1.0. The [structured output](https://docs.baseten.co/inference/structured-outputs.md) schema puts `reasoning` and `evidence` before `verdict`:

```python judge.py expandable theme={"system"}
import os
from typing import Literal

from openai import LengthFinishReasonError, OpenAI
from pydantic import BaseModel, ValidationError

client = OpenAI(
    base_url="https://inference.baseten.co/v1", api_key=os.environ["BASETEN_API_KEY"]
)

JUDGE_MODEL = "moonshotai/Kimi-K3"
JUDGE_SETTINGS = {"temperature": 1.0, "reasoning_effort": "low", "max_tokens": 8192}

AXES = {
    "grounded": {
        "question": "Is every fact in the summary stated in the ticket?",
        "pass": "Paraphrases of facts the ticket states. A fact the summary leaves "
        "out isn't a grounding failure, and neither is calling the company's product "
        "by a generic name.",
        "fail": "Any error, ID, number, time, or impact that the ticket doesn't "
        "state, including a claim about workarounds or about what isn't affected that "
        "the ticket doesn't make. A cause the ticket doesn't give, or one the merchant "
        "only guessed at, stated as fact or as likely.",
    },
    "coverage": {
        "question": "Does the summary name the affected service, the error, and the "
        "customer impact?",
        "pass": "Each of the three is named, or the summary says the ticket doesn't "
        "state it.",
        "fail": "The summary leaves out one of the three that the ticket states.",
    },
}


class Verdict(BaseModel):
    reasoning: str
    evidence: str
    verdict: Literal["pass", "fail"]


def judge(axis, ticket, summary, attempts=3):
    rule = AXES[axis]
    system = (
        "You grade one aspect of a summary of a support ticket.\n"
        f"Aspect: {rule['question']}\n"
        "Grade only this aspect. Ignore style, length, and everything else.\n"
        f"Passes at the edge: {rule['pass']}\n"
        f"Fails at the edge: {rule['fail']}\n"
        "Explain your reasoning, quote the part of the summary that decides it, "
        "then give the verdict."
    )
    for _ in range(attempts):
        try:
            response = client.beta.chat.completions.parse(
                model=JUDGE_MODEL,
                messages=[
                    {"role": "system", "content": system},
                    {
                        "role": "user",
                        "content": f"Ticket:\n{ticket}\n\nSummary:\n{summary}",
                    },
                ],
                response_format=Verdict,
                **JUDGE_SETTINGS,
            )
        except (LengthFinishReasonError, ValidationError):
            continue
        verdict = response.choices[0].message.parsed
        if verdict is not None:
            return verdict.model_dump()
    raise RuntimeError(f"No parseable {axis} verdict after {attempts} attempts")
```

`judge()` retries a reply that doesn't parse instead of dropping the row or scoring it as a fail: dropped rows skew the score toward easy cases, and a parse failure says nothing about the summary. Each axis is one judge call per summary, so two axes on the 37 holdout tickets cost 74 judge calls per model. To lower the cost of each call, set a lower [`reasoning_effort`](https://docs.baseten.co/inference/model-apis/reasoning).

## Run the evaluation

Save the following as `evaluate.py`. It sends each ticket to one candidate with the system prompt, runs the checks, judges each summary that parses, and writes one result per ticket to `runs/`:

```python evaluate.py expandable theme={"system"}
import json
import os
import sys
from concurrent.futures import ThreadPoolExecutor

from openai import OpenAI

from checks import check, parse
from judge import AXES, judge
from prompt import SYSTEM_PROMPT

MODEL_APIS = "https://inference.baseten.co/v1"
GLM_SETTINGS = {
    "temperature": 1.0,
    "top_p": 0.95,
    "reasoning_effort": "low",
    "max_tokens": 4096,
}
CANDIDATES = {
    "glm-5.3": (MODEL_APIS, "zai-org/GLM-5.3", GLM_SETTINGS),
    "glm-5.3-flash": (MODEL_APIS, "zai-org/GLM-5.3-Flash", GLM_SETTINGS),
}

split_file, label = sys.argv[1], sys.argv[2]
base_url, model, settings = CANDIDATES[label]
client = OpenAI(base_url=base_url, api_key=os.environ["BASETEN_API_KEY"])


def score(row):
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": row["ticket"]},
        ],
        **settings,
    )
    output = response.choices[0].message.content
    passed = check(row, output)
    judgments = {}
    if passed["valid_json"]:
        summary = parse(output)["summary"]
        judgments = {axis: judge(axis, row["ticket"], summary) for axis in AXES}
    for axis in AXES:
        passed[axis] = axis in judgments and judgments[axis]["verdict"] == "pass"
    return {"id": row["id"], "output": output, "passed": passed, "judgments": judgments}


with open(split_file, encoding="utf-8") as source:
    rows = [json.loads(line) for line in source]
with ThreadPoolExecutor(max_workers=8) as pool:
    results = list(pool.map(score, rows))

os.makedirs("runs", exist_ok=True)
run_file = f"runs/{split_file.removesuffix('.jsonl')}-{label}.jsonl"
with open(run_file, "w", encoding="utf-8") as output:
    for result in results:
        output.write(json.dumps(result) + "\n")

print(f"{label} on {split_file}: {len(results)} tickets, written to {run_file}")
for metric in results[0]["passed"]:
    rate = sum(result["passed"][metric] for result in results) / len(results)
    print(f"{metric:16} {rate:7.1%}")
```

`CANDIDATES` gives each candidate an OpenAI-compatible base URL, a model name, and the request settings to send with it. For a model on Model APIs, the base URL is `https://inference.baseten.co/v1` and the model name is its slug. Send every setting explicitly rather than relying on defaults, so every run uses the same ones.

While you iterate on the rules and the judge, run on the dev set:

```bash theme={"system"}
uv run python evaluate.py dev.jsonl glm-5.3
```

```text Output theme={"system"}
glm-5.3 on dev.jsonl: 32 tickets, written to runs/dev-glm-5.3.jsonl
valid_json         96.9%
length             96.9%
ids_from_ticket    96.9%
severity           90.6%
grounded           65.6%
coverage           96.9%
```

The rates differ between runs: the models and the judge sample at temperature 1.0.

Read the failing rows before you change anything. One reply isn't valid JSON: its summary quotes an event name without escaping the quotation marks. Eleven tickets fail `grounded`: that reply, plus ten judged summaries, most of which share two patterns. Four state an impact the ticket never mentions, such as "No workaround available", and four give a likely cause the merchant didn't, such as "Likely an issue with the webhook dispatcher". The severity policy in the prompt asks the model to weigh impact and workarounds, and the summaries report that reasoning as fact. The other two invent details, including one that attributes the merchant's logs to another payment company.

The rules are right to catch these, so they stay. A prompt that keeps the severity reasoning out of the summary is a fix to try before training.

A reply that doesn't parse fails every metric, not just `valid_json`, and a format failure is the request's problem rather than the model's. [JSON mode](https://docs.baseten.co/inference/json-mode.md) constrains the reply to a JSON object. Add a candidate for Flash that requests it to `CANDIDATES`, and freeze the set before you score the holdout:

```python evaluate.py theme={"system"}
    "glm-5.3-flash-json": (
        MODEL_APIS,
        "zai-org/GLM-5.3-Flash",
        {**GLM_SETTINGS, "response_format": {"type": "json_object"}},
    ),
```

## Check the judge before you trust it

A judge is a measuring instrument. Measure it on dev rows before it decides anything:

1. **Three known cases.** On real rows, confirm a known-good output passes, a known-bad output fails, and an empty or errored output neither crashes your parser nor scores as a pass.
2. **Stability.** Judge the same outputs several times and compute agreement per axis. Report a chance-corrected statistic such as Cohen's kappa. On an axis where 95% of outputs pass, a judge that always says pass reaches 95% raw agreement and a kappa near 0.
3. **Planted violations.** Take outputs the judge passed and add one violation the rules forbid. A judge that misses planted violations is too lenient to gate a release or filter training data.
4. **Agreement with people.** Have a person label a sample of dev outputs on each axis, then compare the judge's verdicts with theirs using kappa. Where they disagree, read the row: either the rule or the label needs fixing.

Save the following as `check_judge.py`. It judges each dev summary three times per axis, then changes a number in each summary that passed `grounded` and judges it again:

```python check_judge.py expandable theme={"system"}
import json
import re
import sys
from concurrent.futures import ThreadPoolExecutor

from checks import parse
from judge import AXES, judge

REPLAYS = 3

with open("dev.jsonl", encoding="utf-8") as source:
    tickets = {row["id"]: row["ticket"] for row in map(json.loads, source)}
with open(sys.argv[1], encoding="utf-8") as run:
    rows = [row for row in map(json.loads, run) if row["passed"]["valid_json"]]


def kappa(a, b):
    """Cohen's kappa for two lists of pass (True) or fail (False) verdicts."""
    observed = sum(x == y for x, y in zip(a, b)) / len(a)
    rate_a, rate_b = sum(a) / len(a), sum(b) / len(b)
    expected = rate_a * rate_b + (1 - rate_a) * (1 - rate_b)
    return None if expected == 1 else (observed - expected) / (1 - expected)


def plant(summary, ticket):
    """Change one number in the summary so that it states a fact the ticket doesn't."""
    for match in re.finditer(r"\b\d+\b", summary):
        changed = str(int(match.group()) + 3)
        if changed not in ticket:
            return summary[: match.start()] + changed + summary[match.end() :]
    return None


def replay(row):
    summary = parse(row["output"])["summary"]
    return {
        axis: [
            judge(axis, tickets[row["id"]], summary)["verdict"] == "pass"
            for _ in range(REPLAYS)
        ]
        for axis in AXES
    }


def judge_planted(row):
    planted = plant(parse(row["output"])["summary"], tickets[row["id"]])
    if planted is None:
        return None
    return judge("grounded", tickets[row["id"]], planted)["verdict"] == "fail"


with ThreadPoolExecutor(max_workers=8) as pool:
    replays = list(pool.map(replay, rows))
    grounded = [row for row in rows if row["passed"]["grounded"]]
    caught = [c for c in pool.map(judge_planted, grounded) if c is not None]

for axis in AXES:
    verdicts = [r[axis] for r in replays]
    stable = sum(len(set(v)) == 1 for v in verdicts) / len(verdicts)
    first, second = [v[0] for v in verdicts], [v[1] for v in verdicts]
    k = kappa(first, second)
    k_text = "undefined (one verdict only)" if k is None else f"{k:.2f}"
    passing = sum(first) / len(first)
    print(
        f"{axis}: {len(verdicts)} summaries, {passing:.0%} pass, "
        f"{stable:.0%} identical across {REPLAYS} replays, kappa {k_text}"
    )
print(f"grounded caught {sum(caught)} of {len(caught)} planted violations")
```

To check the judge against the dev run, run the following command:

```bash theme={"system"}
uv run python check_judge.py runs/dev-glm-5.3.jsonl
```

```text Output theme={"system"}
grounded: 31 summaries, 71% pass, 90% identical across 3 replays, kappa 0.92
coverage: 31 summaries, 100% pass, 100% identical across 3 replays, kappa undefined (one verdict only)
grounded caught 20 of 20 planted violations
```

The first version of the `grounded` boundary didn't say whether a claim like "no workaround" or a hedged cause like "likely an issue with the webhook dispatcher" fails. In two checks on the dev set, its verdicts were identical across three replays on only 72% and 81% of summaries, with a kappa between the first two replays of 0.14 and 0.45. Most of the summaries whose verdicts changed between replays were those cases. Naming both cases in the `fail` text produced this output. A change to a rule makes a new version of the evaluation, so score every candidate again after one.

`coverage` passed every summary, so its kappa is undefined and it can't show a difference between models on these tickets. An axis that nothing fails costs judge calls without measuring anything: harden it with harder rows, or drop it. The planted violations show that `grounded` catches a changed number. Plant violations of each rule you rely on, and have a person label a sample of dev summaries before the judge gates a release or filters training data.

## Score a baseline

Before you train, score the models you can already use: the model that serves the task today, cheaper candidates, and the strongest model you can call. Score them on the holdout with the same prompt each time:

```bash theme={"system"}
uv run python evaluate.py holdout.jsonl glm-5.3
uv run python evaluate.py holdout.jsonl glm-5.3-flash
```

Save the following as `compare.py`. It prints each metric for two runs on the same split, with the tickets the candidate won and lost against the baseline:

```python compare.py theme={"system"}
import json
import sys


def load(run_file):
    with open(run_file, encoding="utf-8") as run:
        return {row["id"]: row["passed"] for row in map(json.loads, run)}


baseline, candidate = load(sys.argv[1]), load(sys.argv[2])
if baseline.keys() != candidate.keys():
    raise ValueError("Score both models on the same split before comparing them")

ids = sorted(baseline)
print(f"{'metric':16} {'baseline':>8} {'candidate':>9} {'wins':>5} {'losses':>6}")
for metric in baseline[ids[0]]:
    before = [baseline[i][metric] for i in ids]
    after = [candidate[i][metric] for i in ids]
    wins = sum(a and not b for b, a in zip(before, after))
    losses = sum(b and not a for b, a in zip(before, after))
    print(
        f"{metric:16} {sum(before) / len(ids):8.1%} {sum(after) / len(ids):9.1%}"
        f" {wins:5} {losses:6}"
    )
```

To compare GLM 5.3 Flash with GLM 5.3, run the following command:

```bash theme={"system"}
uv run python compare.py runs/holdout-glm-5.3.jsonl runs/holdout-glm-5.3-flash.jsonl
```

```text Output theme={"system"}
metric           baseline candidate  wins losses
valid_json         100.0%     64.9%     0     13
length             100.0%     64.9%     0     13
ids_from_ticket    100.0%     64.9%     0     13
severity            83.8%     62.2%     3     11
grounded            73.0%     43.2%     4     15
coverage           100.0%     64.9%     0     13
```

Start with the rows, not the ranking. All 13 tickets Flash lost on `valid_json` are valid JSON wrapped in a Markdown code fence, which the tool can't parse, so those tickets fail every other metric too. The JSON-mode candidate you added on dev targets exactly that failure. To score it on the holdout and compare it with GLM 5.3, run the following commands:

```bash theme={"system"}
uv run python evaluate.py holdout.jsonl glm-5.3-flash-json
uv run python compare.py runs/holdout-glm-5.3.jsonl runs/holdout-glm-5.3-flash-json.jsonl
```

```text Output theme={"system"}
metric           baseline candidate  wins losses
valid_json         100.0%    100.0%     0      0
length             100.0%    100.0%     0      0
ids_from_ticket    100.0%    100.0%     0      0
severity            83.8%     89.2%     2      0
grounded            73.0%     54.1%     3     10
coverage           100.0%    100.0%     0      0
```

JSON mode closes the format gap. To tell a real difference from noise, look at wins and losses, because only the tickets where the two models disagree carry information. As a rule of thumb, a difference stands out from noise when wins minus losses is more than twice the square root of wins plus losses. The threshold assumes independent rows, so with account-grouped tickets it screens for a difference rather than certifying one. Flash's 2 `severity` wins with no losses are within noise. On `grounded`, it wins 3 tickets and loses 10, a gap of 7 against a threshold of about 7.2. Its losses repeat the dev set's patterns, such as "Likely a dashboard display bug", so Flash may state more unsupported causes, but 37 tickets can't confirm it. Score a larger holdout and try the prompt fix before you decide. If the gap holds, it's the gap a trained model has to close.

Read the spread as well as the ranking. If clearly different models all land within the judge's run-to-run variation, the evaluation is saturated: it can't show a training gain, so make the task harder or the rules sharper first. If a smaller model beats a larger one, read the rows where it won before you decide whether the evaluation or your expectation is wrong.

If an existing model already meets your bar, skip post-training. For that decision, see [Decide whether to post-train](https://docs.baseten.co/post-training/decide.md).

## Compare a checkpoint with its baseline

[Deploy the checkpoint](https://docs.baseten.co/loops/deploy-checkpoints.md), then add it to `CANDIDATES` with its deployment URL and the settings you'll serve it with. For a deployed Loops checkpoint, the model name is the checkpoint name, such as `step-124`:

```python evaluate.py theme={"system"}
    "trained": (
        "https://model-<model_id>.api.baseten.co/deployment/<deployment_id>/sync/v1",
        "step-124",
        {"temperature": 1.0, "top_p": 0.95, "max_tokens": 4096},
    ),
```

To score the checkpoint and compare it with the model it would replace, run the following commands:

```bash theme={"system"}
uv run python evaluate.py holdout.jsonl trained
uv run python compare.py runs/holdout-glm-5.3.jsonl runs/holdout-trained.jsonl
```

The comparison uses the same columns as the baseline comparison. The deployment bills for its GPU while it's live; when you're done, [delete it](https://docs.baseten.co/loops/deploy-checkpoints.md#delete-the-deployment).

Scoring through the deployment measures what you'd serve, including its chat template. To see what training added, also score the untrained base model with the same prompt and settings. The gain over the base model is what training added; the comparison with GLM 5.3 tells you whether to ship.

Apply the same wins-and-losses test to the checkpoint, and read its losses row by row: a checkpoint can gain overall and still regress on the rows that matter most.

Break each metric down by the slices that matter before you decide, such as `sev1` tickets for the support tool; each run file holds one row per ticket, so a slice is a filter over `runs/*.jsonl`. The [supervised fine-tuning (SFT) example](https://docs.baseten.co/examples/sft-tool-calling.md) shows why: its adapter's overall tool-call decision score rose from 88.0% to 99.0%, and on the 30 requests no offered tool could serve it rose from 73.3% to 100%, because the base model called an unrelated tool and the adapter answered directly.

Release thresholds are a product decision. Set them from what the task needs, such as "`grounded` and `ids_from_ticket` don't drop, and `severity` is right on at least 95% of `sev1` tickets", not from the baseline's numbers. Keep the holdout, the checks, the judge prompts, and the judge model fixed across candidates; changing any of them makes earlier scores incomparable.

## Next steps

* [Decide whether to post-train](https://docs.baseten.co/post-training/decide.md): Compare what training would save with what it costs.
* [Prepare SFT data](https://docs.baseten.co/loops/sft-data.md): Use the same rules to choose training examples.
* [Check whether an RL run is learning](https://docs.baseten.co/loops/rl-run-health.md): Read reward and rollout metrics during training.
* [Deploy a checkpoint](https://docs.baseten.co/loops/deploy-checkpoints.md): Serve the checkpoint you evaluate.