> ## Documentation Index
> Fetch the complete documentation index at: https://docs.selftune.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Portable evaluations

> Define realistic skill cases, run paired comparisons, and grade outputs with concrete evidence.

An evaluation answers a narrow question: did this skill produce a better result
under comparable conditions?

Routing tests and output-quality tests answer different questions. Routing asks
whether the skill should be used. Output-quality testing asks whether using it
improves the completed work.

## Package contract

Keep versioned definitions with the skill:

```text theme={null}
my-skill/
├── SKILL.md
└── evals/
    ├── evals.json
    ├── routing.json
    └── files/
```

`evals/evals.json` follows the Agent Skills contract:

```json theme={null}
{
  "skill_name": "support-triage",
  "evals": [
    {
      "id": "refund-escalation",
      "prompt": "Review the attached support request and propose the next action.",
      "expected_output": "Identifies the refund exception and escalates it with evidence.",
      "files": ["evals/files/refund-request.txt"],
      "assertions": [
        "The response names the applicable refund exception.",
        "The response includes the evidence needed for escalation."
      ]
    }
  ]
}
```

SelfTune also supports `selftune_assertions` for deterministic checks such as
required text, forbidden text, regular expressions, tool calls, and JSON paths.

## Paired runs

```bash theme={null}
selftune eval run --skill-path /path/to/my-skill
```

Every case runs twice in a fresh context:

* `with_skill` uses the current `SKILL.md`;
* `without_skill` receives no evaluated skill; or
* `old_skill` uses `--baseline-skill-path`.

The runner copies input fixtures into each arm and sets that arm as the agent's
working directory. It records the response, duration, token measurement, and
assertion evidence separately.

## Reading the benchmark

Each iteration contains:

* per-case outputs and grading evidence;
* mean and standard deviation for pass rate, time, and tokens;
* the delta between the current skill and baseline;
* a blind A/B comparison for every case;
* assertions that always pass or always fail across both arms; and
* `feedback.json` for human review.

Do not select a winner from one aggregate number. Read the failures, check the
grader's evidence, and revise weak assertions before changing the skill.

<Warning>
  A score from a different model, test set, fixture, or grading rule is not a
  valid before-and-after comparison.
</Warning>