Portable evaluations

An evaluation answers a narrow question: did this skill produce a better result under comparable conditions? Routing tests and output-quality tests answer different questions. Routing asks whether the skill should be used. Output-quality testing asks whether using it improves the completed work.

Package contract

Keep versioned definitions with the skill:

my-skill/
├── SKILL.md
└── evals/
    ├── evals.json
    ├── routing.json
    └── files/

evals/evals.json follows the Agent Skills contract:

{
  "skill_name": "support-triage",
  "evals": [
    {
      "id": "refund-escalation",
      "prompt": "Review the attached support request and propose the next action.",
      "expected_output": "Identifies the refund exception and escalates it with evidence.",
      "files": ["evals/files/refund-request.txt"],
      "assertions": [
        "The response names the applicable refund exception.",
        "The response includes the evidence needed for escalation."
      ]
    }
  ]
}

SelfTune also supports selftune_assertions for deterministic checks such as required text, forbidden text, regular expressions, tool calls, and JSON paths.

Paired runs

selftune eval run --skill-path /path/to/my-skill

Every case runs twice in a fresh context:

with_skill uses the current SKILL.md;
without_skill receives no evaluated skill; or
old_skill uses --baseline-skill-path.

The runner copies input fixtures into each arm and sets that arm as the agent’s working directory. It records the response, duration, token measurement, and assertion evidence separately.

Reading the benchmark

Each iteration contains:

per-case outputs and grading evidence;
mean and standard deviation for pass rate, time, and tokens;
the delta between the current skill and baseline;
a blind A/B comparison for every case;
assertions that always pass or always fail across both arms; and
feedback.json for human review.

Do not select a winner from one aggregate number. Read the failures, check the grader’s evidence, and revise weak assertions before changing the skill.

A score from a different model, test set, fixture, or grading rule is not a valid before-and-after comparison.

Start

Manage your skills

Improve and validate

Run it your way

For agents

Concepts

Operate safely

Reference

Portable evaluations

Package contract

Paired runs

Reading the benchmark

​Package contract

​Paired runs

​Reading the benchmark

Package contract

Paired runs

Reading the benchmark