> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.astropods.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.astropods.com/_mcp/server. # RFC-4: Evaluator Spec ## Abstract The Evaluator Spec defines a declarative YAML format for an agent's evaluation set: the automated checks that run against an agent's production traces before a reviewer curates them into an evaluation dataset. The spec is consumed by the CLI, which validates it and publishes it to the platform; it intentionally excludes review-queue behavior, dataset storage, and how a reviewer curates traces, which the platform owns. Astro runs a default evaluation set for every agent; a builder who needs different checks activates a custom set with this file. ## Changelog | Version | Date | Changes | | ------- | ---------- | -------------- | | v1.0 | 2026-09-02 | Initial draft. | ## Conventions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119). --- ## 1. Introduction An Evaluator Spec file (`EVALUATION.yaml` or `EVALUATION.yml`) is a YAML document that declares an agent's evaluation set: an ordered list of evaluators, each either a reference to an Astro-maintained preset or a complete custom LLM evaluator definition. The file lives beside `astropods.yml`, at the root of the agent's project directory. Discovery is exact and case-sensitive: only `EVALUATION.yaml` or `EVALUATION.yml` is recognized. An agent that has never activated a custom set uses Astro's default evaluation set (see [Appendix A](#appendix-a-preset-registry-non-normative)). Activating `EVALUATION.yaml` replaces the active set entirely; it is not merged with the default. Every evaluator returns one typed value for one trace: `boolean`, `enum`, `number`, or `string`. Astro owns the execution mechanics: the model, temperature, retries, timeouts, and response validation. The only per-evaluator configuration a builder controls is its prompt, its output schema, and which additional trace context it receives. The spec does **not** cover: the review queue, dataset storage, or how a reviewer curates traces into a dataset. These are platform concerns configured and operated separately. The document format is YAML. Implementations MUST accept files named `EVALUATION.yaml` or `EVALUATION.yml`. --- ## 2. Top-Level Structure A conforming document MUST contain the following top-level fields: | Field | Type | Required | Description | | ------------ | -------- | ------------ | ----------------------------------------------------------------------- | | `schema` | string | **REQUIRED** | Evaluation document contract version. MUST be `evaluation/v1`. | | `evaluators` | Entry\[] | **REQUIRED** | Ordered evaluators to run for each trace. MUST contain 1 to 10 entries. | Evaluator order controls display and execution order. It does not create dependencies between evaluators — each evaluator makes its own independent model call. --- ## 3. Evaluator Entries Each entry in `evaluators` is either a **preset reference** or a **custom evaluator**. These are mutually exclusive: an entry MUST be one or the other, never both. ### 3.1 Preset Reference A preset reference selects one of Astro's maintained evaluators (Appendix A) by name: ```yaml evaluators: - ref: preset/user-sentiment ``` | Field | Type | Required | Description | | ----- | ------ | ------------ | ---------------------------------------------------------------- | | `ref` | string | **REQUIRED** | Preset evaluator reference. MUST be a name listed in Appendix A. | A preset reference entry MUST NOT set `key`, `label`, `description`, `type`, `config`, `prompt`, `prompt_file`, or `output`. A builder who needs different behavior from a preset defines a custom evaluator with its own key instead. ### 3.2 Custom Evaluator A custom evaluator defines its own prompt and output schema: ```yaml evaluators: - key: has_secrets label: Contains secrets description: Flags credentials, API keys, tokens, or other secrets in the output. type: llm prompt: Determine whether the agent output exposes credentials, API keys, tokens, or other secrets. output: type: boolean ``` | Field | Type | Required | Description | | ------------- | ------ | ------------ | ------------------------------------------------------------------------------- | | `key` | string | **REQUIRED** | Stable machine identifier. MUST match `^[a-z][a-z0-9_]{0,63}$`. | | `label` | string | **REQUIRED** | Human-readable name. MUST contain 1 to 50 characters. | | `description` | string | OPTIONAL | Human-readable explanation shown alongside the evaluator's results. | | `type` | string | **REQUIRED** | Execution mechanism. MUST be `llm` (the only type this version supports). | | `config` | Config | OPTIONAL | Additional trace context to include. See [Section 4](#4-context-configuration). | | `prompt` | string | Conditional | Inline evaluation instructions. | | `prompt_file` | string | Conditional | Path to a Markdown rubric file, relative to the project root. | | `output` | Output | **REQUIRED** | Schema of the evaluator's returned value. See [Section 5](#5-output-schema). | An entry MUST specify exactly one of `prompt` or `prompt_file`. Whichever is used, the resolved prompt text MUST contain 1 to 8,000 characters. `prompt_file` MUST be a relative path within the agent project, MUST NOT start with `/`, and MUST end in `.md`. During normalization, the file's UTF-8 contents replace `prompt_file` as the evaluator's `prompt`; the path is not retained in the published definition. ```yaml prompt_file: evaluation/unsupported-claims.md ``` --- ## 4. Context Configuration Every evaluator always receives the target trace's `input` and `output`. Additional context is opt-in per evaluator via `config.context`, and defaults to excluded: | Field | Type | Required | Description | | ------------------- | --------- | -------- | --------------------------------------------------------------------------------------------------------------------------- | | `previous_turns` | boolean | OPTIONAL | Include up to 3 preceding completed traces from the same session. Default: `false`. | | `next_user_message` | boolean | OPTIONAL | Include the next user-authored message, when available. Default: `false`. | | `user_feedback` | boolean | OPTIONAL | Include structured user feedback (thumbs-up or thumbs-down). Default: `false`. | | `steps` | boolean | OPTIONAL | Include up to 20 of the trace's tool-call and generation steps. Default: `false`. | | `step_types` | string\[] | OPTIONAL | Restrict included steps to these types (for example `tool`, `generation`). Requires `steps: true`. Default: all step types. | ```yaml config: context: previous_turns: true next_user_message: true user_feedback: true ``` Context is supplied to the model as data. It cannot alter the system-owned execution instructions or the required output schema. --- ## 5. Output Schema Each evaluator returns one value. Version 1 supports `boolean`, `enum`, `number`, and `string`. ### 5.1 Boolean ```yaml output: type: boolean ``` Boolean outputs accept no type-specific fields. ### 5.2 Enum ```yaml output: type: enum options: [positive, neutral, negative, unclear] ``` | Field | Type | Required | Description | | --------- | --------- | ------------ | ----------------------------------------------------------------------------- | | `options` | string\[] | **REQUIRED** | Allowed values. MUST contain 2 to 20 unique entries, each 1 to 50 characters. | ### 5.3 Number ```yaml output: type: number minimum: 0 maximum: 1 ``` | Field | Type | Required | Description | | --------- | ------ | -------- | ---------------------------------------------------------------------------------- | | `minimum` | number | OPTIONAL | Lower bound. MUST be finite. | | `maximum` | number | OPTIONAL | Upper bound. MUST be finite. MUST be greater than `minimum` when both are present. | ### 5.4 String ```yaml output: type: string max_length: 1000 ``` | Field | Type | Required | Description | | ------------ | ------- | -------- | --------------------------------------------------------------------- | | `max_length` | integer | OPTIONAL | Maximum characters. MUST be 1 to 4,000. Default: `1000` when omitted. | --- ## 6. Publishing and Activation `ast eval validate` checks `EVALUATION.yaml` against this spec without changing the agent's active set. `ast eval push` validates and activates it: the file becomes the agent's active evaluation set for every one of its deployments immediately, with no redeploy required. An agent that has never run `ast eval push` uses Astro's default evaluation set (Appendix A). --- ## 7. Validation Rules Implementations MUST enforce the following validation rules: 1. `schema` MUST equal `evaluation/v1`. 2. `evaluators` MUST contain 1 to 10 entries. 3. Each entry MUST be either a preset reference or a complete custom definition, never both. 4. A preset reference MUST name a preset from Appendix A and MUST NOT set any other field. 5. Evaluator keys MUST be unique after preset references are resolved. 6. A custom evaluator's `key` MUST match `^[a-z][a-z0-9_]{0,63}$`. 7. `label` MUST contain 1 to 50 characters. 8. `type` MUST be `llm`. 9. A custom evaluator MUST specify exactly one of `prompt` or `prompt_file`. The resolved prompt MUST contain 1 to 8,000 characters. 10. `prompt_file` MUST be a relative path within the project ending in `.md`. 11. `step_types` MUST NOT be set unless `steps` is `true`. 12. Each `output` MUST satisfy its type-specific contract (Section 5): boolean accepts no extra fields; enum requires 2–20 unique options of 1–50 characters each; number bounds MUST be finite, with `minimum` less than `maximum` when both are present; string `max_length` MUST be 1–4,000. 13. The document MUST contain exactly one YAML document. Unknown fields anywhere in the document are rejected. 14. The YAML document plus all referenced prompt files MUST NOT exceed 128 KiB combined. --- ## Appendix A: Preset Registry (Non-Normative) The following table documents Astro's built-in evaluator presets as of this specification version. | Ref | Key | Output | Description | | -------------------------------------- | ------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | | `preset/exposed-pii` | `exposed_pii` | boolean | Flags personal data in the output, such as names, emails, phone numbers, or addresses. | | `preset/leaked-credentials` | `leaked_credentials` | boolean | Flags credentials in the output, such as API keys, tokens, or passwords. | | `preset/disclosed-system-instructions` | `disclosed_system_instructions` | boolean | Flags output that reveals the agent's instructions, guardrails, or tool definitions. | | `preset/unnecessary-tool-call` | `unnecessary_tool_call` | boolean | Flags tool calls that had no bearing on the agent's response or actions. | | `preset/claim-grounding` | `claim_grounding` | enum: `grounded`, `unsupported`, `contradicted`, `no_claims` | Rates how well factual claims in the output are supported by the agent's tool calls and observations. | | `preset/user-sentiment` | `user_sentiment` | enum: `positive`, `neutral`, `negative`, `unclear` | Rates the user's tone across the conversation. | --- ## Appendix B: Complete Example ```yaml schema: evaluation/v1 evaluators: - ref: preset/exposed-pii - ref: preset/leaked-credentials - key: response_quality label: Response quality type: llm prompt_file: evaluation/response-quality.md output: type: number minimum: 0 maximum: 1 - key: unnecessary_escalation label: Unnecessary escalation description: Flags responses that hand off to a human when the agent could have resolved the request itself. type: llm config: context: steps: true step_types: [tool] prompt: Determine whether the agent escalated to a human when its available tools could have resolved the request. output: type: boolean - key: user_sentiment label: User sentiment type: llm config: context: previous_turns: true next_user_message: true user_feedback: true prompt: Determine the user's sentiment toward the agent response. output: type: enum options: - positive - neutral - negative - unclear ```