RFC-4: Evaluator Spec

Abstract

The Evaluator Spec defines a declarative YAML format for an agent’s evaluation set: the automated checks that run against an agent’s production traces before a reviewer curates them into an evaluation dataset. The spec is consumed by the CLI, which validates it and publishes it to the platform; it intentionally excludes review-queue behavior, dataset storage, and how a reviewer curates traces, which the platform owns. Astro runs a default evaluation set for every agent; a builder who needs different checks activates a custom set with this file.

Changelog

Conventions

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.

1. Introduction

An Evaluator Spec file (EVALUATION.yaml or EVALUATION.yml) is a YAML document that declares an agent’s evaluation set: an ordered list of evaluators, each either a reference to an Astro-maintained preset or a complete custom LLM evaluator definition.

The file lives beside astropods.yml, at the root of the agent’s project directory. Discovery is exact and case-sensitive: only EVALUATION.yaml or EVALUATION.yml is recognized.

An agent that has never activated a custom set uses Astro’s default evaluation set (see Appendix A). Activating EVALUATION.yaml replaces the active set entirely; it is not merged with the default.

Every evaluator returns one typed value for one trace: boolean, enum, number, or string. Astro owns the execution mechanics: the model, temperature, retries, timeouts, and response validation. The only per-evaluator configuration a builder controls is its prompt, its output schema, and which additional trace context it receives.

The spec does not cover: the review queue, dataset storage, or how a reviewer curates traces into a dataset. These are platform concerns configured and operated separately.

The document format is YAML. Implementations MUST accept files named EVALUATION.yaml or EVALUATION.yml.

2. Top-Level Structure

A conforming document MUST contain the following top-level fields:

Evaluator order controls display and execution order. It does not create dependencies between evaluators — each evaluator makes its own independent model call.

3. Evaluator Entries

Each entry in evaluators is either a preset reference or a custom evaluator. These are mutually exclusive: an entry MUST be one or the other, never both.

3.1 Preset Reference

A preset reference selects one of Astro’s maintained evaluators (Appendix A) by name:

evaluators:
  - ref: preset/user-sentiment

A preset reference entry MUST NOT set key, label, description, type, config, prompt, prompt_file, or output. A builder who needs different behavior from a preset defines a custom evaluator with its own key instead.

3.2 Custom Evaluator

A custom evaluator defines its own prompt and output schema:

evaluators:
  - key: has_secrets
    label: Contains secrets
    description: Flags credentials, API keys, tokens, or other secrets in the output.
    type: llm
    prompt: Determine whether the agent output exposes credentials, API keys, tokens, or other secrets.
    output:
      type: boolean

An entry MUST specify exactly one of prompt or prompt_file. Whichever is used, the resolved prompt text MUST contain 1 to 8,000 characters. prompt_file MUST be a relative path within the agent project, MUST NOT start with /, and MUST end in .md. During normalization, the file’s UTF-8 contents replace prompt_file as the evaluator’s prompt; the path is not retained in the published definition.

prompt_file: evaluation/unsupported-claims.md

4. Context Configuration

Every evaluator always receives the target trace’s input and output. Additional context is opt-in per evaluator via config.context, and defaults to excluded:

config:
  context:
    previous_turns: true
    next_user_message: true
    user_feedback: true

Context is supplied to the model as data. It cannot alter the system-owned execution instructions or the required output schema.

5. Output Schema

Each evaluator returns one value. Version 1 supports boolean, enum, number, and string.

5.1 Boolean

output:
  type: boolean

Boolean outputs accept no type-specific fields.

5.2 Enum

output:
  type: enum
  options: [positive, neutral, negative, unclear]

5.3 Number

output:
  type: number
  minimum: 0
  maximum: 1

5.4 String

output:
  type: string
  max_length: 1000

6. Publishing and Activation

ast eval validate checks EVALUATION.yaml against this spec without changing the agent’s active set. ast eval push validates and activates it: the file becomes the agent’s active evaluation set for every one of its deployments immediately, with no redeploy required.

An agent that has never run ast eval push uses Astro’s default evaluation set (Appendix A).

7. Validation Rules

Implementations MUST enforce the following validation rules:

1. schema MUST equal evaluation/v1.
2. evaluators MUST contain 1 to 10 entries.
3. Each entry MUST be either a preset reference or a complete custom definition, never both.
4. A preset reference MUST name a preset from Appendix A and MUST NOT set any other field.
5. Evaluator keys MUST be unique after preset references are resolved.
6. A custom evaluator’s key MUST match ^[a-z][a-z0-9_]{0,63}$.
7. label MUST contain 1 to 50 characters.
8. type MUST be llm.
9. A custom evaluator MUST specify exactly one of prompt or prompt_file. The resolved prompt MUST contain 1 to 8,000 characters.
10. prompt_file MUST be a relative path within the project ending in .md.
11. step_types MUST NOT be set unless steps is true.
12. Each output MUST satisfy its type-specific contract (Section 5): boolean accepts no extra fields; enum requires 2–20 unique options of 1–50 characters each; number bounds MUST be finite, with minimum less than maximum when both are present; string max_length MUST be 1–4,000.
13. The document MUST contain exactly one YAML document. Unknown fields anywhere in the document are rejected.
14. The YAML document plus all referenced prompt files MUST NOT exceed 128 KiB combined.

Appendix A: Preset Registry (Non-Normative)

The following table documents Astro’s built-in evaluator presets as of this specification version.

Appendix B: Complete Example

schema: evaluation/v1
evaluators:
  - ref: preset/exposed-pii
  - ref: preset/leaked-credentials
  - key: response_quality
    label: Response quality
    type: llm
    prompt_file: evaluation/response-quality.md
    output:
      type: number
      minimum: 0
      maximum: 1
  - key: unnecessary_escalation
    label: Unnecessary escalation
    description: Flags responses that hand off to a human when the agent could have resolved the request itself.
    type: llm
    config:
      context:
        steps: true
        step_types: [tool]
    prompt: Determine whether the agent escalated to a human when its available tools could have resolved the request.
    output:
      type: boolean
  - key: user_sentiment
    label: User sentiment
    type: llm
    config:
      context:
        previous_turns: true
        next_user_message: true
        user_feedback: true
    prompt: Determine the user's sentiment toward the agent response.
    output:
      type: enum
      options:
        - positive
        - neutral
        - negative
        - unclear