ast eval

Manage a blueprint’s evaluators and evaluate a deployed agent’s traces.

ast eval <subcommand>

status, run, and review target a deployment and require exactly one of --name, --id, or --env, as in ast agent. With --env, the blueprint comes from --blueprint, or from astropods.yml in the current directory. ast agent trace shows each trace’s evaluation.

eval push

ast eval push [name] [--file <path>]

Reads EVALUATION.yaml (or EVALUATION.yml) beside astropods.yml and activates it as the blueprint’s evaluation set. This does not build or push an image. name defaults to the blueprint name in the spec file. --file (-f) sets the spec file path.

eval validate

ast eval validate [--file <path>]

Validates EVALUATION.yaml without activating it.

eval get

ast eval get <blueprint> [--json]

Lists the blueprint’s active evaluators with each one’s key, label, type, and accepted values. Use the keys with eval review.

eval status

ast eval status --name <display-or-blueprint-name> [--json]
ast eval status --id <deployment-id> [--json]
ast eval status --env <environment> [--blueprint <name>] [--json]

Shows how many of the deployment’s traces are queued, in progress, completed, failed, and outdated. A trace is outdated when it was evaluated against an earlier evaluation set.

eval run

ast eval run --name <display-or-blueprint-name> [--include-outdated] [--json]
ast eval run --name <display-or-blueprint-name> --trace-id <trace-id> [--json]

Queues evaluations for up to 50 recent traces that have not been evaluated or whose last run failed. When it queues 50, run it again to queue more. --include-outdated also queues traces evaluated against an earlier evaluation set.

--trace-id (-t) queues one trace, even one that already has a current result, and prints the run ID. It cannot be combined with --include-outdated.

Runs finish in the background. Check progress with eval status.

eval review

ast eval review --name <display-or-blueprint-name> --trace-id <trace-id> \
  --set <key>=<value> [--set <key>=<value> ...] \
  [--set-string <key>=<value> ...] [--update-dataset] [--json]

Saves your values for a trace’s evaluators as its human review. The review replaces any earlier one, and evaluators you leave out are not part of it.

• --set reads true and false as booleans, numbers as numbers, and anything else as text.
• --set-string always reads the value as text. Use it for text that looks like a number or boolean.
• --update-dataset also updates the trace’s dataset item.

The command prints the reason when a value is rejected.

ast eval review --name my-agent --trace-id 2dc10d31ac3ebec7 --set helpful=true --set tone=warm

Next steps

• ast dataset: curate the dataset that reviewed traces feed
• ast agent: list traces and see each trace’s evaluation