AI Gateway

Set one line in your spec and your agent gets a managed API key for calling supported models. The gateway is OpenAI-API-compatible, so it works with the SDKs and frameworks you already use.

Quick start

Scaffold a new agent wired to the gateway in one step:

$
ast create my-agent --model gateway

This generates a gateway model in the spec and agent code that reads the injected env vars — no provider key to configure. To enable the gateway on an existing agent, declare a model with provider: gateway in astropods.yml and list the models you want to choose between:

1
models:
2
  default:
3
    provider: gateway
4
    models: [claude-sonnet-4-6, gpt-4o]

The models list is a menu of options. At deploy time — in the web console and CLI — you pick one, and it is injected as MODEL_<NAME> (here MODEL_DEFAULT). Your agent code reads the model id from that env var.

On ast deploy (and ast dev), your agent container receives:

Calling the gateway

The OpenAI-compatible API lives under the /v1 path, so set your client’s base URL to ${ASTRO_GATEWAY_URL}/v1. Everything else is a standard OpenAI-style request — pass one of the supported models as model and authenticate with ASTRO_GATEWAY_API_KEY as a bearer token.

1
from openai import OpenAI
2
import os
3
4
client = OpenAI(
5
    api_key=os.environ["ASTRO_GATEWAY_API_KEY"],
6
    base_url=f"{os.environ['ASTRO_GATEWAY_URL']}/v1",
7
)
8
9
response = client.chat.completions.create(
10
    model="claude-sonnet-4-6",
11
    messages=[{"role": "user", "content": "Hello"}],
12
)
13
print(response.choices[0].message.content)

For Mastra: the AI-SDK provider returned by createOpenAI is what Agent.model: accepts, so the rest of your Agent / Workflow / Tool definitions stay the same.

Supported models

Pass any of these strings as model on the SDK call:

All chat models are called the same way — swap the model string. titan-embed-text-v2 is an embeddings model (use the embeddings endpoint, below).

Streaming

Set stream: true to receive tokens incrementally as they’re generated.

1
stream = client.chat.completions.create(
2
    model="claude-sonnet-4-6",
3
    messages=[{"role": "user", "content": "Write a haiku about the sea."}],
4
    stream=True,
5
)
6
for chunk in stream:
7
    delta = chunk.choices[0].delta.content
8
    if delta:
9
        print(delta, end="", flush=True)

Embeddings

Use titan-embed-text-v2 on the embeddings endpoint for RAG and vector search. It returns a 1024-dimension vector per input.

1
resp = client.embeddings.create(
2
    model="titan-embed-text-v2",
3
    input="Astro AI deploys agents as containers.",
4
)
5
vector = resp.data[0].embedding   # list[float], length 1024

Images (vision)

pixtral-large accepts image input alongside text. Pass the image as a base64 data URI in an image_url content block — the gateway forwards the image bytes to the model, so remote URLs are not fetched server-side.

1
import base64
2
3
with open("chart.png", "rb") as f:
4
    data_uri = "data:image/png;base64," + base64.b64encode(f.read()).decode()
5
6
response = client.chat.completions.create(
7
    model="pixtral-large",
8
    messages=[{
9
        "role": "user",
10
        "content": [
11
            {"type": "text", "text": "What does this chart show?"},
12
            {"type": "image_url", "image_url": {"url": data_uri}},
13
        ],
14
    }],
15
)
16
print(response.choices[0].message.content)

Structured output

1
import json
2
3
response = client.chat.completions.create(
4
    model="claude-sonnet-4-6",
5
    messages=[
6
        {"role": "system", "content": "Reply with only a JSON object: {\"sentiment\": \"positive|negative|neutral\"}."},
7
        {"role": "user", "content": "I love this product."},
8
    ],
9
)
10
raw = response.choices[0].message.content
11
try:
12
    result = json.loads(raw)
13
except json.JSONDecodeError:
14
    result = {"sentiment": "unknown"}

Native Anthropic API (Claude Code, Claude Agent SDK)

Agents built on the native Anthropic Messages API — Claude Code, the Claude Agent SDK, or the @anthropic-ai/sdk client — can’t use the OpenAI-compatible surface above. Use the gateway’s /anthropic passthrough endpoint instead. This is the only way to run a Claude-Code-based agent on the gateway.

The platform injects only ASTRO_GATEWAY_URL and ASTRO_GATEWAY_API_KEY — it does not set any ANTHROPIC_* variables. You map them yourself, and there are three gateway-specific details:

$
export ANTHROPIC_BASE_URL="$ASTRO_GATEWAY_URL/anthropic"
$
# The gateway reads the virtual key from x-bf-vk; ANTHROPIC_AUTH_TOKEN is set only so
$
# the client has *a* credential (the gateway ignores it).
$
export ANTHROPIC_CUSTOM_HEADERS="x-bf-vk: $ASTRO_GATEWAY_API_KEY"
$
export ANTHROPIC_AUTH_TOKEN="$ASTRO_GATEWAY_API_KEY"
$
# Bedrock-served model ids (bare claude-* returns 401):
$
export ANTHROPIC_MODEL="bedrock/claude-opus-4-8"
$
export ANTHROPIC_DEFAULT_SONNET_MODEL="bedrock/claude-sonnet-4-6"
$
export ANTHROPIC_DEFAULT_HAIKU_MODEL="bedrock/claude-haiku-4-5"
$
# Bedrock rejects Claude Code's first-party pre-release beta flags:
>
export CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1
>
# then run `claude` (or the Agent SDK) as usual

Streaming, client-defined tool use, and adaptive thinking all work over this path. Anthropic’s server-side tools (web_search, web_fetch, code execution, computer use) are not available — the models run on Amazon Bedrock, which doesn’t host them. If your agent needs web search, wire up your own tool (call a search API and feed the result back as a tool result).

Local development

ast dev handles the gateway automatically. Run:

$
ast login
$
ast dev

Your local agent container receives the same ASTRO_GATEWAY_* env vars it would have in production. Code written against those env vars works identically in dev and prod.

Mixing with your own keys

astro_ai_gateway: true is independent of any models you declare. You can use the gateway for some calls and your own provider key for others:

1
agent:
2
  image: my-agent:latest
3
  astro_ai_gateway: true        # gateway-managed models
4
models:
5
  custom-fine-tune:
6
    provider: openai      # your own OpenAI key for a specific use case

Your agent code reads ASTRO_GATEWAY_API_KEY for gateway calls and OPENAI_API_KEY for the bring-your-own-key (BYOK) provider.

Errors and limits

The gateway returns standard OpenAI-style error responses. Common cases:

What it doesn’t cover

• Provider-native SDKs and prefixes (except Anthropic). Call models through the OpenAI-compatible API by their gateway id (for example claude-sonnet-4-6), not with a provider prefix such as anthropic/. The one exception is the native Anthropic Messages API, served on the /anthropic passthrough for Claude Code / Claude Agent SDK agents — see Native Anthropic API.
• Anthropic server-side tools. web_search, web_fetch, code execution, and computer use aren’t available — models run on Amazon Bedrock, which doesn’t host them. Client-defined tools work normally.
• Image generation. No image-generation models are offered. Image input (vision) is supported by pixtral-large.
• Bring-your-own-model. Adding new models or fine-tunes to the gateway isn’t self-serve; contact support.

Next steps

• Deploy your first agent: ship an agent that calls the gateway
• Monitor your agents: trace the model calls your agent makes