AI Gateway

Chat completions

Call the gateway from any OpenAI-compatible client, with streaming and structured output

View as Markdown

The OpenAI-compatible API lives under the /v1 path, so set your client’s base URL to ${ASTRO_GATEWAY_URL}/v1. Everything else is a standard OpenAI-style request: pass one of the supported models as model and authenticate with ASTRO_GATEWAY_API_KEY as a bearer token.

Append /v1 to ASTRO_GATEWAY_URL. The env var is the host only (e.g. https://aig.example); the OpenAI-compatible endpoints are served under /v1 (/v1/chat/completions, /v1/embeddings, /v1/responses). Pointing a client at the bare host returns 404.

from openai import OpenAI
import os
client = OpenAI(
    api_key=os.environ["ASTRO_GATEWAY_API_KEY"],
    base_url=f"{os.environ['ASTRO_GATEWAY_URL']}/v1",
)
response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

For Mastra: the AI-SDK provider returned by createOpenAI is what Agent.model: accepts, so the rest of your Agent / Workflow / Tool definitions stay the same.

Mastra needs an AI SDK v5+ model. Add @ai-sdk/openai at v2 or later to your own package.json. Version 1.x produces an AI SDK v4 model (specificationVersion: "v1") that Mastra’s stream() rejects with AGENT_STREAM_V1_MODEL_NOT_SUPPORTED. This can be newer than the @ai-sdk/openai version Mastra pins internally, so install it explicitly rather than relying on the transitive one.

Streaming

Set stream: true to receive tokens incrementally as they’re generated.

stream = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Write a haiku about the sea."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Structured output

Prompting for a JSON object and parsing it (with a fallback for the occasional non-JSON reply) is a portable way to get structured output that works regardless of which structured-output features a given model exposes through the gateway.

Python
import json
response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[
        {"role": "system", "content": "Reply with only a JSON object: {\"sentiment\": \"positive|negative|neutral\"}."},
        {"role": "user", "content": "I love this product."},
    ],
)
raw = response.choices[0].message.content
try:
    result = json.loads(raw)
except json.JSONDecodeError:
    result = {"sentiment": "unknown"}

Next steps