Skip to navigation
AI Gateway

Chat completions

Call the gateway from any OpenAI-compatible client, with streaming and structured output

View as Markdown

The OpenAI-compatible API lives under the /v1 path, so set your client’s base URL to ${ASTRO_GATEWAY_URL}/v1. Everything else is a standard OpenAI-style request: pass one of the supported models as model and authenticate with ASTRO_GATEWAY_API_KEY as a bearer token.

Append /v1 to ASTRO_GATEWAY_URL. The env var is the host only (e.g. https://aig.example); the OpenAI-compatible endpoints are served under /v1 (/v1/chat/completions, /v1/embeddings, /v1/responses). Pointing a client at the bare host returns 404.

from openai import OpenAI
import os
client = OpenAI(
    api_key=os.environ["ASTRO_GATEWAY_API_KEY"],
    base_url=f"{os.environ['ASTRO_GATEWAY_URL']}/v1",
)
response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

For Mastra: the AI-SDK provider returned by createOpenAI is what Agent.model: accepts, so the rest of your Agent / Workflow / Tool definitions stay the same.

Call gateway.chat(model), not gateway(model). Calling the provider directly selects the Responses API, which the AI Gateway doesn’t fully support:

  • The AI SDK sends the agent’s earlier replies as references to stored responses. The AI Gateway doesn’t store responses, so the model never sees what it said before, and it may answer every earlier question again.
  • On a Claude model, a turn fails once a tool returns a result, with Expected toolResult blocks.

Web search is the exception: it needs the Responses API, and it runs on the gpt-* models.

Mastra needs an AI SDK v5+ model. Add @ai-sdk/openai at v2 or later to your own package.json. Version 1.x produces an AI SDK v4 model (specificationVersion: "v1") that Mastra’s stream() rejects with AGENT_STREAM_V1_MODEL_NOT_SUPPORTED. This can be newer than the @ai-sdk/openai version Mastra pins internally, so install it explicitly rather than relying on the transitive one.

Streaming

Set stream: true to receive tokens incrementally as they’re generated.

stream = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Write a haiku about the sea."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Structured output

Prompting for a JSON object and parsing it (with a fallback for the occasional non-JSON reply) is a portable way to get structured output that works regardless of which structured-output features a given model exposes through the gateway.

Python
import json
response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[
        {"role": "system", "content": "Reply with only a JSON object: {\"sentiment\": \"positive|negative|neutral\"}."},
        {"role": "user", "content": "I love this product."},
    ],
)
raw = response.choices[0].message.content
try:
    result = json.loads(raw)
except json.JSONDecodeError:
    result = {"sentiment": "unknown"}

Next steps