Get Started with TrueFoundry AI Gateway

​

What is TrueFoundry AI Gateway?

• Comprehensive Observability - Track all API calls, latencies, and errors in one place with detailed request logging
• Cost Management - Monitor and control spending across models and teams with real-time cost tracking
• Access Control - Manage API keys and permissions centrally with role-based access
• Rate Limiting - Protect your applications from unexpected usage spikes with configurable limits
• Analytics Dashboard - Visualize usage patterns and performance metrics across your organization

​

Prerequisites

• Cerebras API Key - Get a free API key here
• TrueFoundry Account - Visit TrueFoundry and create an account or log in
• Python 3.12 or higher - For running the code examples

​

Configure TrueFoundry AI Gateway

1

Navigate to Cerebras in AI Gateway

From the TrueFoundry dashboard, navigate to the Cerebras models section:

1. Go to AI Gateway > Models > Cerebras

This opens the Cerebras configuration panel where you’ll add your account and models.

2

Add your Cerebras account

Click Add Cerebras Account to configure your Cerebras API credentials:

1. Click the Add Cerebras Account button
2. Enter your Account Name (e.g., “Production” or “Development”)
3. Enter your Cerebras API Key from the prerequisites step
4. Optionally add Collaborators who should have access
5. Click Save

You can configure multiple Cerebras accounts with different access controls. This is useful for separating production and development environments or managing different teams. See Access Control for more details.

3

Add Cerebras models

Click + Add Model to add Cerebras models to your gateway. Unlike other providers, you need to get the Model ID directly from the Cerebras documentation.To add a model:

1. Click + Add Model
2. Enter the Model ID exactly as shown in the Cerebras Models documentation
3. Configure any model-specific settings like rate limits or access controls
4. Click Save to activate the model

For Cerebras, you don’t select from a dropdown list. Instead, copy the exact Model ID from the Cerebras docs and paste it in the Model ID field.

4

Get your TrueFoundry API credentials

After configuring your models, TrueFoundry will provide you with gateway credentials. These credentials authenticate your application to the TrueFoundry gateway, which then routes requests to Cerebras.Find your credentials in the AI Gateway settings:

1. Navigate to AI Gateway > API Credentials
2. Copy your Gateway Base URL (e.g., https://gateway.truefoundry.ai)
3. Copy your Gateway API Key (a JWT token)

Keep these credentials secure - they provide access to all models configured in your gateway.

5

Install required dependencies

Install the OpenAI Python SDK, which is compatible with Cerebras through TrueFoundry’s OpenAI-compatible API:

Report incorrect code

Copy

Ask AI

pip install openai python-dotenv

The python-dotenv package helps manage environment variables securely.

6

Configure environment variables

Create a .env file in your project directory to store your TrueFoundry credentials securely:

Report incorrect code

Copy

Ask AI

TRUEFOUNDRY_API_KEY=your-truefoundry-gateway-key-here
TRUEFOUNDRY_BASE_URL=https://gateway.truefoundry.ai

Replace the placeholder values with your actual credentials from Step 4.

Never commit your .env file to version control. Add it to your .gitignore file to keep your credentials secure.

7

Initialize the client

Set up the OpenAI client to route requests through TrueFoundry’s gateway. The gateway intercepts your requests, captures observability data, and forwards them to Cerebras.

Python

cURL

Report incorrect code

Copy

Ask AI

import os
from openai import OpenAI
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

# Initialize client with TrueFoundry gateway
client = OpenAI(
    api_key=os.getenv("TRUEFOUNDRY_API_KEY"),
    base_url=os.getenv("TRUEFOUNDRY_BASE_URL"),
    default_headers={
        "X-Cerebras-3rd-Party-Integration": "TrueFoundry"
    }
)

The base_url parameter routes all requests through TrueFoundry’s gateway, which then forwards them to Cerebras while collecting metrics and logs.

8

Make your first request

Now you can make requests to Cerebras models through TrueFoundry. The gateway will automatically track this request in your analytics dashboard.

Python

cURL

Report incorrect code

Copy

Ask AI

import os
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

client = OpenAI(
    api_key=os.getenv("TRUEFOUNDRY_API_KEY"),
    base_url=os.getenv("TRUEFOUNDRY_BASE_URL"),
    default_headers={
        "X-Cerebras-3rd-Party-Integration": "TrueFoundry"
    }
)

# Make a chat completion request
response = client.chat.completions.create(
    model="cbrs/gpt-oss-120b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing in simple terms."}
    ],
    max_tokens=500,
    temperature=0.7
)

print(response.choices[0].message.content)

Use the format cbrs/MODEL_NAME when specifying models through TrueFoundry (e.g., cbrs/gpt-oss-120b). This prefix tells the gateway to route the request to your configured Cerebras account.

​

Advanced Features

​

Streaming Responses

import os
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

client = OpenAI(
    api_key=os.getenv("TRUEFOUNDRY_API_KEY"),
    base_url=os.getenv("TRUEFOUNDRY_BASE_URL"),
    default_headers={
        "X-Cerebras-3rd-Party-Integration": "TrueFoundry"
    }
)

# Stream the response
stream = client.chat.completions.create(
    model="cbrs/gpt-oss-120b",
    messages=[
        {"role": "user", "content": "Write a short story about a robot."}
    ],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

​

Custom Metadata and Tagging

import os
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

client = OpenAI(
    api_key=os.getenv("TRUEFOUNDRY_API_KEY"),
    base_url=os.getenv("TRUEFOUNDRY_BASE_URL"),
    default_headers={
        "X-Cerebras-3rd-Party-Integration": "TrueFoundry",
        "X-TrueFoundry-User-ID": "user-123",
        "X-TrueFoundry-Session-ID": "session-456",
        "X-TrueFoundry-Environment": "production"
    }
)

response = client.chat.completions.create(
    model="cbrs/gpt-oss-120b",
    messages=[
        {"role": "user", "content": "Summarize this article..."}
    ]
)

​

Rate Limiting and Budget Controls

1. Navigate to AI Gateway > Rate Limiting
2. Configure limits per user, API key, or model
3. Set daily or monthly budget caps to prevent unexpected costs
4. Configure alerts to notify you when limits are approached

​

Virtual Models

​

Monitoring and Analytics

​

View Request Logs

1. Go to AI Gateway > Observability > Request Logging
2. Filter by model, user, time range, or custom metadata
3. View request and response payloads, latencies, and errors
4. Export logs for further analysis or compliance requirements

​

Cost Tracking

1. Navigate to AI Gateway > Cost Tracking
2. View costs broken down by model, user, or time period
3. Set up alerts for budget thresholds
4. Export cost reports for billing or analysis

​

Analytics Dashboard

• Request Volume - Total requests over time, broken down by model
• Latency - P50, P95, and P99 latency percentiles
• Error Rates - Track errors by type and model
• Token Usage - Monitor input and output tokens across models
• Cost Trends - Visualize spending patterns over time

​

Export Metrics

• OpenTelemetry - Export traces and metrics to your observability platform
• Prometheus - Scrape metrics for custom dashboards
• Grafana - Visualize performance and cost data

​

Troubleshooting

​

Next Steps