# Authentication Source: https://inference-docs.cerebras.ai/api-reference/authentication The Cerebras API uses API keys for authentication. Create and manage API keys from our [Inference Cloud Console](https://cloud.cerebras.ai?utm_source=3pi_authentication\&utm_campaign=api_reference). **Keep your API key secure.** Never share it publicly or include it in client-side code (such as browsers or mobile apps). Instead, load it safely from an environment variable or a server-side key management service. API keys are passed using HTTP Bearer authentication: ```text theme={null} Authorization: Bearer CEREBRAS_API_KEY ``` ## Example Request ```bash theme={null} curl --location 'https://api.cerebras.ai/v1/chat/completions' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" \ --data '{ "model": "qwen-3.8-27b", "messages": [ {"role": "user", "content": "Tell me a fun fact about space."} ] }' ``` ## Using Official SDKs You can also authenticate automatically when using the official SDKs for Python and Node.js by passing your API key during client initialization: ```python Python theme={null} import os from cerebras.cloud.sdk import Cerebras client = Cerebras( # This is the default and can be omitted api_key=os.environ.get("CEREBRAS_API_KEY"), ) ``` ```javascript Node.js theme={null} import Cerebras from 'cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], // This is the default and can be omitted }); ``` ## Set Your API Key For security reasons, and to avoid configuring your API key each time, we recommend setting your API key as an environment variable. You can do this by running the following command in your terminal: ```bash macOS / Linux theme={null} export CEREBRAS_API_KEY="your-api-key" ``` ```bash Windows theme={null} setx CEREBRAS_API_KEY "your-api-key" ``` # Cancel batch Source: https://inference-docs.cerebras.ai/api-reference/batch/cancel-batch DELETE https://api.cerebras.ai/v1/batches/{batch_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Cancels a batch job that is currently queued or running. Completed requests remain saved in the results and can be retrieved through the retrieve batch endpoint. ## Path Parameters The ID of the batch job to cancel. ## Response A unique identifier for the batch job. The object type, which is always `batch`. The API endpoint used for batch processing. Information about any errors that occurred during batch processing. The ID of the input file containing batch requests. The time window for batch completion. Always `24h`. The status of the batch job. Will be `cancelling` immediately after calling cancel. Once cancellation is complete (up to 10 minutes), the status changes to `cancelled`. The ID of the file containing batch results with any completed requests. The ID of the file containing errors (if any errors occurred). The Unix timestamp (in seconds) of when the batch job was created. The Unix timestamp (in seconds) of when the batch job started processing. The Unix timestamp (in seconds) of when the batch job will expire. The Unix timestamp (in seconds) of when the batch job started finalizing. The Unix timestamp (in seconds) of when the batch job completed. The Unix timestamp (in seconds) of when the batch job failed. The Unix timestamp (in seconds) of when the batch job expired. The Unix timestamp (in seconds) of when the batch job started cancelling. The Unix timestamp (in seconds) of when the batch job was cancelled. Statistics about the requests in the batch (if available). Custom metadata associated with the batch job. When you cancel an in-progress batch, it will be in `cancelling` status for up to 10 minutes before changing to `cancelled`, where it will have partial results (if any) available in the output file. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) batch = client.batches.cancel("batch_abc123") print(batch) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const batch = await client.batches.cancel("batch_abc123"); console.log(batch); } main(); ``` ```bash cURL theme={null} curl --location --request DELETE 'https://api.cerebras.ai/v1/batches/batch_abc123' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" ``` ```json Response theme={null} { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file-abc123", "completion_window": "24h", "status": "calling", "output_file_id": null, "error_file_id": null, "created_at": 1765492780, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": 1765493188, "failed_at": null, "expired_at": null, "cancelling_at": 1765492797, "cancelled_at":null, "request_counts": null, "metadata": null } ``` # Create batch Source: https://inference-docs.cerebras.ai/api-reference/batch/create-batch POST https://api.cerebras.ai/v1/batches This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. See the [Batch](/capabilities/batch) guide for more information. ## Request The ID of an uploaded file that contains requests for the batch job. The file must follow these constraints: * Max size of 200 MB * Uploaded with `purpose: "batch"` * Max 50,000 requests The API endpoint for batch processing. Currently only `/v1/chat/completions` is supported. The time window for batch completion. Currently, only `24h` is supported. Key-value pairs that can be attached to an object. Optional custom metadata for the batch job. Useful for organizing and tracking batch jobs. ## Response A unique identifier for the batch job. The object type, which is always `batch`. The API endpoint used for batch processing. Information about any errors that occurred during batch processing. The ID of the input file containing batch requests. The time window for batch completion. Always `24h`. The current status of the batch job. Possible values: `queued`, `in_progress`, `finalizing`, `completed`, `expired`, `failed`, `cancelled`, `cancelling`. The ID of the file containing batch results (available once processing is complete). The ID of the file containing errors (if any errors occurred). The Unix timestamp (in seconds) of when the batch job was created. The Unix timestamp (in seconds) of when the batch job started processing. The Unix timestamp (in seconds) of when the batch job will expire. The Unix timestamp (in seconds) of when the batch job started finalizing. The Unix timestamp (in seconds) of when the batch job completed. The Unix timestamp (in seconds) of when the batch job failed. The Unix timestamp (in seconds) of when the batch job expired. The Unix timestamp (in seconds) of when the batch job started cancelling. The Unix timestamp (in seconds) of when the batch job was cancelled. Statistics about the requests in the batch. Total number of requests in the batch. Number of requests that completed successfully. Number of requests that failed. Custom metadata associated with the batch job. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) batch = client.batches.create( input_file_id="file_abc123", endpoint="/v1/chat/completions", completion_window="24h", metadata={ "custom_tags": ["legal-review", "q3"] } ) print(batch) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const batch = await client.batches.create({ input_file_id: "file_abc123", endpoint: "/v1/chat/completions", completion_window: "24h", metadata: { custom_tags: ["legal-review", "q3"] } }); console.log(batch); } main(); ``` ```bash cURL theme={null} curl --location 'https://api.cerebras.ai/v1/batches' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" \ --data '{ "input_file_id": "file_abc123", "endpoint": "/v1/chat/completions", "completion_window": "24h", "metadata": { "custom_tags": ["legal-review", "q3"] } }' ``` ```json Response theme={null} { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file_abc123", "completion_window": "24h", "status": "validating", "output_file_id": null, "error_file_id": null, "created_at": 1766003277, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 0, "completed": 0, "failed": 0 }, "metadata": null } ``` # List batch Source: https://inference-docs.cerebras.ai/api-reference/batch/list-batch GET https://api.cerebras.ai/v1/batches This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. List all batch jobs associated with your organization. Supports pagination. ## Query Parameters The maximum number of files to return. Default: `20`, Maximum: `100`. A cursor for pagination. Pass a batch ID to retrieve all files created after that file. For example, if you have files `[batch-abc123, batch-def456, batch-jkl789, batch-mno123]` and pass `after=batch-def456`, the response will return `[batch-jkl789, batch-mno123]`. ## Response The object type, which is always `list`. An array of file objects that show file properties. Indicates whether there are more files available beyond this page. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) batch = client.batches.list() print(batch) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const list = await client.batches.list(); for await (const batch of list) { console.log(batch); } } main(); ``` ```bash cURL theme={null} curl --location --request GET 'https://api.cerebras.ai/v1/batches' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" ``` ```json Response theme={null} { "object": "list", "data": [ { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": "Model 'cognition-devstral-small-2508-pc' is not supported for batch processing. Supported models: gpt-oss-120b", "input_file_id": "file-abc123", "completion_window": "24h", "status": "failed", "output_file_id": null, "error_file_id": null, "created_at": 1765490209, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": 1765490211, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 0, "completed": 0, "failed": 0 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file-abc123", "completion_window": "24h", "status": "completed", "output_file_id": "file-abc123", "error_file_id": null, "created_at": 1765227145, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": 1765227301, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 656, "completed": 656, "failed": 0 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file-abc123", "completion_window": "24h", "status": "cancelled", "output_file_id": null, "error_file_id": null, "created_at": 1765492137, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": 1765492173, "request_counts": { "total": 40, "completed": 0, "failed": 0 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": "File service error: Failed to retrieve file content: Access denied to file file-8zOP2tNFQW2b7G4u3hsKDQ", "input_file_id": "file-abc123", "completion_window": "24h", "status": "failed", "output_file_id": null, "error_file_id": null, "created_at": 1765344102, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": 1765344103, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 0, "completed": 0, "failed": 0 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file-abc123", "completion_window": "24h", "status": "validating", "output_file_id": null, "error_file_id": null, "created_at": 1765211147, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 0, "completed": 0, "failed": 0 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file-abc123", "completion_window": "24h", "status": "completed", "output_file_id": "file-abc123", "error_file_id": "file-abc123", "created_at": 1765269860, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": 1765270045, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 4, "completed": 1, "failed": 3 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file-abc123", "completion_window": "24h", "status": "completed", "output_file_id": null, "error_file_id": null, "created_at": 1765223136, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": 1765223175, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 13, "completed": 0, "failed": 13 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": "Model 'LLAMA3.1-8b' is not supported for batch processing. Supported models: gpt-oss-120b", "input_file_id": "file-abc123", "completion_window": "24h", "status": "failed", "output_file_id": null, "error_file_id": null, "created_at": 1765344330, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": 1765344332, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 0, "completed": 0, "failed": 0 }, "metadata": null } ], "first_id": "batch_abc123", "last_id": "batch_abc123", "has_more": true } ``` # Retrieve batch Source: https://inference-docs.cerebras.ai/api-reference/batch/retrieve-batch GET https://api.cerebras.ai/v1/batches/{batch_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Retrieves the status and progress of a batch job. Use this endpoint to monitor your batch job as it processes. ## Path Parameters The ID of the batch job to retrieve. ## Response A unique identifier for the batch job. The object type, which is always `batch`. The API endpoint used for batch processing. Information about any errors that occurred during batch processing. The ID of the input file containing batch requests. The time window for batch completion. Always `24h`. The current status of the batch job. Possible values: `validating`, `queued`, `in_progress`, `finalizing`, `completed`, `expired`, `failed`, `cancelled`, `cancelling`. The ID of the file containing batch results (available once processing is complete). The ID of the file containing errors (if any errors occurred). The Unix timestamp (in seconds) of when the batch job was created. The Unix timestamp (in seconds) of when the batch job started processing. The Unix timestamp (in seconds) of when the batch job will expire. The Unix timestamp (in seconds) of when the batch job started finalizing. The Unix timestamp (in seconds) of when the batch job completed. The Unix timestamp (in seconds) of when the batch job failed. The Unix timestamp (in seconds) of when the batch job expired. The Unix timestamp (in seconds) of when the batch job started cancelling. The Unix timestamp (in seconds) of when the batch job was cancelled. Statistics about the requests in the batch. Total number of requests in the batch. Number of requests that completed successfully. Number of requests that failed. Custom metadata associated with the batch job. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) batch = client.batches.retrieve("batch_abc123") print(batch) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const batch = await client.batches.retrieve("batch_abc123"); console.log(batch); } main(); ``` ```bash cURL theme={null} curl --location 'https://api.cerebras.ai/v1/batches/batch_abc123' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" ``` ```json Response theme={null} { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file_abc123", "completion_window": "24h", "status": "in_progress", "output_file_id": null, "error_file_id": null, "created_at": 1766003277, "in_progress_at": 1766003300, "expires_at": 1766089677, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 100, "completed": 75, "failed": 2 }, "metadata": { "custom_tags": ["legal-review", "q3"] } } ``` # Chat Completions Source: https://inference-docs.cerebras.ai/api-reference/chat-completions POST /v1/chat/completions Generate conversational responses using a structured message format with roles (system, user, assistant, developer, tool). Best for chatbots, assistants, and multi-turn conversations. Parameter support can differ depending on the model used to generate the response, particularly for newer reasoning models. For details about parameters in reasoning models, refer to the [Reasoning Guide](/capabilities/reasoning). # Completions Source: https://inference-docs.cerebras.ai/api-reference/completions POST https://api.cerebras.ai/v1/completions Generate text continuations from one or more prompts. Best for simple text generation, autocomplete, and single-turn tasks. Image inputs, structured chat messages, reasoning controls, structured outputs, and tools require [Chat Completions](/api-reference/chat-completions). ## Request ### Headers The media type of the request body. **Possible values:** `application/json`, `application/vnd.msgpack` **Default:** `application/json` See [Payload Optimization](/capabilities/payload-optimization) for details. The compression encoding applied to the request body. **Possible values:** `gzip` When set, the request body must be gzip-compressed. Can be combined with any supported `Content-Type`. See [Payload Optimization](/capabilities/payload-optimization) for details. ### Body The prompt(s) to generate completions for, encoded as a string, array of strings, array of tokens, or array of token arrays. The model to use for completion. See [Supported Models](/models/overview) for a list of available models. Streams partial completion data as data-only server-sent events as tokens become available. The stream ends with a `data: [DONE]` message. Default: `false` Returns raw tokens instead of text. Default: `false` The maximum number of tokens that can be generated in the completion. The total length of input tokens and generated tokens is limited by the model's context length. Default: `null` The minimum number of tokens to generate for a completion. If not specified or set to 0, the model will generate as many tokens as it deems necessary. Setting to -1 sets to max sequence length. Default: `null` Makes a best effort to sample deterministically so repeated requests with the same `seed` and parameters return the same result. Determinism is not guaranteed. Default: `null` Up to four sequences that cause the API to stop generating further tokens. The returned text does not contain the stop sequence. Default: `null` Sampling temperature between 0 and 2. Higher values, such as 0.8, make output more random; lower values, such as 0.2, make it more focused and deterministic. We recommend changing either this value or `top_p`, but not both. Constraints: min: `0`, max: `2` Default: `1` Nucleus sampling parameter between 0 and 1. The model considers only tokens comprising the top `top_p` probability mass. For example, 0.1 means only tokens comprising the top 10% probability mass are considered. We recommend changing either this value or `temperature`, but not both. Constraints: min: `0`, max: `1` Default: `1` Returns the prompt in addition to the completion. This parameter is incompatible with `return_raw_tokens: true`. Default: `false` A unique identifier representing your end-user, which can help Cerebras to monitor and detect abuse. Default: `null` An opaque identifier that groups related requests so they reuse the same [prompt cache](/capabilities/prompt-caching). Requests sharing the same `prompt_cache_key` are routed together, which increases cache hits and reduces time to first token. Set it to a stable conversation, session, or workflow ID. Maximum length: `1024` characters. Default: `null` Returns log probabilities of the output tokens. For example, if `logprobs` is `5`, the API returns the five most likely tokens. The API always returns the log probability of the sampled token, so there may be up to `logprobs + 1` elements in the response. Constraints: min: `0`, max: `20` Default: `null` Setting `logprobs` to `0` is different from setting it to `null`. A value of `null` disables log probabilities. A value of `0` returns the sampled token's log probability without alternative token log probabilities. ## Completion Response The list of completion choices the model generated for the input prompt. The reason the model stopped generating tokens. This will be `stop` if the model hit a natural stop point or a provided stop sequence, `length` if the maximum number of tokens specified in the request was reached, or `content_filter` if content was omitted due to a flag from our content filters. Number of characters since the prompt. Log probability for each token. The tokens. The most likely tokens and their log probabilities at this token position. In rare cases, the response may contain fewer alternatives than requested. The generated completion text. The raw tokens of the completion, returned when `return_raw_tokens` is set to `true`. The Unix timestamp (in seconds) of when the completion was created. A unique identifier for the completion. The model used for completion. The object type, which is always `text_completion`. Identifies the backend configuration used by the model. Use it with the `seed` request parameter to determine whether backend changes might affect determinism. Usage statistics for the completion request. Number of tokens in the prompt. Number of tokens in the generated completion. Total number of tokens used in the request (prompt + completion). Detailed breakdown of prompt token usage. Number of prompt tokens that were served from the cache and reused from a previous request. See [Prompt Caching](/capabilities/prompt-caching) for more information. Performance timing information for the request. Time spent in queue waiting for processing (in seconds). Time spent processing the prompt/input tokens (in seconds). Time spent generating the completion/output tokens (in seconds). Total time for the entire request from submission to completion (in seconds). Unix timestamp (in seconds) of when the time\_info was recorded. ```python Python theme={null} import os from cerebras.cloud.sdk import Cerebras client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), # This is the default and can be omitted ) completion = client.completions.create( prompt="It was a dark and stormy night", max_tokens=100, model="qwen-3.8-27b", logprobs=5, ) print(completion) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], // This is the default and can be omitted }); async function main() { const completion = await client.completions.create({ prompt: "It was a dark and stormy night", model: 'qwen-3.8-27b', logprobs: 5, }); console.log(completion?.choices[0]?.text); } main(); ``` ```cli cURL theme={null} curl -X POST https://api.cerebras.ai/v1/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "prompt": "It was a dark and stormy night", "max_tokens": 100, "model": "qwen-3.8-27b", "logprobs": 5 }' ``` ```json Response theme={null} { "id": "chatcmpl-b1743f2d-8c20-4ad5-a77c-426521ae1b1c", "choices": [ { "index": 0, "finish_reason": "length", "logprobs": { "text_offset": [30, 34, 40, 44, 47, 52], "token_logprobs": [ -3.3912997245788574, -4.322617053985596, -0.3532562553882599, -1.7001153230667114, -2.187103271484375, -2.4667720794677734 ], "tokens": [" and", " there", " was", " no", " moon", ","], "top_logprobs": [ { ".": -1.1334871053695679, ",": -1.3366121053695679, " when": -2.3366122245788574, " in": -2.9381747245788574, "...": -3.3287997245788574, " and": -3.3912997245788574 }, { " I": -0.9554294943809509, " the": -1.7679295539855957, " a": -2.2366795539855957, " all": -3.1898045539855957, " we": -3.4241795539855957, " there": -4.322617053985596 }, { " was": -0.3532562553882599, " were": -1.6813812255859375, " I": -4.0251312255859375, " wasn": -4.1345062255859375, " had": -4.1970062255859375 }, { " a": -0.3719903528690338, " no": -1.7001153230667114, " an": -3.614177942276001, " this": -4.129802703857422, " nothing": -4.184490203857422 }, { " electricity": -1.3433531522750854, " one": -1.6949156522750854, " moon": -2.187103271484375, " way": -3.038665771484375, " sign": -3.054290771484375 }, { ".": -0.8495846390724182, " to": -2.1386470794677734, " in": -2.2870845794677734, ",": -2.4667720794677734, ".\n": -3.2714595794677734 } ] }, "text": " and there was no moon,", "tokens": null } ], "created": 1769297019, "model": "qwen-3.8-27b", "object": "text_completion", "system_fingerprint": "fp_feb5e1faa8274e54bef0", "time_info": { "completion_time": 0.002412253, "prompt_time": 0.000347391, "queue_time": 0.000245702, "total_time": 0.004331350326538086, "created": 1769297019.2632425 }, "usage": { "completion_tokens": 6, "prompt_tokens": 8, "prompt_tokens_details": { "cached_tokens": 0 }, "total_tokens": 14 } } ``` # Delete model version Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/delete-model-version delete /management/v1/orgs/{org_name}/models/{model_arch_id}/versions/{version_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Delete a specific model version and its associated artifacts from Cerebras. # Deploy model to endpoint Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/deploy-model-to-endpoint post /management/v1/endpoints/{endpoint_id}:deployModel This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Deploy a model version to a dedicated endpoint running a model with the same underlying architecture. The endpoint queues the deployment operation and returns a deployment ID for tracking status. # List endpoints Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/list-endpoints get /management/v1/orgs/{org_name}/endpoints This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. List all endpoints accessible to an organization for management. # List model architectures Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/list-model-architectures get /management/v1/orgs/{org_name}/models This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. List model architectures accessible to an organization. # List model versions Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/list-model-versions get /management/v1/orgs/{org_name}/models/{model_arch_id}/versions This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Get all uploaded model versions for a particular supported architecture. # Retrieve endpoint status Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/retrieve-endpoint-status get /management/v1/endpoints/{endpoint_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Describes the status of an endpoint, including deployment status and active model versions. # Retrieve model version status Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/retrieve-model-version-status get /management/v1/orgs/{org_name}/models/{model_arch_id}/versions/{version_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Retrieve the status and details of a specific model version, including upload status. # Update model version aliases Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/update-model-version-aliases patch /management/v1/orgs/{org_name}/models/{model_arch_id}/versions/{version_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Assign unique aliases to model versions instead of using autogenerated IDs. Use these aliases anywhere you would normally use an integer version ID. # Upload model version Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/upload-model-version post /management/v1/orgs/{org_name}/models:upload This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Upload a new finetuned model version to Cerebras. Model versions are custom variants of Cerebras-supported model architectures. Before using this endpoint, you must configure an S3 bucket with cross-account access to Cerebras. See [S3 Bucket Setup](/dedicated/management-api#s3-bucket-setup) for instructions. # Delete file Source: https://inference-docs.cerebras.ai/api-reference/file/delete-file DELETE https://api.cerebras.ai/v1/files/{file_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Deletes a file before its automatic expiration. This removes the file from storage immediately. ## Path Parameters The ID of the file to delete. ## Response The ID of the deleted file. The object type, which is always `file`. Confirms whether the file was successfully deleted. Deleting a file removes it from storage before its automatic expiration date. Files are automatically deleted after their retention period expires (7 days by default). ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) result = client.files.delete("file_123") print(result) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const result = await client.files.delete("file_123"); console.log(result); } main(); ``` ```bash cURL theme={null} curl --location --request DELETE 'https://api.cerebras.ai/v1/files/file_123' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" ``` ```json Response theme={null} { "id": "file_123", "object": "file", "deleted": true } ``` # List files Source: https://inference-docs.cerebras.ai/api-reference/file/list-files GET https://api.cerebras.ai/v1/files/ This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Lists all files that have been uploaded. Returns metadata for each file including size, status, and expiration. ## Query Parameters The maximum number of files to return. Default: `20`\ Maximum: `100` A cursor for pagination. Pass a file ID to retrieve all files created after that file. For example, if you have files `[file-abc123, file-def456, file-jkl789, file-mno123]` and pass `after=file-def456`, the response will return `[file-jkl789, file-mno123]`. ## Response The object type, which is always `list`. An array of file objects. A unique identifier for the file. The object type, which is always `file`. The size of the file in bytes. The name of the uploaded file. The Unix timestamp (in seconds) of when the file was created. The Unix timestamp (in seconds) of when the file will be automatically deleted. The intended purpose of the file (e.g., `batch`). Whether there are more files available beyond this page. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) files = client.files.list(limit=20) print(files) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const files = await client.files.list({ limit: 20 }); console.log(files); } main(); ``` ```bash cURL theme={null} curl --location 'https://api.cerebras.ai/v1/files?limit=20/' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" ``` ```json Response theme={null} { "object": "list", "data": [ { "id": "file_123", "object": "file", "bytes": 1048576, "filename": "batch-requests.jsonl", "created_at": 1726315200, "expires_at": 1726920000, "purpose": "batch" }, { "id": "file_124", "object": "file", "bytes": 2097152, "created_at": 1726315200, "expires_at": 1726920000, "filename": "batch-results.jsonl", "purpose": "batch" } ], "has_more": false } ``` # Retrieve file Source: https://inference-docs.cerebras.ai/api-reference/file/retrieve-file GET https://api.cerebras.ai/v1/files/{file_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Retrieves metadata for a specific file, including its size, status, and expiration date. ## Path Parameters The ID of the file to retrieve. ## Response A unique identifier for the file. The object type, which is always `file`. The name of the uploaded file. The intended purpose of the file. The size of the file in bytes. The Unix timestamp (in seconds) of when the file was created. The Unix timestamp (in seconds) of when the file will be automatically deleted. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) file = client.files.retrieve("file_123") print(file) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const file = await client.files.retrieve("file_123"); console.log(file); } main(); ``` ```bash cURL theme={null} curl --location 'https://api.cerebras.ai/v1/files/file_123' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" ``` ```json Response theme={null} { "id": "file_123", "object": "file", "bytes": 1048576, "filename": "batch-requests.jsonl", "created_at": 1726315200, "expires_at": 1726920000, "purpose": "batch" } ``` # Retrieve file content Source: https://inference-docs.cerebras.ai/api-reference/file/retrieve-file-content GET https://api.cerebras.ai/v1/files/{file_id}/content This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Downloads the raw content of a file. Use this endpoint to retrieve the actual file data rather than just metadata. ## Path Parameters The ID of the file whose content to retrieve. ## Response Returns the raw file contents. The `Content-Type` header indicates the file type: * JSONL files: `application/x-ndjson` ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) content = client.files.retrieve_content("file_123") # Save to file with open("downloaded-file.jsonl", "wb") as f: f.write(content) print("File downloaded successfully") ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; import * as fs from 'fs'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const content = await client.files.retrieveContent("file_123"); // Save to file fs.writeFileSync("downloaded-file.jsonl", content); console.log("File downloaded successfully"); } main(); ``` ```bash cURL theme={null} curl --location 'https://api.cerebras.ai/v1/files/file_123/content' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" \ --output downloaded-file.jsonl ``` ```text Response theme={null} Returns raw file content with appropriate Content-Type header. For JSONL files, returns newline-delimited JSON. ``` # Upload file Source: https://inference-docs.cerebras.ai/api-reference/file/upload-file POST https://api.cerebras.ai/v1/files This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Uploads a file for use with the [Batch API](/api-reference/batch/create-batch). Files can be up to **200 MB** and must be **JSONL** format. Files are retained for 7 days by default. ## Request The file to upload. Must be a binary file. The intended purpose of the file. Currently supported values: * `batch`: Input file for batch processing The expiration policy for the file. Files with `purpose=batch` expire after 7 days by default. Anchor timestamp after which the expiration policy applies. Must be `created_at`. The number of seconds after the anchor point when the file expires. Must be between 3600 (1 hour) and 2592000 (30 days). ## Response A unique identifier for the file. The object type, which is always `file`. The name of the uploaded file. The intended purpose of the file. The size of the file in bytes. The Unix timestamp (in seconds) of when the file was created. The Unix timestamp (in seconds) of when the file will be automatically deleted. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) file = client.files.create( file=open("batch-requests.jsonl", "rb"), purpose="batch" ) print(file) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; import * as fs from 'fs'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const file = await client.files.create({ file: fs.createReadStream("batch-requests.jsonl"), purpose: "batch" }); console.log(file); } main(); ``` ```bash cURL theme={null} curl --location 'https://api.cerebras.ai/v1/files' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" \ --form 'file=@"batch-requests.jsonl"' \ --form 'purpose="batch"' ``` ```json Response theme={null} { "id": "fileI72lQshwSO6Z7LIQ850CvA", "object": "file", "filename": "test.jsonl", "bytes": 4213, "purpose": "batch", "created_at": 1765172961, "expires_at": 1767764961 } ``` # Retrieve metrics Source: https://inference-docs.cerebras.ai/api-reference/metrics/retrieve-metrics GET https://cloud.cerebras.ai/api/v1/metrics/organizations/{organization_id} Retrieve operational metrics for your organization's inference endpoints in Prometheus format. See the [Metrics](/capabilities/metrics) guide for more info. ## Path Parameters The unique identifier for your organization (e.g., `org_abc123`) ## Response Returns metrics in Prometheus text-based exposition format. ```bash cURL theme={null} curl -H "Authorization: Bearer $CEREBRAS_API_KEY" \ https://cloud.cerebras.ai/api/v1/metrics/organizations/org_abc123 ``` ```python Python theme={null} import requests import os response = requests.get( "https://cloud.cerebras.ai/api/v1/metrics/organizations/org_abc123", headers={ "Authorization": f"Bearer {os.environ.get('CEREBRAS_API_KEY')}" } ) print(response.text) ``` ```javascript Node.js theme={null} const response = await fetch( 'https://cloud.cerebras.ai/api/v1/metrics/organizations/org_abc123', { headers: { 'Authorization': `Bearer ${process.env.CEREBRAS_API_KEY}` } } ); const metrics = await response.text(); console.log(metrics); ``` ```prometheus Response theme={null} # HELP inference_endpoint_status Status of inference endpoint (-1=error calculating status, 0=down, 1=up) # TYPE inference_endpoint_status gauge inference_endpoint_status{endpoint="model",organization_id="org_abc123"} 1.0 # HELP requests_count_total Total request count (all HTTP codes) in the last complete minute # TYPE requests_count_total gauge requests_count_total{endpoint="model",organization_id="org_abc123"} 1.0 # HELP requests_success_total Total successful requests (HTTP 200) in the last complete minute # TYPE requests_success_total gauge requests_success_total{endpoint="model",organization_id="org_abc123"} 1.0 # HELP requests_failure_total Total failed requests by HTTP code in the last complete minute # TYPE requests_failure_total gauge requests_failure_total{endpoint="model",organization_id="org_abc123"} 0.0 # HELP input_tokens_total Total input tokens for successful requests in the last complete minute # TYPE input_tokens_total gauge input_tokens_total{endpoint="model",organization_id="org_abc123"} 123456.0 # HELP output_tokens_total Total output tokens for successful requests in the last complete minute # TYPE output_tokens_total gauge output_tokens_total{endpoint="model",organization_id="org_abc123"} 12345.0 # HELP queue_time_seconds Queue time percentiles in seconds # TYPE queue_time_seconds gauge queue_time_seconds{endpoint="model",organization_id="org_abc123",percentile="avg"} 0.55 queue_time_seconds{endpoint="model",organization_id="org_abc123",percentile="p50"} 0.55 queue_time_seconds{endpoint="model",organization_id="org_abc123",percentile="p90"} 0.55 queue_time_seconds{endpoint="model",organization_id="org_abc123",percentile="p95"} 0.55 queue_time_seconds{endpoint="model",organization_id="org_abc123",percentile="p99"} 0.55 # HELP e2e_latency_seconds End-to-end API latency percentiles in seconds # TYPE e2e_latency_seconds gauge e2e_latency_seconds{endpoint="model",organization_id="org_abc123",statistic="avg"} 0.9 e2e_latency_seconds{endpoint="model",organization_id="org_abc123",statistic="p50"} 0.9 e2e_latency_seconds{endpoint="model",organization_id="org_abc123",statistic="p90"} 0.9 e2e_latency_seconds{endpoint="model",organization_id="org_abc123",statistic="p95"} 0.9 e2e_latency_seconds{endpoint="model",organization_id="org_abc123",statistic="p99"} 0.9 # HELP ttft_seconds Time To First Token percentiles in seconds # TYPE ttft_seconds gauge ttft_seconds{endpoint="model",organization_id="org_abc123",statistic="avg"} 0.9 ttft_seconds{endpoint="model",organization_id="org_abc123",statistic="p50"} 0.9 ttft_seconds{endpoint="model",organization_id="org_abc123",statistic="p90"} 0.9 ttft_seconds{endpoint="model",organization_id="org_abc123",statistic="p95"} 0.9 ttft_seconds{endpoint="model",organization_id="org_abc123",statistic="p99"} 0.9 # HELP cache_reads_total Total input tokens read from cache for successful requests in the last complete minute # TYPE cache_reads_total gauge cache_reads_total{endpoint="model",organization_id="org_abc123"} 1234.0 # HELP cache_rate Ratio of input tokens read from cache, to total input tokens, for successful requests in the last complete minute # TYPE cache_rate gauge cache_rate{endpoint="model",organization_id="org_abc123"} 0.01 # HELP tpot Time Per Output Token (TPOT) percentiles # TYPE tpot gauge tpot{endpoint="model",organization_id="org_abc123",statistic="avg"} 0.0001 tpot{endpoint="model",organization_id="org_abc123",statistic="p50"} 0.0001 tpot{endpoint="model",organization_id="org_abc123",statistic="p90"} 0.0001 tpot{endpoint="model",organization_id="org_abc123",statistic="p95"} 0.0001 tpot{endpoint="model",organization_id="org_abc123",statistic="p99"} 0.0001 # HELP latency_generation_seconds Completion time percentiles in seconds # TYPE latency_generation_seconds gauge latency_generation_seconds{endpoint="model",organization_id="org_abc123",statistic="avg"} 1.1 latency_generation_seconds{endpoint="model",organization_id="org_abc123",statistic="p50"} 1.1 latency_generation_seconds{endpoint="model",organization_id="org_abc123",statistic="p90"} 1.1 latency_generation_seconds{endpoint="model",organization_id="org_abc123",statistic="p95"} 1.1 latency_generation_seconds{endpoint="model",organization_id="org_abc123",statistic="p99"} 1.1 ``` ## Available Metrics The following metrics are available on an opt-in basis. Contact your Cerebras account representative to enable specific metrics for your organization. ### Endpoint Health Status of inference endpoint **Values:** * `-1` = Error calculating status * `0` = Down * `1` = Up ### Request Metrics Total request count (all HTTP codes) in the last complete minute Total successful requests (HTTP 200) in the last complete minute Total failed requests by HTTP code in the last complete minute ### Token Metrics Total input tokens for successful requests in the last complete minute Total output tokens for successful requests in the last complete minute Total input tokens read from cache for successful requests in the last complete minute Ratio of input tokens read from cache, to total input tokens, for successful requests in the last complete minute ### Latency Metrics Queue time percentiles in seconds for successful requests (avg/p50/p90/p95/p99) (e.g. time a request spends waiting for resources at runtime) End-to-end API latency percentiles in seconds for successful requests (avg/p50/p90/p95/p99). Includes overall latency from requests received at the API gateway to the response output from API gateway, inclusive of `latency_generation_seconds`. Time To First Token percentiles in seconds for successful requests (avg/p50/p90/p95/p99) Time per output tokens percentiles (avg/p50/p90/p95/p99), excluding time to first token, averaged across successful requests Time to generate all output tokens (e.g. time from last prompt to last output token) percentiles in seconds for successful requests (avg/p50/p90/p95/p99) ## Error Codes For information about possible error responses, see the [Error Codes](/support/error) documentation. # List models Source: https://inference-docs.cerebras.ai/api-reference/models/list-models GET https://api.cerebras.ai/v1/models Lists all currently available models and provides essential details about each, including the owner and availability. ## Response The object type, which is always `list`. An array of model objects. The model identifier, which can be referenced in the API endpoints. The object type, which is always `model`. The Unix timestamp (in seconds) of when the model was created. The organization that owns the model. ```python Python theme={null} import os from cerebras.cloud.sdk import Cerebras client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) models = client.models.list() print(models) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const models = await client.models.list(); console.log(models); } main(); ``` ```bash cURL theme={null} curl https://api.cerebras.ai/v1/models \ -H "Authorization: Bearer $CEREBRAS_API_KEY" ``` ```json Response theme={null} { "object": "list", "data": [ { "id": "gpt-oss-120b", "object": "model", "created": 0, "owned_by": "Cerebras" }, { "id": "qwen-3.8-27b", "object": "model", "created": 0, "owned_by": "Cerebras" } ] } ``` # Public models Source: https://inference-docs.cerebras.ai/api-reference/models/public-models GET https://api.cerebras.ai/public/v1/models Retrieve details about publicly available Cerebras models without an API key, including context length, pricing, and supported features. This endpoint supports multiple response formats for compatibility with different platforms. This endpoint is **public** and does not require an API key. ## Supported Formats The endpoint supports three response formats via the `format` query parameter: * **Default (Cerebras)** - Native Cerebras format * **OpenRouter** - OpenRouter-compatible format * **HuggingFace** - HuggingFace-compatible format ## Query Parameters Response format. Options: `openrouter`, `huggingface`. Omit for default Cerebras format. ## Response ### Default Format The object type, which is always `list` for list responses or `model` for single model responses. Array of model objects (only in list responses). The model identifier (e.g., `gpt-oss-120b`). The object type, which is always `model`. The Unix timestamp (in seconds) of when the model was created. The organization that owns the model. Human-readable model name. Description of the model's capabilities and recommended use cases. The HuggingFace repository identifier for this model. Pricing per token in USD. Cost per prompt token. Cost per completion token. Boolean flags for supported features. Whether streaming is supported. Whether function calling is supported. Whether structured outputs are supported. Whether vision/image input is supported. Whether JSON mode is supported. Whether tool use is supported. Whether tool choice is supported. Whether parallel tool calls are supported. Whether response format control is supported. Whether the model supports reasoning. Boolean flags for supported API parameters. Model architecture details. Input/output modality (e.g., `text`). Tokenizer type. Instruction tuning format. Model limits. Maximum context window in tokens. Maximum completion tokens. Requests per minute limit (null if unlimited). Tokens per minute limit (null if unlimited). Whether this model is deprecated. Whether this model is in preview. Quantization method used for the model. ### OpenRouter Format Array of model objects with extended metadata. The model identifier. The HuggingFace repository identifier. Human-readable model name. Supported input types (e.g., `["text"]`). Supported output types (e.g., `["text"]`). Maximum context window size in tokens. Maximum output length in tokens. Pricing information per token. Cost per prompt token. Cost per completion token. Cost per request (typically `"0"`). Cost per image token (typically `"0"` for text-only models). Cost per cached input token read (typically `"0"`). Cost per cached input token write (typically `"0"`). List of supported sampling parameters. List of supported features (e.g., `["tools", "json_mode", "structured_outputs"]`). ### HuggingFace Format Array of model objects. The model identifier. The HuggingFace repository identifier. The object type, which is always `model`. The Unix timestamp (in seconds) of when the model was created. The organization that owns the model. Pricing in USD per million tokens. Price per million input tokens. Price per million output tokens. Maximum context window size in tokens. Boolean flags for supported features. ```bash cURL (Default) theme={null} curl -sS 'https://api.cerebras.ai/public/v1/models' | jq ``` ```bash cURL (OpenRouter) theme={null} curl -sS 'https://api.cerebras.ai/public/v1/models?format=openrouter' | jq ``` ```bash cURL (HuggingFace) theme={null} curl -sS 'https://api.cerebras.ai/public/v1/models?format=huggingface' | jq ``` ```python Python theme={null} import httpx # Default format response = httpx.get("https://api.cerebras.ai/public/v1/models") models = response.json() # OpenRouter format response = httpx.get( "https://api.cerebras.ai/public/v1/models", params={"format": "openrouter"} ) models = response.json() # HuggingFace format response = httpx.get( "https://api.cerebras.ai/public/v1/models", params={"format": "huggingface"} ) models = response.json() ``` ```javascript JavaScript theme={null} // Default format const response = await fetch('https://api.cerebras.ai/public/v1/models'); const models = await response.json(); // OpenRouter format const response = await fetch( 'https://api.cerebras.ai/public/v1/models?format=openrouter' ); const models = await response.json(); // HuggingFace format const response = await fetch( 'https://api.cerebras.ai/public/v1/models?format=huggingface' ); const models = await response.json(); ``` ```json Default Format theme={null} { "object": "list", "data": [ { "id": "gpt-oss-120b", "object": "model", "created": 1754438400, "owned_by": "OpenAI", "name": "OpenAI GPT OSS", "description": "This model excels at efficient reasoning across science, math, and coding applications. It's ideal for real-time coding assistance, processing large documents for Q&A and summarization, agentic research workflows, and regulated on-premises workloads.", "hugging_face_id": "openai/gpt-oss-120b", "pricing": { "prompt": "0.00000035", "completion": "0.00000075" }, "capabilities": { "streaming": true, "function_calling": true, "structured_outputs": true, "vision": false, "json_mode": true, "tools": true, "tool_choice": true, "parallel_tool_calls": false, "response_format": true, "reasoning": true }, "supported_parameters": { "temperature": true, "top_p": true, "seed": true, "stop": true, "max_completion_tokens": true, "logprobs": true, "top_logprobs": true, "frequency_penalty": true, "presence_penalty": true, "logit_bias": true, "repetition_penalty": false }, "architecture": { "modality": "text", "tokenizer": "GPT", "instruct_type": "harmony" }, "limits": { "max_context_length": 131072, "max_completion_tokens": 40960, "requests_per_minute": null, "tokens_per_minute": null }, "datacenter_locations": [], "deprecated": false, "preview": false, "quantization": "FP16/8 (weights only)" } ] } ``` ```json OpenRouter Format theme={null} { "data": [ { "id": "gpt-oss-120b", "hugging_face_id": "openai/gpt-oss-120b", "name": "OpenAI GPT OSS", "created": 1754438400, "input_modalities": ["text"], "output_modalities": ["text"], "quantization": "fp16", "context_length": 131072, "max_output_length": 40960, "pricing": { "prompt": "0.00000035", "completion": "0.00000075", "request": "0", "image": "0", "input_cache_read": "0", "input_cache_write": "0" }, "supported_sampling_parameters": [ "temperature", "top_p", "stop", "seed", "frequency_penalty", "presence_penalty", "max_tokens", "logprobs", "top_logprobs", "logit_bias" ], "supported_features": [ "tools", "json_mode", "structured_outputs", "reasoning" ] } ] } ``` ```json HuggingFace Format theme={null} { "data": [ { "id": "gpt-oss-120b", "hugging_face_id": "openai/gpt-oss-120b", "object": "model", "created": 1754438400, "owned_by": "OpenAI", "context_length": 131072, "pricing": { "input": 0.35, "output": 0.75 }, "capabilities": { "streaming": true, "function_calling": true, "structured_outputs": true, "vision": false } } ] } ``` ## Retrieve Specific Model You can also retrieve information about a specific model by appending the model ID to the endpoint: ``` GET https://api.cerebras.ai/public/v1/models/{model_id} ``` ### Path Parameters The ID of the model to retrieve (for example, `gpt-oss-120b` or `qwen-3.8-27b`). ```bash cURL theme={null} curl -sS 'https://api.cerebras.ai/public/v1/models/gpt-oss-120b' | jq ``` ```python Python theme={null} import httpx model_id = "gpt-oss-120b" response = httpx.get(f"https://api.cerebras.ai/public/v1/models/{model_id}") model = response.json() ``` ```javascript JavaScript theme={null} const modelId = 'gpt-oss-120b'; const response = await fetch( `https://api.cerebras.ai/public/v1/models/${modelId}` ); const model = await response.json(); ``` ```json Response theme={null} { "id": "gpt-oss-120b", "object": "model", "created": 1754438400, "owned_by": "OpenAI", "name": "OpenAI GPT OSS", "description": "This model excels at efficient reasoning across science, math, and coding applications. It's ideal for real-time coding assistance, processing large documents for Q&A and summarization, agentic research workflows, and regulated on-premises workloads.", "hugging_face_id": "openai/gpt-oss-120b", "pricing": { "prompt": "0.00000035", "completion": "0.00000075" }, "capabilities": { "streaming": true, "function_calling": true, "structured_outputs": true, "vision": false, "json_mode": true, "tools": true, "tool_choice": true, "parallel_tool_calls": false, "response_format": true, "reasoning": true }, "limits": { "max_context_length": 131072, "max_completion_tokens": 40960, "requests_per_minute": null, "tokens_per_minute": null }, "deprecated": false, "preview": false, "quantization": "FP16/8 (weights only)" } ``` ## Use Cases **Platform Integration** - Use the OpenRouter or HuggingFace formats to integrate Cerebras models into existing platforms that support these standards. **Model Discovery** - Programmatically discover available models and their capabilities without authentication. **Pricing Comparison** - Compare pricing across different models using the structured pricing information. **Feature Detection** - Check which features (streaming, tools, JSON mode) are supported by each model. # Retrieve model Source: https://inference-docs.cerebras.ai/api-reference/models/retrieve-model GET https://api.cerebras.ai/v1/models/{model_id} Fetches a model instance, offering key details about the model, including its owner and permissions. ## Path Parameters The ID of the model to retrieve. Available options: * `gpt-oss-120b` * `qwen-3.8-27b` ## Response The model identifier. The object type, which is always `model`. The Unix timestamp (in seconds) of when the model was created. The organization that owns the model. ```python Python theme={null} import os from cerebras.cloud.sdk import Cerebras client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) model = client.models.retrieve("qwen-3.8-27b") print(model) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const model = await client.models.retrieve("qwen-3.8-27b"); console.log(model); } main(); ``` ```bash cURL theme={null} curl https://api.cerebras.ai/v1/models/qwen-3.8-27b \ -H "Authorization: Bearer $CEREBRAS_API_KEY" ``` ```json Response theme={null} { "id": "qwen-3.8-27b", "object": "model", "created": 1721692800, "owned_by": "Cerebras" } ``` # Versions Source: https://inference-docs.cerebras.ai/api-reference/versions Understand how Cerebras uses versioning to manage breaking changes. Cerebras uses API versioning to manage breaking changes to request validation and response formats. Version 2 is now the default for API requests. **This versioning does not change the API interface.** You'll continue using the same endpoints (e.g., `/v1/chat/completions`) and the same SDKs. New versions only affect validation rules and response field behavior. No new endpoints or SDK updates required. Review [what changed](#what-changed-in-version-2). The `X-Cerebras-Version-Patch: 2` header remains supported, but is no longer required to receive version 2 behavior. ## What's Guaranteed Within a given API version, we guarantee: * Existing request parameters continue to work as documented * Existing response fields remain present with the same types Non-breaking changes that may occur: * New optional request parameters may be added * New fields may be added to responses * Error message text may change (but not error types within a version) Breaking changes only happen in new API versions, giving you time to test and migrate. This includes stricter validation, removed parameters, or changed defaults. *** ## Setting the API Version Override the default API version per-request by passing the `X-Cerebras-Version-Patch` header: ```bash cURL theme={null} curl https://api.cerebras.ai/v1/chat/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "X-Cerebras-Version-Patch: 2" \ -H "Content-Type: application/json" \ -d '{ "model": "qwen-3.8-27b", "messages": [{"role": "user", "content": "Hello!"}] }' ``` ```python OpenAI SDK theme={null} import os from openai import OpenAI client = OpenAI( base_url="https://api.cerebras.ai/v1", api_key=os.environ.get("CEREBRAS_API_KEY") ) completion = client.chat.completions.create( model="qwen-3.8-27b", messages=[{"role": "user", "content": "Hello!"}], extra_headers={"X-Cerebras-Version-Patch": "2"} ) ``` ```python Cerebras SDK (Python) theme={null} import os from cerebras.cloud.sdk import Cerebras client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY") ) completion = client.chat.completions.create( model="qwen-3.8-27b", messages=[{"role": "user", "content": "Hello!"}], extra_headers={"X-Cerebras-Version-Patch": "2"} ) ``` ```javascript Cerebras SDK (Node.js) theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const completion = await client.chat.completions.create( { model: 'qwen-3.8-27b', messages: [{ role: 'user', content: 'Hello!' }], }, { headers: { 'X-Cerebras-Version-Patch': '2' }, } ); console.log(completion); } main(); ``` *** ## Version 2 Rollout Timeline | Date | What Happens | | -------------------- | ----------------------------------------------------------------------------------- | | **January 21, 2026** | Version 2 available for testing via header | | **July 22, 2026** | Phased rollout of Version 2 as the default begins; older versions reach end-of-life | *** ## What Changed in Version 2 Version 2 introduces stricter validation for structured outputs and tool calling, refines reasoning model behavior, and fixes edge cases. If your application uses JSON schemas, tool calls, or reasoning models, review these changes carefully. ### Structured Outputs: Stricter Schema Validation When using `strict: true`, Version 2 requires explicit, strictly-typed schemas. `additionalProperties: false` is required at every level of nested objects: ```json theme={null} // ❌ Fails: missing additionalProperties { "type": "object", "properties": { "user": { "type": "object", "properties": { "name": { "type": "string" } }, "required": ["name"] } }, "required": ["user"] } ``` ```json theme={null} // ✅ Works: additionalProperties at all levels { "type": "object", "properties": { "user": { "type": "object", "properties": { "name": { "type": "string" } }, "required": ["name"], "additionalProperties": false } }, "required": ["user"], "additionalProperties": false } ``` ### Tool Calling: Stricter Message Validation Version 2 validates the structure of multi-turn conversations involving tool calls: | Validation | What it checks | | -------------------------- | --------------------------------------------------------------------------------------------------------- | | Tool response completeness | Every `tool_call_id` in an assistant message must have a corresponding tool message immediately following | | Orphan tool messages | Tool messages cannot reference a `tool_call_id` that doesn't exist in a prior assistant message | | Tool choice consistency | `tool_choice` cannot be set unless `tools` is also provided | | Unique tool call IDs | Duplicate `tool_call.id` values in the same request are rejected | ### Reasoning Logprobs Added For reasoning models, version 2 adds a separate `reasoning_logprobs` field that contains logprobs for the reasoning tokens. Previously, reasoning logprobs were included in the main `logprobs` field alongside content logprobs. See [Reasoning](/capabilities/reasoning) for details on reasoning format options. ```json theme={null} // Before: reasoning logprobs included in main logprobs field { "message": { "content": "Here is the answer...", "reasoning": "Let me think..." }, "logprobs": { "content": [ {"token": "Let ", "logprob": -0.3}, {"token": "me", "logprob": -0.4}, ... {"token": "Here", "logprob": -0.1}, {"token": " is", "logprob": -0.2}, ... ] } } ``` ```json theme={null} // After (version 2): reasoning logprobs are separated { "message": { "content": "Here is the answer...", "reasoning": "Let me think..." }, "logprobs": { "content": [ {"token": "Here", "logprob": -0.1}, {"token": " is", "logprob": -0.2}, ... ] }, "reasoning_logprobs": { "content": [ {"token": "Let ", "logprob": -0.3}, {"token": "me", "logprob": -0.4}, ... ] } } ``` ### Unicode Handling Fix Logprobs now reflect partial Unicode tokens as they appear in the model's vocabulary. Previously, the Unicode replacement character (`\uFFFD`) was not handled correctly in logprobs output. This fix is backward-compatible and requires no code changes. *** ## Migration Checklist Use this checklist to validate your integration with version 2: Add `X-Cerebras-Version-Patch: 2` to your requests and run your test suite. ```python theme={null} extra_headers={"X-Cerebras-Version-Patch": "2"} ``` * Add all properties to the `required` array * Add `additionalProperties: false` to all object definitions * Ensure enum values match their property types If you parse reasoning content, update your code to read from the `reasoning` field in the response instead of extracting it from `content`. ```python theme={null} # Before reasoning = extract_reasoning_from_content(response.choices[0].message.content) # After (version 2) reasoning = response.choices[0].message.reasoning content = response.choices[0].message.content ``` Once your tests pass, no further action is required. Version 2 is now the default, and the header is no longer needed. *** ## Future Versions We strive to maintain backward compatibility whenever possible. In rare cases where breaking changes are necessary, a new API version will be released and made available for testing via the `X-Cerebras-Version-Patch` header for a minimum of 6 months before taking effect. Cerebras Cloud users will be notified via email before each version transition. # Batch Source: https://inference-docs.cerebras.ai/capabilities/batch Run large-scale inference workloads asynchronously. This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. The Batch API lets you process groups of requests asynchronously, making it perfect for workloads where you don't need immediate results: * **Evaluation pipelines**: Test model performance across thousands of test cases * **Data labeling**: Classify or annotate large datasets for training * **Content generation**: Create product descriptions, summaries, or translations in bulk * **Research and analysis**: Process scientific data or run experiments at scale Batch requests are guaranteed to complete in 24 hours. ## How It Works To process a batch request: 1. **Prepare a JSONL file** containing all your requests 2. **Upload the file** using the [Files API](/api-reference/file/upload-file) 3. **Submit a batch job** referencing your uploaded file 4. **Download results** once processing completes Behind the scenes, Cerebras processes your requests during periods of lower demand. ### Understanding Batch IDs The Batch API generates different IDs at each stage of the workflow: | ID Type | When is it generated? | Purpose | | ---------------- | ------------------------------- | -------------------------------------------------------- | | `input_file_id` | When you upload your JSONL file | Reference your uploaded input file when creating a batch | | `batch_id` | When you create a batch job | Track status and manage the batch job | | `output_file_id` | When batch completes | Download successful results | | `error_file_id` | When batch completes | Download failed request details | Store each ID as you receive it. You'll need the input file ID to create a batch, the batch ID to check progress, and the output/error file IDs to download results. ## Create a Batch Request SDK support for Batch is not yet available during Private Preview. Use cURL or direct HTTP requests for now—SDK support will be added at GA. Start by creating a `.jsonl` file where each line represents one API request. Every request needs a unique `custom_id` so you can match inputs to outputs later. The available endpoint is currently `/v1/chat/completions`. Here's what two requests look like: ```jsonl theme={null} {"custom_id": "eval-001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "user", "content": "Summarize the water cycle"}], "max_completion_tokens": 500}} {"custom_id": "eval-002", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "user", "content": "Explain photosynthesis"}], "max_completion_tokens": 500}} ``` Each line contains the same parameters you'd use in a regular chat completion request, just wrapped in the batch format. **Important constraints:** * Maximum 200 MB file size * Minimum 10 requests * Up to 50,000 requests per file * Each line limited to 1 MB * UTF-8 encoding with LF line endings * All requests must use the same model Use the [Files API](/api-reference/file/upload-file) to upload your prepared input file: ```python Python theme={null} from cerebras.cloud.sdk import Cerebras client = Cerebras(api_key="your-api-key") input_file = client.files.create( file=open("my_batch_requests.jsonl", "rb"), purpose="batch" ) print(f"Uploaded file: {input_file.id}") ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; import fs from 'fs'; const client = new Cerebras({ apiKey: process.env.CEREBRAS_API_KEY }); const file = await client.files.create({ file: fs.createReadStream("my_batch_requests.jsonl"), purpose: "batch" }); console.log(`Uploaded file: ${file.id}`); ``` ```bash cURL theme={null} curl https://api.cerebras.ai/v1/files \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -F purpose="batch" \ -F file="@my_batch_requests.jsonl" ``` Once your file is uploaded, create the batch job: ```python Python theme={null} batch = client.batches.create( input_file_id=input_file.id, endpoint="/v1/chat/completions", completion_window="24h", metadata={"description": "Evaluation run - December 2025"} ) print(f"Batch started: {batch.id}") print(f"Status: {batch.status}") ``` ```javascript Node.js theme={null} const batch = await client.batches.create({ input_file_id: file.id, endpoint: "/v1/chat/completions", completion_window: "24h", metadata: { description: "Evaluation run - December 2025" } }); console.log(`Batch started: ${batch.id}`); console.log(`Status: ${batch.status}`); ``` ```bash cURL theme={null} curl https://api.cerebras.ai/v1/batches \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "input_file_id": "", "endpoint": "/v1/chat/completions", "completion_window": "24h", "metadata": {"description": "Evaluation run - December 2025"} }' ``` The batch starts in `queued` status, then moves to `in_progress`, `finalizing`, and finally `completed`. Check on your batch job anytime to see how it's progressing: ```python Python theme={null} batch = client.batches.retrieve("") print(f"Status: {batch.status}") print(f"Completed: {batch.request_counts.completed}/{batch.request_counts.total}") ``` ```javascript Node.js theme={null} const batch = await client.batches.retrieve(""); console.log(`Status: ${batch.status}`); console.log(`Completed: ${batch.request_counts.completed}/${batch.request_counts.total}`); ``` ```bash cURL theme={null} curl https://api.cerebras.ai/v1/batches/ \ -H "Authorization: Bearer $CEREBRAS_API_KEY" ``` The response includes detailed request count metrics showing how many requests have completed and failed. When the status shows `completed`, the response object will include `output_file_id` and `error_file_id`: ```json Response highlight={6-7} theme={null} { "id": "", "object": "batch", "endpoint": "/v1/chat/completions", "status": "completed", "output_file_id": "", "error_file_id": "", "created_at": 1768244812, "completed_at": 1768244962, "request_counts": { "total": 10, "completed": 6, "failed": 4 } } ``` Use the `output_file_id` and `error_file_id` to download your results through the [Files API](/api-reference/file/retrieve-file-content): ```python Python theme={null} results = client.files.retrieve_content("") with open("batch_results.jsonl", "wb") as f: f.write(results) ``` ```javascript Node.js theme={null} const results = await client.files.retrieveContent(""); fs.writeFileSync("batch_results.jsonl", results); ``` ```bash cURL theme={null} curl https://api.cerebras.ai/v1/files//content \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -o batch_results.jsonl ``` Your results file contains one line per request. Successful requests include the full completion response, while failed requests include error details: ```jsonl theme={null} {"custom_id":"eval-001","status":"succeeded","response":{"id":"cmpl_1","object":"chat.completion","created":1699999999,"model":"gpt-oss-120b","choices":[{"index":0,"message":{"role":"assistant","content":"The water cycle consists of..."},"finish_reason":"stop"}],"usage":{"prompt_tokens":15,"completion_tokens":85,"total_tokens":100}}} {"custom_id":"eval-002","status":"succeeded","response":{"id":"cmpl_2","object":"chat.completion","created":1700000000,"model":"gpt-oss-120b","choices":[{"index":0,"message":{"role":"assistant","content":"Photosynthesis is the process..."},"finish_reason":"stop"}],"usage":{"prompt_tokens":12,"completion_tokens":92,"total_tokens":104}}} ``` Results aren't necessarily in the same order as your input. Always use `custom_id` to match requests to responses. ## Batch States Your batch progresses through these states: | State | What's happening | | ------------- | --------------------------------- | | `in_progress` | Actively processing requests | | `finalizing` | Preparing output file | | `completed` | Results ready to download | | `failed` | System error prevented completion | | `expired` | Exceeded 24-hour window | | `cancelled` | You stopped the batch | | `cancelling` | Cancellation in progress | Most batches complete well before the 24-hour limit, often within a few hours depending on size and load. ### Expired Batches If your batch doesn't finish within 24 hours, unprocessed requests are marked as expired. You'll still get results for any completed requests, and expired ones appear in your error\_file like this: ```json theme={null} {"custom_id":"eval-999","status":"expired","error":{"code":"timeout","message":"Batch expired before this request completed."}} ``` You're only charged for requests that completed. ## Cancel a Batch Use the cancel endpoint: ```python Python theme={null} cancelled = client.batches.cancel("") print(f"Status: {cancelled.status}") ``` ```javascript Node.js theme={null} const cancelled = await client.batches.cancel(""); console.log(`Status: ${cancelled.status}`); ``` ```bash cURL theme={null} curl -X DELETE https://api.cerebras.ai/v1/batches/ \ -H "Authorization: Bearer $CEREBRAS_API_KEY" ``` Any completed requests remain available in your results. Unfinished requests will be deleted. ## Limits and Quotas * 50,000 requests maximum per batch * 200 MB maximum file size * 1 MB maximum per request line * 10 concurrent active batches * Configurable rate limits separate from real-time API * Results retained for 7 days after completion * Automatic deletion after expiration (download promptly!) ## Organize Batches Use `metadata` to keep track of different batch runs. Metadata is a set of key-value pairs that you can attach to a batch job for your own organizational purposes. For example, to track which environment, dataset, or version a batch belongs to. ```python theme={null} batch = client.batches.create( input_file_id="file_xyz", endpoint="/v1/chat/completions", completion_window="24h", metadata={ "environment": "production", "dataset": "customer_feedback_q4", "version": "v2.1" } ) ``` The metadata you provide is stored with the batch object and returned in all API responses, making it easy to filter, search, and organize your batch jobs. Common uses include: * **Environment tracking**: Tag batches with `production`, `staging`, or `development` * **Dataset identification**: Link batches to specific datasets or experiments * **Version control**: Track which version of your prompts or models you're testing ## Handling Large Datasets Need to process more than 50,000 items? Split them across multiple batches: ```python theme={null} def split_into_batches(items, batch_size=50000): for i in range(0, len(items), batch_size): yield items[i:i + batch_size] # Process each chunk as a separate batch for i, chunk in enumerate(split_into_batches(all_items)): file = create_batch_file(chunk, f"batch_{i}.jsonl") # Submit each batch... ``` # Image Inputs Source: https://inference-docs.cerebras.ai/capabilities/image-inputs Pass images to vision-capable models. This feature is in [Public Preview](/support/preview-releases#public-preview). Vision-capable models can process objects, diagrams, screenshots, and text within images alongside a text prompt. Send images through the [Chat Completions](/api-reference/chat-completions) API as base64-encoded data URIs in the `messages` array. See [Limitations](#limitations) for known constraints. Image inputs are available for [`qwen-3.8-27b`](/models/qwen-3.8-27b) on the public shared tier. [`gemma-4-31b`](/dedicated/overview#supported-models) and other image-capable models are available through Dedicated Endpoints. `kimi-k2.7-code` supports image inputs on customer-trial endpoints but is not available on the public shared tier. ## Usage To send an image, add an `image_url` object to the `content` array in a user message. The image must be base64-encoded and passed as a data URI. Use the [encoder in the Token Usage section](#estimate-token-count) to convert your image to a base64 data URI. It also shows the estimated token count and encoded payload size. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os import base64 client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) def encode_image(image_path): with open(image_path, "rb") as image_file: return base64.b64encode(image_file.read()).decode("utf-8") base64_image = encode_image("screenshot.png") response = client.chat.completions.create( model="qwen-3.8-27b", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Describe this image in one concise sentence."}, { "type": "image_url", "image_url": { "url": f"data:image/png;base64,{base64_image}" }, }, ], } ], ) print(response.choices[0].message.content) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; import fs from 'fs'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); const base64Image = fs.readFileSync('screenshot.png').toString('base64'); const response = await client.chat.completions.create({ model: 'qwen-3.8-27b', messages: [ { role: 'user', content: [ { type: 'text', text: 'Describe this image in one concise sentence.' }, { type: 'image_url', image_url: { url: `data:image/png;base64,${base64Image}`, }, }, ], }, ], }); console.log(response.choices[0].message.content); ``` ```bash cURL theme={null} # Encode image to base64 (macOS/Linux) BASE64_IMAGE=$(base64 -i screenshot.png) # Windows PowerShell: # $BASE64_IMAGE = [Convert]::ToBase64String([IO.File]::ReadAllBytes("screenshot.png")) # If you run this example in PowerShell, use curl.exe and replace ${BASE64_IMAGE} with $BASE64_IMAGE. curl https://api.cerebras.ai/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${CEREBRAS_API_KEY}" \ -d "{ \"model\": \"qwen-3.8-27b\", \"messages\": [ { \"role\": \"user\", \"content\": [ {\"type\": \"text\", \"text\": \"Describe this image in one concise sentence.\"}, { \"type\": \"image_url\", \"image_url\": { \"url\": \"data:image/png;base64,${BASE64_IMAGE}\" } } ] } ] }" ``` Free Trial accounts can include up to 2 images in a single request. Developer and Enterprise accounts on the shared tier can include up to 10 images. Add each image as an additional `image_url` content part in the `content` array. The model considers all images together when generating its response. Each image counts toward your [token usage](#token-usage). Higher image limits may be available with [Dedicated Endpoints](/dedicated) or explicit organization configurations. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os import base64 client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) def encode_image(image_path): with open(image_path, "rb") as image_file: return base64.b64encode(image_file.read()).decode("utf-8") base64_image_1 = encode_image("image1.jpeg") base64_image_2 = encode_image("image2.png") response = client.chat.completions.create( model="qwen-3.8-27b", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Compare these two images."}, { "type": "image_url", "image_url": { "url": f"data:image/jpeg;base64,{base64_image_1}" }, }, { "type": "image_url", "image_url": { "url": f"data:image/png;base64,{base64_image_2}" }, }, ], } ], ) print(response.choices[0].message.content) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; import fs from 'fs'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); const base64Image1 = fs.readFileSync('image1.jpeg').toString('base64'); const base64Image2 = fs.readFileSync('image2.png').toString('base64'); const response = await client.chat.completions.create({ model: 'qwen-3.8-27b', messages: [ { role: 'user', content: [ { type: 'text', text: 'Compare these two images.' }, { type: 'image_url', image_url: { url: `data:image/jpeg;base64,${base64Image1}`, }, }, { type: 'image_url', image_url: { url: `data:image/png;base64,${base64Image2}`, }, }, ], }, ], }); console.log(response.choices[0].message.content); ``` ```bash cURL theme={null} # Encode images to base64 (macOS/Linux) BASE64_IMAGE_1=$(base64 -i image1.jpeg) BASE64_IMAGE_2=$(base64 -i image2.png) curl https://api.cerebras.ai/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${CEREBRAS_API_KEY}" \ -d "{ \"model\": \"qwen-3.8-27b\", \"messages\": [ { \"role\": \"user\", \"content\": [ {\"type\": \"text\", \"text\": \"Compare these two images.\"}, { \"type\": \"image_url\", \"image_url\": { \"url\": \"data:image/jpeg;base64,${BASE64_IMAGE_1}\" } }, { \"type\": \"image_url\", \"image_url\": { \"url\": \"data:image/png;base64,${BASE64_IMAGE_2}\" } } ] } ] }" ``` ## Input requirements | Requirement | Details | | ---------------------- | ------------------------------------------------------------------------------------------------------------------ | | Supported formats | PNG (`.png`), JPEG (`.jpeg`, `.jpg`) | | Encoding | Base64 data URI, such as `data:image/png;base64,...` | | Message role | `user` messages | | External image URLs | Not supported | | `image_url.detail` | Not supported | | Max payload size | `10 MiB` total request payload 1 | | Max image dimensions | Width and height must each be `15,000` pixels or smaller | | Max images per request | `2` for Free Trial; `10` for Developer and Enterprise accounts on the shared tier 1 | | Decompressed images | Images that require excessive decompressed memory return `413 Content Too Large` with error code `image_too_large` |
1 These limits apply to the shared tier during Public Preview. Higher image limits may be available with [Dedicated Endpoints](/dedicated) or explicit organization configurations.
`qwen-3.8-27b` currently accepts image content only in `user` messages. Images in `tool` messages aren't supported. ## Token usage Image token usage depends on the selected model and its processed image dimensions, not the uploaded file size alone. The estimator below implements the current preprocessing behavior for the listed models and shows both processed resolution and estimated image tokens. ### Estimate token count Select a model and upload an image below to copy its base64 data URI, check the encoded size, and view an estimate. The estimator defaults to `qwen-3.8-27b`. ### Model preprocessing | Model | Processed grid | Resize behavior | Maximum image tokens | | ----------------------------------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | [`qwen-3.8-27b`](/models/qwen-3.8-27b) | 32 × 32 pixels per image token | Preserves aspect ratio; upscales very small images to a minimum processed area; downscales large images to the configured maximum area | 2,304 | | `kimi-k2.7-code` (customer trials) | 28 × 28 pixels per image token | Does not upscale; downscales to its patch budget; pads processed dimensions to multiples of 28 | 2,304 | | [`gemma-4-31b`](/dedicated/overview#supported-models) | 48 × 48 pixels per image token | Preserves aspect ratio and resizes to its configured image area | 280 | For each model, estimate tokens from the processed dimensions: ```text theme={null} image_tokens = (processed_width / grid_width) × (processed_height / grid_height) ``` To validate image token usage, inspect `usage.image_tokens` in the API response. This field reports the total number of image tokens used by the request. `usage.prompt_tokens` includes text tokens, image tokens, and message-formatting tokens. The difference between image and text-only `prompt_tokens` can include message-formatting tokens and might not match `usage.image_tokens`. Keep the following in mind: * The estimator is an approximation based on the current model preprocessing configuration. * Compressed file size does not directly determine token count. Processed image dimensions matter more than PNG or JPEG byte size. * Image tokens are included in `usage.prompt_tokens` and are also reported in `usage.image_tokens`. * Image tokens occupy part of the model context window, just like text prompt tokens. ## Limitations * **Medical images**: Not suitable for interpreting specialized medical images such as CT scans or MRIs. Do not use for medical diagnosis or advice. * **Small text**: May have difficulty reading small or low-resolution text. Enlarging text within the image before sending can improve results. * **Rotated content**: May misinterpret text or images that are rotated or upside-down. * **Graphs and charts**: May struggle to distinguish visual elements that differ only in color or line style, such as solid versus dashed lines. * **Spatial reasoning**: Not reliable for tasks requiring precise spatial localization, such as identifying positions on a map or board game. * **Object counting**: The model may give approximate counts for objects in images. * **Image shape**: May perform less accurately on panoramic or fisheye images. * **Preprocessing**: The model cannot access original filenames or metadata. Images may be resized before analysis. See [Token usage](#token-usage) for details. * **Accuracy**: The model may generate inaccurate descriptions or captions in some scenarios. Verify outputs for high-stakes use cases. * **CAPTCHAs**: CAPTCHA images are not supported. * **Indirect prompt injection**: Text embedded in an image is included in the model's prompt context alongside the user's text. If an image contains adversarial instructions (for example, text that says "ignore all previous instructions") and the user prompt asks the model to answer based on the image, the model may follow those embedded instructions. Treat image content from untrusted sources as untrusted input, and use a system prompt to constrain the model's behavior when processing images you don't control. * **Untrusted output**: The model may transcribe or describe text from an image verbatim, including HTML, script tags, URLs, or control characters. The API returns this content unmodified. Treat it the same as any other untrusted input before rendering, logging, or executing it in your application. ## FAQs Yes. Cerebras Chat Completions is stateless. If a follow-up request depends on an earlier image, include that image-bearing turn in the conversation history you send with the new request. Continue to include that turn for as long as the model needs the visual context. No, only image input is supported. The model returns text only and does not generate images. Yes. Prompt caching can help with repeated images and repeated multimodal context within your organization. Prompt caches are never shared between organizations and remain ephemeral. See [Prompt Caching](/capabilities/prompt-caching). No. Image support uses the same rate limit framework as text. The same request and token limits still apply based on your organization and tier. For current details, see [Rate Limits](/support/rate-limits). Image inputs are processed as soon as they are received, and the original image payloads are not persisted. After preprocessing, image tokens and image embeddings may be cached ephemerally within your organization to support prompt caching. # Metrics Source: https://inference-docs.cerebras.ai/capabilities/metrics Monitor your dedicated inference endpoints with Prometheus-compatible metrics for requests, tokens, latency, and endpoint health. The Metrics API is designed for customers on dedicated endpoints who need granular observability into their inference workloads. Metrics are: * Aggregated at the minute level - Data represents the last complete minute * Pull-based - Your monitoring system queries the API on demand * Rate-limited - Up to 6 requests per minute For complete API details including request format, authentication, and error codes, see the [Metrics API reference](/api-reference/metrics/retrieve-metrics). ## Set Up Metrics Collection **Prerequisites:** You need a dedicated Cerebras inference endpoint, your organization ID, and a valid API key. Find your organization ID from your Cerebras account dashboard's Settings page. It will look like `org_abc123`. Use this URL format, replacing `` with your actual organization ID: ``` https://cloud.cerebras.ai/api/v1/metrics/organizations/ ``` This endpoint can be directly used by monitoring tools like Prometheus, Grafana Cloud, or Datadog. Configure your monitoring tool to include the Authorization header with your Cerebras API key: ``` Authorization: Bearer YOUR_API_KEY ``` Set your monitoring system to poll the endpoint every 60 seconds (1 minute). This matches the metric aggregation window and stays within rate limits. **Rate limit:** Maximum 6 requests per minute per organization Verify the connection by making a test request: ```bash cURL theme={null} curl -H "Authorization: Bearer $CEREBRAS_API_KEY" \ https://cloud.cerebras.ai/api/v1/metrics/organizations/org_abc123 ``` ```python Python theme={null} import requests import os response = requests.get( "https://cloud.cerebras.ai/api/v1/metrics/organizations/org_abc123", headers={ "Authorization": f"Bearer {os.environ.get('CEREBRAS_API_KEY')}" } ) # Parse Prometheus text format print(response.text) ``` ```javascript Node.js theme={null} const response = await fetch( 'https://cloud.cerebras.ai/api/v1/metrics/organizations/org_abc123', { headers: { 'Authorization': `Bearer ${process.env.CEREBRAS_API_KEY}` } } ); const metrics = await response.text(); console.log(metrics); ``` ## Direct Prometheus Integration To integrate directly with Prometheus, specify the Cerebras metrics endpoint in your scrape config: ```yaml prometheus.yml theme={null} global: scrape_interval: 60s scrape_configs: - job_name: 'cerebras-inference' metrics_path: '/api/v1/metrics/organizations/' authorization: type: "Bearer" credentials: "YOUR_API_KEY" static_configs: - targets: ['cloud.cerebras.ai'] scheme: https ``` For more details on Prometheus configuration, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/). ## Available Metrics For a complete list of available metrics including endpoint health, requests, tokens, and latency percentiles, see the [Available Metrics](/api-reference/metrics/retrieve-metrics#available-metrics) section in the API reference. ### Understanding Percentiles Latency metrics include multiple percentiles to give you a complete picture: * **avg** - Mean latency across all requests * **p50** - Median latency (50th percentile) * **p90** - 90% of requests complete faster than this * **p95** - 95% of requests complete faster than this * **p99** - 99% of requests complete faster than this **Example:** If `ttft_seconds{statistic="p95"} = 0.5`, then 95% of your requests receive their first token within 500ms. ## Example PromQL Queries ```promql theme={null} # Average end-to-end latency avg(e2e_latency_seconds{statistic="avg"}) # Request success rate rate(requests_success_total[5m]) / rate(requests_count_total[5m]) # Token throughput (tokens per second) rate(output_tokens_total[5m]) # Cache hit rate rate(cache_rate[5m]) # P95 time to first token ttft_seconds{statistic="p95"} ``` ## Use Cases ### Performance Monitoring Track latency percentiles (avg, p50, p90, p95, p99) across multiple dimensions: * **Time to First Token (TTFT)** - Measure initial response latency * **Time Per Output Token (TPOT)** - Monitor generation speed * **End-to-end latency** - Track total request duration * **Queue time** - Identify capacity constraints ### Usage Analytics Monitor token consumption and request patterns: * Track input and output tokens for cost analysis * Measure cache hit rates with `cache_rate` * Analyze request success and failure rates * Monitor endpoint availability ### Alerting and SLAs Set up alerts based on: * Endpoint health status changes * Latency threshold breaches * Error rate spikes * Request volume anomalies ## Troubleshooting If `queue_time_seconds` percentiles are elevated: 1. Check if you're hitting capacity limits on your dedicated endpoint by checking the Analytics tab on cloud.cerebras.ai 2. Review request patterns for traffic spikes 3. Contact your Cerebras representative about scaling options If `requests_failure_total` is increasing: 1. Check the `http_code` label to identify specific error types 2. Review the [Error Codes](/support/error) documentation 3. Verify authentication tokens are valid and not expired 4. Check [Rate Limits](/support/rate-limits) if seeing 429 errors If `cache_rate` is lower than expected: 1. Verify [Prompt Caching](/capabilities/prompt-caching) is enabled 2. Check that prompts have sufficient shared prefixes 3. Review cache configuration with your Cerebras representative If metrics appear stale: 1. Verify your API key has correct permissions 2. Check that metrics are enabled for your organization 3. Ensure you're querying the correct `organization_id` 4. Check for [error responses](/support/error) from the API # Payload Optimization Source: https://inference-docs.cerebras.ai/capabilities/payload-optimization Reduce latency by compressing request payloads with msgpack encoding and gzip. For requests with large payloads, you can reduce the size of the request body sent to the Cerebras API by using **[msgpack](https://msgpack.org/) encoding**, **[gzip](https://docs.python.org/3/library/gzip.html) compression**, or both. This can meaningfully improve time-to-first-token (TTFT) for requests with long prompts. Payload optimization is supported on [/v1/chat/completions](/api-reference/chat-completions) and [/v1/completions](/api-reference/completions). Support on [Dedicated Endpoints](/dedicated/overview) may vary by model. ## Encoding Options The Cerebras API accepts the following request body encodings. The table shows the expected payload size reduction for each option: | Content-Type | Description | Chat Completions1 | Completions2 | | ---------------------------------------------------- | -------------------------- | ---------------------------- | ----------------------- | | `application/json` | Default JSON encoding | Baseline | Baseline | | `application/vnd.msgpack` | msgpack binary encoding | up to \~5% | up to \~56% | | `application/json` + `Content-Encoding: gzip` | JSON with gzip compression | up to \~98% | up to \~68% | | `application/vnd.msgpack` + `Content-Encoding: gzip` | msgpack + gzip | up to \~98% | up to \~69% | 1 Measured against a 50k-token chat completions payload (206 KB JSON baseline).
2 Measured against a 50k token-ID completions payload (331 KB JSON baseline). You can use msgpack encoding or gzip compression independently, or combine them for maximum compression. Smaller request payloads reduce network transfer time, a contributing factor to TTFT. Actual TTFT improvement will vary, as network transfer is one of several factors that contribute to overall latency. ## When to Use Payload Optimization Optimizing payload size with request compression is most beneficial for: * **Long prompts** – requests with long system prompts, extensive conversation history, or large code blocks. Gzip compression is the most effective option for these payloads. * **Token-ID completions** – `/v1/completions` payloads using integer token arrays benefit from both msgpack encoding and gzip compression * **Tool call-heavy payloads** – requests with many tool definitions or deeply nested JSON structures. Both msgpack encoding and gzip compression provide savings. For small requests (under a few KB), the overhead of compression may outweigh the savings. Standard JSON encoding is fine for typical chat interactions. Size reductions depend on payload content. The benchmarks above used a token-ID completions payload, where msgpack's integer encoding provides the greatest benefit. String-heavy chat payloads may see smaller msgpack reductions, while payloads with deeply nested structures (e.g., tool calls) may see greater savings. Gzip benefits are more consistent across payload types. ## msgpack Encoding [msgpack](https://msgpack.org/) is a binary serialization format that produces smaller payloads than equivalent JSON. To use it, serialize your request body with msgpack and set the `Content-Type` header to `application/vnd.msgpack`. ```python Python theme={null} import msgpack import requests import os url = "https://api.cerebras.ai/v1/chat/completions" api_key = os.environ.get("CEREBRAS_API_KEY") payload = { "model": "qwen-3.8-27b", "messages": [ {"role": "user", "content": "Explain quantum computing in one paragraph."} ] } response = requests.post( url, data=msgpack.packb(payload), headers={ "Content-Type": "application/vnd.msgpack", "Authorization": f"Bearer {api_key}" } ) print(response.json()["choices"][0]["message"]["content"]) ``` ```javascript Node.js theme={null} const { encode } = require("@msgpack/msgpack"); const apiKey = process.env.CEREBRAS_API_KEY; const payload = { model: "qwen-3.8-27b", messages: [ { role: "user", content: "Explain quantum computing in one paragraph." } ] }; const response = await fetch("https://api.cerebras.ai/v1/chat/completions", { method: "POST", body: encode(payload), headers: { "Content-Type": "application/vnd.msgpack", "Authorization": `Bearer ${apiKey}` } }); const data = await response.json(); console.log(data.choices[0].message.content); ``` ```bash cURL theme={null} # Create a msgpack-encoded payload python3 -c " import msgpack, sys payload = { 'model': 'qwen-3.8-27b', 'messages': [{'role': 'user', 'content': 'Explain quantum computing in one paragraph.'}] } sys.stdout.buffer.write(msgpack.packb(payload)) " > /tmp/request.msgpack curl https://api.cerebras.ai/v1/chat/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/vnd.msgpack" \ --data-binary @/tmp/request.msgpack ``` ## Gzip Compression You can gzip-compress any request body and set the `Content-Encoding: gzip` header. This works with both JSON and msgpack payloads. ```python Python theme={null} import gzip import json import requests import os url = "https://api.cerebras.ai/v1/chat/completions" api_key = os.environ.get("CEREBRAS_API_KEY") payload = { "model": "qwen-3.8-27b", "messages": [ {"role": "user", "content": "Explain quantum computing in one paragraph."} ] } json_bytes = json.dumps(payload).encode("utf-8") compressed = gzip.compress(json_bytes, compresslevel=5) response = requests.post( url, data=compressed, headers={ "Content-Type": "application/json", "Content-Encoding": "gzip", "Authorization": f"Bearer {api_key}" } ) print(response.json()["choices"][0]["message"]["content"]) ``` ```javascript Node.js theme={null} const { gzipSync } = require("zlib"); const apiKey = process.env.CEREBRAS_API_KEY; const payload = { model: "qwen-3.8-27b", messages: [ { role: "user", content: "Explain quantum computing in one paragraph." } ] }; const jsonBytes = Buffer.from(JSON.stringify(payload)); const compressed = gzipSync(jsonBytes, { level: 5 }); const response = await fetch("https://api.cerebras.ai/v1/chat/completions", { method: "POST", body: compressed, headers: { "Content-Type": "application/json", "Content-Encoding": "gzip", "Authorization": `Bearer ${apiKey}` } }); const data = await response.json(); console.log(data.choices[0].message.content); ``` ```bash cURL theme={null} # Compress the JSON payload with gzip echo '{"model": "qwen-3.8-27b", "messages": [{"role": "user", "content": "Explain quantum computing in one paragraph."}]}' \ | gzip > /tmp/request.json.gz curl https://api.cerebras.ai/v1/chat/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/json" \ -H "Content-Encoding: gzip" \ --data-binary @/tmp/request.json.gz ``` ## Combining Both For maximum compression, use msgpack encoding with gzip compression: ```python Python theme={null} import gzip import msgpack import requests import os url = "https://api.cerebras.ai/v1/chat/completions" api_key = os.environ.get("CEREBRAS_API_KEY") payload = { "model": "qwen-3.8-27b", "messages": [ {"role": "user", "content": "Explain quantum computing in one paragraph."} ] } data = msgpack.packb(payload) compressed = gzip.compress(data, compresslevel=5) response = requests.post( url, data=compressed, headers={ "Content-Type": "application/vnd.msgpack", "Content-Encoding": "gzip", "Authorization": f"Bearer {api_key}" } ) print(response.json()["choices"][0]["message"]["content"]) ``` ```javascript Node.js theme={null} const { encode } = require("@msgpack/msgpack"); const { gzipSync } = require("zlib"); const apiKey = process.env.CEREBRAS_API_KEY; const payload = { model: "qwen-3.8-27b", messages: [ { role: "user", content: "Explain quantum computing in one paragraph." } ] }; const msgpackData = Buffer.from(encode(payload)); const compressed = gzipSync(msgpackData, { level: 5 }); const response = await fetch("https://api.cerebras.ai/v1/chat/completions", { method: "POST", body: compressed, headers: { "Content-Type": "application/vnd.msgpack", "Content-Encoding": "gzip", "Authorization": `Bearer ${apiKey}` } }); const data = await response.json(); console.log(data.choices[0].message.content); ``` ```bash cURL theme={null} # Create a msgpack+gzip payload (requires Python) python3 -c " import msgpack, gzip, sys payload = { 'model': 'qwen-3.8-27b', 'messages': [{'role': 'user', 'content': 'Explain quantum computing in one paragraph.'}] } sys.stdout.buffer.write(gzip.compress(msgpack.packb(payload), compresslevel=5)) " > /tmp/request.msgpack.gz curl https://api.cerebras.ai/v1/chat/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/vnd.msgpack" \ -H "Content-Encoding: gzip" \ --data-binary @/tmp/request.msgpack.gz ``` # Prompt Caching Source: https://inference-docs.cerebras.ai/capabilities/prompt-caching Store and reuse previously processed prompts to reduce latency and increase response times for similar or repeated queries. This feature reduces Time to First Token (TTFT) and improves responsiveness for long-context workloads, such as multi-turn conversations, RAG (Retrieval Augmented Generation), and agentic workflows. ## How It Works Unlike other providers that require manual cache breakpoints or header modifications, Cerebras Prompt Caching works automatically on all supported API requests. No code changes are required. 1. **Prefix Matching**: When you send a request, the system analyzes the beginning of your prompt (the prefix). This includes system prompts, tool definitions, and few-shot examples. 2. **Block-Based Caching**: The system processes prompts in 128-token blocks. If a block matches a segment stored in our ephemeral memory from a recent request within your organization, the system reuses the computation. 3. **Cache Hit**: Reusing cached blocks skips the processing phase for those tokens, resulting in lower latency. 4. **Cache Miss**: If no match is found, the system processes the prompt as normal and stores the prefix in the cache for potential future matches. 5. **Automatic Expiration**: Cached data is ephemeral. We guarantee a Time-To-Live (TTL) of 5 minutes, though caches may persist up to 1 hour depending on system load. To get a cache hit, the entire beginning of your prompt must match *exactly* with a previously cached prefix. Even a single character difference in the first token will result in a cache miss for that block and all subsequent blocks. ## Example: Multi-Turn Conversation with Tool Calling In this scenario, a shopping assistant helps users check order status and cancel orders using two tools: `get_order_status` and `cancel_order`. The system message and tool definitions remain constant across turns and are cached, while the conversation progresses naturally. ```python Python expandable theme={null} import os import json from cerebras.cloud.sdk import Cerebras client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) # Mock order database ORDERS = { "ORD-123456": {"status": "processing", "eta_days": 5}, } def get_order_status(order_id: str): """Look up an order's status""" order = ORDERS.get(order_id) if not order: return {"error": "Order not found"} return {"order_id": order_id, "status": order["status"], "eta_days": order.get("eta_days")} def cancel_order(order_id: str): """Cancel an order""" order = ORDERS.get(order_id) if not order: return {"error": "Order not found"} if order["status"] in ["shipped", "delivered"]: return {"error": f"Cannot cancel - order already {order['status']}"} order["status"] = "cancelled" order.pop("eta_days", None) return {"order_id": order_id, "status": "cancelled"} tools = [ { "type": "function", "function": { "name": "get_order_status", "description": "Look up an order status by order ID", "parameters": { "type": "object", "properties": { "order_id": {"type": "string", "description": "Order ID (e.g., ORD-123456)"} }, "required": ["order_id"] }, "strict": True } }, { "type": "function", "function": { "name": "cancel_order", "description": "Cancel an order by order ID", "parameters": { "type": "object", "properties": { "order_id": {"type": "string", "description": "Order ID (e.g., ORD-123456)"} }, "required": ["order_id"] }, "strict": True } } ] available_functions = { "get_order_status": get_order_status, "cancel_order": cancel_order } messages = [ {"role": "system", "content": "You are a shopping assistant. Help users check order status and cancel orders."}, {"role": "user", "content": "Where is my order ORD-123456?"} ] # Turn 1 - creates cache response = client.chat.completions.create(model="qwen-3.8-27b", messages=messages, tools=tools) print("Turn 1 usage:", response.usage) msg = response.choices[0].message messages.append(msg.model_dump()) if msg.tool_calls: for call in msg.tool_calls: func = available_functions[call.function.name] result = func(**json.loads(call.function.arguments)) messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)}) response = client.chat.completions.create(model="qwen-3.8-27b", messages=messages, tools=tools) messages.append(response.choices[0].message.model_dump()) print("Turn 1 response:", response.choices[0].message.content) # Turn 2 - uses cache messages.append({"role": "user", "content": "Please cancel it, I ordered by mistake."}) response = client.chat.completions.create(model="qwen-3.8-27b", messages=messages, tools=tools) print("\nTurn 2 usage:", response.usage) msg = response.choices[0].message messages.append(msg.model_dump()) if msg.tool_calls: for call in msg.tool_calls: func = available_functions[call.function.name] result = func(**json.loads(call.function.arguments)) messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)}) response = client.chat.completions.create(model="qwen-3.8-27b", messages=messages, tools=tools) print("Turn 2 response:", response.choices[0].message.content) ``` ```javascript Node.js expandable theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env.CEREBRAS_API_KEY }); // Mock order database const ORDERS = { 'ORD-123456': { status: 'processing', eta_days: 5 } }; const getOrderStatus = (order_id) => { const order = ORDERS[order_id]; if (!order) return { error: 'Order not found' }; return { order_id, status: order.status, eta_days: order.eta_days }; }; const cancelOrder = (order_id) => { const order = ORDERS[order_id]; if (!order) return { error: 'Order not found' }; if (['shipped', 'delivered'].includes(order.status)) { return { error: `Cannot cancel - order already ${order.status}` }; } order.status = 'cancelled'; delete order.eta_days; return { order_id, status: 'cancelled' }; }; const tools = [ { type: 'function', function: { name: 'get_order_status', description: 'Look up an order status by order ID', parameters: { type: 'object', properties: { order_id: { type: 'string', description: 'Order ID (e.g., ORD-123456)' } }, required: ['order_id'] }, strict: true } }, { type: 'function', function: { name: 'cancel_order', description: 'Cancel an order by order ID', parameters: { type: 'object', properties: { order_id: { type: 'string', description: 'Order ID (e.g., ORD-123456)' } }, required: ['order_id'] }, strict: true } } ]; // Helper function to handle tool calls in a more reusable way async function handleToolCalls(toolCalls, messages) { for (const call of toolCalls) { const args = JSON.parse(call.function.arguments); const orderId = args.order_id; let result; if (call.function.name === 'get_order_status') { result = getOrderStatus(orderId); } else if (call.function.name === 'cancel_order') { result = cancelOrder(orderId); } messages.push({ role: 'tool', tool_call_id: call.id, content: JSON.stringify(result) }); } } async function run() { const messages = [ { role: 'system', content: 'You are a shopping assistant. Help users check order status and cancel orders.' }, { role: 'user', content: 'Where is my order ORD-123456?' } ]; // Turn 1 - creates cache let response = await client.chat.completions.create({ model: 'qwen-3.8-27b', messages, tools }); console.log('Turn 1 usage:', response.usage); let msg = response.choices[0].message; messages.push(msg); if (msg.tool_calls) { await handleToolCalls(msg.tool_calls, messages); response = await client.chat.completions.create({ model: 'qwen-3.8-27b', messages, tools }); messages.push(response.choices[0].message); console.log('Turn 1 response:', response.choices[0].message.content); } // Turn 2 - uses cache messages.push({ role: 'user', content: 'Please cancel it, I ordered by mistake.' }); response = await client.chat.completions.create({ model: 'qwen-3.8-27b', messages, tools }); console.log('Turn 2 usage:', response.usage); msg = response.choices[0].message; messages.push(msg); if (msg.tool_calls) { await handleToolCalls(msg.tool_calls, messages); response = await client.chat.completions.create({ model: 'qwen-3.8-27b', messages, tools }); console.log('Turn 2 response:', response.choices[0].message.content); } } run().catch(console.error); ``` ```bash cURL expandable theme={null} #!/bin/bash API_KEY="${CEREBRAS_API_KEY}" BASE_URL="https://api.cerebras.ai/v1" if [ -z "$API_KEY" ]; then echo "Error: CEREBRAS_API_KEY environment variable is not set" exit 1 fi TOOLS='[ { "type": "function", "function": { "name": "get_order_status", "description": "Look up an order status by order ID", "parameters": { "type": "object", "properties": { "order_id": {"type": "string", "description": "Order ID (e.g., ORD-123456)"} }, "required": ["order_id"] }, "strict": true } }, { "type": "function", "function": { "name": "cancel_order", "description": "Cancel an order by order ID", "parameters": { "type": "object", "properties": { "order_id": {"type": "string", "description": "Order ID (e.g., ORD-123456)"} }, "required": ["order_id"] }, "strict": true } } ]' SYSTEM="You are a shopping assistant. Help users check order status and cancel orders." echo "=== Turn 1 (Creates Cache) ===" curl -s -X POST "$BASE_URL/chat/completions" \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d "$(jq -n --arg sys "$SYSTEM" --argjson tools "$TOOLS" '{ model: "qwen-3.8-27b", messages: [{role: "system", content: $sys}, {role: "user", content: "Where is my order ORD-123456?"}], tools: $tools }')" | jq '{content: .choices[0].message.content, usage: .usage}' echo "" echo "=== Turn 2 (Uses Cache) ===" curl -s -X POST "$BASE_URL/chat/completions" \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d "$(jq -n --arg sys "$SYSTEM" --argjson tools "$TOOLS" '{ model: "qwen-3.8-27b", messages: [ {role: "system", content: $sys}, {role: "user", content: "Where is my order ORD-123456?"}, {role: "assistant", content: "Your order ORD-123456 is currently processing with an estimated delivery in 5 days."}, {role: "user", content: "Please cancel it, I ordered by mistake."} ], tools: $tools }')" | jq '{content: .choices[0].message.content, usage: .usage}' ``` During each turn, the system automatically caches the longest matching prefix from previous requests. In this example: * **System message**: The shopping assistant instructions remain identical across all turns * **Tool definitions**: Both order management tool schemas (including parameters and descriptions) stay constant * **Conversation history**: Previous user messages, assistant responses, and tool results are all cached as the conversation grows Only the new content at the end of each request requires fresh processing: * New user messages (the latest question) * New tool execution results * The model's reasoning and decision-making for the current turn As the conversation grows, the cache hit rate increases dramatically. The static prefix (system + tools) remains cached, and the expanding conversation history also gets cached, meaning only the newest user message and the model's fresh response require full processing. ## Structuring Prompts for Caching To maximize cache hits and minimize latency, organize your prompts with static content first and dynamic content last. The system caches prompts from the **beginning** of the message. If you place dynamic content (like a timestamp or a unique User ID) at the start of the prompt, the prefix will differ for every request and the cache will never be triggered. Place content that remains the same across multiple requests at the beginning: * System instructions ("You are a helpful assistant...") * Tool definitions and schemas * Few-shot examples * Large context documents (e.g., a legal agreement or code base) Place content that changes with each request at the end: * User-specific questions * Session variables * Timestamps The "You are a coding assistant..." instruction block remains static and can be cached in subsequent requests. Only the short timestamp and user query are processed fresh. ```json theme={null} [ { "role": "system", "content": "You are a coding assistant... Current Time: 12:01 PM" }, { "role": "user", "content": "Debug this code." } ] ``` **Result:** The static portion of the system prompt is cached. Subsequent requests reuse the cache and only process the timestamp and user query. In this example, the time is included at the start of the system instructions. Because the time changes every minute, the prefix never matches. Subsequent requests will always be fully processed. ```json theme={null} [ { "role": "system", "content": "Current Time: 12:01 PM. You are a coding assistant..." }, { "role": "user", "content": "Debug this code." } ] ``` **Result:** Cache miss on every request because the timestamp changes the prefix. ## Maximize Cache Hits with `prompt_cache_key` For the best cache hit rate within a single conversation or workflow, include a `prompt_cache_key` on each request that shares the same prompt prefix. This optional opaque string tells the system which requests are likely to share a common prompt prefix, so it can route those requests to the same prompt cache. Prompt caching and prefix matching work automatically on every request, so you benefit from caching even without setting `prompt_cache_key`. Use it when you want to give the system an explicit routing hint. This keeps requests from the same conversation or workflow on the same prompt cache. Under load, turn 1 of a session can be routed to one prompt cache and turn 2 to another, causing a cache miss even though the prefixes match. Don't set `prompt_cache_key` for prefixes that are shared across many users, such as a common system prompt or shared RAG context. This would funnel all of those requests to a single backend, creating a bottleneck that limits your deployment's throughput. These shared prefixes are already handled by automatic prefix caching. ### When to Set a Cache Key Choose a stable identifier for one conversation or workflow whose requests actually share a common prompt prefix: * **Multi-turn chat session:** Use the conversation ID so every turn in the same conversation uses the same prompt cache. * **Agentic or RAG workflow:** Use one workflow ID only for a small set of steps that reuse the same prompt prefix. Reuse the same key across all requests in a conversation or workflow — a one-off unique value provides no improvement over omitting the field. When a new conversation or workflow begins, start a new key. If later requests no longer share the same prompt prefix, reusing the old key only adds routing noise and does not create a cache hit. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras client = Cerebras() response = client.chat.completions.create( model="qwen-3.8-27b", messages=[ {"role": "system", "content": "You are a helpful coding assistant."}, {"role": "user", "content": "Explain recursion in Python."} ], prompt_cache_key="conversation-abc-123", ) ``` ```javascript Node.js theme={null} import Cerebras from "@cerebras/cerebras_cloud_sdk"; const client = new Cerebras(); const response = await client.chat.completions.create({ model: "qwen-3.8-27b", messages: [ { role: "system", content: "You are a helpful coding assistant." }, { role: "user", content: "Explain recursion in Python." }, ], prompt_cache_key: "conversation-abc-123", }); ``` ```bash cURL theme={null} curl -s -X POST https://api.cerebras.ai/v1/chat/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "qwen-3.8-27b", "messages": [ {"role": "system", "content": "You are a helpful coding assistant."}, {"role": "user", "content": "Explain recursion in Python."} ], "prompt_cache_key": "conversation-abc-123" }' ``` `prompt_cache_key` values have a maximum length of 1024 characters. Longer values are rejected with a 400 error. `prompt_cache_key` is optional and does not affect billing. Cached tokens are priced the same whether or not the key is set. The key value is hashed before it is written to internal logs or metrics. ## Track Cache Usage Verify if your requests are hitting the cache by viewing the `cached_tokens` field within the [`usage.prompt_token_details`](/api-reference/chat-completions#param-prompt-tokens-details) response object. This indicates the number of prompt tokens that were found in the cache. ```json theme={null} "usage": { "prompt_tokens": 3000, "completion_tokens": 150, "total_tokens": 3150, "prompt_tokens_details": { "cached_tokens": 2800 } } ``` In this example, the cache served 2,800 of the 3,000 prompt tokens, resulting in faster processing. Additionally, log in to [cloud.cerebras.ai](https://cloud.cerebras.ai) and click **Analytics** to track your cache usage. ## FAQs Cached tokens count toward your **total** TPM limit, but **not** your uncached TPM limit. Since the uncached limit is the primary constraint on throughput, a higher cache hit rate gives you more effective capacity — the same uncached TPM limit can serve significantly more total tokens. **Calculation:** `uncached_tokens + cached_tokens` = Total TPM usage for that request. See [Rate Limits](/support/rate-limits#uncached-vs-total-tokens) for details on the dual-bucket model. There is no additional fee for using prompt caching. Input tokens, whether served from the cache or processed fresh, are billed at the standard input token rate for the respective model. There are three common reasons for a cache miss on identical requests: 1. **Block Size:** We cache in 128-token blocks. If a request or prefix is shorter than 128 tokens, it may not be cached. 2. **Data Center Routing:** While we make a best effort to route you to the same data center, traffic profiles may occasionally route you to a different location where your cache does not exist. 3. **TTL Expiration:** If requests are sent more than 5 minutes apart, the cache may have been evicted. Yes. Prompt caching is automatically enabled for all customers and models. Prompt caching is enabled by default for all models. Yes, it is fully ZDR-compliant. All cached context remains ephemeral in memory and never persisted. Cached tokens are stored in key-value stores colocated in the same data center as the model instance serving your traffic. Prompt caches are never shared between organizations. Only members of your organization can benefit from caches created by identical prompts within your team. No. Prompt caching works automatically. `prompt_cache_key` is an optional routing hint for requests in the same conversation or workflow that share a common prompt prefix. Do not set it just to share one large system prompt or RAG context across many users. See [Maximize Cache Hits with `prompt_cache_key`](#maximize-cache-hits-with-prompt_cache_key). Caching only affects the input processing phase (how we read your prompt). The output generation phase remains exactly the same speed and quality. You will receive the same quality response, just with faster prompt processing. No manual cache management is required or available. The system automatically manages cache eviction based on the TTL (5 minutes to 1 hour). Guaranteed TTL is 5 minutes, but up to 1 hour max depending on system load. Check the `usage.prompt_tokens_details.cached_tokens` field in your API response. When it's greater than 0, caching was used for that request. Additionally, log in to [cloud.cerebras.ai](https://cloud.cerebras.ai) and click **Analytics** to track your cache usage. # Reasoning Source: https://inference-docs.cerebras.ai/capabilities/reasoning Control reasoning effort, response format, and multi-turn reasoning behavior. Reasoning models generate intermediate thinking tokens before their final response. Model families differ in whether reasoning can be disabled, how effort levels behave, and how reasoning is returned. | Model | Default | `reasoning_effort` | Disable reasoning | Availability | | ----------------------------------------------------- | -------------- | ------------------------------- | -------------------------------- | -------------------- | | [`qwen-3.8-27b`](/models/qwen-3.8-27b) | `high` | `none`, `low`, `medium`, `high` | Set `reasoning_effort` to `none` | Public shared tier | | `kimi-k2.7-code` | Always enabled | Accepted but ignored | Not supported | Customer trials only | | [`gpt-oss-120b`](/models/openai-oss) | `medium` | `low`, `medium`, `high` | Not supported | Public shared tier | | [`gemma-4-31b`](/dedicated/overview#supported-models) | Disabled | `none`, `low`, `medium`, `high` | Set `reasoning_effort` to `none` | Dedicated endpoints | Reasoning tokens count toward `max_completion_tokens` and the completion-token usage reported by the API. ## Reasoning format Use `reasoning_format` to control how supported models return reasoning. | Format | Behavior | | -------- | -------------------------------------------------------------------------------------------------------- | | `parsed` | Returns reasoning separately in `choices[0].message.reasoning` or streaming `choices[0].delta.reasoning` | | `raw` | Prepends reasoning to the final content when the model supports it | | `hidden` | Omits reasoning text while still generating and counting reasoning tokens | | `none` | Uses the model's default response format | Format support is model-specific: | Model | Default response behavior | `raw` | `hidden` | | ---------------- | -------------------------------- | --------------------------------------------- | ------------- | | `qwen-3.8-27b` | Reasoning returned separately | Does not change the separated response format | Not supported | | `kimi-k2.7-code` | `parsed` | Supported | Not supported | | `gpt-oss-120b` | Parsed reasoning | Supported | Supported | | `gemma-4-31b` | Reasoning disabled until enabled | Not supported | Not supported | For `kimi-k2.7-code`, do not combine `reasoning_format` set to `raw` with `response_format` types `json_object` or `json_schema`. ### Parsed response ```python theme={null} from cerebras.cloud.sdk import Cerebras client = Cerebras() response = client.chat.completions.create( model="qwen-3.8-27b", messages=[{"role": "user", "content": "Which is larger, 9.11 or 9.9? Explain."}], reasoning_format="parsed", ) print(response.choices[0].message.reasoning) print(response.choices[0].message.content) ``` For streaming responses, reasoning arrives in `choices[0].delta.reasoning` and final-answer text arrives in `choices[0].delta.content`. ## Qwen 3.8 27B `qwen-3.8-27b` enables reasoning by default at `high` effort. | Value | Behavior | | -------- | ----------------------------------------------------- | | `none` | Disables reasoning | | `low` | Selects Qwen's low reasoning mode | | `medium` | Selects Qwen's medium reasoning mode | | `high` | Selects Qwen's native `xhigh` mode and is the default | Effort levels select reasoning modes. They do not reserve or guarantee an exact reasoning-token budget. ```python theme={null} response = client.chat.completions.create( model="qwen-3.8-27b", messages=[{"role": "user", "content": "Find the bug in this concurrency design."}], reasoning_effort="high", ) ``` To disable reasoning for a simpler request: ```python theme={null} response = client.chat.completions.create( model="qwen-3.8-27b", messages=[{"role": "user", "content": "Return only the word ready."}], reasoning_effort="none", ) ``` Do not send Qwen-native parameters such as `disable_reasoning`, `enable_thinking`, `preserve_thinking`, or `thinking_budget`. Use `reasoning_effort` and `clear_thinking` instead. ### Multi-turn reasoning and `clear_thinking` Chat Completions is stateless, so your application must resend conversation history. When you include an earlier assistant message with its `reasoning` field, `clear_thinking` controls whether Qwen receives that historical reasoning: | Value | Behavior | | ------------------ | --------------------------------------------------------------------------------------------------------------------------- | | `false` or omitted | Preserves historical assistant reasoning. This is the default. | | `true` | Removes historical assistant reasoning before the model is prompted. Assistant final-answer content remains in the history. | ```python theme={null} from cerebras.cloud.sdk import Cerebras client = Cerebras() messages = [{"role": "user", "content": "Design a retry strategy for this job queue."}] first = client.chat.completions.create( model="qwen-3.8-27b", messages=messages, ) assistant = first.choices[0].message messages.extend([ { "role": "assistant", "content": assistant.content, "reasoning": assistant.reasoning, }, {"role": "user", "content": "Now account for duplicate delivery."}, ]) # Preserve the earlier reasoning. This is also the default when omitted. preserved = client.chat.completions.create( model="qwen-3.8-27b", messages=messages, clear_thinking=False, ) # Use the same visible conversation, but remove earlier reasoning from the prompt. cleared = client.chat.completions.create( model="qwen-3.8-27b", messages=messages, clear_thinking=True, ) ``` ## Kimi K2.7 Code `kimi-k2.7-code` is currently available only for customer trials. Reasoning is always enabled. The API accepts `reasoning_effort` for client compatibility but ignores its value, including `none`. ```python theme={null} response = client.chat.completions.create( model="kimi-k2.7-code", messages=[{"role": "user", "content": "Review this patch for race conditions."}], reasoning_effort="low", # Accepted, but does not change Kimi's reasoning behavior. reasoning_format="parsed", ) ``` Kimi defaults to parsed reasoning and also supports `raw`. It does not support `hidden`. `clear_thinking` is not implemented and should not be sent. ## GPT OSS 120B Use `reasoning_effort` to select `low`, `medium`, or `high`. The default is `medium`. ```python theme={null} response = client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": "Explain quantum entanglement."}], reasoning_effort="medium", ) ``` GPT OSS supports parsed, raw, and hidden reasoning formats. When using `raw`, reasoning and final content are concatenated without separators. ## Gemma 4 31B Reasoning is disabled by default for `gemma-4-31b`. | Value | Behavior | | -------------------------- | ------------------------------------------------------------------------- | | `none` | Reasoning disabled and is the default | | `low`, `medium`, or `high` | Reasoning enabled. The three active values currently behave equivalently. | Gemma 4 does not support `raw`, `hidden`, or `clear_thinking`. ```python theme={null} response = client.chat.completions.create( model="gemma-4-31b", messages=[{"role": "user", "content": "Solve this step by step."}], reasoning_effort="medium", ) ``` ## OpenAI client parameters `reasoning_effort` is a standard OpenAI client parameter. Pass Cerebras-specific parameters such as `reasoning_format` and `clear_thinking` through `extra_body`. See [OpenAI Compatibility](/resources/openai#pass-non-standard-parameters). # Service Tiers Source: https://inference-docs.cerebras.ai/capabilities/service-tiers Control request prioritization with service tiers. This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Prioritize requests on the Cerebras Inference API to balance latency sensitivity and resource allocation across your workloads. Service tiers are not available on shared endpoints. The examples on this page assume a dedicated endpoint with service-tier support enabled. ## Service Tiers Service tiers determine the processing priority of your requests. You can specify a tier using the [`service_tier`](/api-reference/chat-completions#param-service-tier) parameter in your API requests. | Tier | Description | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `priority` | Highest priority - requests are processed first. Use for time-critical, user-facing requests that require immediate processing. Only available for dedicated endpoints, not shared endpoints. | | `default` | Standard priority processing. Use for standard production workloads with normal latency requirements. | | `auto` | Automatically uses the highest available service tier. Use when you want to maximize requests served while allowing flexibility in processing priority. | | `flex` | Lowest priority - requests are processed towards the end. Use for overflow requests that cannot fit in higher service tier rate limits or for experiments. | When no `service_tier` is specified, requests default to the `default` tier. ## Usage Add the `service_tier` parameter to your chat completions request to specify the priority level. ```python Python highlight={13} theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) response = client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "system", "content": "Best pastries in San Francisco?"}], stream=True, max_tokens=20000, temperature=0.7, top_p=0.8, service_tier="auto" ) ``` ```javascript Node.js highlight={14} theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); const response = await client.chat.completions.create({ model: "gpt-oss-120b", messages: [{ role: "system", content: "Best pastries in San Francisco?" }], stream: true, max_tokens: 20000, temperature: 0.7, top_p: 0.8, service_tier: "auto" }); ``` ```bash cURL highlight={10} theme={null} curl --location 'https://api.cerebras.ai/v1/chat/completions' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" \ --data '{ "model": "llama-3.3-70b", "stream": true, "max_tokens": 20000, "temperature": 0.7, "top_p": 0.8, "service_tier": "auto", "messages": [ { "role": "system", "content": "Best pastries in San Francisco?" } ] }' ``` When using `auto`, the response will include a `service_tier_used` field that indicates the effective service tier used for processing. ## Queue Threshold Control Only applies to requests using the `flex` or `auto` service tiers. The [`queue_threshold`](/api-reference/chat-completions#param-queue-threshold) header allows you to set a maximum acceptable queue time for flex tier requests. If the expected queue time exceeds your threshold, the request is preemptively rejected rather than waiting in the queue. **Valid range:** 50-20000 milliseconds ```python Python highlight={10} theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) response = client.chat.completions.create( model="gpt-oss-120b", service_tier="flex", messages=[{"role": "user", "content": "What are the latest AI trends?"}], extra_headers={"queue_threshold": "100"} ) ``` ```javascript Node.js highlight={12} theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); const response = await client.chat.completions.create({ model: "gpt-oss-120b", service_tier: "flex", messages: [{ role: "user", content: "What are the latest AI trends?" }] }, { headers: { "queue_threshold": "100" } }); ``` ```bash cURL highlight={4} theme={null} curl https://api.cerebras.ai/v1/chat/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/json" \ -H "queue_threshold: 100" \ -d '{ "model": "llama-3.3-70b", "service_tier": "flex", "messages": [ { "role": "user", "content": "What are the latest AI trends?" } ] }' ``` If no threshold is specified, a system default is used. ## FAQ `priority` and `default` rate limits are the same, while `flex` rate limits are tracked independently and are several multiples of `default` rate limits. Yes. Log in to [cloud.cerebras.ai](https://cloud.cerebras.ai) and click **Analytics**. Graphs in the analytics tab display usage across different service tiers, allowing you to monitor consumption by priority level. No, during the preview launch all service tiers are billed equally. No, only requests set to `auto` can be processed on a lower service tier. The queue time threshold only applies once a request is being processed on the `flex` service tier. You can set it on requests using `auto` or `flex`, but it will only be evaluated if the request is processed on the flex tier. # Streaming Responses Source: https://inference-docs.cerebras.ai/capabilities/streaming Learn how to enable streaming responses in the Cerebras API. **[Get a free API key](https://cloud.cerebras.ai?utm_source=3pi_streaming\&utm_campaign=capabilities) to get started.** The Cerebras API supports streaming responses, which send messages back in chunks and display them incrementally as the model generates them. To enable this feature, set the `stream` parameter to `True` within the `chat.completions.create` method. The API then returns an iterable containing the chunks of the message. Similarly, the same can be done in TypeScript by setting the `stream` property to `true` within the `chat.completions.create` method. Begin by importing the Cerebras SDK and setting up the client. ```python Python theme={null} import os from cerebras.cloud.sdk import Cerebras client = Cerebras( # This is the default and can be omitted api_key=os.environ.get("CEREBRAS_API_KEY"), ) ``` ```javascript Node.js theme={null} import Cerebras from 'cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], // This is the default and can be omitted }); ``` Set the `stream` parameter to `True` within the `chat.completions.create` method to enable streaming responses. ```python Python theme={null} stream = client.chat.completions.create( messages=[ { "role": "user", "content": "Why is fast inference important?", } ], model="qwen-3.8-27b", stream=True, ) for chunk in stream: print(chunk.choices[0].delta.content or "", end="") ``` ```javascript Node.js theme={null} import Cerebras from 'cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], // This is the default and can be omitted }); async function main() { const stream = await client.chat.completions.create({ messages: [{ role: 'user', content: 'Why is fast inference important?' }], model: 'qwen-3.8-27b', stream: true, }); for await (const chunk of stream) { process.stdout.write(chunk.choices[0]?.delta?.content || ''); } } main(); ``` # Structured Outputs Source: https://inference-docs.cerebras.ai/capabilities/structured-outputs Generate structured data with the Cerebras Inference API. [**Get a free API key to get started.**](https://cloud.cerebras.ai?utm_source=inferencedocs) Structured Outputs constrains model responses to a JSON schema so applications can process generated data reliably. Key benefits include: * **Consistent fields**: Responses follow the fields defined in your schema. * **Type safety**: Values use the data types defined in your schema. * **Simpler parsing and integration**: Applications can consume responses without additional format conversion. | Model | Text | JSON object | JSON Schema | `strict: true` | Availability | | ----------------------------------------------------- | ---- | ----------- | ----------- | -------------- | -------------------- | | [`qwen-3.8-27b`](/models/qwen-3.8-27b) | Yes | Yes | Yes | Yes | Public shared tier | | `kimi-k2.7-code` | Yes | Yes | Yes | Yes | Customer trials only | | [`gpt-oss-120b`](/models/openai-oss) | Yes | Yes | Yes | Yes | Public shared tier | | [`gemma-4-31b`](/dedicated/overview#supported-models) | Yes | Yes | Yes | Yes | Dedicated endpoints | The strict-mode rules below apply to `response_format.json_schema.strict: true` and `tools[].function.strict: true`. They do not apply to `strict: false` or legacy `json_object` mode. When using structured outputs with `kimi-k2.7-code`, use parsed reasoning. `reasoning_format: "raw"` is incompatible with `json_object` and `json_schema` response formats. ## Tutorial: Structured Outputs using Cerebras Inference The following steps define a movie-recommendation schema and use the Cerebras Cloud SDK to return a response that conforms to it. Complete steps 1 and 2 of the [Quickstart](/quickstart) to configure your API key and install the Cerebras Cloud SDK. Import the required modules and initialize the client. ```python Python theme={null} import os from cerebras.cloud.sdk import Cerebras import json client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY") ) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'] }); ``` Define a JSON schema that specifies the response fields, their types, and which fields are required. This example defines a movie recommendation with a title, director, and year. For every object in your schema (the root and any nested objects), you must set `additionalProperties` to `false` when using strict mode. See [Required Schema Structure](#required-schema-structure) for details. ```python Python theme={null} movie_schema = { "type": "object", "properties": { "title": {"type": "string"}, "director": {"type": "string"}, "year": {"type": "integer"}, }, "required": ["title", "director", "year"], "additionalProperties": False } ``` ```javascript Node.js theme={null} const movieSchema = { type: "object", properties: { title: { type: "string" }, director: { type: "string" }, year: { type: "integer" }, }, required: ["title", "director", "year"], additionalProperties: false }; ``` Add the schema to `response_format` in the API request. * `strict: true` uses constrained decoding to guarantee that output conforms to schemas in the supported JSON Schema subset. Recommended for production use. * `strict: false` (or omitted) treats the schema as a hint. The model may return additional fields, miss required fields, or use incorrect types, similar to [JSON mode](#json-mode). ```python Python theme={null} completion = client.chat.completions.create( model="qwen-3.8-27b", messages=[ {"role": "system", "content": "You are a helpful assistant that generates movie recommendations."}, {"role": "user", "content": "Suggest a sci-fi movie from the 1990s"} ], response_format={ "type": "json_schema", "json_schema": { "name": "movie_schema", "strict": True, "schema": movie_schema } } ) # Parse the JSON response movie_data = json.loads(completion.choices[0].message.content) print(json.dumps(movie_data, indent=2)) ``` ```javascript Node.js theme={null} async function main() { const schemaCompletion = await client.chat.completions.create({ model: 'qwen-3.8-27b', messages: [ { role: 'system', content: 'You are a helpful assistant that generates movie recommendations.' }, { role: 'user', content: 'Suggest a sci-fi movie from the 1990s' } ], response_format: { type: 'json_schema', json_schema: { name: 'movie_schema', strict: true, schema: movieSchema } } }); // Parse and display the JSON response const schemaMovieData = JSON.parse(schemaCompletion.choices[0].message.content); console.log('Movie Recommendation:'); console.log(JSON.stringify(schemaMovieData, null, 2)); } ``` Sample output: ``` { "title": "Terminator 2: Judgment Day", "director": "James Cameron", "year": 1991 } ``` The returned JSON conforms to the schema and can be used directly by the application. ## Understanding strict mode For schemas that use the supported JSON Schema subset, strict mode guarantees that generated output conforms to the schema. Set `strict` to `true` to enable constrained decoding at the token level. ### Why use strict mode Without strict mode, a response can contain: * Malformed JSON that fails to parse * Missing required fields * Incorrect data types, such as `"16"` instead of `16` * Fields that are not defined in the schema Strict mode provides: * Valid JSON * Fields that conform to the schema * Correct data types for properties * Fewer retries caused by schema violations ### Enable strict mode Set `strict` to `true` in your `response_format` configuration: ```python theme={null} response_format={ "type": "json_schema", "json_schema": { "name": "my_schema", "strict": True, # Enable constrained decoding "schema": your_schema } } ``` ### Schema requirements for strict mode When using strict mode, you must set `additionalProperties: false`. This is required for every object in your schema. ### Limitations in strict mode When strict mode is enabled, your schema must conform to specific requirements. See the [Supported Schemas](#supported-schemas) section for detailed information on constraints, limits, and unsupported features. ## Schema references and definitions Use `$ref` with `$defs` to define reusable components within a JSON schema. Reusable definitions reduce repetition and make schemas easier to maintain. ```python highlight={5,7-8} theme={null} schema_with_defs = { "type": "object", "properties": { "title": {"type": "string"}, "director": {"$ref": "#/$defs/person"}, "year": {"type": "integer"}, "lead_actor": {"$ref": "#/$defs/person"}, "studio": {"$ref": "#/$defs/studio"} }, "required": ["title", "director", "year"], "additionalProperties": False, "$defs": { "person": { "type": "object", "properties": { "name": {"type": "string"}, "age": {"type": "integer"} }, "required": ["name"], "additionalProperties": False }, "studio": { "type": "object", "properties": { "name": {"type": "string"}, "founded": {"type": "integer"}, "headquarters": {"type": "string"} }, "required": ["name"], "additionalProperties": False } } } ``` ## Supported schemas Structured Outputs supports a subset of [JSON Schema](https://json-schema.org/docs). The following sections describe its supported types, properties, and constraints. ### Supported types The following types are supported for Structured Outputs: | Type | Description | | --------- | --------------------------------- | | `string` | Text values | | `number` | Numeric values (floating point) | | `integer` | Whole number values | | `boolean` | True/false values | | `object` | Nested objects with properties | | `array` | Lists of items | | `enum` | Constrained set of allowed values | | `null` | Null values | | `anyOf` | Union types | ### Schema constraints When using strict mode, the following constraints apply: | Constraint | Limit | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | Maximum counted schema text | 5,000 characters across property names, definition names, and string values used by `enum` and `const`. Descriptions are excluded. | | Maximum nesting depth | 10 levels | | Maximum object properties | 500 | | Maximum total enum values | 500 across all enum properties | | Enum string budget | 7,500 characters for a single enum when it contains more than 250 values | ### Required schema structure All schemas must follow these rules: * **Root must be an object**: The top-level schema must have `"type": "object"`. * **Root must define properties**: Include a `properties` object at the root. * **Root unions are not supported**: The root cannot be an array, scalar, or `anyOf` union. * **No additional properties**: You must set `"additionalProperties": false` for every object in your schema. * **Arrays need an item schema**: Every array must define `items` or use `prefixItems` with `items: false`. These requirements are enforced by API version 2. Non-conforming strict schemas return a validation error. See [API Versions](/api-reference/versions). ### Supported features Your schema can include the following JSON Schema features: * **Nested structures**: Define complex objects with nested properties. * **Required fields**: Specify which fields must be present. * **Optional fields**: Properties may be omitted from `required`. * **Enums (value constraints)**: Use the `enum` keyword to whitelist the exact literals a field may take. See `rating` in the example below. * **Constants**: Use `const` with primitive string, number, integer, boolean, or null values. * **Non-root unions**: Use `anyOf` below the root object. * **Schema references**: Use local, nonrecursive `$ref` values with `$defs` to define reusable schema components within your schema. * **Tuple validation**: `items: false` is supported when used with `prefixItems` for tuple-like arrays. * **Number constraints**: Use `minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum`, and `multipleOf` to constrain `number` and `integer` values. * **Annotations**: `description`, `title`, `$schema`, and `default` are accepted but are not enforced by constrained decoding. ### Unsupported features The following JSON Schema features are **not** supported in [strict mode](/capabilities/structured-outputs#understanding-strict-mode): | Feature | Notes | | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | Recursive schemas | Self-referencing schemas are not supported | | External `$ref` | References to external URLs are blocked for security | | `$anchor` keyword | Use relative paths within definitions instead | | Missing or boolean `items` | Arrays must define an item schema. `items: true` is not supported. `items: false` is supported only with `prefixItems`. | | Array-form `items` | Use `prefixItems` for tuple validation instead | | Composition keywords | `oneOf`, `allOf`, and `not` are not supported | | OpenAPI nullable syntax | `nullable: true` is not supported. Represent nullability with a JSON Schema union where supported. | | Conditional and dependent schemas | `if`, `then`, `else`, `dependentRequired`, and `dependentSchemas` are not supported | | Dynamic object-key schemas | `patternProperties` and `unevaluatedProperties` are not supported | | String `pattern` | Regular expression constraints on strings are not supported | | String `format` | Format validation, such as `email`, `date-time`, and `uuid`, is not supported | | Array constraints | `minItems`, `maxItems` are not supported | Unlisted JSON Schema keywords should be treated as unsupported unless the selected model's documentation states otherwise. String-length support is model-dependent. In particular, strict tool schemas for `qwen-3.8-27b` do not support `minLength` or `maxLength`. Do not combine `tools` and `response_format` unless the selected model's contract explicitly documents and validates the combination. For portable workflows, call tools first and format the result in a separate structured-output request. ### Example: Complex schema The following example defines a schema with nested objects and arrays: ```python theme={null} detailed_schema = { "type": "object", "properties": { "title": {"type": "string"}, "director": {"type": "string"}, "year": {"type": "integer"}, "genres": { "type": "array", "items": {"type": "string"} }, "rating": { "type": "string", "enum": ["G", "PG", "PG‑13", "R"] }, "cast": { "type": "array", "items": { "type": "object", "properties": { "name": {"type": "string"}, "role": {"type": "string"} }, "required": ["name"], "additionalProperties": False } } }, "required": ["title", "director", "year", "genres"], "additionalProperties": False } ``` The API can return a response such as: ```json theme={null} { "title": "Jurassic Park", "director": "Steven Spielberg", "year": 1993, "genres": ["Science Fiction", "Adventure", "Thriller"], "cast": [ {"name": "Sam Neill", "role": "Dr. Alan Grant"}, {"name": "Laura Dern", "role": "Dr. Ellie Sattler"}, {"name": "Jeff Goldblum", "role": "Dr. Ian Malcolm"} ] } ``` ### Key ordering The keys in the generated JSON output will appear in the same order as they are defined in your schema. ## Working with Pydantic and Zod You can define a schema with Pydantic for Python or Zod for JavaScript instead of writing JSON Schema manually. Pydantic's `model_json_schema` and Zod's `zodToJsonSchema` methods generate JSON Schema for use in an API request. ```python Python (Pydantic) theme={null} from pydantic import BaseModel import json # Define your schema using Pydantic class Movie(BaseModel): title: str director: str year: int # Convert the Pydantic model to a JSON schema movie_schema = Movie.model_json_schema() # Print the JSON schema to verify it print(json.dumps(movie_schema, indent=2)) ``` ```javascript Node.js (Zod) theme={null} import { z } from 'zod'; import { zodToJsonSchema } from 'zod-to-json-schema'; // Define your schema using Zod const MovieSchema = z.object({ title: z.string(), director: z.string(), year: z.number().int() }); // Convert the Zod schema to a JSON schema const movieJsonSchema = zodToJsonSchema(MovieSchema, { name: 'movie_schema' }); // Print the JSON schema to verify it console.log(JSON.stringify(movieJsonSchema, null, 2)); ``` ## JSON mode JSON mode generates valid JSON without enforcing a specific schema. The model chooses which fields to include based on the prompt. We recommend using structured outputs with `strict` set to `true` instead of JSON mode whenever possible. Structured outputs guarantee schema adherence, while JSON mode only ensures valid JSON without enforcing a specific structure. To use JSON mode, set the `response_format` parameter to `json_object` and include instructions in your message asking the model to respond in JSON format: ```python Python theme={null} completion = client.chat.completions.create( model="qwen-3.8-27b", messages=[ {"role": "system", "content": "You are a helpful assistant that generates movie recommendations. Respond with JSON."}, {"role": "user", "content": "Suggest a sci-fi movie from the 1990s"} ], response_format={"type": "json_object"} ) ``` ```javascript Node.js theme={null} async function main() { const jsonModeCompletion = await client.chat.completions.create({ model: 'qwen-3.8-27b', messages: [ { role: 'system', content: 'You are a helpful assistant that generates movie recommendations. Respond with JSON.' }, { role: 'user', content: 'Suggest a sci-fi movie from the 1990s' } ], response_format: { type: 'json_object' } }); } ``` ### Structured Outputs vs. JSON mode The following table compares Structured Outputs and JSON mode: | Feature | Structured Outputs | JSON Mode | | -------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------ | | Outputs valid JSON | Yes | Yes | | Enforces schema | Yes (when `strict: true`) | No | | Constrained decoding | Yes (when `strict: true`) | No | | Configuration | `response_format: { type: "json_schema", json_schema: { "strict": true, "schema": ... } }` | `response_format: { type: "json_object" }` | Do not combine `tools` and `response_format` unless the selected model's documentation explicitly supports and validates the combination. ## Conclusion Learn how to combine structured responses with other Cerebras capabilities: * [Tool Calling](/capabilities/tool-use): Connect models to external functions and data. * [Streaming](/capabilities/streaming): Process response chunks as they are generated. * [CePO](/capabilities/cepo): Improve reasoning with test-time compute. # Tool Calling Source: https://inference-docs.cerebras.ai/capabilities/tool-use Learn how to connect models to external tools with tool calling. Tool calling, also known as tool use or function calling, lets a model request functions that your application defines. Your application executes each requested function and returns the result to the model. | Model | `tool_choice` | Strict mode | Parallel calls | Multi-turn calling | Availability | | ----------------------------------------------------- | ------------------------------------------ | ----------- | -------------- | ------------------ | -------------------- | | [`qwen-3.8-27b`](/models/qwen-3.8-27b) | `none`, `auto`, `required`, named function | Yes | Yes | Yes | Public shared tier | | `kimi-k2.7-code` | `none`, `auto`, `required`, named function | Yes | Yes | Yes | Customer trials only | | [`gpt-oss-120b`](/models/openai-oss) | `none`, `auto`, `required`, named function | Yes | Yes | Yes | Public shared tier | | [`gemma-4-31b`](/dedicated/overview#supported-models) | `none`, `auto`, `required`, named function | Yes | Yes | Yes | Dedicated endpoints | See [Strict mode for tool calling](#strict-mode-for-tool-calling) for model-specific schema requirements. ## How it works 1. **Define the tool**: Provide a name, description, and input parameters for each tool you want the model to access. 2. **Send the request**: The prompt is sent along with available tool definitions in your API call. 3. **Select a tool**: The model determines whether a tool can help answer the request. If so, it returns the tool name and arguments. 4. **Execute the tool**: The client application receives the model's tool call request, executes the specified tool (such as calling an external API), and retrieves the result. 5. **Generate the final response**: Send the tool result to the model so it can continue the conversation. ## Basic tool calling Import the required libraries and initialize the Cerebras client. If you have not configured an API key, complete the [Quickstart](/quickstart) first. ```python theme={null} import os import json import re from cerebras.cloud.sdk import Cerebras # Initialize Cerebras client client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) ``` Define the function that the application executes. This example uses a calculator that performs basic arithmetic. ```python theme={null} def calculate(expression): expression = re.sub(r'[^0-9+\-*/().]', '', expression) try: result = eval(expression) return str(result) except (SyntaxError, ZeroDivisionError, NameError, TypeError, OverflowError): return "Error: Invalid expression" ``` Define the tool name, description, and parameters that the model can use. For schemas in the supported JSON Schema subset, `strict: true` uses constrained decoding to guarantee that tool-call arguments conform to the schema. ```python theme={null} tools = [ { "type": "function", "function": { "name": "calculate", "strict": True, "description": "A calculator tool that can perform basic arithmetic operations. Use this when you need to compute mathematical expressions or solve numerical problems.", "parameters": { "type": "object", "properties": { "expression": { "type": "string", "description": "The mathematical expression to evaluate" } }, "required": ["expression"], "additionalProperties": False } } } ] ``` Send the messages and tool schema. The response can include a tool call. ```python theme={null} messages = [ {"role": "system", "content": "You are a helpful assistant with access to a calculator. Use the calculator tool to compute mathematical expressions when needed."}, {"role": "user", "content": "What's the result of 15 multiplied by 7?"}, ] response = client.chat.completions.create( model="qwen-3.8-27b", messages=messages, tools=tools, parallel_tool_calls=False, ) ``` Check the response for a tool call. If one is present, execute the requested function and return its result to the model. ```python theme={null} choice = response.choices[0].message if choice.tool_calls: function_call = choice.tool_calls[0].function if function_call.name == "calculate": # Logging that the model is executing a function named "calculate". print(f"Model executing function '{function_call.name}' with arguments {function_call.arguments}") # Parse the arguments from JSON format and perform the requested calculation. arguments = json.loads(function_call.arguments) result = calculate(arguments["expression"]) # Note: This is the result of executing the model's request (the tool call), not the model's own output. print(f"Calculation result sent to model: {result}") # Send the result back to the model to fulfill the request. messages.append({ "role": "tool", "content": json.dumps(result), "tool_call_id": choice.tool_calls[0].id }) # Request the final response from the model, now that it has the calculation result. final_response = client.chat.completions.create( model="qwen-3.8-27b", messages=messages, ) # Handle and display the model's final response. if final_response: print("Final model output:", final_response.choices[0].message.content) else: print("No final response received") else: # Handle cases where the model's response does not include expected tool calls. print("Unexpected response from the model") ``` The example produces output similar to the following: ``` Model executing function 'calculate' with arguments {"expression": "15 * 7"} Calculation result sent to model: 105 Final model output: 15 * 7 = 105 ``` ## Strict mode for tool calling For schemas that use the supported JSON Schema subset, strict mode guarantees that tool-call arguments conform to the schema. ### Why strict mode matters for tools Without strict mode, tool calls can contain: * Incorrect parameter types, such as `"2"` instead of `2` * Missing required parameters * Unexpected parameters * Malformed argument JSON Strict mode prevents these schema violations for supported schemas. ### Enabling strict mode Set `strict` to `true` inside the `function` object of your tool definition: ```python Python theme={null} tools = [ { "type": "function", "function": { "name": "get_weather", "strict": True, # Enable constrained decoding "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "City and country, such as San Francisco, USA" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location", "unit"], "additionalProperties": False } } } ] ``` ### Schema requirements When using strict mode, you must set `additionalProperties: false`. This is required for every object in your schema. For information about schema limitations that apply when using strict mode, see [Limitations in Strict Mode](/capabilities/structured-outputs#limitations-in-strict-mode). For `kimi-k2.7-code`, either set the same `strict` value on every function in the request or omit it from every function. For `qwen-3.8-27b`, do not use `pattern`, `minLength`, or `maxLength` in strict tool schemas. ### Strict mode with parallel tool calling Strict mode works with parallel tool calling. When multiple tools are called simultaneously, each tool call's arguments will conform to its respective schema: ```python Python theme={null} response = client.chat.completions.create( model="qwen-3.8-27b", messages=messages, tools=tools, # Each tool can have strict: true parallel_tool_calls=True, ) ``` ## Multi-turn tool calling Most real-world workflows require more than one tool invocation. Multi-turn tool calling lets a model call a tool, incorporate its output, and then, within the same conversation, decide whether it needs to call the tool (or another tool) again to finish the task. 1. Append each tool result to `messages` and ask the model to continue. 2. Let the model determine whether it needs another tool call. 3. Continue calling `client.chat.completions.create()` until the response does not contain `tool_calls`. The following example extends the calculator example. Complete steps 1 through 3 in [Basic tool calling](#basic-tool-calling) before running it. ```python theme={null} messages = [ { "role": "system", "content": ( "You are a helpful assistant with a calculator tool. " "Use it whenever math is required." ), }, {"role": "user", "content": "First, multiply 15 by 7. Then take that result, add 20, and divide the total by 2. What's the final number?"}, ] # Register every callable tool once available_functions = { "calculate": calculate, } while True: resp = client.chat.completions.create( model="qwen-3.8-27b", messages=messages, tools=tools, ) msg = resp.choices[0].message # If the assistant didn’t ask for a tool, we’re done if not msg.tool_calls: print("Assistant:", msg.content) break # Save the assistant turn exactly as returned messages.append(msg.model_dump()) # Run the requested tool call = msg.tool_calls[0] fname = call.function.name if fname not in available_functions: raise ValueError(f"Unknown tool requested: {fname!r}") args_dict = json.loads(call.function.arguments) # assumes JSON object output = available_functions[fname](**args_dict) # Feed the tool result back messages.append({ "role": "tool", "tool_call_id": call.id, "content": json.dumps(output), }) ``` ## Parallel tool calling Parallel tool calling lets a model request multiple independent tool calls in one response, which can reduce latency. For example, if a user asks "Is Toronto warmer than Montreal?", the model needs to check the weather in both cities. Rather than making two separate requests, parallel tool calling enables the model to request both operations at once, reducing latency and improving efficiency. Use parallel tool calling when: * A request requires multiple independent data points, such as weather in different cities. * Tool calls do not depend on the results of other tool calls. ### Enable parallel tool calling You can explicitly control this behavior using the `parallel_tool_calls` parameter: ```python highlight={5} theme={null} response = client.chat.completions.create( model="qwen-3.8-27b", messages=messages, tools=tools, parallel_tool_calls=True, # Enable parallel calling (default) ) ``` To disable parallel tool calling and force sequential execution: ```python highlight={5} theme={null} response = client.chat.completions.create( model="qwen-3.8-27b", messages=messages, tools=tools, parallel_tool_calls=False, # Disable parallel calling ) ``` ### Example: Weather comparison The following example requests weather data for two cities in parallel. Define a weather function and its tool schema. ```python theme={null} import os import json from cerebras.cloud.sdk import Cerebras client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) def get_weather(location): """ Dummy function that returns mock weather data. In production, this would call a real weather API. """ weather_data = { "location": location, "temperature": 22, "condition": "sunny", "humidity": 45, } return json.dumps(weather_data) tools = [ { "type": "function", "function": { "name": "get_weather", "strict": True, "description": "Get temperature for a given location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "City and country, such as Toronto, Canada" } }, "required": ["location"], "additionalProperties": False } } } ] ``` Send a request that requires weather data for two cities. ```python theme={null} messages = [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "Is Toronto warmer than Montreal?" } ] response = client.chat.completions.create( model="qwen-3.8-27b", messages=messages, tools=tools, parallel_tool_calls=True, ) ``` Iterate through every entry in the response's `tool_calls` array. ```python theme={null} choice = response.choices[0].message if choice.tool_calls: # Add the assistant message with tool_calls first messages.append(choice) # Process all tool calls for tool_call in choice.tool_calls: function_call = tool_call.function print(f"Model executing function '{function_call.name}' with arguments {function_call.arguments}") # Parse arguments and execute the function arguments = json.loads(function_call.arguments) result = get_weather(arguments["location"]) print(f"Weather data sent to model: {result}") # Append each tool result to messages messages.append({ "role": "tool", "content": result, "tool_call_id": tool_call.id }) # Get final response after all tool calls are processed final_response = client.chat.completions.create( model="qwen-3.8-27b", messages=messages, ) if final_response: print("Final model output:", final_response.choices[0].message.content) else: print("No final response received") else: print("No tool calls in response") ``` # Account & Billing Source: https://inference-docs.cerebras.ai/console/account-billing Manage team members, billing, and organization settings in the Cloud Console. This page documents the Members, Billing, and Settings sections. To view them, open the Cloud Console: [cloud.cerebras.ai](https://cloud.cerebras.ai) → **Members**, **Billing**, or **Settings** in the left nav. ## Members The Members page is only visible to organization admins. Invite new users and manage access to your organization. When you invite a user, assign one of the following roles: | Role | Description | | ---------------------- | -------------------------------------------------------- | | **Organization Admin** | Can manage all members, settings, and projects | | **Project Admin** | Can manage API keys and members within assigned projects | | **Project Member** | Can view and use API keys within assigned projects | For a full breakdown of what each role can do, see [Roles & Permissions](/console/projects#roles--permissions). ### Invite Users Click **New Invite** on the Members page. Enter one or more email addresses, select a role, and click **Invite Users**. Each person will receive an email with a link to accept the invite. ### Manage Invitations The Invites section lists pending and rejected invitations from the last 30 days. You can **resend** an expired or missed invitation, or **revoke** one before it's accepted. ## Billing ### Free Trial Credits New accounts receive **\$5 in free credits** after adding a verified payment method. Credits expire 30 days after they're granted and can be used across all public models. There is no charge for adding a payment method — you only pay if you choose to purchase additional credits. ### Check Your Credit Balance Your remaining credits are shown on the **Overview** tab. To add credits, click **Add Credits** or go to the **Pay as you go** tab. Your full credit grant history is also available there. ### Set Up Auto-Recharge Auto-recharge is off by default. Enable it on the **Pay as you go** tab when you add credits or by clicking **Enable**. You can configure the threshold that triggers a recharge and the amount you're topped off to. ### Manage Subscriptions The **Subscriptions** tab lets you manage inference plans on a per-model basis. Each model offers multiple tiers at different monthly rates. If a tier shows **Temporarily Sold Out**, check back later or [contact us](https://www.cerebras.ai/contact) for availability. ### Update Payment Method or Billing Address Use the **Payment** and **Billing address** tabs to manage these. ## Delete Your Account The Cloud Console does not offer a self-service option to delete an organization or user account. To request account deletion, contact us at [support@cerebras.net](mailto:support@cerebras.net). Deleting an [API key](/console/api-keys#rotate-or-delete-a-key) or archiving a [project](/console/projects) does not delete the underlying account. ## Settings Open the [Cloud Console](https://cloud.cerebras.ai) and click **Settings** in the left navigation sidebar. Settings is a top-level page, separate from **Billing**. The **Settings** page lets you: * View your **Organization ID** — copy it from the top of the page when support asks for it or when you need to identify your org programmatically. * Update your **Organization name**. * See your **Organization type**. To change your organization type, contact support. If you don't see **Settings** in the left nav, you likely don't have the Organization Admin role. Ask an admin in your org, or contact support. # API Keys Source: https://inference-docs.cerebras.ai/console/api-keys Authenticate to the Cerebras API with API keys. API keys are static secrets you generate in the Cloud Console and include in every request for authentication. This page documents how API keys work. To manage them, open the Cloud Console: [cloud.cerebras.ai](https://cloud.cerebras.ai) → **API Keys** in the left nav. ## Create a Key 1. Open the [Cloud Console](https://cloud.cerebras.ai) and go to **API Keys** in the left nav. 2. **On paid accounts**, select a project first — the key inherits that project's rate limits and usage tracking. **On Free Trial accounts**, [projects](/console/projects) aren't available, so skip this step and go straight to **Generate API Key**. 3. Click **Generate API Key**, give it a descriptive name, and confirm. 4. The key is created and appears in the list. The list shows a truncated preview, but you can copy the full key at any time by clicking the copy icon next to it. ## Use the Key Set the `CEREBRAS_API_KEY` environment variable, or pass the key in the `Authorization: Bearer ` header on direct HTTP requests. See the [Authentication guide](/api-reference/authentication) for SDK initialization and shell setup. ## Rotate or Delete a Key API keys do not expire. Store them in a secrets manager and rotate them periodically. To rotate a key, create a new one, update your application to use it, then delete the old one. To delete a key, click the delete icon next to it on the **API Keys** page. Deletion is immediate and permanent — any application still using the key will receive `401 Unauthorized` errors. # Cloud Console Source: https://inference-docs.cerebras.ai/console/overview Your dashboard for managing Cerebras Inference projects, API keys, usage, billing, and team access.
Cloud Console
Your dashboard for managing Cerebras Inference projects, API keys, usage, billing, and team access.
Open the Console Sign up or log in to get started.
Features
Organize workloads, isolate environments, and control access with project-level rate limits and membership. Interactively test models and prompts without writing any code. Create, rotate, and manage org-level and project-level API keys. View your current rate limits at the org and project level. Track requests, token usage, and trends across models and projects. Monitor HTTP status code trends and inspect individual requests for debugging and auditing. Invite team members and assign org and project roles. Manage credits, payment methods, subscriptions, and invoices. Update org name, view org ID, and other organization-level configuration.
# Playground Source: https://inference-docs.cerebras.ai/console/playground Interactively test models and prompts in the Cloud Console without writing any code. The Playground lets you experiment with Cerebras models directly in the browser. Use it to evaluate models, iterate on prompts, test tool/function calling, and tune parameters, then export a working request to code when you're ready. This page documents how the Playground works. To use it, open the Playground in the Cloud Console: [cloud.cerebras.ai](https://cloud.cerebras.ai) → **Playground** in the left nav. Your playground and API requests are never used to train models. ## Message Roles | Role | Purpose | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **System** | Sets the behavior and context for the model. Use this to define a persona, provide background information, or constrain responses. | | **User** | Represents input from the human turn. This is the prompt the model responds to. | | **Assistant** | Represents a prior model response. Insert assistant messages to simulate a multi-turn conversation or prime the model toward a particular style or format. | Use the **Add** button to add a new User or Assistant message to the conversation without running inference. Use it to build multi-turn conversations manually. Click **Run** to send the full conversation to the model. After each response, the Playground displays token usage, inference time, speed (tokens per second), and round trip time in the upper-right corner of the response. ## Configuration Select a model from the dropdown at the top right. See the [Models overview](/models/overview) to learn more about the available options. | Parameter | What it controls | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | **Temperature** | Randomness of the output. Lower values produce more focused, predictable responses; higher values produce more varied responses. | | **Max Completion Tokens** | Maximum number of tokens the model will generate in a single response. | | **Top P** | Nucleus sampling threshold. Limits the model to sampling from the top portion of the probability distribution. | | **Format** | Output format. `text` (default), `json_object`, or `json_schema`. | | **Functions** | Define tool/function schemas the model can call. See [Tool Calling](/capabilities/tool-use) for the full reference. | | **Reasoning Effort** | Controls how much effort the model spends reasoning before responding. Only available for select models. | | **Seed** | Set an integer to produce deterministic outputs across requests with the same input. | | **Stream** | Stream tokens as they're generated rather than returning the full response at once. | | **Stop Sequence** | A string that causes the model to stop generating when produced. Useful for structured outputs. | ## View Code Once you have a prompt and parameters you're happy with, click **View Code** to get a ready-to-run code snippet for your application. # Projects Source: https://inference-docs.cerebras.ai/console/projects Organize workloads, isolate environments, control costs, and manage access using Projects in the Cerebras Cloud console. Projects are workspaces within your organization that group API keys, rate limits, and members. Use them to isolate environments, segment usage analytics, and control who can access what. Every organization starts with a **Default Project** that contains all existing API keys, members, and quotas. If you don't create additional projects, nothing about your setup changes. This page documents how Projects work. To manage them, open the Cloud Console: [cloud.cerebras.ai](https://cloud.cerebras.ai) → **Projects** in the left nav. ## Concepts & Hierarchy **Organization** — Your top-level account. Billing, global rate-limit ceilings, and project management live here. **Project** — A workspace within an organization. Every API key belongs to exactly one project, and all keys in a project share that project's rate limits. ## Roles & Permissions In addition to organization roles, each project has its own set of roles. Roles are assigned from the [Members page](/console/account-billing#members). **Limited** - applies only to projects the user is assigned to. | Action | Org Admin | Project Admin | Project Member | | ------------------------- | --------------------- | --------------------- | --------------------- | | Create / archive projects | | | | | Set project rate limits | | | | | View all projects in org | | Limited | Limited | | Create API keys | | Limited | | | Delete API keys | | Limited | | | View API keys | | Limited | Limited | | Use API keys | | Limited | Limited | | View usage & logs | | Limited | Limited | | Manage org members | | | | | Manage project members | | Limited | | **Key behaviors:** * Members are invited at the organization level, then assigned to one or more projects. * Org Admins are automatically added to all projects with write access. * If a user is only assigned to one project and that project is archived, their organization access will be revoked. An Org Admin can restore it by adding them to another project. ## Navigate Projects in the Console Use the **Organization / Project dropdown** in the top-left of the console to switch context: * **Select a specific project** — all pages (Playground, API Keys, Limits, Analytics, Members, Audit Logs, Settings) filter to that project. * **Select "All projects"** — org-level view. Available to Org Admins only. Screenshot of cloud console navigation bar showing the org dropdown and the project dropdown ## Create a Project Only Org Admins can create projects. 1. In the console, open the **Projects** tab or select it from the Projects dropdown. 2. Click **Create project**. 3. Enter a project name. 4. Click **Save**. The project is created under your organization with no custom quotas, so it inherits org-level limits by default. ### Recommended Project Structures Separate projects for each deployment stage: ``` dev, test, prod ``` Separate projects for each surface or tool: ``` frontend, internal-tools ``` Separate projects per team: ``` team-console, team-platform ``` Try to keep the number of projects in your organization to a minimum. Each project adds overhead for monitoring and rate limits management. ## Configure Rate Limits Cerebras uses a two-level quota model. Every request must satisfy both levels to proceed. | Level | Description | | ----------------------- | ---------------------------------------------------------------------- | | **Org-level quota** | Hard limit for the entire organization (e.g., 1M TPM on a given model) | | **Project-level quota** | Per-project limit; cannot exceed the org-level quota | By default, projects have no explicit quotas and inherit org-level limits. For example: | Project | Configured limit | | ------- | ------------------------------------------ | | `prod` | 700,000 TPM | | `test` | 600,000 TPM | | `dev` | None (inherits org limit of 1,000,000 TPM) | If `prod` and `test` together attempt 1,300,000 TPM, some requests are throttled at the organization level — even though each project is individually within its own configured limit. ### Set Project-Level Rate Limits Only Org Admins can set project rate limits. 1. From the **All Projects** view, open the **Projects** tab. 2. Identify the project and click **Project Settings**. 3. For each model, set the desired rate limits. 4. Click **Save**. **We recommend:** * Starting with conservative limits for new projects * Monitoring 429 responses and usage graphs per project * Adjusting as you gain confidence in your workload patterns ## Manage Service Tiers If service tiers are enabled for your organization, use the `service_tier` request parameter to prioritize traffic across projects — `flex` for lower-priority projects and `priority` for production. See the [Service Tier guide](/capabilities/service-tiers) for details. ## Archive Projects Archived projects cannot be restored. Only Org Admins can archive projects. You cannot archive your last remaining project. Archiving invalidates all API keys in the project, preserves usage logs for auditing, and revokes member access to the project. ## FAQs They've been migrated into a Default Project under your organization, with the same quotas and members. Behavior is unchanged unless you configure additional projects or project-level limits. No. Using only the Default Project is fully supported. Create additional projects only if you need isolated rate limits, access control, or separate analytics by use case. Projects can be archived but not permanently deleted. Archiving disables all API keys in the project and preserves logs and usage history. No. Each API key is scoped to exactly one project. To move a workload, create a new key in the target project and update your application to use it. Projects don't create separate billing accounts. All usage accrued within a project is billed through your top-level organization. Projects provide logical segmentation for quotas, keys, and usage, but billing is always aggregated and invoiced at the organization level. You may be hitting the project-level limit for that project, or the org-level ceiling across all projects. Check the **Limits** page for both the project and the org, then review the project's analytics. # Usage & Monitoring Source: https://inference-docs.cerebras.ai/console/usage-monitoring Track API usage, inspect request logs, and view rate limits across your organization and projects. This page documents the Analytics, Logs, and Limits sections. To view them, open the Cloud Console: [cloud.cerebras.ai](https://cloud.cerebras.ai) → **Analytics** in the left nav. ## Analytics The **Analytics** page has three tabs: **Usage**, **Cached-Usage**, and **Cost**. All dates and timestamps are displayed in UTC. Track request volume and token consumption over a selected date range. Toggle **Show quotas** to overlay your rate limit thresholds and see how close you are to your limits. Use **Download Report** to export the data as a CSV. See how your input tokens break down between cache hits and fresh processing. A high cache hit rate means your prompts share a consistent prefix and your [prompt caching](/capabilities/prompt-caching) setup is working effectively. Use **Download Report** to export the data as a CSV. Review your spending for a selected calendar month, broken down by model and token type (input/output). Cost data may be delayed by up to 10 minutes. Requests made under an active monthly subscription are excluded from usage-based billing. ### Tips **Monitor quota headroom** — Enable **Show quotas** on the Usage tab to see how close you are to your rate limits. If you're consistently near the ceiling, consider distributing traffic across projects or [requesting a limit increase](https://www.cerebras.ai/contact). **Optimize caching** — If cache hits are low on the Cached-Usage tab, review whether your prompts have a stable, shared prefix. Effective caching reduces Time to First Token (TTFT) for long-context workloads. **Track costs by model** — Filter the Cost tab by model to compare spend. This helps when deciding whether a smaller, faster model is sufficient for a given use case. **Debug usage spikes** — Narrow the date range to isolate when a spike started, then cross-reference with Logs to identify the source. ## Logs The **Logs** page has two tabs: **Request Logs** and **Audit Logs**. All dates and timestamps are displayed in UTC. Inspect individual API calls by filtering on model, API key, date range, or HTTP status code. The status code chart at the top gives you a quick visual of error rates over time. Use **Download Report** to export the current view as a CSV. When contacting support about a failed request, include the **Request ID** from the log entry. Audit logs are only visible to admin users. Review actions taken in the console, such as adding members, managing billing, and changing organization settings. Click **View Payload Details** to see the full payload for any entry. ## Limits The **Limits** page displays a table of models available to your organization or project, along with their associated rate limits. The table shows each model's name, context length, limit type (requests or tokens), and quota by minute and day. Hourly limits are shown in the org-level view only. If you need higher limits, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. The values shown on the Limits page are **specific to your plan and project**. Your limits may differ from examples in the documentation. ### Per-Org vs. Per-Project Limits The limits shown depend on your current console context: | Console context | Limits shown | | ----------------------------- | ------------------------------------- | | **All Projects** view | Org-level limits | | **Specific project** selected | Project-level limits for that project | Cerebras uses a two-level quota model — requests are checked against both the project-level limit and the org-level ceiling. See [Projects](/console/projects) for more on how the two levels interact. # Inference Cookbook Source: https://inference-docs.cerebras.ai/cookbook
# Inference Cookbook

Practical guides and examples to help you build with Cerebras

## Latest Cookbooks ## All Cookbooks
# Academic Research Agent Source: https://inference-docs.cerebras.ai/cookbook/agents/academic-research-agent Generate arXiv search queries, analyze academic papers, download and process PDFs, and synthesize research insights with a conversational AI research assistant powered by PydanticAI + Cerebras + Unstructured. This cookbook demonstrates how to build a conversational agent that: * Generates diverse arXiv search queries * Searches and analyzes academic papers * Downloads and processes PDFs with Unstructured * Performs deep analysis and synthesizes research insights and saves them as reports ## What You'll Learn 1. **PydanticAI Agent Architecture** - Building conversational agents with tools 2. **Cerebras Integration** - Using Cerebras LLMs with PydanticAI 3. **Pydantic Schemas** - Type-safe structured outputs from LLMs 4. **Unstructured** - High-quality PDF text extraction 5. **Tool Design** - Creating effective agent tools with RunContext ## Setup ### Install Dependencies ```python theme={null} %pip install -q pydantic-ai cerebras-cloud-sdk python-dotenv requests feedparser unstructured-client pydantic ``` ### Load API Keys Get API keys to get started with super fast inference, and Unstructured's powerful document procesing: * **Cerebras**: [https://cloud.cerebras.ai](https://cloud.cerebras.ai?utm_source=3pi_academic-research-agent\&utm_campaign=docs) (free tier available) * **Unstructured**: [https://unstructured.io](https://unstructured.io?utm_source=cerebras\&utm_campaign=academic-research-agent) (free tier available) Next, we suggest to add the secrets in the Google Collab Password service, or via a .env file, if you cloned the repository. ```bash theme={null} CEREBRAS_API_KEY=your-key-here UNSTRUCTURED_API_KEY=your-key-here ``` ```python theme={null} import os from dotenv import load_dotenv load_dotenv() required = ["CEREBRAS_API_KEY", "UNSTRUCTURED_API_KEY"] missing = [k for k in required if not os.getenv(k)] if missing: raise RuntimeError( f"Missing API keys: {', '.join(missing)}. Add them to .env file." ) print("✅ API keys loaded") ``` ## Part 1: Pydantic Schemas We are using Pydantic models for type safety. Pydantic is a production grade typing framework, that helps to create reliable LLM responses. These schemas: * Guide the LLM on expected output structure * Validate responses automatically * Provide type hints throughout the codebase ```python theme={null} from typing import List, Optional, Dict, Any from pydantic import BaseModel, Field class ArxivQueries(BaseModel): """Structured output for arXiv query generation""" queries: List[str] = Field(description="List of diverse search queries") reasoning: str = Field(description="Why these queries were chosen") class AbstractAnalysis(BaseModel): """Analysis of paper abstracts""" key_themes: List[str] = Field(description="Main themes across papers") top_papers_for_deep_analysis: List[str] = Field( description="arXiv IDs of most relevant papers" ) reasoning: str = Field(description="Why these papers were selected") class PaperAnalysis(BaseModel): """Deep analysis of a single paper""" arxiv_id: str methods: str = Field(description="Methods and architectures used") contributions: str = Field(description="Novel contributions") limitations: Optional[str] = Field(default=None) class ResearchDirection(BaseModel): """A future research direction""" direction: str = Field(description="The research direction") rationale: str = Field(description="Why this is important") class ResearchOutput(BaseModel): """Final comprehensive research output""" research_landscape_summary: str = Field( description="Overview of the research landscape" ) key_innovations: List[str] = Field( description="Major innovations identified" ) future_research_directions: List[ResearchDirection] = Field( description="Suggested future research directions" ) papers_analyzed: int = Field(description="Total papers analyzed") queries_used: List[str] = Field(description="Search queries used") ``` ### Example: Using Schemas for Type Safety Here's how schemas validate LLM outputs: ```python theme={null} # Example: Creating a validated ArxivQueries object example_queries: ArxivQueries = ArxivQueries( queries=["vision language models", "multimodal reasoning"], reasoning="These queries cover both architecture and capability aspects" ) print(f"Queries: {example_queries.queries}") print(f"Reasoning: {example_queries.reasoning}") # Example: Creating a validated ResearchOutput example_output: ResearchOutput = ResearchOutput( research_landscape_summary="The field is rapidly evolving...", key_innovations=["Cross-modal attention", "Chain-of-thought prompting"], future_research_directions=[ ResearchDirection(direction="Video reasoning", rationale="Temporal understanding is key") ], papers_analyzed=10, queries_used=["vision language models"] ) print(f"\nPapers analyzed: {example_output.papers_analyzed}") print(f"Innovations: {example_output.key_innovations}") ``` ## Part 2: Dependencies & Configuration The agent uses **dependency injection** via PydanticAI's `RunContext`. This allows tools to access shared resources like API clients and caches. ```python theme={null} from cerebras.cloud.sdk import AsyncCerebras from unstructured_client import UnstructuredClient class ResearchDeps(BaseModel): """Dependencies for the research agent""" cerebras_client: Any unstructured_client: Any papers_cache: Dict[str, Dict[str, Any]] = Field(default_factory=dict) start_year: int = 2020 max_papers_per_query: int = 15 max_papers_for_deep_analysis: int = 3 fulltext_excerpt_chars: int = 12000 model_config = {"arbitrary_types_allowed": True} def create_research_deps( start_year: int = 2020, max_papers_for_deep_analysis: int = 3 ) -> ResearchDeps: """Create research dependencies with API clients""" return ResearchDeps( cerebras_client=AsyncCerebras( api_key=os.getenv("CEREBRAS_API_KEY"), default_headers={"X-Cerebras-3rd-Party-Integration": "academic-research-agent"} ), unstructured_client=UnstructuredClient( api_key_auth=os.getenv("UNSTRUCTURED_API_KEY") ), start_year=start_year, max_papers_for_deep_analysis=max_papers_for_deep_analysis ) ``` ## Part 3: Cerebras in Strict Mode **Important**: Cerebras requires all tools to have the same `strict` parameter value. PydanticAI may generate tools with mixed values, which causes errors. We proactively avoid this with a `prepare_tools` hook that normalizes all tools to `strict=False`: ```python theme={null} from dataclasses import replace from pydantic_ai.tools import ToolDefinition async def set_consistent_strict_param( ctx: Any, tool_defs: List[ToolDefinition] ) -> List[ToolDefinition]: """ Enforce consistent strict=False for all tools. This addresses the error: "Tools with mixed values for 'strict' are not allowed" """ return [replace(tool_def, strict=False) for tool_def in tool_defs] ``` ## Part 4: Create the Agent Now we instantiate the PydanticAI agent with: * Cerebras `gpt-oss-120b` model * `ResearchDeps` for dependency injection * `prepare_tools` hook for strict mode * System prompt defining the agent's role ```python theme={null} from pydantic_ai import Agent, RunContext agent = Agent( 'cerebras:gpt-oss-120b', deps_type=ResearchDeps, prepare_tools=set_consistent_strict_param, system_prompt="""You are an expert academic research assistant specializing in literature reviews. You help researchers by: 1. Generating effective arXiv search queries 2. Searching and analyzing academic papers 3. Identifying key themes and innovations 4. Suggesting future research directions You have access to tools for each step of the research process. Use them strategically to conduct comprehensive literature reviews. When asked to research a topic: 1. First generate diverse search queries 2. Search arXiv with those queries 3. Analyze abstracts to identify most relevant papers 4. Download and analyze full papers 5. Synthesize findings into a comprehensive report Be thorough, cite specific papers, and provide actionable insights.""" ) ``` ## Part 5: Define the 7 Research Tools In PydanticAI, each tool is decorated with `@agent.tool` and receives `RunContext[ResearchDeps]` for dependency access. ### Tool 1: Generate arXiv Search Queries ```python theme={null} import json @agent.tool async def generate_arxiv_queries( ctx: RunContext[ResearchDeps], topic: str, num_queries: int = 5 ) -> str: """ Generate diverse arXiv search queries for a research topic. Args: topic: The research topic to generate queries for num_queries: Number of queries to generate (default: 5) Returns: JSON string with queries and reasoning """ print(f"\n🔍 Generating {num_queries} search queries for: {topic}") prompt = f"""Generate {num_queries} diverse arXiv search queries for researching: "{topic}" Make queries: - Specific and targeted - Cover different aspects/angles - Use relevant technical terms - Suitable for arXiv API search Return JSON: {{ "queries": ["query1", "query2", ...], "reasoning": "why these queries cover the topic well" }}""" response = await ctx.deps.cerebras_client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": prompt}], response_format={"type": "json_object"}, temperature=1.0, max_completion_tokens=12000 ) content = response.choices[0].message.content data = json.loads(content) # Validate with Pydantic schema result: ArxivQueries = ArxivQueries(**data) print(f"✓ Generated {len(result.queries)} queries") for i, q in enumerate(result.queries, 1): print(f" {i}. {q}") return json.dumps(data) ``` ### Tool 2: Search arXiv Papers ```python theme={null} import asyncio import requests import feedparser @agent.tool async def search_arxiv_papers( ctx: RunContext[ResearchDeps], queries: List[str] ) -> str: """ Search arXiv with multiple queries and cache results. Args: queries: List of search query strings Returns: Summary of papers found """ print(f"\n📚 Searching arXiv with {len(queries)} queries...") all_papers: Dict[str, Dict[str, Any]] = {} for query in queries: search_url = "http://export.arxiv.org/api/query" params = { "search_query": f"all:{query}", "start": 0, "max_results": ctx.deps.max_papers_per_query, "sortBy": "relevance", "sortOrder": "descending" } try: response = requests.get(search_url, params=params, timeout=30) feed = feedparser.parse(response.content) for entry in feed.entries: # Skip entries without required fields if not hasattr(entry, 'id') or not hasattr(entry, 'published'): continue if not hasattr(entry, 'title') or not hasattr(entry, 'summary'): continue arxiv_id = entry.id.split("/abs/")[-1] if arxiv_id not in all_papers: try: year = int(entry.published[:4]) except (ValueError, TypeError): continue if year >= ctx.deps.start_year: authors = [] if hasattr(entry, 'authors'): authors = [author.name for author in entry.authors if hasattr(author, 'name')] all_papers[arxiv_id] = { "arxiv_id": arxiv_id, "title": entry.title, "authors": authors, "year": year, "abstract": entry.summary, "link": getattr(entry, 'link', f"https://arxiv.org/abs/{arxiv_id}") } except Exception as e: print(f" ⚠️ Query failed: {query[:50]}... ({str(e)[:50]})") continue await asyncio.sleep(1) # Rate limiting # Cache papers in dependencies ctx.deps.papers_cache.update(all_papers) summary = f"Found {len(all_papers)} unique papers from {ctx.deps.start_year} onwards\n\n" summary += "Top papers:\n" for i, (arxiv_id, paper) in enumerate(list(all_papers.items())[:10], 1): summary += f"{i}. [{paper['year']}] {arxiv_id} — {paper['title'][:80]}...\n" print(f"✓ Found {len(all_papers)} papers") return summary ``` ### Tool 3: Analyze Paper Abstracts ```python theme={null} @agent.tool async def analyze_paper_abstracts( ctx: RunContext[ResearchDeps], topic: str, max_papers: int = 20 ) -> str: """ Analyze paper abstracts to identify key themes and select papers for deep analysis. Args: topic: The research topic max_papers: Maximum papers to analyze (default: 20) Returns: JSON string with analysis results """ print(f"\n📊 Analyzing abstracts for: {topic}") papers = list(ctx.deps.papers_cache.values())[:max_papers] if not papers: return json.dumps({ "key_themes": [], "top_papers_for_deep_analysis": [], "reasoning": "No papers in cache. Run search_arxiv_papers first." }) abstracts_text = "\n\n---\n\n".join([ f"Paper {i+1} (arXiv:{p['arxiv_id']})\nTitle: {p['title']}\nAbstract: {p['abstract']}" for i, p in enumerate(papers) ]) prompt = f"""Analyze these {len(papers)} paper abstracts for research on: "{topic}" {abstracts_text} Identify: 1. Key themes across papers 2. Top {ctx.deps.max_papers_for_deep_analysis} most relevant papers for deep analysis (by arXiv ID) 3. Reasoning for selections Return JSON: {{ "key_themes": ["theme1", "theme2", ...], "top_papers_for_deep_analysis": ["arxiv_id1", "arxiv_id2", ...], "reasoning": "explanation" }}""" response = await ctx.deps.cerebras_client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": prompt}], response_format={"type": "json_object"}, temperature=1.0, max_completion_tokens=12000 ) content = response.choices[0].message.content data = json.loads(content) # Validate with Pydantic schema result: AbstractAnalysis = AbstractAnalysis(**data) print(f"✓ Identified {len(result.key_themes)} key themes") print(f"✓ Selected {len(result.top_papers_for_deep_analysis)} papers for deep analysis") return json.dumps(data) ``` ### Tool 4: Download and Process PDF Next, we create a tool that downloads PDFs from arXiv and uses Unstructured's `hi_res` partitioning strategy to detect document layout and extract structured elements like tables, images, and text. You can also swap this out for VLM partitioning, add chunking, enrichment (like table descriptions or NER), and embedding nodes to your workflow. Check out this [notebook](https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_API_On_Demand_Jobs_Walkthrough.ipynb) for a hands-on tutorial. ```python theme={null} from unstructured_client.models.operations import CreateJobRequest, DownloadJobOutputRequest from unstructured_client.models.shared import BodyCreateJob, InputFiles import asyncio import time import json @agent.tool async def download_and_process_pdf( ctx: RunContext[ResearchDeps], arxiv_id: str ) -> str: """ Download and extract text from an arXiv paper PDF using Unstructured. Args: arxiv_id: The arXiv ID (e.g., "2301.12345") Returns: Extracted text excerpt """ print(f"\n📄 Processing PDF: {arxiv_id}") # Check cache first if arxiv_id in ctx.deps.papers_cache and "fulltext" in ctx.deps.papers_cache[arxiv_id]: print(f"✓ Using cached fulltext") return ctx.deps.papers_cache[arxiv_id]["fulltext"] try: # Download PDF from arXiv pdf_url = f"https://arxiv.org/pdf/{arxiv_id}.pdf" response = requests.get(pdf_url, timeout=120) response.raise_for_status() # Define the hi-res partitioning workflow node # Create the job job_response = ctx.deps.unstructured_client.jobs.create_job( request=CreateJobRequest( body_create_job=BodyCreateJob( request_data=json.dumps({ "template_id": "hi_res_partition" # Use a template }), input_files=[ InputFiles( content=response.content, file_name=f"{arxiv_id}.pdf", content_type="application/pdf" ) ] ) ) ) job_id = job_response.job_information.id file_id = job_response.job_information.input_file_ids[0] print(f" Job ID: {job_id}") # Poll job status until complete while True: status_response = ctx.deps.unstructured_client.jobs.get_job( request={"job_id": job_id} ) job = status_response.job_information if job.status == "SCHEDULED": print(" Job is scheduled, polling again in 10 seconds...") await asyncio.sleep(10) elif job.status == "IN_PROGRESS": print(" Job is in progress, polling again in 10 seconds...") await asyncio.sleep(10) elif job.status == "COMPLETED": print(" ✓ Job completed") break elif job.status in ["FAILED", "STOPPED"]: raise Exception(f"Job {job.status.lower()}") else: await asyncio.sleep(10) # Download job output output_response = ctx.deps.unstructured_client.jobs.download_job_output( request=DownloadJobOutputRequest( job_id=job_id, file_id=file_id ) ) # Extract text from elements text_parts: List[str] = [] for element in output_response.any: if isinstance(element, dict): text = element.get("text", "") elif hasattr(element, "text"): text = element.text else: text = "" if text: text_parts.append(text) fulltext = "\n".join(text_parts) # Limit length to stay within context window excerpt = fulltext[:ctx.deps.fulltext_excerpt_chars] # Cache for reuse if arxiv_id in ctx.deps.papers_cache: ctx.deps.papers_cache[arxiv_id]["fulltext"] = excerpt print(f"✓ Extracted {len(fulltext):,} chars (using {len(excerpt):,} char excerpt)") return excerpt except Exception as e: error_msg = f"Failed to process {arxiv_id}: {str(e)}" print(f"⚠️ {error_msg}") return error_msg ``` ### Tool 5: Deep Analyze Papers ```python theme={null} @agent.tool async def deep_analyze_papers( ctx: RunContext[ResearchDeps], topic: str, arxiv_ids: List[str] ) -> str: """ Perform deep analysis of papers using their full text. Args: topic: The research topic arxiv_ids: List of arXiv IDs to analyze Returns: JSON string with deep analysis results """ print(f"\n🔬 Deep analyzing {len(arxiv_ids)} papers...") analyses: List[PaperAnalysis] = [] for arxiv_id in arxiv_ids: # Get fulltext (from cache or download) if arxiv_id in ctx.deps.papers_cache and "fulltext" in ctx.deps.papers_cache[arxiv_id]: fulltext = ctx.deps.papers_cache[arxiv_id]["fulltext"] else: fulltext = await download_and_process_pdf(ctx, arxiv_id) if "Failed to process" in fulltext: continue prompt = f"""Analyze this paper in the context of research on: "{topic}" Paper ID: {arxiv_id} Full text excerpt: {fulltext[:8000]} Extract: 1. Methods and architectures used 2. Novel contributions 3. Limitations (if mentioned) Return JSON: {{ "arxiv_id": "{arxiv_id}", "methods": "description", "contributions": "description", "limitations": "description or null" }}""" response = await ctx.deps.cerebras_client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": prompt}], response_format={"type": "json_object"}, temperature=1.0, max_completion_tokens=12000 ) content = response.choices[0].message.content data = json.loads(content) # Validate with Pydantic schema analysis: PaperAnalysis = PaperAnalysis(**data) analyses.append(analysis) print(f" ✓ Analyzed {arxiv_id}") result = { "papers": [a.model_dump() for a in analyses], "count": len(analyses) } print(f"✓ Completed deep analysis of {len(analyses)} papers") return json.dumps(result) ``` ### Tool 6: Synthesize Research Findings ```python theme={null} @agent.tool async def synthesize_research_findings( ctx: RunContext[ResearchDeps], topic: str, deep_analysis_json: str, queries_used: List[str] ) -> str: """ Synthesize all research findings into a comprehensive report. Args: topic: The research topic deep_analysis_json: JSON string from deep_analyze_papers queries_used: List of search queries that were used Returns: JSON string with final research output """ print(f"\n🎯 Synthesizing research findings...") deep_analysis = json.loads(deep_analysis_json) # Safely get papers list with fallback papers_list = deep_analysis.get('papers', []) papers_count = deep_analysis.get('count', len(papers_list)) if not papers_list: # If no papers key, the JSON might be a single paper analysis or different format # Try to handle it gracefully print("⚠️ No 'papers' key found in deep_analysis_json, attempting to parse as single paper") if 'arxiv_id' in deep_analysis: # It's a single paper analysis papers_list = [deep_analysis] papers_count = 1 else: # Return a minimal synthesis return json.dumps({ "research_landscape_summary": "Unable to synthesize - no paper analysis data available.", "key_innovations": [], "future_research_directions": [], "papers_analyzed": 0, "queries_used": queries_used }) papers_text = "\n\n".join([ f"Paper {i+1} ({p.get('arxiv_id', 'unknown')}):\n" f"Methods: {p.get('methods', 'Not specified')}\n" f"Contributions: {p.get('contributions', 'Not specified')}\n" f"Limitations: {p.get('limitations', 'Not specified')}" for i, p in enumerate(papers_list) ]) prompt = f"""Synthesize research findings on: "{topic}" Deep analysis of {papers_count} papers: {papers_text} Create a comprehensive research summary with: 1. Research landscape overview (2-3 paragraphs) 2. Key innovations (3-5 items) 3. Future research directions (3-5 items with rationale) Return JSON: {{ "research_landscape_summary": "overview text", "key_innovations": ["innovation1", "innovation2", ...], "future_research_directions": [ {{"direction": "direction1", "rationale": "why"}}, ... ] }}""" response = await ctx.deps.cerebras_client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": prompt}], response_format={"type": "json_object"}, temperature=1.0, max_completion_tokens=12000 ) content = response.choices[0].message.content data = json.loads(content) # Add metadata data["papers_analyzed"] = papers_count data["queries_used"] = queries_used # Validate with Pydantic schema result: ResearchOutput = ResearchOutput(**data) print(f"✓ Synthesis complete!") return json.dumps(data) ``` ### Tool 7: Save Research Report ```python theme={null} from datetime import datetime from pathlib import Path @agent.tool def save_research_report( ctx: RunContext[ResearchDeps], topic: str, research_output_json: str ) -> str: """ Save the research report to a file. Args: topic: The research topic research_output_json: JSON string from synthesize_research_findings Returns: Path to saved file """ try: print(f"\n💾 Saving research report...") output = json.loads(research_output_json) # Create output directory output_dir = Path("research_exports") output_dir.mkdir(exist_ok=True) # Generate filename timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") filename = f"research_analysis_{timestamp}.txt" filepath = output_dir / filename # Safely get values with defaults papers_analyzed = output.get('papers_analyzed', 'N/A') research_landscape_summary = output.get('research_landscape_summary', 'No summary available.') key_innovations = output.get('key_innovations', []) future_research_directions = output.get('future_research_directions', []) queries_used = output.get('queries_used', []) # Format report report = f""" {'=' * 80} ACADEMIC RESEARCH ANALYSIS {'=' * 80} Topic: {topic} Date: {datetime.now().strftime("%Y-%m-%d %H:%M:%S")} Papers Analyzed: {papers_analyzed} {'=' * 80} RESEARCH LANDSCAPE {'=' * 80} {research_landscape_summary} {'=' * 80} KEY INNOVATIONS {'=' * 80} """ for i, innovation in enumerate(key_innovations, 1): report += f"{i}. {innovation}\n" report += f"\n{'=' * 80}\nFUTURE RESEARCH DIRECTIONS\n{'=' * 80}\n\n" for i, direction in enumerate(future_research_directions, 1): if isinstance(direction, dict): report += f"{i}. {direction.get('direction', 'Unknown')}\n" report += f" Rationale: {direction.get('rationale', 'Not specified')}\n\n" else: report += f"{i}. {direction}\n\n" report += f"{'=' * 80}\nSEARCH QUERIES USED\n{'=' * 80}\n\n" for i, query in enumerate(queries_used, 1): report += f"{i}. {query}\n" report += f"\n{'=' * 80}\n" # Save filepath.write_text(report) print(f"✓ Report saved: {filepath}") return str(filepath) except Exception as e: error_msg = f"Failed to save report: {str(e)}" print(f"⚠️ {error_msg}") return error_msg ``` ## Part 6: Conversational Interface This function handles the conversation with the agent, including extracting the response from PydanticAI's message structure: ```python theme={null} async def chat_with_agent(research_question: str, deps: ResearchDeps) -> str: """ Have a conversation with the research agent. Args: research_question: The research question or instruction deps: Research dependencies Returns: Agent's response text """ print(f"\n📋 Your request: {research_question}\n") result = await agent.run(research_question, deps=deps) # Extract response from PydanticAI result # The result contains new_messages() with TextPart and ThinkingPart objects new_msgs = result.new_messages() if new_msgs: last_msg = new_msgs[-1] if hasattr(last_msg, 'parts'): # Extract only TextPart content, skip ThinkingPart text_parts: List[str] = [] for part in last_msg.parts: if hasattr(part, 'content') and 'TextPart' in str(type(part)): text_parts.append(part.content) response = ' '.join(text_parts) if text_parts else str(last_msg) else: response = str(last_msg) else: response = str(result) print("💬 AGENT RESPONSE") print(f"\n{response}\n") return response ``` ## Part 7: Run the Agent! ### Instantiate Our Previously Created Dependencies ```python theme={null} deps = create_research_deps( start_year=2023, max_papers_for_deep_analysis=1 ) print(f" Start year: {deps.start_year}") print(f" Max papers for deep analysis: {deps.max_papers_for_deep_analysis}") print(f" Fulltext excerpt: {deps.fulltext_excerpt_chars:,} chars") ``` ### Example 1: Full Research Workflow The agent will autonomously: 1. Generate search queries 2. Search arXiv 3. Analyze abstracts 4. Download and process PDFs 5. Perform deep analysis 6. Synthesize findings 7. Save the report ```python theme={null} research_question = """ Please conduct a comprehensive literature review on "vision-language models for multimodal reasoning". Follow these steps: 1. Generate 3 diverse arXiv search queries 2. Search arXiv with those queries 3. Analyze the abstracts to identify key themes 4. Select the top 1 most relevant paper 5. Download and analyze that paper in depth 6. Synthesize the findings into a comprehensive report 7. Save the report to a file Provide a summary of your findings at the end. """ response = await chat_with_agent(research_question, deps) ``` ### Example 2: Quick Abstract-Only Analysis The agent adapts to simpler requests: ```python theme={null} quick_question = """ What are the key themes in recent papers about "persuasive natural language generation"? Just analyze abstracts, don't download full papers. """ response = await chat_with_agent(quick_question, deps) ``` ### Example 3: Follow-up Questions The agent can answer follow-up questions: ```python theme={null} followup = "What were the most innovative methods you found in those papers?" response = await chat_with_agent(followup, deps) ``` ## Part 8: Inspect Results ### View Cached Papers ```python theme={null} print(f"Papers in cache: {len(deps.papers_cache)}") print("\nCached papers:") for i, (arxiv_id, paper) in enumerate(list(deps.papers_cache.items())[:5], 1): print(f"{i}. {arxiv_id} — {paper['title'][:60]}...") if 'fulltext' in paper: print(f" ✓ Full text cached ({len(paper['fulltext']):,} chars)") ``` ### View Saved Reports ```python theme={null} export_dir = Path("research_exports") if export_dir.exists(): reports = sorted(export_dir.glob("*.txt"), key=lambda p: p.stat().st_mtime, reverse=True) print(f"Saved reports ({len(reports)}):") for report in reports[:5]: size = report.stat().st_size print(f" • {report.name} ({size:,} bytes)") else: print("No reports saved yet") ``` ## Summary ### What We Built A **conversational academic research agent** with: * **tools** for a complete research workflow * **PydanticAI** for agent orchestration and tool management * **Cerebras** `gpt-oss-120b` for fast, high-quality reasoning * **Unstructured** for PDF text extraction * **Pydantic schemas** for type-safe structured outputs ### Key Patterns 1. **Cerebras Strict Mode**: Use `prepare_tools` hook to normalize all tools to `strict=False` 2. **Dependency Injection**: Use `RunContext[ResearchDeps]` to share API clients and caches 3. **Schema Validation**: Validate all LLM outputs with Pydantic models 4. **Error Resilience**: Tools return error messages instead of raising exceptions 5. **Caching**: Cache papers and full text to avoid redundant API calls ### Next Steps * Add semantic search with vector embeddings, rather than different API calls to arxiv's API * Add a citation graph analysis * Add multi-source search (PubMed, Semantic Scholar) ### Resources * [Cerebras Inference Docs](https://inference-docs.cerebras.ai) * [PydanticAI Docs](https://ai.pydantic.dev) * [Unstructured Docs](https://docs.unstructured.io) * [arXiv API](https://info.arxiv.org/help/api/basics.html) ### Acknowledgement Thank you team from Pydantic AI and Unstructured for incredibly helpful inputs during the creation of this cookbook. Also a shoutout to my colleagues Zhenwei Gao, Ryan Loney and Sarah Chieng for great feedback on initial versions. # Automate User Research with LangChain Source: https://inference-docs.cerebras.ai/cookbook/agents/automate-user-research Learn how to build an AI-powered user research system that can automatically generate user personas, conduct interviews, and synthesize insights using LangGraph's multi-agent workflow in under 60 seconds.