# Authentication Source: https://inference-docs.cerebras.ai/api-reference/authentication The Cerebras API uses API keys for authentication. Create and manage API keys from our [Inference Cloud Console](https://cloud.cerebras.ai?utm_source=3pi_authentication\&utm_campaign=api_reference). **Keep your API key secure.** Never share it publicly or include it in client-side code (such as browsers or mobile apps). Instead, load it safely from an environment variable or a server-side key management service. API keys are passed using HTTP Bearer authentication: ``` Authorization: Bearer CEREBRAS_API_KEY ``` ## Example Request ```bash theme={null} curl --location 'https://api.cerebras.ai/v1/chat/completions' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" \ --data '{ "model": "gpt-oss-120b", "messages": [ {"role": "user", "content": "Tell me a fun fact about space."} ] }' ``` ## Using Official SDKs You can also authenticate automatically when using the official SDKs for Python and Node.js by passing your API key during client initialization: ```python Python theme={null} import os from cerebras.cloud.sdk import Cerebras client = Cerebras( # This is the default and can be omitted api_key=os.environ.get("CEREBRAS_API_KEY"), ) ``` ```javascript Node.js theme={null} import Cerebras from 'cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], // This is the default and can be omitted }); ``` ## Set Your API Key For security reasons, and to avoid configuring your API key each time, we recommend setting your API key as an environment variable. You can do this by running the following command in your terminal: ```bash macOS / Linux theme={null} export CEREBRAS_API_KEY="your-api-key" ``` ```bash Windows theme={null} setx CEREBRAS_API_KEY "your-api-key" ``` # Cancel batch Source: https://inference-docs.cerebras.ai/api-reference/batch/cancel-batch DELETE https://api.cerebras.ai/v1/batches/{batch_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Cancels a batch job that is currently queued or running. Completed requests remain saved in the results and can be retrieved through the retrieve batch endpoint. ## Path Parameters The ID of the batch job to cancel. ## Response A unique identifier for the batch job. The object type, which is always `batch`. The API endpoint used for batch processing. Information about any errors that occurred during batch processing. The ID of the input file containing batch requests. The time window for batch completion. Always `24h`. The status of the batch job. Will be `cancelling` immediately after calling cancel. Once cancellation is complete (up to 10 minutes), the status changes to `cancelled`. The ID of the file containing batch results with any completed requests. The ID of the file containing errors (if any errors occurred). The Unix timestamp (in seconds) of when the batch job was created. The Unix timestamp (in seconds) of when the batch job started processing. The Unix timestamp (in seconds) of when the batch job will expire. The Unix timestamp (in seconds) of when the batch job started finalizing. The Unix timestamp (in seconds) of when the batch job completed. The Unix timestamp (in seconds) of when the batch job failed. The Unix timestamp (in seconds) of when the batch job expired. The Unix timestamp (in seconds) of when the batch job started cancelling. The Unix timestamp (in seconds) of when the batch job was cancelled. Statistics about the requests in the batch (if available). Custom metadata associated with the batch job. When you cancel an in-progress batch, it will be in `cancelling` status for up to 10 minutes before changing to `cancelled`, where it will have partial results (if any) available in the output file. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) batch = client.batches.cancel("batch_abc123") print(batch) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const batch = await client.batches.cancel("batch_abc123"); console.log(batch); } main(); ``` ```bash cURL theme={null} curl --location --request DELETE 'https://api.cerebras.ai/v1/batches/batch_abc123' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" ``` ```json Response theme={null} { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file-abc123", "completion_window": "24h", "status": "calling", "output_file_id": null, "error_file_id": null, "created_at": 1765492780, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": 1765493188, "failed_at": null, "expired_at": null, "cancelling_at": 1765492797, "cancelled_at":null, "request_counts": null, "metadata": null } ``` # Create batch Source: https://inference-docs.cerebras.ai/api-reference/batch/create-batch POST https://api.cerebras.ai/v1/batches This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. See the [Batch](/capabilities/batch) guide for more information. ## Request The ID of an uploaded file that contains requests for the batch job. The file must follow these constraints: * Max size of 200 MB * Uploaded with `purpose: "batch"` * Max 50,000 requests The API endpoint for batch processing. Currently only `/v1/chat/completions` is supported. The time window for batch completion. Currently, only `24h` is supported. Key-value pairs that can be attached to an object. Optional custom metadata for the batch job. Useful for organizing and tracking batch jobs. ## Response A unique identifier for the batch job. The object type, which is always `batch`. The API endpoint used for batch processing. Information about any errors that occurred during batch processing. The ID of the input file containing batch requests. The time window for batch completion. Always `24h`. The current status of the batch job. Possible values: `queued`, `in_progress`, `finalizing`, `completed`, `expired`, `failed`, `cancelled`, `cancelling`. The ID of the file containing batch results (available once processing is complete). The ID of the file containing errors (if any errors occurred). The Unix timestamp (in seconds) of when the batch job was created. The Unix timestamp (in seconds) of when the batch job started processing. The Unix timestamp (in seconds) of when the batch job will expire. The Unix timestamp (in seconds) of when the batch job started finalizing. The Unix timestamp (in seconds) of when the batch job completed. The Unix timestamp (in seconds) of when the batch job failed. The Unix timestamp (in seconds) of when the batch job expired. The Unix timestamp (in seconds) of when the batch job started cancelling. The Unix timestamp (in seconds) of when the batch job was cancelled. Statistics about the requests in the batch. Total number of requests in the batch. Number of requests that completed successfully. Number of requests that failed. Custom metadata associated with the batch job. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) batch = client.batches.create( input_file_id="file_abc123", endpoint="/v1/chat/completions", completion_window="24h", metadata={ "custom_tags": ["legal-review", "q3"] } ) print(batch) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const batch = await client.batches.create({ input_file_id: "file_abc123", endpoint: "/v1/chat/completions", completion_window: "24h", metadata: { custom_tags: ["legal-review", "q3"] } }); console.log(batch); } main(); ``` ```bash cURL theme={null} curl --location 'https://api.cerebras.ai/v1/batches' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" \ --data '{ "input_file_id": "file_abc123", "endpoint": "/v1/chat/completions", "completion_window": "24h", "metadata": { "custom_tags": ["legal-review", "q3"] } }' ``` ```json Response theme={null} { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file_abc123", "completion_window": "24h", "status": "validating", "output_file_id": null, "error_file_id": null, "created_at": 1766003277, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 0, "completed": 0, "failed": 0 }, "metadata": null } ``` # List batch Source: https://inference-docs.cerebras.ai/api-reference/batch/list-batch GET https://api.cerebras.ai/v1/batches This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. List all batch jobs associated with your organization. Supports pagination. ## Query Parameters The maximum number of files to return. Default: `20`, Maximum: `100`. A cursor for pagination. Pass a batch ID to retrieve all files created after that file. For example, if you have files `[batch-abc123, batch-def456, batch-jkl789, batch-mno123]` and pass `after=batch-def456`, the response will return `[batch-jkl789, batch-mno123]`. ## Response The object type, which is always `list`. An array of file objects that show file properties. Indicates whether there are more files available beyond this page. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) batch = client.batches.list() print(batch) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const list = await client.batches.list(); for await (const batch of list) { console.log(batch); } } main(); ``` ```bash cURL theme={null} curl --location --request GET 'https://api.cerebras.ai/v1/batches' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" ``` ```json Response theme={null} { "object": "list", "data": [ { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": "Model 'cognition-devstral-small-2508-pc' is not supported for batch processing. Supported models: gpt-oss-120b", "input_file_id": "file-abc123", "completion_window": "24h", "status": "failed", "output_file_id": null, "error_file_id": null, "created_at": 1765490209, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": 1765490211, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 0, "completed": 0, "failed": 0 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file-abc123", "completion_window": "24h", "status": "completed", "output_file_id": "file-abc123", "error_file_id": null, "created_at": 1765227145, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": 1765227301, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 656, "completed": 656, "failed": 0 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file-abc123", "completion_window": "24h", "status": "cancelled", "output_file_id": null, "error_file_id": null, "created_at": 1765492137, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": 1765492173, "request_counts": { "total": 40, "completed": 0, "failed": 0 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": "File service error: Failed to retrieve file content: Access denied to file file-8zOP2tNFQW2b7G4u3hsKDQ", "input_file_id": "file-abc123", "completion_window": "24h", "status": "failed", "output_file_id": null, "error_file_id": null, "created_at": 1765344102, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": 1765344103, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 0, "completed": 0, "failed": 0 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file-abc123", "completion_window": "24h", "status": "validating", "output_file_id": null, "error_file_id": null, "created_at": 1765211147, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 0, "completed": 0, "failed": 0 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file-abc123", "completion_window": "24h", "status": "completed", "output_file_id": "file-abc123", "error_file_id": "file-abc123", "created_at": 1765269860, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": 1765270045, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 4, "completed": 1, "failed": 3 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file-abc123", "completion_window": "24h", "status": "completed", "output_file_id": null, "error_file_id": null, "created_at": 1765223136, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": 1765223175, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 13, "completed": 0, "failed": 13 }, "metadata": null }, { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": "Model 'LLAMA3.1-8b' is not supported for batch processing. Supported models: gpt-oss-120b", "input_file_id": "file-abc123", "completion_window": "24h", "status": "failed", "output_file_id": null, "error_file_id": null, "created_at": 1765344330, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": 1765344332, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 0, "completed": 0, "failed": 0 }, "metadata": null } ], "first_id": "batch_abc123", "last_id": "batch_abc123", "has_more": true } ``` # Retrieve batch Source: https://inference-docs.cerebras.ai/api-reference/batch/retrieve-batch GET https://api.cerebras.ai/v1/batches/{batch_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Retrieves the status and progress of a batch job. Use this endpoint to monitor your batch job as it processes. ## Path Parameters The ID of the batch job to retrieve. ## Response A unique identifier for the batch job. The object type, which is always `batch`. The API endpoint used for batch processing. Information about any errors that occurred during batch processing. The ID of the input file containing batch requests. The time window for batch completion. Always `24h`. The current status of the batch job. Possible values: `validating`, `queued`, `in_progress`, `finalizing`, `completed`, `expired`, `failed`, `cancelled`, `cancelling`. The ID of the file containing batch results (available once processing is complete). The ID of the file containing errors (if any errors occurred). The Unix timestamp (in seconds) of when the batch job was created. The Unix timestamp (in seconds) of when the batch job started processing. The Unix timestamp (in seconds) of when the batch job will expire. The Unix timestamp (in seconds) of when the batch job started finalizing. The Unix timestamp (in seconds) of when the batch job completed. The Unix timestamp (in seconds) of when the batch job failed. The Unix timestamp (in seconds) of when the batch job expired. The Unix timestamp (in seconds) of when the batch job started cancelling. The Unix timestamp (in seconds) of when the batch job was cancelled. Statistics about the requests in the batch. Total number of requests in the batch. Number of requests that completed successfully. Number of requests that failed. Custom metadata associated with the batch job. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) batch = client.batches.retrieve("batch_abc123") print(batch) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const batch = await client.batches.retrieve("batch_abc123"); console.log(batch); } main(); ``` ```bash cURL theme={null} curl --location 'https://api.cerebras.ai/v1/batches/batch_abc123' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" ``` ```json Response theme={null} { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file_abc123", "completion_window": "24h", "status": "in_progress", "output_file_id": null, "error_file_id": null, "created_at": 1766003277, "in_progress_at": 1766003300, "expires_at": 1766089677, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 100, "completed": 75, "failed": 2 }, "metadata": { "custom_tags": ["legal-review", "q3"] } } ``` # Chat Completions Source: https://inference-docs.cerebras.ai/api-reference/chat-completions POST /v1/chat/completions Generate conversational responses using a structured message format with roles (system, user, assistant, developer, tool). Best for chatbots, assistants, and multi-turn conversations. Parameter support can differ depending on the model used to generate the response, particularly for newer reasoning models. For details about parameters in reasoning models, refer to the [Reasoning Guide](/capabilities/reasoning). # Completions Source: https://inference-docs.cerebras.ai/api-reference/completions POST https://api.cerebras.ai/v1/completions Generate text continuations from a single prompt string. Best for simple text generation, autocomplete, and single-turn tasks. ## Request ### Headers The media type of the request body. **Supported values:** `application/json`, `application/vnd.msgpack` **Default:** `application/json` See [Payload Optimization](/capabilities/payload-optimization) for details. The compression encoding applied to the request body. **Supported values:** `gzip` When set, the request body must be gzip-compressed. Can be combined with any supported `Content-Type`. See [Payload Optimization](/capabilities/payload-optimization) for details. ### Body The prompt(s) to generate completions for, encoded as a string, array of strings, array of tokens, or array of token arrays. Default: `""` The model to use for completion. See [Supported Models](/models/overview) for a list of available models. If set, partial message deltas will be sent. Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a `data: [DONE]` message. Default: `false` Return raw tokens instead of text. Default: `false` The maximum number of tokens that can be generated in the chat completion. The total length of input tokens and generated tokens is limited by the model's context length. Default: `null` The minimum number of tokens to generate for a completion. If not specified or set to 0, the model will generate as many tokens as it deems necessary. Setting to -1 sets to max sequence length. Default: `null` The grammar root used for structured output generation. Supported values: `root`, `fcall`, `nofcall`, `insidevalue`, `value`, `object`, `array`, `string`, `number`, `funcarray`, `func`, `ws`. Default: `null` If specified, our system will make a best effort to sample deterministically, such that repeated requests with the same `seed` and parameters should return the same result. Determinism is not guaranteed. Default: `null` Up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence. Default: `null` What sampling temperature to use, between 0 and 1.5. Higher values (e.g., 0.8) will make the output more random, while lower values (e.g., 0.2) will make it more focused and deterministic. We generally recommend altering this or `top_p` but not both. Minimum: `0`, Maximum: `1.5` Default: `1.0` An alternative to sampling with temperature, called nucleus sampling, where the model considers the tokens with top\_p probability mass. For example, 0.1 means only the tokens comprising the top 10% probability mass are considered. We generally recommend altering this or `temperature` but not both. Minimum: `0`, Maximum: `1` Default: `1.0` Echo back the prompt in addition to the completion. Incompatible with `return_raw_tokens=True`. Default: `false` A unique identifier representing your end-user, which can help Cerebras to monitor and detect abuse. Default: `null` An opaque identifier that groups related requests so they reuse the same [prompt cache](/capabilities/prompt-caching). Requests sharing the same `prompt_cache_key` are routed together, which increases cache hits and reduces time to first token. Set it to a stable identifier like a conversation ID, user ID, or session ID. Maximum length: `1024` characters Default: `null` `prompt_cache_key` must be enabled on your account before you can use it. [Contact us](https://www.cerebras.ai/contact) or reach out to your account representative to request access. Return log probabilities of the output tokens. For example, if `logprobs` is 5, the API will return a list of the 5 most likely tokens. The API will always return the logprob of the sampled token, so there may be up to `logprobs+1` elements in the response. Minimum: `0`, Maximum: `20` Default: `null` Setting `logprobs` to 0 is different than `null`. When set to `null`, log probabilities are disabled entirely. When set to 0, log probabilities are enabled but it does not return `top_logprobs`. ## Completion Response The list of completion choices the model generated for the input prompt. The reason the model stopped generating tokens. This will be `stop` if the model hit a natural stop point or a provided stop sequence, `length` if the maximum number of tokens specified in the request was reached, or `content_filter` if content was omitted due to a flag from our content filters. Number of characters since the prompt. Logprob value for each token. The tokens. List of the most likely tokens and their log probability, at this token position. In rare cases, there may be fewer than the number of requested `top_logprobs` returned. The generated completion text. The raw tokens of the completion, returned when `return_raw_tokens` is set to `true`. The Unix timestamp (in seconds) of when the completion was created. A unique identifier for the completion. The model used for completion. The object type, which is always "text\_completion" This fingerprint represents the backend configuration that the model runs with. Can be used in conjunction with the `seed` request parameter to understand when backend changes have been made that might impact determinism. Usage statistics for the completion request. Number of tokens in the prompt. Number of tokens in the generated completion. Total number of tokens used in the request (prompt + completion). Detailed breakdown of prompt token usage. Number of prompt tokens that were served from the cache and reused from a previous request. See [Prompt Caching](/capabilities/prompt-caching) for more information. Performance timing information for the request. Time spent in queue waiting for processing (in seconds). Time spent processing the prompt/input tokens (in seconds). Time spent generating the completion/output tokens (in seconds). Total time for the entire request from submission to completion (in seconds). Unix timestamp (in seconds) of when the time\_info was recorded. ```python Python theme={null} import os from cerebras.cloud.sdk import Cerebras client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), # This is the default and can be omitted ) completion = client.completions.create( prompt="It was a dark and stormy night", max_tokens=100, model="gpt-oss-120b", logprobs=5, ) print(completion) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], // This is the default and can be omitted }); async function main() { const completion = await client.completions.create({ prompt: "It was a dark and stormy night", model: 'gpt-oss-120b', logprobs: 5, }); console.log(completion?.choices[0]?.text); } main(); ``` ```cli cURL theme={null} curl -X POST https://api.cerebras.ai/v1/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "prompt": "It was a dark and stormy night", "max_tokens": 100, "model": "gpt-oss-120b", "logprobs": 5 }' ``` ```json Response theme={null} { "id": "chatcmpl-b1743f2d-8c20-4ad5-a77c-426521ae1b1c", "choices": [ { "index": 0, "finish_reason": "length", "logprobs": { "text_offset": [30, 34, 40, 44, 47, 52], "token_logprobs": [ -3.3912997245788574, -4.322617053985596, -0.3532562553882599, -1.7001153230667114, -2.187103271484375, -2.4667720794677734 ], "tokens": [" and", " there", " was", " no", " moon", ","], "top_logprobs": [ { ".": -1.1334871053695679, ",": -1.3366121053695679, " when": -2.3366122245788574, " in": -2.9381747245788574, "...": -3.3287997245788574, " and": -3.3912997245788574 }, { " I": -0.9554294943809509, " the": -1.7679295539855957, " a": -2.2366795539855957, " all": -3.1898045539855957, " we": -3.4241795539855957, " there": -4.322617053985596 }, { " was": -0.3532562553882599, " were": -1.6813812255859375, " I": -4.0251312255859375, " wasn": -4.1345062255859375, " had": -4.1970062255859375 }, { " a": -0.3719903528690338, " no": -1.7001153230667114, " an": -3.614177942276001, " this": -4.129802703857422, " nothing": -4.184490203857422 }, { " electricity": -1.3433531522750854, " one": -1.6949156522750854, " moon": -2.187103271484375, " way": -3.038665771484375, " sign": -3.054290771484375 }, { ".": -0.8495846390724182, " to": -2.1386470794677734, " in": -2.2870845794677734, ",": -2.4667720794677734, ".\n": -3.2714595794677734 } ] }, "text": " and there was no moon,", "tokens": null } ], "created": 1769297019, "model": "gpt-oss-120b", "object": "text_completion", "system_fingerprint": "fp_feb5e1faa8274e54bef0", "time_info": { "completion_time": 0.002412253, "prompt_time": 0.000347391, "queue_time": 0.000245702, "total_time": 0.004331350326538086, "created": 1769297019.2632425 }, "usage": { "completion_tokens": 6, "prompt_tokens": 8, "prompt_tokens_details": { "cached_tokens": 0 }, "total_tokens": 14 } } ``` # Delete model version Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/delete-model-version delete /management/v1/orgs/{org_name}/models/{model_arch_id}/versions/{version_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Delete a specific model version and its associated artifacts from Cerebras. # Deploy model to endpoint Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/deploy-model-to-endpoint post /management/v1/endpoints/{endpoint_id}:deployModel This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Deploy a model version to a dedicated endpoint running a model with the same underlying architecture. The endpoint queues the deployment operation and returns a deployment ID for tracking status. # List endpoints Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/list-endpoints get /management/v1/orgs/{org_name}/endpoints This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. List all endpoints accessible to an organization for management. # List model architectures Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/list-model-architectures get /management/v1/orgs/{org_name}/models This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. List model architectures accessible to an organization. # List model versions Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/list-model-versions get /management/v1/orgs/{org_name}/models/{model_arch_id}/versions This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Get all uploaded model versions for a particular supported architecture. # Retrieve endpoint status Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/retrieve-endpoint-status get /management/v1/endpoints/{endpoint_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Describes the status of an endpoint, including deployment status and active model versions. # Retrieve model version status Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/retrieve-model-version-status get /management/v1/orgs/{org_name}/models/{model_arch_id}/versions/{version_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Retrieve the status and details of a specific model version, including upload status. # Update model version aliases Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/update-model-version-aliases patch /management/v1/orgs/{org_name}/models/{model_arch_id}/versions/{version_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Assign unique aliases to model versions instead of using autogenerated IDs. Use these aliases anywhere you would normally use an integer version ID. # Upload model version Source: https://inference-docs.cerebras.ai/api-reference/customer_management_api/upload-model-version post /management/v1/orgs/{org_name}/models:upload This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Upload a new finetuned model version to Cerebras. Model versions are custom variants of Cerebras-supported model architectures. Before using this endpoint, you must configure an S3 bucket with cross-account access to Cerebras. See [S3 Bucket Setup](/dedicated/management-api#s3-bucket-setup) for instructions. # Delete file Source: https://inference-docs.cerebras.ai/api-reference/file/delete-file DELETE https://api.cerebras.ai/v1/files/{file_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Deletes a file before its automatic expiration. This removes the file from storage immediately. ## Path Parameters The ID of the file to delete. ## Response The ID of the deleted file. The object type, which is always `file`. Confirms whether the file was successfully deleted. Deleting a file removes it from storage before its automatic expiration date. Files are automatically deleted after their retention period expires (7 days by default). ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) result = client.files.delete("file_123") print(result) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const result = await client.files.delete("file_123"); console.log(result); } main(); ``` ```bash cURL theme={null} curl --location --request DELETE 'https://api.cerebras.ai/v1/files/file_123' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" ``` ```json Response theme={null} { "id": "file_123", "object": "file", "deleted": true } ``` # List files Source: https://inference-docs.cerebras.ai/api-reference/file/list-files GET https://api.cerebras.ai/v1/files/ This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Lists all files that have been uploaded. Returns metadata for each file including size, status, and expiration. ## Query Parameters The maximum number of files to return. Default: `20`\ Maximum: `100` A cursor for pagination. Pass a file ID to retrieve all files created after that file. For example, if you have files `[file-abc123, file-def456, file-jkl789, file-mno123]` and pass `after=file-def456`, the response will return `[file-jkl789, file-mno123]`. ## Response The object type, which is always `list`. An array of file objects. A unique identifier for the file. The object type, which is always `file`. The size of the file in bytes. The name of the uploaded file. The Unix timestamp (in seconds) of when the file was created. The Unix timestamp (in seconds) of when the file will be automatically deleted. The intended purpose of the file (e.g., `batch`). Whether there are more files available beyond this page. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) files = client.files.list(limit=20) print(files) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const files = await client.files.list({ limit: 20 }); console.log(files); } main(); ``` ```bash cURL theme={null} curl --location 'https://api.cerebras.ai/v1/files?limit=20/' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" ``` ```json Response theme={null} { "object": "list", "data": [ { "id": "file_123", "object": "file", "bytes": 1048576, "filename": "batch-requests.jsonl", "created_at": 1726315200, "expires_at": 1726920000, "purpose": "batch" }, { "id": "file_124", "object": "file", "bytes": 2097152, "created_at": 1726315200, "expires_at": 1726920000, "filename": "batch-results.jsonl", "purpose": "batch" } ], "has_more": false } ``` # Retrieve file Source: https://inference-docs.cerebras.ai/api-reference/file/retrieve-file GET https://api.cerebras.ai/v1/files/{file_id} This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Retrieves metadata for a specific file, including its size, status, and expiration date. ## Path Parameters The ID of the file to retrieve. ## Response A unique identifier for the file. The object type, which is always `file`. The name of the uploaded file. The intended purpose of the file. The size of the file in bytes. The Unix timestamp (in seconds) of when the file was created. The Unix timestamp (in seconds) of when the file will be automatically deleted. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) file = client.files.retrieve("file_123") print(file) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const file = await client.files.retrieve("file_123"); console.log(file); } main(); ``` ```bash cURL theme={null} curl --location 'https://api.cerebras.ai/v1/files/file_123' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" ``` ```json Response theme={null} { "id": "file_123", "object": "file", "bytes": 1048576, "filename": "batch-requests.jsonl", "created_at": 1726315200, "expires_at": 1726920000, "purpose": "batch" } ``` # Retrieve file content Source: https://inference-docs.cerebras.ai/api-reference/file/retrieve-file-content GET https://api.cerebras.ai/v1/files/{file_id}/content This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Downloads the raw content of a file. Use this endpoint to retrieve the actual file data rather than just metadata. ## Path Parameters The ID of the file whose content to retrieve. ## Response Returns the raw file contents. The `Content-Type` header indicates the file type: * JSONL files: `application/x-ndjson` ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) content = client.files.retrieve_content("file_123") # Save to file with open("downloaded-file.jsonl", "wb") as f: f.write(content) print("File downloaded successfully") ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; import * as fs from 'fs'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const content = await client.files.retrieveContent("file_123"); // Save to file fs.writeFileSync("downloaded-file.jsonl", content); console.log("File downloaded successfully"); } main(); ``` ```bash cURL theme={null} curl --location 'https://api.cerebras.ai/v1/files/file_123/content' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" \ --output downloaded-file.jsonl ``` ```text Response theme={null} Returns raw file content with appropriate Content-Type header. For JSONL files, returns newline-delimited JSON. ``` # Upload file Source: https://inference-docs.cerebras.ai/api-reference/file/upload-file POST https://api.cerebras.ai/v1/files This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Uploads a file for use with the [Batch API](/api-reference/batch/create-batch). Files can be up to **200 MB** and must be **JSONL** format. Files are retained for 7 days by default. ## Request The file to upload. Must be a binary file. The intended purpose of the file. Currently supported values: * `batch`: Input file for batch processing The expiration policy for the file. Files with `purpose=batch` expire after 7 days by default. Anchor timestamp after which the expiration policy applies. Must be `created_at`. The number of seconds after the anchor point when the file expires. Must be between 3600 (1 hour) and 2592000 (30 days). ## Response A unique identifier for the file. The object type, which is always `file`. The name of the uploaded file. The intended purpose of the file. The size of the file in bytes. The Unix timestamp (in seconds) of when the file was created. The Unix timestamp (in seconds) of when the file will be automatically deleted. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) file = client.files.create( file=open("batch-requests.jsonl", "rb"), purpose="batch" ) print(file) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; import * as fs from 'fs'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const file = await client.files.create({ file: fs.createReadStream("batch-requests.jsonl"), purpose: "batch" }); console.log(file); } main(); ``` ```bash cURL theme={null} curl --location 'https://api.cerebras.ai/v1/files' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" \ --form 'file=@"batch-requests.jsonl"' \ --form 'purpose="batch"' ``` ```json Response theme={null} { "id": "fileI72lQshwSO6Z7LIQ850CvA", "object": "file", "filename": "test.jsonl", "bytes": 4213, "purpose": "batch", "created_at": 1765172961, "expires_at": 1767764961 } ``` # Retrieve metrics Source: https://inference-docs.cerebras.ai/api-reference/metrics/retrieve-metrics GET https://cloud.cerebras.ai/api/v1/metrics/organizations/{organization_id} Retrieve operational metrics for your organization's inference endpoints in Prometheus format. See the [Metrics](/capabilities/metrics) guide for more info. ## Path Parameters The unique identifier for your organization (e.g., `org_abc123`) ## Response Returns metrics in Prometheus text-based exposition format. ```bash cURL theme={null} curl -H "Authorization: Bearer $CEREBRAS_API_KEY" \ https://cloud.cerebras.ai/api/v1/metrics/organizations/org_abc123 ``` ```python Python theme={null} import requests import os response = requests.get( "https://cloud.cerebras.ai/api/v1/metrics/organizations/org_abc123", headers={ "Authorization": f"Bearer {os.environ.get('CEREBRAS_API_KEY')}" } ) print(response.text) ``` ```javascript Node.js theme={null} const response = await fetch( 'https://cloud.cerebras.ai/api/v1/metrics/organizations/org_abc123', { headers: { 'Authorization': `Bearer ${process.env.CEREBRAS_API_KEY}` } } ); const metrics = await response.text(); console.log(metrics); ``` ```prometheus Response theme={null} # HELP inference_endpoint_status Status of inference endpoint (-1=error calculating status, 0=down, 1=up) # TYPE inference_endpoint_status gauge inference_endpoint_status{endpoint="model",organization_id="org_abc123"} 1.0 # HELP requests_count_total Total request count (all HTTP codes) in the last complete minute # TYPE requests_count_total gauge requests_count_total{endpoint="model",organization_id="org_abc123"} 1.0 # HELP requests_success_total Total successful requests (HTTP 200) in the last complete minute # TYPE requests_success_total gauge requests_success_total{endpoint="model",organization_id="org_abc123"} 1.0 # HELP requests_failure_total Total failed requests by HTTP code in the last complete minute # TYPE requests_failure_total gauge requests_failure_total{endpoint="model",organization_id="org_abc123"} 0.0 # HELP input_tokens_total Total input tokens for successful requests in the last complete minute # TYPE input_tokens_total gauge input_tokens_total{endpoint="model",organization_id="org_abc123"} 123456.0 # HELP output_tokens_total Total output tokens for successful requests in the last complete minute # TYPE output_tokens_total gauge output_tokens_total{endpoint="model",organization_id="org_abc123"} 12345.0 # HELP queue_time_seconds Queue time percentiles in seconds # TYPE queue_time_seconds gauge queue_time_seconds{endpoint="model",organization_id="org_abc123",percentile="avg"} 0.55 queue_time_seconds{endpoint="model",organization_id="org_abc123",percentile="p50"} 0.55 queue_time_seconds{endpoint="model",organization_id="org_abc123",percentile="p90"} 0.55 queue_time_seconds{endpoint="model",organization_id="org_abc123",percentile="p95"} 0.55 queue_time_seconds{endpoint="model",organization_id="org_abc123",percentile="p99"} 0.55 # HELP e2e_latency_seconds End-to-end API latency percentiles in seconds # TYPE e2e_latency_seconds gauge e2e_latency_seconds{endpoint="model",organization_id="org_abc123",statistic="avg"} 0.9 e2e_latency_seconds{endpoint="model",organization_id="org_abc123",statistic="p50"} 0.9 e2e_latency_seconds{endpoint="model",organization_id="org_abc123",statistic="p90"} 0.9 e2e_latency_seconds{endpoint="model",organization_id="org_abc123",statistic="p95"} 0.9 e2e_latency_seconds{endpoint="model",organization_id="org_abc123",statistic="p99"} 0.9 # HELP ttft_seconds Time To First Token percentiles in seconds # TYPE ttft_seconds gauge ttft_seconds{endpoint="model",organization_id="org_abc123",statistic="avg"} 0.9 ttft_seconds{endpoint="model",organization_id="org_abc123",statistic="p50"} 0.9 ttft_seconds{endpoint="model",organization_id="org_abc123",statistic="p90"} 0.9 ttft_seconds{endpoint="model",organization_id="org_abc123",statistic="p95"} 0.9 ttft_seconds{endpoint="model",organization_id="org_abc123",statistic="p99"} 0.9 # HELP cache_reads_total Total input tokens read from cache for successful requests in the last complete minute # TYPE cache_reads_total gauge cache_reads_total{endpoint="model",organization_id="org_abc123"} 1234.0 # HELP cache_rate Ratio of input tokens read from cache, to total input tokens, for successful requests in the last complete minute # TYPE cache_rate gauge cache_rate{endpoint="model",organization_id="org_abc123"} 0.01 # HELP tpot Time Per Output Token (TPOT) percentiles # TYPE tpot gauge tpot{endpoint="model",organization_id="org_abc123",statistic="avg"} 0.0001 tpot{endpoint="model",organization_id="org_abc123",statistic="p50"} 0.0001 tpot{endpoint="model",organization_id="org_abc123",statistic="p90"} 0.0001 tpot{endpoint="model",organization_id="org_abc123",statistic="p95"} 0.0001 tpot{endpoint="model",organization_id="org_abc123",statistic="p99"} 0.0001 # HELP latency_generation_seconds Completion time percentiles in seconds # TYPE latency_generation_seconds gauge latency_generation_seconds{endpoint="model",organization_id="org_abc123",statistic="avg"} 1.1 latency_generation_seconds{endpoint="model",organization_id="org_abc123",statistic="p50"} 1.1 latency_generation_seconds{endpoint="model",organization_id="org_abc123",statistic="p90"} 1.1 latency_generation_seconds{endpoint="model",organization_id="org_abc123",statistic="p95"} 1.1 latency_generation_seconds{endpoint="model",organization_id="org_abc123",statistic="p99"} 1.1 ``` ## Available Metrics The following metrics are available on an opt-in basis. Contact your Cerebras account representative to enable specific metrics for your organization. ### Endpoint Health Status of inference endpoint **Values:** * `-1` = Error calculating status * `0` = Down * `1` = Up ### Request Metrics Total request count (all HTTP codes) in the last complete minute Total successful requests (HTTP 200) in the last complete minute Total failed requests by HTTP code in the last complete minute ### Token Metrics Total input tokens for successful requests in the last complete minute Total output tokens for successful requests in the last complete minute Total input tokens read from cache for successful requests in the last complete minute Ratio of input tokens read from cache, to total input tokens, for successful requests in the last complete minute ### Latency Metrics Queue time percentiles in seconds for successful requests (avg/p50/p90/p95/p99) (e.g. time a request spends waiting for resources at runtime) End-to-end API latency percentiles in seconds for successful requests (avg/p50/p90/p95/p99). Includes overall latency from requests received at the API gateway to the response output from API gateway, inclusive of `latency_generation_seconds`. Time To First Token percentiles in seconds for successful requests (avg/p50/p90/p95/p99) Time per output tokens percentiles (avg/p50/p90/p95/p99), excluding time to first token, averaged across successful requests Time to generate all output tokens (e.g. time from last prompt to last output token) percentiles in seconds for successful requests (avg/p50/p90/p95/p99) ## Error Codes For information about possible error responses, see the [Error Codes](/support/error) documentation. # List models Source: https://inference-docs.cerebras.ai/api-reference/models/list-models GET https://api.cerebras.ai/v1/models Lists all currently available models and provides essential details about each, including the owner and availability. ## Response The object type, which is always `list`. An array of model objects. The model identifier, which can be referenced in the API endpoints. The object type, which is always `model`. The Unix timestamp (in seconds) of when the model was created. The organization that owns the model. ```python Python theme={null} import os from cerebras.cloud.sdk import Cerebras client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) models = client.models.list() print(models) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const models = await client.models.list(); console.log(models); } main(); ``` ```bash cURL theme={null} curl https://api.cerebras.ai/v1/models \ -H "Authorization: Bearer $CEREBRAS_API_KEY" ``` ```json Response theme={null} { "object": "list", "data": [ { "id": "zai-glm-4.7", "object": "model", "created": 0, "owned_by": "Cerebras" }, { "id": "gpt-oss-120b", "object": "model", "created": 0, "owned_by": "Cerebras" } ] } ``` # Public models Source: https://inference-docs.cerebras.ai/api-reference/models/public-models GET https://api.cerebras.ai/public/v1/models Retrieve details about publicly available Cerebras models without an API key, including context length, pricing, and supported features. This endpoint supports multiple response formats for compatibility with different platforms. This endpoint is **public** and does not require an API key. ## Supported Formats The endpoint supports three response formats via the `format` query parameter: * **Default (Cerebras)** - Native Cerebras format * **OpenRouter** - OpenRouter-compatible format * **HuggingFace** - HuggingFace-compatible format ## Query Parameters Response format. Options: `openrouter`, `huggingface`. Omit for default Cerebras format. ## Response ### Default Format The object type, which is always `list` for list responses or `model` for single model responses. Array of model objects (only in list responses). The model identifier (e.g., `gpt-oss-120b`). The object type, which is always `model`. The Unix timestamp (in seconds) of when the model was created. The organization that owns the model. Human-readable model name. Description of the model's capabilities and recommended use cases. The HuggingFace repository identifier for this model. Pricing per token in USD. Cost per prompt token. Cost per completion token. Boolean flags for supported features. Whether streaming is supported. Whether function calling is supported. Whether structured outputs are supported. Whether vision/image input is supported. Whether JSON mode is supported. Whether tool use is supported. Whether tool choice is supported. Whether parallel tool calls are supported. Whether response format control is supported. Whether the model supports reasoning. Boolean flags for supported API parameters. Model architecture details. Input/output modality (e.g., `text`). Tokenizer type. Instruction tuning format. Model limits. Maximum context window in tokens. Maximum completion tokens. Requests per minute limit (null if unlimited). Tokens per minute limit (null if unlimited). Whether this model is deprecated. Whether this model is in preview. Quantization method used for the model. ### OpenRouter Format Array of model objects with extended metadata. The model identifier. The HuggingFace repository identifier. Human-readable model name. Supported input types (e.g., `["text"]`). Supported output types (e.g., `["text"]`). Maximum context window size in tokens. Maximum output length in tokens. Pricing information per token. Cost per prompt token. Cost per completion token. Cost per request (typically `"0"`). Cost per image token (typically `"0"` for text-only models). Cost per cached input token read (typically `"0"`). Cost per cached input token write (typically `"0"`). List of supported sampling parameters. List of supported features (e.g., `["tools", "json_mode", "structured_outputs"]`). ### HuggingFace Format Array of model objects. The model identifier. The HuggingFace repository identifier. The object type, which is always `model`. The Unix timestamp (in seconds) of when the model was created. The organization that owns the model. Pricing in USD per million tokens. Price per million input tokens. Price per million output tokens. Maximum context window size in tokens. Boolean flags for supported features. ```bash cURL (Default) theme={null} curl -sS 'https://api.cerebras.ai/public/v1/models' | jq ``` ```bash cURL (OpenRouter) theme={null} curl -sS 'https://api.cerebras.ai/public/v1/models?format=openrouter' | jq ``` ```bash cURL (HuggingFace) theme={null} curl -sS 'https://api.cerebras.ai/public/v1/models?format=huggingface' | jq ``` ```python Python theme={null} import httpx # Default format response = httpx.get("https://api.cerebras.ai/public/v1/models") models = response.json() # OpenRouter format response = httpx.get( "https://api.cerebras.ai/public/v1/models", params={"format": "openrouter"} ) models = response.json() # HuggingFace format response = httpx.get( "https://api.cerebras.ai/public/v1/models", params={"format": "huggingface"} ) models = response.json() ``` ```javascript JavaScript theme={null} // Default format const response = await fetch('https://api.cerebras.ai/public/v1/models'); const models = await response.json(); // OpenRouter format const response = await fetch( 'https://api.cerebras.ai/public/v1/models?format=openrouter' ); const models = await response.json(); // HuggingFace format const response = await fetch( 'https://api.cerebras.ai/public/v1/models?format=huggingface' ); const models = await response.json(); ``` ```json Default Format theme={null} { "object": "list", "data": [ { "id": "gpt-oss-120b", "object": "model", "created": 1754438400, "owned_by": "OpenAI", "name": "OpenAI GPT OSS", "description": "This model excels at efficient reasoning across science, math, and coding applications. It's ideal for real-time coding assistance, processing large documents for Q&A and summarization, agentic research workflows, and regulated on-premises workloads.", "hugging_face_id": "openai/gpt-oss-120b", "pricing": { "prompt": "0.00000035", "completion": "0.00000075" }, "capabilities": { "streaming": true, "function_calling": true, "structured_outputs": true, "vision": false, "json_mode": true, "tools": true, "tool_choice": true, "parallel_tool_calls": false, "response_format": true, "reasoning": true }, "supported_parameters": { "temperature": true, "top_p": true, "seed": true, "stop": true, "max_completion_tokens": true, "logprobs": true, "top_logprobs": true, "frequency_penalty": true, "presence_penalty": true, "logit_bias": true, "repetition_penalty": false }, "architecture": { "modality": "text", "tokenizer": "GPT", "instruct_type": "harmony" }, "limits": { "max_context_length": 131072, "max_completion_tokens": 40960, "requests_per_minute": null, "tokens_per_minute": null }, "datacenter_locations": [], "deprecated": false, "preview": false, "quantization": "FP16/8 (weights only)" } ] } ``` ```json OpenRouter Format theme={null} { "data": [ { "id": "gpt-oss-120b", "hugging_face_id": "openai/gpt-oss-120b", "name": "OpenAI GPT OSS", "created": 1754438400, "input_modalities": ["text"], "output_modalities": ["text"], "quantization": "fp16", "context_length": 131072, "max_output_length": 40960, "pricing": { "prompt": "0.00000035", "completion": "0.00000075", "request": "0", "image": "0", "input_cache_read": "0", "input_cache_write": "0" }, "supported_sampling_parameters": [ "temperature", "top_p", "stop", "seed", "frequency_penalty", "presence_penalty", "max_tokens", "logprobs", "top_logprobs", "logit_bias" ], "supported_features": [ "tools", "json_mode", "structured_outputs", "reasoning" ] } ] } ``` ```json HuggingFace Format theme={null} { "data": [ { "id": "gpt-oss-120b", "hugging_face_id": "openai/gpt-oss-120b", "object": "model", "created": 1754438400, "owned_by": "OpenAI", "context_length": 131072, "pricing": { "input": 0.35, "output": 0.75 }, "capabilities": { "streaming": true, "function_calling": true, "structured_outputs": true, "vision": false } } ] } ``` ## Retrieve Specific Model You can also retrieve information about a specific model by appending the model ID to the endpoint: ``` GET https://api.cerebras.ai/public/v1/models/{model_id} ``` ### Path Parameters The ID of the model to retrieve (e.g., `gpt-oss-120b`, `zai-glm-4.7`). ```bash cURL theme={null} curl -sS 'https://api.cerebras.ai/public/v1/models/gpt-oss-120b' | jq ``` ```python Python theme={null} import httpx model_id = "gpt-oss-120b" response = httpx.get(f"https://api.cerebras.ai/public/v1/models/{model_id}") model = response.json() ``` ```javascript JavaScript theme={null} const modelId = 'gpt-oss-120b'; const response = await fetch( `https://api.cerebras.ai/public/v1/models/${modelId}` ); const model = await response.json(); ``` ```json Response theme={null} { "id": "gpt-oss-120b", "object": "model", "created": 1754438400, "owned_by": "OpenAI", "name": "OpenAI GPT OSS", "description": "This model excels at efficient reasoning across science, math, and coding applications. It's ideal for real-time coding assistance, processing large documents for Q&A and summarization, agentic research workflows, and regulated on-premises workloads.", "hugging_face_id": "openai/gpt-oss-120b", "pricing": { "prompt": "0.00000035", "completion": "0.00000075" }, "capabilities": { "streaming": true, "function_calling": true, "structured_outputs": true, "vision": false, "json_mode": true, "tools": true, "tool_choice": true, "parallel_tool_calls": false, "response_format": true, "reasoning": true }, "limits": { "max_context_length": 131072, "max_completion_tokens": 40960, "requests_per_minute": null, "tokens_per_minute": null }, "deprecated": false, "preview": false, "quantization": "FP16/8 (weights only)" } ``` ## Use Cases **Platform Integration** - Use the OpenRouter or HuggingFace formats to integrate Cerebras models into existing platforms that support these standards. **Model Discovery** - Programmatically discover available models and their capabilities without authentication. **Pricing Comparison** - Compare pricing across different models using the structured pricing information. **Feature Detection** - Check which features (streaming, tools, JSON mode) are supported by each model. # Retrieve model Source: https://inference-docs.cerebras.ai/api-reference/models/retrieve-model GET https://api.cerebras.ai/v1/models/{model_id} Fetches a model instance, offering key details about the model, including its owner and permissions. ## Path Parameters The ID of the model to retrieve. Available options: * `gpt-oss-120b` * `zai-glm-4.7` (preview) * `gemma-4-31b` (preview) ## Response The model identifier. The object type, which is always `model`. The Unix timestamp (in seconds) of when the model was created. The organization that owns the model. ```python Python theme={null} import os from cerebras.cloud.sdk import Cerebras client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) model = client.models.retrieve("gpt-oss-120b") print(model) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const model = await client.models.retrieve("gpt-oss-120b"); console.log(model); } main(); ``` ```bash cURL theme={null} curl https://api.cerebras.ai/v1/models/gpt-oss-120b \ -H "Authorization: Bearer $CEREBRAS_API_KEY" ``` ```json Response theme={null} { "id": "gpt-oss-120b", "object": "model", "created": 1721692800, "owned_by": "Cerebras" } ``` # Versions Source: https://inference-docs.cerebras.ai/api-reference/versions Understand how Cerebras uses versioning to manage breaking changes. Cerebras uses API versioning to manage breaking changes to request validation and response formats. The latest version is available for testing via the `X-Cerebras-Version-Patch` header and becomes the default on July 21, 2026. **This versioning does not change the API interface.** You'll continue using the same endpoints (e.g., `/v1/chat/completions`) and the same SDKs. New versions only affect validation rules and response field behavior. No new endpoints or SDK updates required. Review [what changed](#what-changed-in-version-2), then set `X-Cerebras-Version-Patch: 2` to test your integration before the July 21, 2026 cutover. ## What's Guaranteed Within a given API version, we guarantee: * Existing request parameters continue to work as documented * Existing response fields remain present with the same types Non-breaking changes that may occur: * New optional request parameters may be added * New fields may be added to responses * Error message text may change (but not error types within a version) Breaking changes only happen in new API versions, giving you time to test and migrate. This includes stricter validation, removed parameters, or changed defaults. *** ## Setting the API Version Override the default API version per-request by passing the `X-Cerebras-Version-Patch` header: ```bash cURL theme={null} curl https://api.cerebras.ai/v1/chat/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "X-Cerebras-Version-Patch: 2" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-oss-120b", "messages": [{"role": "user", "content": "Hello!"}] }' ``` ```python OpenAI SDK theme={null} import os from openai import OpenAI client = OpenAI( base_url="https://api.cerebras.ai/v1", api_key=os.environ.get("CEREBRAS_API_KEY") ) completion = client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": "Hello!"}], extra_headers={"X-Cerebras-Version-Patch": "2"} ) ``` ```python Cerebras SDK (Python) theme={null} import os from cerebras.cloud.sdk import Cerebras client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY") ) completion = client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": "Hello!"}], extra_headers={"X-Cerebras-Version-Patch": "2"} ) ``` ```javascript Cerebras SDK (Node.js) theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); async function main() { const completion = await client.chat.completions.create( { model: 'gpt-oss-120b', messages: [{ role: 'user', content: 'Hello!' }], }, { headers: { 'X-Cerebras-Version-Patch': '2' }, } ); console.log(completion); } main(); ``` *** ## Version 2 Rollout Timeline | Date | What Happens | | -------------------- | --------------------------------------------------------- | | **January 21, 2026** | Version 2 available for testing via header | | **July 21, 2026** | Version 2 becomes the default; older versions end-of-life | After July 21, 2026, all requests will use version 2. *** ## What Changed in Version 2 Version 2 introduces stricter validation for structured outputs and tool calling, refines reasoning model behavior, and fixes edge cases. If your application uses JSON schemas, tool calls, or reasoning models, review these changes carefully. ### Structured Outputs: Stricter Schema Validation When using `strict: true`, Version 2 requires explicit, strictly-typed schemas. `additionalProperties: false` is required at every level of nested objects: ```json theme={null} // ❌ Fails: missing additionalProperties { "type": "object", "properties": { "user": { "type": "object", "properties": { "name": { "type": "string" } }, "required": ["name"] } }, "required": ["user"] } ``` ```json theme={null} // ✅ Works: additionalProperties at all levels { "type": "object", "properties": { "user": { "type": "object", "properties": { "name": { "type": "string" } }, "required": ["name"], "additionalProperties": false } }, "required": ["user"], "additionalProperties": false } ``` ### Tool Calling: Stricter Message Validation Version 2 validates the structure of multi-turn conversations involving tool calls: | Validation | What it checks | | -------------------------- | --------------------------------------------------------------------------------------------------------- | | Tool response completeness | Every `tool_call_id` in an assistant message must have a corresponding tool message immediately following | | Orphan tool messages | Tool messages cannot reference a `tool_call_id` that doesn't exist in a prior assistant message | | Tool choice consistency | `tool_choice` cannot be set unless `tools` is also provided | | Unique tool call IDs | Duplicate `tool_call.id` values in the same request are rejected | ### Reasoning Logprobs Added For reasoning models, version 2 adds a separate `reasoning_logprobs` field that contains logprobs for the reasoning tokens. Previously, reasoning logprobs were included in the main `logprobs` field alongside content logprobs. See [Reasoning](/capabilities/reasoning) for details on reasoning format options. ```json theme={null} // Before: reasoning logprobs included in main logprobs field { "message": { "content": "Here is the answer...", "reasoning": "Let me think..." }, "logprobs": { "content": [ {"token": "Let ", "logprob": -0.3}, {"token": "me", "logprob": -0.4}, ... {"token": "Here", "logprob": -0.1}, {"token": " is", "logprob": -0.2}, ... ] } } ``` ```json theme={null} // After (version 2): reasoning logprobs are separated { "message": { "content": "Here is the answer...", "reasoning": "Let me think..." }, "logprobs": { "content": [ {"token": "Here", "logprob": -0.1}, {"token": " is", "logprob": -0.2}, ... ] }, "reasoning_logprobs": { "content": [ {"token": "Let ", "logprob": -0.3}, {"token": "me", "logprob": -0.4}, ... ] } } ``` ### Unicode Handling Fix Logprobs now reflect partial Unicode tokens as they appear in the model's vocabulary. Previously, the Unicode replacement character (`\uFFFD`) was not handled correctly in logprobs output. This fix is backward-compatible and requires no code changes. *** ## Migration Checklist Use this checklist to prepare for version 2: Add `X-Cerebras-Version-Patch: 2` to your requests and run your test suite. ```python theme={null} extra_headers={"X-Cerebras-Version-Patch": "2"} ``` * Add all properties to the `required` array * Add `additionalProperties: false` to all object definitions * Ensure enum values match their property types If you parse reasoning content, update your code to read from the `reasoning` field in the response instead of extracting it from `content`. ```python theme={null} # Before reasoning = extract_reasoning_from_content(response.choices[0].message.content) # After (version 2) reasoning = response.choices[0].message.reasoning content = response.choices[0].message.content ``` Once your tests pass, no further action is required. On July 21, 2026, version 2 takes effect and the header is no longer needed. *** ## Future Versions We strive to maintain backward compatibility whenever possible. In rare cases where breaking changes are necessary, a new API version will be released and made available for testing via the `X-Cerebras-Version-Patch` header for a minimum of 6 months before taking effect. Cerebras Cloud users will be notified via email before each version transition. # Batch Source: https://inference-docs.cerebras.ai/capabilities/batch Run large-scale inference workloads asynchronously. This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. The Batch API lets you process groups of requests asynchronously, making it perfect for workloads where you don't need immediate results: * **Evaluation pipelines**: Test model performance across thousands of test cases * **Data labeling**: Classify or annotate large datasets for training * **Content generation**: Create product descriptions, summaries, or translations in bulk * **Research and analysis**: Process scientific data or run experiments at scale Batch requests are guaranteed to complete in 24 hours. ## How It Works To process a batch request: 1. **Prepare a JSONL file** containing all your requests 2. **Upload the file** using the [Files API](/api-reference/file/upload-file) 3. **Submit a batch job** referencing your uploaded file 4. **Download results** once processing completes Behind the scenes, Cerebras processes your requests during periods of lower demand. ### Understanding Batch IDs The Batch API generates different IDs at each stage of the workflow: | ID Type | When is it generated? | Purpose | | ---------------- | ------------------------------- | -------------------------------------------------------- | | `input_file_id` | When you upload your JSONL file | Reference your uploaded input file when creating a batch | | `batch_id` | When you create a batch job | Track status and manage the batch job | | `output_file_id` | When batch completes | Download successful results | | `error_file_id` | When batch completes | Download failed request details | Store each ID as you receive it. You'll need the input file ID to create a batch, the batch ID to check progress, and the output/error file IDs to download results. ## Create a Batch Request SDK support for Batch is not yet available during Private Preview. Use cURL or direct HTTP requests for now—SDK support will be added at GA. Start by creating a `.jsonl` file where each line represents one API request. Every request needs a unique `custom_id` so you can match inputs to outputs later. The available endpoint is currently `/v1/chat/completions`. Here's what two requests look like: ```jsonl theme={null} {"custom_id": "eval-001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "user", "content": "Summarize the water cycle"}], "max_completion_tokens": 500}} {"custom_id": "eval-002", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "user", "content": "Explain photosynthesis"}], "max_completion_tokens": 500}} ``` Each line contains the same parameters you'd use in a regular chat completion request, just wrapped in the batch format. **Important constraints:** * Maximum 200 MB file size * Minimum 10 requests * Up to 50,000 requests per file * Each line limited to 1 MB * UTF-8 encoding with LF line endings * All requests must use the same model Use the [Files API](/api-reference/file/upload-file) to upload your prepared input file: ```python Python theme={null} from cerebras.cloud.sdk import Cerebras client = Cerebras(api_key="your-api-key") input_file = client.files.create( file=open("my_batch_requests.jsonl", "rb"), purpose="batch" ) print(f"Uploaded file: {input_file.id}") ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; import fs from 'fs'; const client = new Cerebras({ apiKey: process.env.CEREBRAS_API_KEY }); const file = await client.files.create({ file: fs.createReadStream("my_batch_requests.jsonl"), purpose: "batch" }); console.log(`Uploaded file: ${file.id}`); ``` ```bash cURL theme={null} curl https://api.cerebras.ai/v1/files \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -F purpose="batch" \ -F file="@my_batch_requests.jsonl" ``` Once your file is uploaded, create the batch job: ```python Python theme={null} batch = client.batches.create( input_file_id=input_file.id, endpoint="/v1/chat/completions", completion_window="24h", metadata={"description": "Evaluation run - December 2025"} ) print(f"Batch started: {batch.id}") print(f"Status: {batch.status}") ``` ```javascript Node.js theme={null} const batch = await client.batches.create({ input_file_id: file.id, endpoint: "/v1/chat/completions", completion_window: "24h", metadata: { description: "Evaluation run - December 2025" } }); console.log(`Batch started: ${batch.id}`); console.log(`Status: ${batch.status}`); ``` ```bash cURL theme={null} curl https://api.cerebras.ai/v1/batches \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "input_file_id": "", "endpoint": "/v1/chat/completions", "completion_window": "24h", "metadata": {"description": "Evaluation run - December 2025"} }' ``` The batch starts in `queued` status, then moves to `in_progress`, `finalizing`, and finally `completed`. Check on your batch job anytime to see how it's progressing: ```python Python theme={null} batch = client.batches.retrieve("") print(f"Status: {batch.status}") print(f"Completed: {batch.request_counts.completed}/{batch.request_counts.total}") ``` ```javascript Node.js theme={null} const batch = await client.batches.retrieve(""); console.log(`Status: ${batch.status}`); console.log(`Completed: ${batch.request_counts.completed}/${batch.request_counts.total}`); ``` ```bash cURL theme={null} curl https://api.cerebras.ai/v1/batches/ \ -H "Authorization: Bearer $CEREBRAS_API_KEY" ``` The response includes detailed request count metrics showing how many requests have completed and failed. When the status shows `completed`, the response object will include `output_file_id` and `error_file_id`: ```json Response highlight={6-7} theme={null} { "id": "", "object": "batch", "endpoint": "/v1/chat/completions", "status": "completed", "output_file_id": "", "error_file_id": "", "created_at": 1768244812, "completed_at": 1768244962, "request_counts": { "total": 10, "completed": 6, "failed": 4 } } ``` Use the `output_file_id` and `error_file_id` to download your results through the [Files API](/api-reference/file/retrieve-file-content): ```python Python theme={null} results = client.files.retrieve_content("") with open("batch_results.jsonl", "wb") as f: f.write(results) ``` ```javascript Node.js theme={null} const results = await client.files.retrieveContent(""); fs.writeFileSync("batch_results.jsonl", results); ``` ```bash cURL theme={null} curl https://api.cerebras.ai/v1/files//content \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -o batch_results.jsonl ``` Your results file contains one line per request. Successful requests include the full completion response, while failed requests include error details: ```jsonl theme={null} {"custom_id":"eval-001","status":"succeeded","response":{"id":"cmpl_1","object":"chat.completion","created":1699999999,"model":"gpt-oss-120b","choices":[{"index":0,"message":{"role":"assistant","content":"The water cycle consists of..."},"finish_reason":"stop"}],"usage":{"prompt_tokens":15,"completion_tokens":85,"total_tokens":100}}} {"custom_id":"eval-002","status":"succeeded","response":{"id":"cmpl_2","object":"chat.completion","created":1700000000,"model":"gpt-oss-120b","choices":[{"index":0,"message":{"role":"assistant","content":"Photosynthesis is the process..."},"finish_reason":"stop"}],"usage":{"prompt_tokens":12,"completion_tokens":92,"total_tokens":104}}} ``` Results aren't necessarily in the same order as your input. Always use `custom_id` to match requests to responses. ## Batch States Your batch progresses through these states: | State | What's happening | | ------------- | --------------------------------- | | `in_progress` | Actively processing requests | | `finalizing` | Preparing output file | | `completed` | Results ready to download | | `failed` | System error prevented completion | | `expired` | Exceeded 24-hour window | | `cancelled` | You stopped the batch | | `cancelling` | Cancellation in progress | Most batches complete well before the 24-hour limit, often within a few hours depending on size and load. ### Expired Batches If your batch doesn't finish within 24 hours, unprocessed requests are marked as expired. You'll still get results for any completed requests, and expired ones appear in your error\_file like this: ```json theme={null} {"custom_id":"eval-999","status":"expired","error":{"code":"timeout","message":"Batch expired before this request completed."}} ``` You're only charged for requests that completed. ## Cancel a Batch Use the cancel endpoint: ```python Python theme={null} cancelled = client.batches.cancel("") print(f"Status: {cancelled.status}") ``` ```javascript Node.js theme={null} const cancelled = await client.batches.cancel(""); console.log(`Status: ${cancelled.status}`); ``` ```bash cURL theme={null} curl -X DELETE https://api.cerebras.ai/v1/batches/ \ -H "Authorization: Bearer $CEREBRAS_API_KEY" ``` Any completed requests remain available in your results. Unfinished requests will be deleted. ## Limits and Quotas * 50,000 requests maximum per batch * 200 MB maximum file size * 1 MB maximum per request line * 10 concurrent active batches * Configurable rate limits separate from real-time API * Results retained for 7 days after completion * Automatic deletion after expiration (download promptly!) ## Organize Batches Use `metadata` to keep track of different batch runs. Metadata is a set of key-value pairs that you can attach to a batch job for your own organizational purposes. For example, to track which environment, dataset, or version a batch belongs to. ```python theme={null} batch = client.batches.create( input_file_id="file_xyz", endpoint="/v1/chat/completions", completion_window="24h", metadata={ "environment": "production", "dataset": "customer_feedback_q4", "version": "v2.1" } ) ``` The metadata you provide is stored with the batch object and returned in all API responses, making it easy to filter, search, and organize your batch jobs. Common uses include: * **Environment tracking**: Tag batches with `production`, `staging`, or `development` * **Dataset identification**: Link batches to specific datasets or experiments * **Version control**: Track which version of your prompts or models you're testing ## Handling Large Datasets Need to process more than 50,000 items? Split them across multiple batches: ```python theme={null} def split_into_batches(items, batch_size=50000): for i in range(0, len(items), batch_size): yield items[i:i + batch_size] # Process each chunk as a separate batch for i, chunk in enumerate(split_into_batches(all_items)): file = create_batch_file(chunk, f"batch_{i}.jsonl") # Submit each batch... ``` # Image Inputs Source: https://inference-docs.cerebras.ai/capabilities/image-inputs Pass images to vision-capable models. This feature is in [Public Preview](/support/preview-releases#public-preview). Vision-capable models can understand visual content alongside text — including objects, diagrams, screenshots, and any text that appears within an image (see [Limitations](#limitations) for exceptions). Images are sent through the [Chat Completions](/api-reference/chat-completions) API as base64-encoded data URIs in the `messages` array. Currently, image support is only available with `gemma-4-31b`. ## Usage To send an image, add an `image_url` object to the `content` array in a user message. The image must be base64-encoded and passed as a data URI. Use the [encoder in the Token Usage section](#estimate-token-count) to convert your image to a base64 data URI. It also shows the estimated token count and encoded payload size. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os import base64 client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) def encode_image(image_path): with open(image_path, "rb") as image_file: return base64.b64encode(image_file.read()).decode("utf-8") base64_image = encode_image("screenshot.png") response = client.chat.completions.create( model="gemma-4-31b", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Describe this image in one concise sentence."}, { "type": "image_url", "image_url": { "url": f"data:image/png;base64,{base64_image}" }, }, ], } ], ) print(response.choices[0].message.content) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; import fs from 'fs'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); const base64Image = fs.readFileSync('screenshot.png').toString('base64'); const response = await client.chat.completions.create({ model: 'gemma-4-31b', messages: [ { role: 'user', content: [ { type: 'text', text: 'Describe this image in one concise sentence.' }, { type: 'image_url', image_url: { url: `data:image/png;base64,${base64Image}`, }, }, ], }, ], }); console.log(response.choices[0].message.content); ``` ```bash cURL theme={null} # Encode image to base64 (macOS/Linux) BASE64_IMAGE=$(base64 -i screenshot.png) # Windows PowerShell: # $BASE64_IMAGE = [Convert]::ToBase64String([IO.File]::ReadAllBytes("screenshot.png")) # If you run this example in PowerShell, use curl.exe and replace ${BASE64_IMAGE} with $BASE64_IMAGE. curl https://api.cerebras.ai/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${CEREBRAS_API_KEY}" \ -d "{ \"model\": \"gemma-4-31b\", \"messages\": [ { \"role\": \"user\", \"content\": [ {\"type\": \"text\", \"text\": \"Describe this image in one concise sentence.\"}, { \"type\": \"image_url\", \"image_url\": { \"url\": \"data:image/png;base64,${BASE64_IMAGE}\" } } ] } ] }" ``` Include up to 5 images in a single request by adding additional `image_url` content parts to the `content` array. The model considers all images together when generating its response. Each image counts toward your [token usage](#token-usage). ```python Python theme={null} from cerebras.cloud.sdk import Cerebras import os import base64 client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) def encode_image(image_path): with open(image_path, "rb") as image_file: return base64.b64encode(image_file.read()).decode("utf-8") base64_image_1 = encode_image("image1.jpeg") base64_image_2 = encode_image("image2.png") response = client.chat.completions.create( model="gemma-4-31b", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Compare these two images."}, { "type": "image_url", "image_url": { "url": f"data:image/jpeg;base64,{base64_image_1}" }, }, { "type": "image_url", "image_url": { "url": f"data:image/png;base64,{base64_image_2}" }, }, ], } ], ) print(response.choices[0].message.content) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; import fs from 'fs'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); const base64Image1 = fs.readFileSync('image1.jpeg').toString('base64'); const base64Image2 = fs.readFileSync('image2.png').toString('base64'); const response = await client.chat.completions.create({ model: 'gemma-4-31b', messages: [ { role: 'user', content: [ { type: 'text', text: 'Compare these two images.' }, { type: 'image_url', image_url: { url: `data:image/jpeg;base64,${base64Image1}`, }, }, { type: 'image_url', image_url: { url: `data:image/png;base64,${base64Image2}`, }, }, ], }, ], }); console.log(response.choices[0].message.content); ``` ```bash cURL theme={null} # Encode images to base64 (macOS/Linux) BASE64_IMAGE_1=$(base64 -i image1.jpeg) BASE64_IMAGE_2=$(base64 -i image2.png) curl https://api.cerebras.ai/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${CEREBRAS_API_KEY}" \ -d "{ \"model\": \"gemma-4-31b\", \"messages\": [ { \"role\": \"user\", \"content\": [ {\"type\": \"text\", \"text\": \"Compare these two images.\"}, { \"type\": \"image_url\", \"image_url\": { \"url\": \"data:image/jpeg;base64,${BASE64_IMAGE_1}\" } }, { \"type\": \"image_url\", \"image_url\": { \"url\": \"data:image/png;base64,${BASE64_IMAGE_2}\" } } ] } ] }" ``` ## Input Requirements | Requirement | Details | | ---------------------- | ---------------------------------------------------- | | Supported formats | PNG (`.png`), JPEG (`.jpeg`, `.jpg`) | | Encoding | Base64 data URI (e.g., `data:image/png;base64,...`) | | External image URLs | Not supported during Public Preview | | Max payload size | `10 MB` total image payload per request 1 | | Max images per request | `5` 1 |
1 These limits apply to the shared tier during Public Preview. Higher limits may be available for [Dedicated Endpoints](/dedicated).
## Token Usage `gemma-4-31b` uses the default preprocessing setting of up to 280 image tokens per image. The model preserves image aspect ratio during preprocessing. Depending on the input dimensions, the image may be downscaled or upscaled before tokenization. The processed height and width are then rounded down to the nearest multiple of 48. As a result, token usage depends on the **processed image dimensions**, not the uploaded file size or original resolution. ### Estimate Token Count Upload an image below to copy its base64 data URI, check the encoded size, and view a token estimate. You can also estimate the token count manually with the following steps: 1. Start with the input width and height. 2. Compute the scale factor: ```text theme={null} scale = sqrt(645120 / (width × height)) ``` 3. Multiply the width and height by the scale factor. ```text theme={null} scaled_width = width × scale scaled_height = height × scale ``` 4. Round each processed dimension down to the nearest multiple of 48. 5. Compute the token count: ```text theme={null} image_tokens = (processed_width / 48) × (processed_height / 48) ``` 6. Cap the result at 280. This means smaller images do not always use fewer image tokens. For example, a `336 × 226` image is upscaled during preprocessing to `960 × 624`, which uses `260` image tokens. | Input resolution | Processed resolution | Image tokens used | | ---------------- | -------------------- | ----------------- | | 336 × 226 | 960 × 624 | 260 | | 512 × 512 | 768 × 768 | 256 | | 672 × 672 | 768 × 768 | 256 | | 1024 × 1024 | 768 × 768 | 256 | | 1280 × 720 | 1056 × 576 | 264 | | 1920 × 1080 | 1056 × 576 | 264 | | 2560 × 1440 | 1056 × 576 | 264 | | 3840 × 2160 | 1056 × 576 | 264 | | 336 × 480 | 672 × 960 | 280 | | 480 × 336 | 960 × 672 | 280 | To validate image token usage, inspect `usage.image_tokens` in the API response. This field reports the total number of image tokens used by the request. `usage.prompt_tokens` includes text tokens, image tokens, and message-formatting tokens. The difference between image and text-only `prompt_tokens` can include message-formatting tokens and might not match `usage.image_tokens`. Keep the following in mind: * 280 is the maximum image tokens per image for `gemma-4-31b` on Cerebras. * Compressed file size does not directly determine token count. Processed image dimensions matter more than PNG or JPEG byte size. * Image tokens are included in `usage.prompt_tokens` and are also reported in `usage.image_tokens`. * Image tokens occupy part of the model context window, just like text prompt tokens. ## Limitations * **Medical images** — not suitable for interpreting specialized medical images such as CT scans or MRIs. Do not use for medical diagnosis or advice. * **Small text** — may have difficulty reading small or low-resolution text. Enlarging text within the image before sending can improve results. * **Rotated content** — may misinterpret text or images that are rotated or upside-down. * **Graphs and charts** — may struggle to distinguish visual elements that differ only in color or line style, such as solid versus dashed lines. * **Spatial reasoning** — not reliable for tasks requiring precise spatial localization, such as identifying positions on a map or board game. * **Object counting** — the model may give approximate counts for objects in images. * **Image shape** — may perform less accurately on panoramic or fisheye images. * **Preprocessing** — the model cannot access original filenames or metadata. Images may be resized before analysis — see [Token Usage](#token-usage) for details. * **Accuracy** — the model may generate inaccurate descriptions or captions in some scenarios. Verify outputs for high-stakes use cases. * **CAPTCHAs** — CAPTCHA images are not supported. * **Indirect prompt injection** — text embedded in an image is included in the model's prompt context alongside the user's text. If an image contains adversarial instructions (for example, text that says "ignore all previous instructions") and the user prompt asks the model to answer based on the image, the model may follow those embedded instructions. Treat image content from untrusted sources as untrusted input, and use a system prompt to constrain the model's behavior when processing images you don't control. * **Untrusted output** — the model may transcribe or describe text from an image verbatim, including HTML, script tags, URLs, or control characters. The API returns this content unmodified. Treat it the same as any other untrusted input before rendering, logging, or executing it in your application. ## FAQs Yes. Cerebras Chat Completions is stateless. If a follow-up request depends on an earlier image, include that image-bearing turn in the conversation history you send with the new request. Continue to include that turn for as long as the model needs the visual context. No, only image input is supported. The model returns text only and does not generate images. Yes. Prompt caching can help with repeated images and repeated multimodal context within your organization. Prompt caches are never shared between organizations and remain ephemeral. See [Prompt Caching](/capabilities/prompt-caching). No. Image support uses the same rate limit framework as text. The same request and token limits still apply based on your organization and tier. For current details, see [Rate Limits](/support/rate-limits). Image inputs are processed as soon as they are received, and the original image payloads are not persisted. After preprocessing, image tokens and image embeddings may be cached ephemerally within your organization to support prompt caching. # Metrics Source: https://inference-docs.cerebras.ai/capabilities/metrics Monitor your dedicated inference endpoints with Prometheus-compatible metrics for requests, tokens, latency, and endpoint health. The Metrics API is designed for customers on dedicated endpoints who need granular observability into their inference workloads. Metrics are: * Aggregated at the minute level - Data represents the last complete minute * Pull-based - Your monitoring system queries the API on demand * Rate-limited - Up to 6 requests per minute For complete API details including request format, authentication, and error codes, see the [Metrics API reference](/api-reference/metrics/retrieve-metrics). ## Set Up Metrics Collection **Prerequisites:** You need a dedicated Cerebras inference endpoint, your organization ID, and a valid API key. Find your organization ID from your Cerebras account dashboard's Settings page. It will look like `org_abc123`. Use this URL format, replacing `` with your actual organization ID: ``` https://cloud.cerebras.ai/api/v1/metrics/organizations/ ``` This endpoint can be directly used by monitoring tools like Prometheus, Grafana Cloud, or Datadog. Configure your monitoring tool to include the Authorization header with your Cerebras API key: ``` Authorization: Bearer YOUR_API_KEY ``` Set your monitoring system to poll the endpoint every 60 seconds (1 minute). This matches the metric aggregation window and stays within rate limits. **Rate limit:** Maximum 6 requests per minute per organization Verify the connection by making a test request: ```bash cURL theme={null} curl -H "Authorization: Bearer $CEREBRAS_API_KEY" \ https://cloud.cerebras.ai/api/v1/metrics/organizations/org_abc123 ``` ```python Python theme={null} import requests import os response = requests.get( "https://cloud.cerebras.ai/api/v1/metrics/organizations/org_abc123", headers={ "Authorization": f"Bearer {os.environ.get('CEREBRAS_API_KEY')}" } ) # Parse Prometheus text format print(response.text) ``` ```javascript Node.js theme={null} const response = await fetch( 'https://cloud.cerebras.ai/api/v1/metrics/organizations/org_abc123', { headers: { 'Authorization': `Bearer ${process.env.CEREBRAS_API_KEY}` } } ); const metrics = await response.text(); console.log(metrics); ``` ## Direct Prometheus Integration To integrate directly with Prometheus, specify the Cerebras metrics endpoint in your scrape config: ```yaml prometheus.yml theme={null} global: scrape_interval: 60s scrape_configs: - job_name: 'cerebras-inference' metrics_path: '/api/v1/metrics/organizations/' authorization: type: "Bearer" credentials: "YOUR_API_KEY" static_configs: - targets: ['cloud.cerebras.ai'] scheme: https ``` For more details on Prometheus configuration, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/). ## Available Metrics For a complete list of available metrics including endpoint health, requests, tokens, and latency percentiles, see the [Available Metrics](/api-reference/metrics/retrieve-metrics#available-metrics) section in the API reference. ### Understanding Percentiles Latency metrics include multiple percentiles to give you a complete picture: * **avg** - Mean latency across all requests * **p50** - Median latency (50th percentile) * **p90** - 90% of requests complete faster than this * **p95** - 95% of requests complete faster than this * **p99** - 99% of requests complete faster than this **Example:** If `ttft_seconds{statistic="p95"} = 0.5`, then 95% of your requests receive their first token within 500ms. ## Example PromQL Queries ```promql theme={null} # Average end-to-end latency avg(e2e_latency_seconds{statistic="avg"}) # Request success rate rate(requests_success_total[5m]) / rate(requests_count_total[5m]) # Token throughput (tokens per second) rate(output_tokens_total[5m]) # Cache hit rate rate(cache_rate[5m]) # P95 time to first token ttft_seconds{statistic="p95"} ``` ## Use Cases ### Performance Monitoring Track latency percentiles (avg, p50, p90, p95, p99) across multiple dimensions: * **Time to First Token (TTFT)** - Measure initial response latency * **Time Per Output Token (TPOT)** - Monitor generation speed * **End-to-end latency** - Track total request duration * **Queue time** - Identify capacity constraints ### Usage Analytics Monitor token consumption and request patterns: * Track input and output tokens for cost analysis * Measure cache hit rates with `cache_rate` * Analyze request success and failure rates * Monitor endpoint availability ### Alerting and SLAs Set up alerts based on: * Endpoint health status changes * Latency threshold breaches * Error rate spikes * Request volume anomalies ## Troubleshooting If `queue_time_seconds` percentiles are elevated: 1. Check if you're hitting capacity limits on your dedicated endpoint by checking the Analytics tab on cloud.cerebras.ai 2. Review request patterns for traffic spikes 3. Contact your Cerebras representative about scaling options If `requests_failure_total` is increasing: 1. Check the `http_code` label to identify specific error types 2. Review the [Error Codes](/support/error) documentation 3. Verify authentication tokens are valid and not expired 4. Check [Rate Limits](/support/rate-limits) if seeing 429 errors If `cache_rate` is lower than expected: 1. Verify [Prompt Caching](/capabilities/prompt-caching) is enabled 2. Check that prompts have sufficient shared prefixes 3. Review cache configuration with your Cerebras representative If metrics appear stale: 1. Verify your API key has correct permissions 2. Check that metrics are enabled for your organization 3. Ensure you're querying the correct `organization_id` 4. Check for [error responses](/support/error) from the API # Payload Optimization Source: https://inference-docs.cerebras.ai/capabilities/payload-optimization Reduce latency by compressing request payloads with msgpack encoding and gzip. For requests with large payloads, you can reduce the size of the request body sent to the Cerebras API by using **[msgpack](https://msgpack.org/) encoding**, **[gzip](https://docs.python.org/3/library/gzip.html) compression**, or both. This can meaningfully improve time-to-first-token (TTFT) for requests with long prompts. Payload optimization is supported on [/v1/chat/completions](/api-reference/chat-completions) and [/v1/completions](/api-reference/completions). Support on [Dedicated Endpoints](/dedicated/overview) may vary by model. ## Encoding Options The Cerebras API accepts the following request body encodings. The table shows the expected payload size reduction for each option: | Content-Type | Description | Chat Completions1 | Completions2 | | ---------------------------------------------------- | -------------------------- | ---------------------------- | ----------------------- | | `application/json` | Default JSON encoding | Baseline | Baseline | | `application/vnd.msgpack` | msgpack binary encoding | up to \~5% | up to \~56% | | `application/json` + `Content-Encoding: gzip` | JSON with gzip compression | up to \~98% | up to \~68% | | `application/vnd.msgpack` + `Content-Encoding: gzip` | msgpack + gzip | up to \~98% | up to \~69% | 1 Measured against a 50k-token chat completions payload (206 KB JSON baseline).
2 Measured against a 50k token-ID completions payload (331 KB JSON baseline). You can use msgpack encoding or gzip compression independently, or combine them for maximum compression. Smaller request payloads reduce network transfer time, a contributing factor to TTFT. Actual TTFT improvement will vary, as network transfer is one of several factors that contribute to overall latency. ## When to Use Payload Optimization Optimizing payload size with request compression is most beneficial for: * **Long prompts** – requests with long system prompts, extensive conversation history, or large code blocks. Gzip compression is the most effective option for these payloads. * **Token-ID completions** – `/v1/completions` payloads using integer token arrays benefit from both msgpack encoding and gzip compression * **Tool call-heavy payloads** – requests with many tool definitions or deeply nested JSON structures. Both msgpack encoding and gzip compression provide savings. For small requests (under a few KB), the overhead of compression may outweigh the savings. Standard JSON encoding is fine for typical chat interactions. Size reductions depend on payload content. The benchmarks above used a token-ID completions payload, where msgpack's integer encoding provides the greatest benefit. String-heavy chat payloads may see smaller msgpack reductions, while payloads with deeply nested structures (e.g., tool calls) may see greater savings. Gzip benefits are more consistent across payload types. ## msgpack Encoding [msgpack](https://msgpack.org/) is a binary serialization format that produces smaller payloads than equivalent JSON. To use it, serialize your request body with msgpack and set the `Content-Type` header to `application/vnd.msgpack`. ```python Python theme={null} import msgpack import requests import os url = "https://api.cerebras.ai/v1/chat/completions" api_key = os.environ.get("CEREBRAS_API_KEY") payload = { "model": "gpt-oss-120b", "messages": [ {"role": "user", "content": "Explain quantum computing in one paragraph."} ] } response = requests.post( url, data=msgpack.packb(payload), headers={ "Content-Type": "application/vnd.msgpack", "Authorization": f"Bearer {api_key}" } ) print(response.json()["choices"][0]["message"]["content"]) ``` ```javascript Node.js theme={null} const { encode } = require("@msgpack/msgpack"); const apiKey = process.env.CEREBRAS_API_KEY; const payload = { model: "gpt-oss-120b", messages: [ { role: "user", content: "Explain quantum computing in one paragraph." } ] }; const response = await fetch("https://api.cerebras.ai/v1/chat/completions", { method: "POST", body: encode(payload), headers: { "Content-Type": "application/vnd.msgpack", "Authorization": `Bearer ${apiKey}` } }); const data = await response.json(); console.log(data.choices[0].message.content); ``` ```bash cURL theme={null} # Create a msgpack-encoded payload python3 -c " import msgpack, sys payload = { 'model': 'gpt-oss-120b', 'messages': [{'role': 'user', 'content': 'Explain quantum computing in one paragraph.'}] } sys.stdout.buffer.write(msgpack.packb(payload)) " > /tmp/request.msgpack curl https://api.cerebras.ai/v1/chat/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/vnd.msgpack" \ --data-binary @/tmp/request.msgpack ``` ## Gzip Compression You can gzip-compress any request body and set the `Content-Encoding: gzip` header. This works with both JSON and msgpack payloads. ```python Python theme={null} import gzip import json import requests import os url = "https://api.cerebras.ai/v1/chat/completions" api_key = os.environ.get("CEREBRAS_API_KEY") payload = { "model": "gpt-oss-120b", "messages": [ {"role": "user", "content": "Explain quantum computing in one paragraph."} ] } json_bytes = json.dumps(payload).encode("utf-8") compressed = gzip.compress(json_bytes, compresslevel=5) response = requests.post( url, data=compressed, headers={ "Content-Type": "application/json", "Content-Encoding": "gzip", "Authorization": f"Bearer {api_key}" } ) print(response.json()["choices"][0]["message"]["content"]) ``` ```javascript Node.js theme={null} const { gzipSync } = require("zlib"); const apiKey = process.env.CEREBRAS_API_KEY; const payload = { model: "gpt-oss-120b", messages: [ { role: "user", content: "Explain quantum computing in one paragraph." } ] }; const jsonBytes = Buffer.from(JSON.stringify(payload)); const compressed = gzipSync(jsonBytes, { level: 5 }); const response = await fetch("https://api.cerebras.ai/v1/chat/completions", { method: "POST", body: compressed, headers: { "Content-Type": "application/json", "Content-Encoding": "gzip", "Authorization": `Bearer ${apiKey}` } }); const data = await response.json(); console.log(data.choices[0].message.content); ``` ```bash cURL theme={null} # Compress the JSON payload with gzip echo '{"model": "gpt-oss-120b", "messages": [{"role": "user", "content": "Explain quantum computing in one paragraph."}]}' \ | gzip > /tmp/request.json.gz curl https://api.cerebras.ai/v1/chat/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/json" \ -H "Content-Encoding: gzip" \ --data-binary @/tmp/request.json.gz ``` ## Combining Both For maximum compression, use msgpack encoding with gzip compression: ```python Python theme={null} import gzip import msgpack import requests import os url = "https://api.cerebras.ai/v1/chat/completions" api_key = os.environ.get("CEREBRAS_API_KEY") payload = { "model": "gpt-oss-120b", "messages": [ {"role": "user", "content": "Explain quantum computing in one paragraph."} ] } data = msgpack.packb(payload) compressed = gzip.compress(data, compresslevel=5) response = requests.post( url, data=compressed, headers={ "Content-Type": "application/vnd.msgpack", "Content-Encoding": "gzip", "Authorization": f"Bearer {api_key}" } ) print(response.json()["choices"][0]["message"]["content"]) ``` ```javascript Node.js theme={null} const { encode } = require("@msgpack/msgpack"); const { gzipSync } = require("zlib"); const apiKey = process.env.CEREBRAS_API_KEY; const payload = { model: "gpt-oss-120b", messages: [ { role: "user", content: "Explain quantum computing in one paragraph." } ] }; const msgpackData = Buffer.from(encode(payload)); const compressed = gzipSync(msgpackData, { level: 5 }); const response = await fetch("https://api.cerebras.ai/v1/chat/completions", { method: "POST", body: compressed, headers: { "Content-Type": "application/vnd.msgpack", "Content-Encoding": "gzip", "Authorization": `Bearer ${apiKey}` } }); const data = await response.json(); console.log(data.choices[0].message.content); ``` ```bash cURL theme={null} # Create a msgpack+gzip payload (requires Python) python3 -c " import msgpack, gzip, sys payload = { 'model': 'gpt-oss-120b', 'messages': [{'role': 'user', 'content': 'Explain quantum computing in one paragraph.'}] } sys.stdout.buffer.write(gzip.compress(msgpack.packb(payload), compresslevel=5)) " > /tmp/request.msgpack.gz curl https://api.cerebras.ai/v1/chat/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/vnd.msgpack" \ -H "Content-Encoding: gzip" \ --data-binary @/tmp/request.msgpack.gz ``` # Predicted Outputs Source: https://inference-docs.cerebras.ai/capabilities/predicted-outputs Reduce latency by specifying parts of the response that are already known. This feature is in [Public Preview](/support/preview-releases#public-preview). Predicted Outputs is enabled for the following models: * [`gpt-oss-120b`](/models/openai-oss) * [`zai-glm-4.7`](/models/zai-glm-47) Predicted Outputs enable you to speed up response generation when parts of the output are already known. This is most useful when regenerating text or code that requires only minor changes. You can provide your draft using the [`prediction`](../api-reference/chat-completions#param-prediction) request parameter. The model reuses matching tokens and regenerates only those that differ, improving output generation speed. ## When to Use Predicted Outputs Use Predicted Outputs in scenarios where most of the model's response is already known or can be pre-computed. Recommended use cases include: * **Code Refactoring:** Modify known code without regenerating from scratch (e.g., tab/inline completion, full-file edits, structural transformations) * **Document Editing:** Apply small edits to known documents (e.g., grammar fixes, tone adjustments) * **Template Filling:** Update placeholders or small sections in predictable structured text Predicted Outputs improves generation speed only when one or more continuous token sequences from the `prediction` field appear in the model’s response. There’s no performance benefit when the output is completely unpredictable. ## Usage For example, imagine you want to modify a CSS file to change the color of all body text from green to blue. To use Predicted Outputs, include the code snippet below as both part of both your prompt and the predicted output: Begin by importing the Cerebras SDK and setting up the client. ```python Python theme={null} import os from cerebras.cloud.sdk import Cerebras client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env.CEREBRAS_API_KEY, }); ``` Include the `prediction` parameter in your chat completions request in addition to your normal `messages`. Set the `prediction` field to include the content you expect to be reused. ```python Python theme={null} code = """ html { margin: 0; padding: 0; box-sizing: border-box; scroll-behavior: smooth; font-size: 16px; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } body { font-family: Georgia, serif; font-size: 14px; line-height: 1.8; background: #000000; margin: 0; padding: 0; color: #00FF00; } """ instructions = "Change the color to blue. Respond only with code. Don't add comments." response = client.chat.completions.create( model="gpt-oss-120b", messages=[ {"role": "user", "content": instructions}, {"role": "user", "content": code} ], prediction={"type": "content", "content": code}, # stream=True, # Uncomment to enable streaming ) print(response) print(response.choices[0].message.content) ``` ```javascript Node.js theme={null} const code = ` html { margin: 0; padding: 0; box-sizing: border-box; scroll-behavior: smooth; font-size: 16px; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } body { font-family: Georgia, serif; font-size: 14px; line-height: 1.8; background: #000000; margin: 0; padding: 0; color: #00FF00; } `; const instructions = "Change the color to blue. Respond only with code. Don't add comments"; const response = await client.chat.completions.create({ model: "gpt-oss-120b", messages: [ { role: "user", content: instructions }, { role: "user", content: code } ], prediction: { type: "content", content: code }, // stream: true // Uncomment to enable streaming }); console.log(response); console.log(response.choices[0].message.content); ``` In this example, most of the code remains unchanged. Only the color value needs to be updated. By providing the original code as the prediction, the model can efficiently reuse the unchanged portions. When providing a prediction, any tokens provided that are not part of the final completion will be charged at completion token rates. ## Token-Reuse Metrics For the best performance, your prediction should have a high token-reuse rate. The response includes usage metrics showing how many prediction tokens were accepted or rejected: ```json theme={null} { "usage": { "completion_tokens": 224, // Number of tokens in your response (billed at output rate) "prompt_tokens": 204, // Number of input tokens (billed at input rate) "total_tokens": 428, // Prompt + Completion tokens "completion_tokens_details": { "accepted_prediction_tokens": 76, // Tokens from prediction successfully reused "rejected_prediction_tokens": 20 // Tokens rejected and regenerated (billed at output rate) } } } ``` These usage stats tell you how efficiently your prediction worked. A high ratio of accepted to rejected tokens indicates efficient prediction reuse, resulting in faster generation with minimal cost for rejected tokens. ## Best Practices * **Use this when most of the output is known**: The larger the known section, the greater the efficiency gain. Predicted Outputs work best when you can anticipate significant portions of the response. * **Set `temperature=0`**: Reduces randomness and increases the likelihood of token acceptance from your prediction. * **Keep predictions accurate**: Misaligned predictions increase rejected tokens and can slow generation speed. Ensure your prediction closely matches the expected output. * **Monitor prediction metrics**: Track accepted vs rejected tokens in the usage metadata to evaluate effectiveness. * **Fallback gracefully**: If the rejection rate is high for a class of prompts or files, fall back to a standard completion request without the prediction field. [Reasoning](/capabilities/reasoning) tokens are treated as completion tokens. When using a reasoning model with Predicted Outputs, the presence of reasoning tokens can generate a few additional `rejected_prediction_tokens`, which slightly increases cost. ## Limitations Please consider the following limitations when using Predicted Outputs. * The following models are currently supported: `gpt-oss-120b`, `zai-glm-4.7` * When you provide a prediction, any tokens that do not appear in the final completion are still billed at completion-token rates. To determine how many predicted tokens were not used, review the [`rejected_prediction_tokens`](../api-reference/chat-completions#param-rejected-prediction-tokens) property in the [`usage`](../api-reference/chat-completions#param-usage) object. * The following API parameters are not supported when using this feature: * `logprobs`: not supported * `n`: values greater than 1 are not supported * `tools`: tool calling is not currently supported with Predicted Outputs ## FAQ Only when predicted tokens are not accepted. Input and output tokens are billed at standard rates, while rejected prediction tokens are billed at the output token rate. Customers with dedicated endpoints are not affected by this pricing. Check `accepted_prediction_tokens` and `rejected_prediction_tokens` in the response's `usage` object. The model rejects mismatched tokens and regenerates them, which may reduce output generation speed and increase costs since rejected tokens are billed as additional output tokens. No, we do not store any prediction data. # Prompt Caching Source: https://inference-docs.cerebras.ai/capabilities/prompt-caching Store and reuse previously processed prompts to reduce latency and increase response times for similar or repeated queries. This feature reduces Time to First Token (TTFT) and improves responsiveness for long-context workloads, such as multi-turn conversations, RAG (Retrieval Augmented Generation), and agentic workflows. ## How It Works Unlike other providers that require manual cache breakpoints or header modifications, Cerebras Prompt Caching works automatically on all supported API requests. No code changes are required. 1. **Prefix Matching**: When you send a request, the system analyzes the beginning of your prompt (the prefix). This includes system prompts, tool definitions, and few-shot examples. 2. **Block-Based Caching**: The system processes prompts in 128-token blocks. If a block matches a segment stored in our ephemeral memory from a recent request within your organization, the system reuses the computation. 3. **Cache Hit**: Reusing cached blocks skips the processing phase for those tokens, resulting in lower latency. 4. **Cache Miss**: If no match is found, the system processes the prompt as normal and stores the prefix in the cache for potential future matches. 5. **Automatic Expiration**: Cached data is ephemeral. We guarantee a Time-To-Live (TTL) of 5 minutes, though caches may persist up to 1 hour depending on system load. To get a cache hit, the entire beginning of your prompt must match *exactly* with a previously cached prefix. Even a single character difference in the first token will result in a cache miss for that block and all subsequent blocks. ## Example: Multi-Turn Conversation with Tool Calling In this scenario, a shopping assistant helps users check order status and cancel orders using two tools: `get_order_status` and `cancel_order`. The system message and tool definitions remain constant across turns and are cached, while the conversation progresses naturally. ```python Python expandable theme={null} import os import json from cerebras.cloud.sdk import Cerebras client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) # Mock order database ORDERS = { "ORD-123456": {"status": "processing", "eta_days": 5}, } def get_order_status(order_id: str): """Look up an order's status""" order = ORDERS.get(order_id) if not order: return {"error": "Order not found"} return {"order_id": order_id, "status": order["status"], "eta_days": order.get("eta_days")} def cancel_order(order_id: str): """Cancel an order""" order = ORDERS.get(order_id) if not order: return {"error": "Order not found"} if order["status"] in ["shipped", "delivered"]: return {"error": f"Cannot cancel - order already {order['status']}"} order["status"] = "cancelled" order.pop("eta_days", None) return {"order_id": order_id, "status": "cancelled"} tools = [ { "type": "function", "function": { "name": "get_order_status", "description": "Look up an order status by order ID", "parameters": { "type": "object", "properties": { "order_id": {"type": "string", "description": "Order ID (e.g., ORD-123456)"} }, "required": ["order_id"] }, "strict": True } }, { "type": "function", "function": { "name": "cancel_order", "description": "Cancel an order by order ID", "parameters": { "type": "object", "properties": { "order_id": {"type": "string", "description": "Order ID (e.g., ORD-123456)"} }, "required": ["order_id"] }, "strict": True } } ] available_functions = { "get_order_status": get_order_status, "cancel_order": cancel_order } messages = [ {"role": "system", "content": "You are a shopping assistant. Help users check order status and cancel orders."}, {"role": "user", "content": "Where is my order ORD-123456?"} ] # Turn 1 - creates cache response = client.chat.completions.create(model="gpt-oss-120b", messages=messages, tools=tools) print("Turn 1 usage:", response.usage) msg = response.choices[0].message messages.append(msg.model_dump()) if msg.tool_calls: for call in msg.tool_calls: func = available_functions[call.function.name] result = func(**json.loads(call.function.arguments)) messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)}) response = client.chat.completions.create(model="gpt-oss-120b", messages=messages, tools=tools) messages.append(response.choices[0].message.model_dump()) print("Turn 1 response:", response.choices[0].message.content) # Turn 2 - uses cache messages.append({"role": "user", "content": "Please cancel it, I ordered by mistake."}) response = client.chat.completions.create(model="gpt-oss-120b", messages=messages, tools=tools) print("\nTurn 2 usage:", response.usage) msg = response.choices[0].message messages.append(msg.model_dump()) if msg.tool_calls: for call in msg.tool_calls: func = available_functions[call.function.name] result = func(**json.loads(call.function.arguments)) messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)}) response = client.chat.completions.create(model="gpt-oss-120b", messages=messages, tools=tools) print("Turn 2 response:", response.choices[0].message.content) ``` ```javascript Node.js expandable theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env.CEREBRAS_API_KEY }); // Mock order database const ORDERS = { 'ORD-123456': { status: 'processing', eta_days: 5 } }; const getOrderStatus = (order_id) => { const order = ORDERS[order_id]; if (!order) return { error: 'Order not found' }; return { order_id, status: order.status, eta_days: order.eta_days }; }; const cancelOrder = (order_id) => { const order = ORDERS[order_id]; if (!order) return { error: 'Order not found' }; if (['shipped', 'delivered'].includes(order.status)) { return { error: `Cannot cancel - order already ${order.status}` }; } order.status = 'cancelled'; delete order.eta_days; return { order_id, status: 'cancelled' }; }; const tools = [ { type: 'function', function: { name: 'get_order_status', description: 'Look up an order status by order ID', parameters: { type: 'object', properties: { order_id: { type: 'string', description: 'Order ID (e.g., ORD-123456)' } }, required: ['order_id'] }, strict: true } }, { type: 'function', function: { name: 'cancel_order', description: 'Cancel an order by order ID', parameters: { type: 'object', properties: { order_id: { type: 'string', description: 'Order ID (e.g., ORD-123456)' } }, required: ['order_id'] }, strict: true } } ]; // Helper function to handle tool calls in a more reusable way async function handleToolCalls(toolCalls, messages) { for (const call of toolCalls) { const args = JSON.parse(call.function.arguments); const orderId = args.order_id; let result; if (call.function.name === 'get_order_status') { result = getOrderStatus(orderId); } else if (call.function.name === 'cancel_order') { result = cancelOrder(orderId); } messages.push({ role: 'tool', tool_call_id: call.id, content: JSON.stringify(result) }); } } async function run() { const messages = [ { role: 'system', content: 'You are a shopping assistant. Help users check order status and cancel orders.' }, { role: 'user', content: 'Where is my order ORD-123456?' } ]; // Turn 1 - creates cache let response = await client.chat.completions.create({ model: 'gpt-oss-120b', messages, tools }); console.log('Turn 1 usage:', response.usage); let msg = response.choices[0].message; messages.push(msg); if (msg.tool_calls) { await handleToolCalls(msg.tool_calls, messages); response = await client.chat.completions.create({ model: 'gpt-oss-120b', messages, tools }); messages.push(response.choices[0].message); console.log('Turn 1 response:', response.choices[0].message.content); } // Turn 2 - uses cache messages.push({ role: 'user', content: 'Please cancel it, I ordered by mistake.' }); response = await client.chat.completions.create({ model: 'gpt-oss-120b', messages, tools }); console.log('Turn 2 usage:', response.usage); msg = response.choices[0].message; messages.push(msg); if (msg.tool_calls) { await handleToolCalls(msg.tool_calls, messages); response = await client.chat.completions.create({ model: 'gpt-oss-120b', messages, tools }); console.log('Turn 2 response:', response.choices[0].message.content); } } run().catch(console.error); ``` ```bash cURL expandable theme={null} #!/bin/bash API_KEY="${CEREBRAS_API_KEY}" BASE_URL="https://api.cerebras.ai/v1" if [ -z "$API_KEY" ]; then echo "Error: CEREBRAS_API_KEY environment variable is not set" exit 1 fi TOOLS='[ { "type": "function", "function": { "name": "get_order_status", "description": "Look up an order status by order ID", "parameters": { "type": "object", "properties": { "order_id": {"type": "string", "description": "Order ID (e.g., ORD-123456)"} }, "required": ["order_id"] }, "strict": true } }, { "type": "function", "function": { "name": "cancel_order", "description": "Cancel an order by order ID", "parameters": { "type": "object", "properties": { "order_id": {"type": "string", "description": "Order ID (e.g., ORD-123456)"} }, "required": ["order_id"] }, "strict": true } } ]' SYSTEM="You are a shopping assistant. Help users check order status and cancel orders." echo "=== Turn 1 (Creates Cache) ===" curl -s -X POST "$BASE_URL/chat/completions" \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d "$(jq -n --arg sys "$SYSTEM" --argjson tools "$TOOLS" '{ model: "gpt-oss-120b", messages: [{role: "system", content: $sys}, {role: "user", content: "Where is my order ORD-123456?"}], tools: $tools }')" | jq '{content: .choices[0].message.content, usage: .usage}' echo "" echo "=== Turn 2 (Uses Cache) ===" curl -s -X POST "$BASE_URL/chat/completions" \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d "$(jq -n --arg sys "$SYSTEM" --argjson tools "$TOOLS" '{ model: "gpt-oss-120b", messages: [ {role: "system", content: $sys}, {role: "user", content: "Where is my order ORD-123456?"}, {role: "assistant", content: "Your order ORD-123456 is currently processing with an estimated delivery in 5 days."}, {role: "user", content: "Please cancel it, I ordered by mistake."} ], tools: $tools }')" | jq '{content: .choices[0].message.content, usage: .usage}' ``` During each turn, the system automatically caches the longest matching prefix from previous requests. In this example: * **System message**: The shopping assistant instructions remain identical across all turns * **Tool definitions**: Both order management tool schemas (including parameters and descriptions) stay constant * **Conversation history**: Previous user messages, assistant responses, and tool results are all cached as the conversation grows Only the new content at the end of each request requires fresh processing: * New user messages (the latest question) * New tool execution results * The model's reasoning and decision-making for the current turn As the conversation grows, the cache hit rate increases dramatically. The static prefix (system + tools) remains cached, and the expanding conversation history also gets cached, meaning only the newest user message and the model's fresh response require full processing. ## Structuring Prompts for Caching To maximize cache hits and minimize latency, organize your prompts with static content first and dynamic content last. The system caches prompts from the **beginning** of the message. If you place dynamic content (like a timestamp or a unique User ID) at the start of the prompt, the prefix will differ for every request and the cache will never be triggered. Place content that remains the same across multiple requests at the beginning: * System instructions ("You are a helpful assistant...") * Tool definitions and schemas * Few-shot examples * Large context documents (e.g., a legal agreement or code base) Place content that changes with each request at the end: * User-specific questions * Session variables * Timestamps The "You are a coding assistant..." instruction block remains static and can be cached in subsequent requests. Only the short timestamp and user query are processed fresh. ```json theme={null} [ { "role": "system", "content": "You are a coding assistant... Current Time: 12:01 PM" }, { "role": "user", "content": "Debug this code." } ] ``` **Result:** The static portion of the system prompt is cached. Subsequent requests reuse the cache and only process the timestamp and user query. In this example, the time is included at the start of the system instructions. Because the time changes every minute, the prefix never matches. Subsequent requests will always be fully processed. ```json theme={null} [ { "role": "system", "content": "Current Time: 12:01 PM. You are a coding assistant..." }, { "role": "user", "content": "Debug this code." } ] ``` **Result:** Cache miss on every request because the timestamp changes the prefix. ## Maximize Cache Hits with `prompt_cache_key` `prompt_cache_key` must be enabled on your account before you can use it. [Contact us](https://www.cerebras.ai/contact) or reach out to your account representative to request access. For the best cache hit rate within a single conversation or workflow, include a `prompt_cache_key` on each request that shares the same prompt prefix. This optional opaque string tells the system which requests are likely to share a common prompt prefix, so it can route those requests to the same prompt cache. Prompt caching and prefix matching work automatically on every request, so you benefit from caching even without setting `prompt_cache_key`. Use it when you want to give the system an explicit routing hint. This keeps requests from the same conversation or workflow on the same prompt cache. Under load, turn 1 of a session can be routed to one prompt cache and turn 2 to another, causing a cache miss even though the prefixes match. Don't set `prompt_cache_key` for prefixes that are shared across many users, such as a common system prompt or shared RAG context. This would funnel all of those requests to a single backend, creating a bottleneck that limits your deployment's throughput. These shared prefixes are already handled by automatic prefix caching. ### When to Set a Cache Key Choose a stable identifier for one conversation or workflow whose requests actually share a common prompt prefix: * **Multi-turn chat session:** Use the conversation ID so every turn in the same conversation uses the same prompt cache. * **Agentic or RAG workflow:** Use one workflow ID only for a small set of steps that reuse the same prompt prefix. Reuse the same key across all requests in a conversation or workflow — a one-off unique value provides no improvement over omitting the field. When a new conversation or workflow begins, start a new key. If later requests no longer share the same prompt prefix, reusing the old key only adds routing noise and does not create a cache hit. ```python Python theme={null} from cerebras.cloud.sdk import Cerebras client = Cerebras() response = client.chat.completions.create( model="gpt-oss-120b", messages=[ {"role": "system", "content": "You are a helpful coding assistant."}, {"role": "user", "content": "Explain recursion in Python."} ], prompt_cache_key="conversation-abc-123", ) ``` ```javascript Node.js theme={null} import Cerebras from "@cerebras/cerebras_cloud_sdk"; const client = new Cerebras(); const response = await client.chat.completions.create({ model: "gpt-oss-120b", messages: [ { role: "system", content: "You are a helpful coding assistant." }, { role: "user", content: "Explain recursion in Python." }, ], prompt_cache_key: "conversation-abc-123", }); ``` ```bash cURL theme={null} curl -s -X POST https://api.cerebras.ai/v1/chat/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-oss-120b", "messages": [ {"role": "system", "content": "You are a helpful coding assistant."}, {"role": "user", "content": "Explain recursion in Python."} ], "prompt_cache_key": "conversation-abc-123" }' ``` `prompt_cache_key` values have a maximum length of 1024 characters. Longer values are rejected with a 400 error. prompt\_cache\_key is optional and does not affect billing — cached tokens are priced the same whether or not the key is set. The key value is hashed before it's written to internal logs or metrics. ## Track Cache Usage Verify if your requests are hitting the cache by viewing the `cached_tokens` field within the [`usage.prompt_token_details`](/api-reference/chat-completions#param-prompt-tokens-details) response object. This indicates the number of prompt tokens that were found in the cache. ```json theme={null} "usage": { "prompt_tokens": 3000, "completion_tokens": 150, "total_tokens": 3150, "prompt_tokens_details": { "cached_tokens": 2800 } } ``` In this example, the cache served 2,800 of the 3,000 prompt tokens, resulting in faster processing. Additionally, log in to [cloud.cerebras.ai](https://cloud.cerebras.ai) and click **Analytics** to track your cache usage. ## FAQs Yes. All cached tokens contribute to your standard Tokens Per Minute (TPM) rate limits. **Calculation:** `cached_tokens + input_tokens` (fresh) = Total TPM usage for that request. There is no additional fee for using prompt caching. Input tokens, whether served from the cache or processed fresh, are billed at the standard input token rate for the respective model. There are three common reasons for a cache miss on identical requests: 1. **Block Size:** We cache in 128-token blocks. If a request or prefix is shorter than 128 tokens, it may not be cached. 2. **Data Center Routing:** While we make a best effort to route you to the same data center, traffic profiles may occasionally route you to a different location where your cache does not exist. 3. **TTL Expiration:** If requests are sent more than 5 minutes apart, the cache may have been evicted. Yes, prompt caching is automatically enabled for all users for the supported models. Prompt caching is enabled by default for the following models: * [`zai-glm-4.7`](/models/zai-glm-47) * [`gpt-oss-120b`](/models/openai-oss) Yes, it is fully ZDR-compliant. All cached context remains ephemeral in memory and never persisted. Cached tokens are stored in key-value stores colocated in the same data center as the model instance serving your traffic. Prompt caches are never shared between organizations. Only members of your organization can benefit from caches created by identical prompts within your team. No. Prompt caching works automatically. `prompt_cache_key` is an optional routing hint for requests in the same conversation or workflow that share a common prompt prefix. Do not set it just to share one large system prompt or RAG context across many users. See [Maximize Cache Hits with `prompt_cache_key`](#maximize-cache-hits-with-prompt_cache_key). Caching only affects the input processing phase (how we read your prompt). The output generation phase remains exactly the same speed and quality. You will receive the same quality response, just with faster prompt processing. No manual cache management is required or available. The system automatically manages cache eviction based on the TTL (5 minutes to 1 hour). Guaranteed TTL is 5 minutes, but up to 1 hour max depending on system load. Check the `usage.prompt_tokens_details.cached_tokens` field in your API response. When it's greater than 0, caching was used for that request. Additionally, log in to [cloud.cerebras.ai](https://cloud.cerebras.ai) and click **Analytics** to track your cache usage. # Reasoning Source: https://inference-docs.cerebras.ai/capabilities/reasoning Reasoning models generate intermediate thinking tokens before their final response, enabling better problem-solving and allowing you to inspect the model's thought process. Reasoning capabilities are currently available for the [OpenAI GPT OSS](/models/openai-oss) (`gpt-oss-120b`), [Z.ai GLM](/models/zai-glm-47) (`zai-glm-4.7`), and [Gemma 4](/models/gemma-4-31b) (`gemma-4-31b`) models. Each model family has slight variations in the parameters used to control reasoning. ## Reasoning Format Control how reasoning text appears in responses using the `reasoning_format` parameter. Format support varies by model. Gemma 4 does not support `raw` or `hidden` reasoning formats today. ### Available Formats | Format | Description | | -------- | ------------------------------------------------------------------------------------------------------------- | | `parsed` | Reasoning returned in separate `reasoning` field; logprobs separated into `reasoning_logprobs` | | `raw` | Reasoning prepended to content; GLM uses `...` tokens, GPT-OSS concatenates without separators | | `hidden` | Reasoning text and logprobs dropped completely (tokens still counted) | | `none` | Uses model's default behavior | ### Default Behavior by Model When `reasoning_format` is set to `none` or omitted, each model uses its default format: | Model | Default Reasoning Format | | -------------- | ------------------------ | | Z.ai GLM 4.7 | `text_parsed` | | OpenAI GPT OSS | `text_parsed` | | Gemma 4 31B | Disabled by default | ### `parsed` Format Reasoning text is returned in a separate `reasoning` field without start/end tokens. When logprobs are enabled, reasoning logprobs are returned in a separate `reasoning_logprobs` field. ```python Request theme={null} from cerebras.cloud.sdk import Cerebras client = Cerebras() response = client.chat.completions.create( model="zai-glm-4.7", messages=[ { "role": "user", "content": "Can you help me with this?" } ], logprobs=True, reasoning_format="parsed" ) print(response) ``` ```json Non-streaming Response theme={null} { "choices": [ { "index": 0, "message": { "role": "assistant", "content": "I can help you with that!", "reasoning": "Let me think..." }, "logprobs": { "content": [ {"token": "I", "logprob": -0.1}, {"token": " can", "logprob": -0.2}, ... ] }, "reasoning_logprobs": { "content": [ {"token": "Let ", "logprob": -0.3}, {"token": "me", "logprob": -0.4}, ... ] }, "finish_reason": "stop" } ] } ``` ```json Streaming Response theme={null} { "choices": [ { "delta": { "reasoning": " should" }, "index": 0 } ] } ``` When streaming, reasoning tokens are delivered in the `reasoning` field of the delta. ### `raw` Format Reasoning text is included in the `content` field, prepended to the response. For GLM, reasoning is wrapped in `...` tokens. All logprobs are returned together in the standard `logprobs` field. Since GPT-OSS does not use thinking tokens, reasoning and content are concatenated without separators when using `raw` format. The `raw` format is not compatible with `json_object` or `json_schema` response formats. Models that default to `raw` will automatically use `hidden` instead when structured output is requested. ```python Request theme={null} from cerebras.cloud.sdk import Cerebras client = Cerebras() response = client.chat.completions.create( model="zai-glm-4.7", messages=[ { "role": "user", "content": "Can you help me with this?" } ], logprobs=True, reasoning_format="raw" ) print(response) ``` ```json Response theme={null} { "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Let me think...I can help you with that!" }, "logprobs": { "content": [ {"token": "Let ", "logprob": -0.3}, {"token": "me", "logprob": -0.4}, {"token": "I", "logprob": -0.1}, {"token": " can", "logprob": -0.2}, ... ] }, "finish_reason": "stop" } ] } ``` ### `hidden` Format Reasoning text and reasoning logprobs are dropped completely from the response. The reasoning tokens are still generated and counted toward total completion tokens. ```python Request theme={null} from cerebras.cloud.sdk import Cerebras client = Cerebras() response = client.chat.completions.create( model="zai-glm-4.7", messages=[ { "role": "user", "content": "Can you help me with this?" } ], logprobs=True, reasoning_format="hidden" ) print(response) ``` ```json Response theme={null} { "choices": [ { "index": 0, "message": { "role": "assistant", "content": "I can help you with that!" }, "logprobs": { "content": [ {"token": "I", "logprob": -0.1}, {"token": " can", "logprob": -0.2}, ... ] }, "finish_reason": "stop" } ] } ``` *** ## Model-Specific Parameters Each model family has its own parameter for controlling reasoning behavior. There are key differences between the OpenAI client and the Cerebras SDK when using non-standard OpenAI parameters. These examples use the Cerebras SDK. For more info, see [Passing Non-Standard Parameters](/resources/openai#passing-non-standard-parameters). ### GPT-OSS-120B: `reasoning_effort` Use `reasoning_effort` to control how much reasoning the model performs: * `"low"` - Minimal reasoning, faster responses * `"medium"` - Moderate reasoning (default) * `"high"` - Extensive reasoning, more thorough analysis ```python Python theme={null} response = client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": "Explain quantum entanglement."}], reasoning_effort="medium" ) ``` ```javascript Node.js theme={null} const response = await client.chat.completions.create({ model: "gpt-oss-120b", messages: [{ role: "user", content: "Explain quantum entanglement." }], reasoning_effort: "medium" }); ``` ### Z.ai GLM 4.7: `reasoning_effort` and `disable_reasoning` Reasoning is enabled by default on `zai-glm-4.7`. Use `reasoning_effort="none"` to disable it: ```python Python theme={null} response = client.chat.completions.create( model="zai-glm-4.7", messages=[{"role": "user", "content": "Explain how photosynthesis works."}], reasoning_effort="none" # Disables reasoning ) ``` ```javascript Node.js theme={null} const response = await client.chat.completions.create({ model: "zai-glm-4.7", messages: [{ role: "user", content: "Explain how photosynthesis works." }], reasoning_effort: "none" // Disables reasoning }); ``` Alternatively, use `disable_reasoning` to toggle reasoning on or off. Set to `true` to disable reasoning, or `false` (default) to enable it. ```python Python theme={null} response = client.chat.completions.create( model="zai-glm-4.7", messages=[{"role": "user", "content": "Explain how photosynthesis works."}], disable_reasoning=False # Set to True to disable reasoning ) ``` ```javascript Node.js theme={null} const response = await client.chat.completions.create({ model: "zai-glm-4.7", messages: [{ role: "user", content: "Explain how photosynthesis works." }], disable_reasoning: false // Set to true to disable reasoning }); ``` `disable_reasoning` is deprecated and will be removed after **July 21, 2026**. Use `reasoning_effort="none"` instead. See the [deprecation notice](/support/deprecation) for details. ### Gemma 4 31B: `reasoning_effort` Reasoning is disabled by default on `gemma-4-31b`. Use `reasoning_effort` to control it: * `"none"` — Reasoning disabled (default) * `"low"`/`"medium"`/`"high"` — Reasoning enabled All three active values are currently equivalent. They enable reasoning without graduated effort control and exist to preserve compatibility across model families. `raw`, `hidden`, `clear_thinking`, and `preserve_thinking` reasoning formats are not supported. ```python Python theme={null} response = client.chat.completions.create( model="gemma-4-31b", messages=[{"role": "user", "content": "Solve this step by step."}], reasoning_effort="medium" ) ``` ```javascript Node.js theme={null} const response = await client.chat.completions.create({ model: "gemma-4-31b", messages: [{ role: "user", content: "Solve this step by step." }], reasoning_effort: "medium" }); ``` ## Reasoning Context Retention Reasoning tokens are not automatically retained across requests. To maintain awareness of prior reasoning in multi-turn conversations, include the reasoning text in the `content` field of the `assistant` message. Use the same format the model outputs: for GLM, include reasoning in `...` tags; for GPT-OSS, prepend reasoning text directly before the answer. ```python Python theme={null} # GPT-OSS: reasoning prepended directly before the answer response = client.chat.completions.create( model="gpt-oss-120b", messages=[ {"role": "user", "content": "What is 25 * 4?"}, {"role": "assistant", "content": "I need to multiply 25 by 4. 25 * 4 = 100. The answer is 100."}, {"role": "user", "content": "Now divide that by 2."} ] ) ``` ```javascript Node.js theme={null} // GPT-OSS: reasoning prepended directly before the answer const response = await client.chat.completions.create({ model: "gpt-oss-120b", messages: [ { role: "user", content: "What is 25 * 4?" }, { role: "assistant", content: "I need to multiply 25 by 4. 25 * 4 = 100. The answer is 100." }, { role: "user", content: "Now divide that by 2." } ] }); ``` ```python Python theme={null} # GLM: reasoning wrapped in tags response = client.chat.completions.create( model="zai-glm-4.7", messages=[ {"role": "user", "content": "What is 25 * 4?"}, {"role": "assistant", "content": "I need to multiply 25 by 4. 25 * 4 = 100.The answer is 100."}, {"role": "user", "content": "Now divide that by 2."} ] ) ``` ```javascript Node.js theme={null} // GLM: reasoning wrapped in tags const response = await client.chat.completions.create({ model: "zai-glm-4.7", messages: [ { role: "user", content: "What is 25 * 4?" }, { role: "assistant", content: "I need to multiply 25 by 4. 25 * 4 = 100.The answer is 100." }, { role: "user", content: "Now divide that by 2." } ] }); ``` # Service Tiers Source: https://inference-docs.cerebras.ai/capabilities/service-tiers Control request prioritization with service tiers. This feature is in [Private Preview](/support/preview-releases). For access or more information, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. Prioritize requests on the Cerebras Inference API to balance latency sensitivity and resource allocation across your workloads. ## Service Tiers Service tiers determine the processing priority of your requests. You can specify a tier using the [`service_tier`](/api-reference/chat-completions#param-service-tier) parameter in your API requests. | Tier | Description | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `priority` | Highest priority - requests are processed first. Use for time-critical, user-facing requests that require immediate processing. Only available for dedicated endpoints, not shared endpoints. | | `default` | Standard priority processing. Use for standard production workloads with normal latency requirements. | | `auto` | Automatically uses the highest available service tier. Use when you want to maximize requests served while allowing flexibility in processing priority. | | `flex` | Lowest priority - requests are processed towards the end. Use for overflow requests that cannot fit in higher service tier rate limits or for experiments. | When no `service_tier` is specified, requests default to the `default` tier. ## Usage Add the `service_tier` parameter to your chat completions request to specify the priority level. ```python Python highlight={13} theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) response = client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "system", "content": "Best pastries in San Francisco?"}], stream=True, max_tokens=20000, temperature=0.7, top_p=0.8, service_tier="auto" ) ``` ```javascript Node.js highlight={14} theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); const response = await client.chat.completions.create({ model: "gpt-oss-120b", messages: [{ role: "system", content: "Best pastries in San Francisco?" }], stream: true, max_tokens: 20000, temperature: 0.7, top_p: 0.8, service_tier: "auto" }); ``` ```bash cURL highlight={10} theme={null} curl --location 'https://api.cerebras.ai/v1/chat/completions' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer ${CEREBRAS_API_KEY}" \ --data '{ "model": "llama-3.3-70b", "stream": true, "max_tokens": 20000, "temperature": 0.7, "top_p": 0.8, "service_tier": "auto", "messages": [ { "role": "system", "content": "Best pastries in San Francisco?" } ] }' ``` When using `auto`, the response will include a `service_tier_used` field that indicates the effective service tier used for processing. ## Queue Threshold Control Only applies to requests using the `flex` or `auto` service tiers. The [`queue_threshold`](/api-reference/chat-completions#param-queue-threshold) header allows you to set a maximum acceptable queue time for flex tier requests. If the expected queue time exceeds your threshold, the request is preemptively rejected rather than waiting in the queue. **Valid range:** 50-20000 milliseconds ```python Python highlight={10} theme={null} from cerebras.cloud.sdk import Cerebras import os client = Cerebras(api_key=os.environ.get("CEREBRAS_API_KEY")) response = client.chat.completions.create( model="gpt-oss-120b", service_tier="flex", messages=[{"role": "user", "content": "What are the latest AI trends?"}], extra_headers={"queue_threshold": "100"} ) ``` ```javascript Node.js highlight={12} theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], }); const response = await client.chat.completions.create({ model: "gpt-oss-120b", service_tier: "flex", messages: [{ role: "user", content: "What are the latest AI trends?" }] }, { headers: { "queue_threshold": "100" } }); ``` ```bash cURL highlight={4} theme={null} curl https://api.cerebras.ai/v1/chat/completions \ -H "Authorization: Bearer $CEREBRAS_API_KEY" \ -H "Content-Type: application/json" \ -H "queue_threshold: 100" \ -d '{ "model": "llama-3.3-70b", "service_tier": "flex", "messages": [ { "role": "user", "content": "What are the latest AI trends?" } ] }' ``` If no threshold is specified, a system default is used. ## FAQ `priority` and `default` rate limits are the same, while `flex` rate limits are tracked independently and are several multiples of `default` rate limits. Yes. Log in to [cloud.cerebras.ai](https://cloud.cerebras.ai) and click **Analytics**. Graphs in the analytics tab display usage across different service tiers, allowing you to monitor consumption by priority level. No, during the preview launch all service tiers are billed equally. No, only requests set to `auto` can be processed on a lower service tier. The queue time threshold only applies once a request is being processed on the `flex` service tier. You can set it on requests using `auto` or `flex`, but it will only be evaluated if the request is processed on the flex tier. # Streaming Responses Source: https://inference-docs.cerebras.ai/capabilities/streaming Learn how to enable streaming responses in the Cerebras API. **[Get a free API key](https://cloud.cerebras.ai?utm_source=3pi_streaming\&utm_campaign=capabilities) to get started.** The Cerebras API supports streaming responses, which send messages back in chunks and display them incrementally as the model generates them. To enable this feature, set the `stream` parameter to `True` within the `chat.completions.create` method. The API then returns an iterable containing the chunks of the message. Similarly, the same can be done in TypeScript by setting the `stream` property to `true` within the `chat.completions.create` method. Begin by importing the Cerebras SDK and setting up the client. ```python Python theme={null} import os from cerebras.cloud.sdk import Cerebras client = Cerebras( # This is the default and can be omitted api_key=os.environ.get("CEREBRAS_API_KEY"), ) ``` ```javascript Node.js theme={null} import Cerebras from 'cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], // This is the default and can be omitted }); ``` Set the `stream` parameter to `True` within the `chat.completions.create` method to enable streaming responses. ```python Python theme={null} stream = client.chat.completions.create( messages=[ { "role": "user", "content": "Why is fast inference important?", } ], model="gpt-oss-120b", stream=True, ) for chunk in stream: print(chunk.choices[0].delta.content or "", end="") ``` ```javascript Node.js theme={null} import Cerebras from 'cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'], // This is the default and can be omitted }); async function main() { const stream = await client.chat.completions.create({ messages: [{ role: 'user', content: 'Why is fast inference important?' }], model: 'gpt-oss-120b', stream: true, }); for await (const chunk of stream) { process.stdout.write(chunk.choices[0]?.delta?.content || ''); } } main(); ``` # Structured Outputs Source: https://inference-docs.cerebras.ai/capabilities/structured-outputs Generate structured data with the Cerebras Inference API [**Get a free API key to get started.**](https://cloud.cerebras.ai?utm_source=inferencedocs) Structured outputs is a feature that can enforce consistent JSON outputs for models in the Cerebras Inference API. This is particularly useful when building applications that need to process AI-generated data programmatically. Some of the key benefits of using structured outputs are: * **Reduced Variability**: Ensures consistent outputs by adhering to predefined fields. * **Type Safety**: Enforces correct data types, preventing mismatches. * **Easier Parsing & Integration**: Enables direct use in applications without extra processing. ## Tutorial: Structured Outputs using Cerebras Inference In this tutorial, we'll explore how to use structured outputs with the Cerebras Cloud SDK. We'll build a simple application that generates movie recommendations and uses structured outputs to ensure the response is in a consistent JSON format. First, ensure that you have completed steps 1 and 2 of our [Quickstart Guide](/quickstart) to set up your API key and install the Cerebras Cloud SDK. Then, initialize the Cerebras client and import the necessary modules we will use in this tutorial. ```python Python theme={null} import os from cerebras.cloud.sdk import Cerebras import json client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY") ) ``` ```javascript Node.js theme={null} import Cerebras from '@cerebras/cerebras_cloud_sdk'; const client = new Cerebras({ apiKey: process.env['CEREBRAS_API_KEY'] }); ``` To ensure structured responses from the model, we'll use a JSON schema to define our output structure. Start by defining your schema, which specifies the fields, their types, and which ones are required. For our example, we'll define a schema for a movie recommendation that includes the title, director, and year: Note: For every `required` array you define in your schema, you must set `additionalProperties` to `false`. ```python Python theme={null} movie_schema = { "type": "object", "properties": { "title": {"type": "string"}, "director": {"type": "string"}, "year": {"type": "integer"}, }, "required": ["title", "director", "year"], "additionalProperties": False } ``` ```javascript Node.js theme={null} const movieSchema = { type: "object", properties: { title: { type: "string" }, director: { type: "string" }, year: { type: "integer" }, }, required: ["title", "director", "year"], additionalProperties: false }; ``` Next, use the schema in your API call by setting the `response_format` parameter to include both the type and your schema. Setting `strict` to `true` will enforce the schema. Setting `strict` to `false` will allow the model to return additional fields that are not specified in the schema, similar to [JSON mode](#json-mode). ```python Python theme={null} completion = client.chat.completions.create( model="gpt-oss-120b", messages=[ {"role": "system", "content": "You are a helpful assistant that generates movie recommendations."}, {"role": "user", "content": "Suggest a sci-fi movie from the 1990s"} ], response_format={ "type": "json_schema", "json_schema": { "name": "movie_schema", "strict": True, "schema": movie_schema } } ) # Parse the JSON response movie_data = json.loads(completion.choices[0].message.content) print(json.dumps(movie_data, indent=2)) ``` ```javascript Node.js theme={null} async function main() { const schemaCompletion = await client.chat.completions.create({ model: 'gpt-oss-120b', messages: [ { role: 'system', content: 'You are a helpful assistant that generates movie recommendations.' }, { role: 'user', content: 'Suggest a sci-fi movie from the 1990s' } ], response_format: { type: 'json_schema', json_schema: { name: 'movie_schema', strict: true, schema: movieSchema } } }); // Parse and display the JSON response const schemaMovieData = JSON.parse(schemaCompletion.choices[0].message.content); console.log('Movie Recommendation:'); console.log(JSON.stringify(schemaMovieData, null, 2)); } ``` Sample output: ``` { "title": "Terminator 2: Judgment Day", "director": "James Cameron", "year": 1991 } ``` Now you have a structured JSON response from the model, which can be used in your application. ## Understanding Strict Mode Strict mode guarantees that the model's output will exactly match the JSON schema you provide. When you set `strict` to `true`, Cerebras employs constrained decoding to ensure schema conformance at the token level, making invalid outputs impossible. ### Why Use Strict Mode Without strict mode, you may encounter: * Malformed JSON that fails to parse * Missing required fields * Incorrect data types (e.g., `"16"` instead of `16`) * Extra fields not defined in your schema With strict mode enabled, you get: * *Guaranteed* valid JSON * Schema compliance: Every field matches your specification * Type safety: Correct data types for all properties * No retries needed: Eliminates error handling for schema violations ### Enable Strict Mode Set `strict` to `true` in your `response_format` configuration: ```python theme={null} response_format={ "type": "json_schema", "json_schema": { "name": "my_schema", "strict": True, # Enable constrained decoding "schema": your_schema } } ``` ### Schema Requirements for Strict Mode When using strict mode, you must set `additionalProperties: false`. This is required for every object in your schema. ### Limitations in Strict Mode When strict mode is enabled, your schema must conform to specific requirements. See the [Supported Schemas](#supported-schemas) section for detailed information on constraints, limits, and unsupported features. ## Schema References and Definitions You can use `$ref` with `$defs` to define reusable schema components within your JSON schema. This is useful for avoiding repetition and creating more maintainable schemas. ```python highlight={5,7-8} theme={null} schema_with_defs = { "type": "object", "properties": { "title": {"type": "string"}, "director": {"$ref": "#/$defs/person"}, "year": {"type": "integer"}, "lead_actor": {"$ref": "#/$defs/person"}, "studio": {"$ref": "#/$defs/studio"} }, "required": ["title", "director", "year"], "additionalProperties": False, "$defs": { "person": { "type": "object", "properties": { "name": {"type": "string"}, "age": {"type": "integer"} }, "required": ["name"], "additionalProperties": False }, "studio": { "type": "object", "properties": { "name": {"type": "string"}, "founded": {"type": "integer"}, "headquarters": {"type": "string"} }, "required": ["name"], "additionalProperties": False } } } ``` ## Supported Schemas Structured Outputs supports a subset of [JSON Schema](https://json-schema.org/docs). This section outlines the supported types, properties, and constraints. ### Supported Types The following types are supported for Structured Outputs: | Type | Description | | --------- | --------------------------------- | | `string` | Text values | | `number` | Numeric values (floating point) | | `integer` | Whole number values | | `boolean` | True/false values | | `object` | Nested objects with properties | | `array` | Lists of items | | `enum` | Constrained set of allowed values | | `null` | Null values | | `anyOf` | Union types | ### Schema Constraints When using strict mode, the following constraints apply: | Constraint | Limit | | ------------------------- | ----------------------------------------------- | | Maximum schema length | 5,000 characters | | Maximum nesting depth | 10 levels | | Maximum object properties | 500 | | Maximum total enum values | 500 across all enum properties | | Single enum string length | 7,500 characters (when > 250 total enum values) | ### Required Schema Structure All schemas must follow these rules: * **Root must be an object**: The top-level schema must have `"type": "object"`. * **No additional properties**: You must set `"additionalProperties": false` for every object in your schema. Schemas that do not follow these rules may be allowed in some cases today. Starting **July 21, 2026**, these requirements will be strictly enforced for all models and non-conforming schemas will return a validation error. To ensure forward compatibility, always follow these rules in your schema definitions. For more information about API versioning and deprecation timelines, see [API Versions](/api-reference/versions). ### Supported Features Your schema can include the following JSON Schema features: * **Nested structures**: Define complex objects with nested properties. * **Required fields**: Specify which fields must be present. * **Enums (value constraints)**: Use the `enum` keyword to whitelist the exact literals a field may take. See `rating` in the example below. * **Schema references**: Use `$ref` with `$defs` to define reusable schema components within your schema. * **Tuple validation**: `items: false` is supported when used with `prefixItems` for tuple-like arrays. * **Number constraints**: Use `minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum`, and `multipleOf` to constrain `number` and `integer` values. ### Unsupported Features The following JSON Schema features are **not** supported in strict mode: | Feature | Notes | | -------------------- | -------------------------------------------------------------------------------- | | Recursive schemas | Self-referencing schemas are not supported | | External `$ref` | References to external URLs are blocked for security | | `$anchor` keyword | Use relative paths within definitions instead | | `items: true` | Not supported for array types | | Informational fields | Additional fields meant as guidelines (not used in validation) are not supported | | String `pattern` | Regular expression constraints on strings are not supported | | String `format` | Format validation (e.g., `email`, `date-time`, `uuid`) is not supported | | Array constraints | `minItems`, `maxItems` are not supported | ### Example: Complex Schema For example, a more complex schema might look like: ```python theme={null} detailed_schema = { "type": "object", "properties": { "title": {"type": "string"}, "director": {"type": "string"}, "year": {"type": "integer"}, "genres": { "type": "array", "items": {"type": "string"} }, "rating": { "type": "string", "enum": ["G", "PG", "PG‑13", "R"] }, "cast": { "type": "array", "items": { "type": "object", "properties": { "name": {"type": "string"}, "role": {"type": "string"} }, "required": ["name"], "additionalProperties": False } } }, "required": ["title", "director", "year", "genres"], "additionalProperties": False } ``` When used with the API, you might get a response like: ```json theme={null} { "title": "Jurassic Park", "director": "Steven Spielberg", "year": 1993, "genres": ["Science Fiction", "Adventure", "Thriller"], "cast": [ {"name": "Sam Neill", "role": "Dr. Alan Grant"}, {"name": "Laura Dern", "role": "Dr. Ellie Sattler"}, {"name": "Jeff Goldblum", "role": "Dr. Ian Malcolm"} ] } ``` ### Key Ordering The keys in the generated JSON output will appear in the same order as they are defined in your schema. ## Working with Pydantic and Zod Besides defining a JSON schema manually, you can use Pydantic (Python) or Zod (JavaScript) to create your schema and convert it to JSON. Pydantic's `model_json_schema` and Zod's `zodToJsonSchema` methods generate the JSON schema, which can then be used in the API call, as demonstrated in the workflow above. ```python Python (Pydantic) theme={null} from pydantic import BaseModel import json # Define your schema using Pydantic class Movie(BaseModel): title: str director: str year: int # Convert the Pydantic model to a JSON schema movie_schema = Movie.model_json_schema() # Print the JSON schema to verify it print(json.dumps(movie_schema, indent=2)) ``` ```javascript Node.js (Zod) theme={null} import { z } from 'zod'; import { zodToJsonSchema } from 'zod-to-json-schema'; // Define your schema using Zod const MovieSchema = z.object({ title: z.string(), director: z.string(), year: z.number().int() }); // Convert the Zod schema to a JSON schema const movieJsonSchema = zodToJsonSchema(MovieSchema, { name: 'movie_schema' }); // Print the JSON schema to verify it console.log(JSON.stringify(movieJsonSchema, null, 2)); ``` ## JSON Mode JSON mode is an alternative to structured outputs that generates JSON responses without enforcing a specific schema. The model decides what fields to include based on the context of your prompt. We recommend using structured outputs with `strict` set to `true` instead of JSON mode whenever possible. Structured outputs guarantee schema adherence, while JSON mode only ensures valid JSON without enforcing a specific structure. To use JSON mode, set the `response_format` parameter to `json_object` and include instructions in your message asking the model to respond in JSON format: ```python Python theme={null} completion = client.chat.completions.create( model="gpt-oss-120b", messages=[ {"role": "system", "content": "You are a helpful assistant that generates movie recommendations. Respond with JSON."}, {"role": "user", "content": "Suggest a sci-fi movie from the 1990s"} ], response_format={"type": "json_object"} ) ``` ```javascript Node.js theme={null} async function main() { const jsonModeCompletion = await client.chat.completions.create({ model: 'gpt-oss-120b', messages: [ { role: 'system', content: 'You are a helpful assistant that generates movie recommendations. Respond with JSON.' }, { role: 'user', content: 'Suggest a sci-fi movie from the 1990s' } ], response_format: { type: 'json_object' } }); } ``` ### Structured Outputs vs JSON Mode The table below summarizes the key differences between Structured Outputs and JSON Mode: | Feature | Structured Outputs | JSON Mode | | -------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------ | | Outputs valid JSON | Yes | Yes | | Enforces schema | Yes (when `strict: true`) | No | | Constrained decoding | Yes (when `strict: true`) | No | | Configuration | `response_format: { type: "json_schema", json_schema: { "strict": true, "schema": ... } }` | `response_format: { type: "json_object" }` | `tools` and `response_format` cannot be used in the same request. ## Conclusion Structured outputs with JSON schema enforcement ensures your AI-generated responses follow a consistent, predictable format. This makes it easier to build reliable applications that process AI outputs programmatically without worrying about unexpected data structures or missing fields. Check out some of our other tutorials to learn more about other features of the Cerebras Inference SDK: * [Tool Use](/capabilities/tool-use): extending models' capabilities to access tools to answer questions and perform actions * [Streaming](/capabilities/streaming): a feature for streaming responses from the model * [CePO](/capabilities/cepo): a reasoning framework for improving reasoning model abilities with test-time compute # Tool Calling Source: https://inference-docs.cerebras.ai/capabilities/tool-use Learn how to connect models to external tools with tool calling. Tool calling (also known as tool use or function calling) enables models to interact with external tools, applications, or APIs to perform various actions and access real-time information beyond their initial training data. ## How It Works 1. **Define the tool**: Provide a name, description, and input parameters for each tool you want the model to access. 2. **Send the request**: The prompt is sent along with available tool definitions in your API call. 3. **Model decides**: The model analyzes the prompt and its available tools to decide if a tool can help answer the question. If it decides to use a tool, it responds with a structured output indicating which tool to call and what arguments to use. 4. **Execute the tool**: The client application receives the model's tool call request, executes the specified tool (such as calling an external API), and retrieves the result. 5. **Generate final response**: The result from the tool is sent back to the model, which can then use this new information to generate a final, accurate response to the user. ## Basic Tool Calling To begin, we need to import the necessary libraries and set up our Cerebras client. If you haven't set up your Cerebras API key yet, please visit our [QuickStart guide](/quickstart) for detailed instructions on how to obtain and configure your API key. ```python theme={null} import os import json import re from cerebras.cloud.sdk import Cerebras # Initialize Cerebras client client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) ``` Our first step is to define the tool that our AI will use. In this example, we're creating a simple calculator function that can perform basic arithmetic operations. ```python theme={null} def calculate(expression): expression = re.sub(r'[^0-9+\-*/().]', '', expression) try: result = eval(expression) return str(result) except (SyntaxError, ZeroDivisionError, NameError, TypeError, OverflowError): return "Error: Invalid expression" ``` Next, we define the tool schema. This schema acts as a blueprint for the AI, describing the tool's functionality, when to use it, and what parameters it expects. It helps the AI understand how to interact with our custom tool effectively. With `strict: true` enabled, tool call arguments are guaranteed to match your schema exactly through constrained decoding. ```python theme={null} tools = [ { "type": "function", "function": { "name": "calculate", "strict": True, "description": "A calculator tool that can perform basic arithmetic operations. Use this when you need to compute mathematical expressions or solve numerical problems.", "parameters": { "type": "object", "properties": { "expression": { "type": "string", "description": "The mathematical expression to evaluate" } }, "required": ["expression"], "additionalProperties": False } } } ] ``` With our tool and its schema defined, we can now set up the conversation for our AI. We will prompt the LLM using natural language to conduct a simple calculation, and make the API call. This call sends our messages and tool schema to the LLM, allowing it to generate a response that may include tool use. ```python theme={null} messages = [ {"role": "system", "content": "You are a helpful assistant with access to a calculator. Use the calculator tool to compute mathematical expressions when needed."}, {"role": "user", "content": "What's the result of 15 multiplied by 7?"}, ] response = client.chat.completions.create( model="gpt-oss-120b", messages=messages, tools=tools, parallel_tool_calls=False, ) ``` Now that we've made the API call, we need to process the response and handle any tool calls the LLM might have made. Note that the LLM determines based on the prompt if it should rely on a tool to respond to the user. Therefore, we need to check for any tool calls and handle them appropriately. In the code below, we first check if there are any tool calls in the model's response. If a tool call is present, we proceed to execute it and ensure that the function is fulfilled correctly. The function call is logged to indicate that the model is requesting a tool call, and the result of the tool call is logged to clarify that this is not the model's final output but rather the result of fulfilling its request. The result is then passed back to the model so it can continue generating a final response. ```python theme={null} choice = response.choices[0].message if choice.tool_calls: function_call = choice.tool_calls[0].function if function_call.name == "calculate": # Logging that the model is executing a function named "calculate". print(f"Model executing function '{function_call.name}' with arguments {function_call.arguments}") # Parse the arguments from JSON format and perform the requested calculation. arguments = json.loads(function_call.arguments) result = calculate(arguments["expression"]) # Note: This is the result of executing the model's request (the tool call), not the model's own output. print(f"Calculation result sent to model: {result}") # Send the result back to the model to fulfill the request. messages.append({ "role": "tool", "content": json.dumps(result), "tool_call_id": choice.tool_calls[0].id }) # Request the final response from the model, now that it has the calculation result. final_response = client.chat.completions.create( model="gpt-oss-120b", messages=messages, ) # Handle and display the model's final response. if final_response: print("Final model output:", final_response.choices[0].message.content) else: print("No final response received") else: # Handle cases where the model's response does not include expected tool calls. print("Unexpected response from the model") ``` In this case, the LLM determined that a tool call was appropriate to answer the users' question of what the result of 15 multiplied by 7 is. See the output below. ``` Model executing function 'calculate' with arguments {"expression": "15 * 7"} Calculation result sent to model: 105 Final model output: 15 * 7 = 105 ``` ## Strict Mode for Tool Calling Strict mode ensures that the model generates tool call arguments that exactly match your defined schema. This is essential for building reliable agentic workflows where invalid parameters could break your application. ### Why Strict Mode Matters for Tools Without strict mode, tool calls might include: * Wrong parameter types (e.g., `"2"` instead of `2`) * Missing required parameters * Unexpected extra parameters * Malformed argument JSON With strict mode, you get guaranteed schema compliance for every tool call. ### Enabling Strict Mode Set `strict` to `true` inside the `function` object of your tool definition: ```python Python theme={null} tools = [ { "type": "function", "function": { "name": "get_weather", "strict": True, # Enable constrained decoding "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "City and country, e.g., 'San Francisco, USA'" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location", "unit"], "additionalProperties": False } } } ] ``` ### Schema Requirements When using strict mode, you must set `additionalProperties: false`. This is required for every object in your schema. For information about schema limitations that apply when using strict mode, see [Limitations in Strict Mode](/capabilities/structured-outputs#limitations-in-strict-mode). ### Strict Mode with Parallel Tool Calling Strict mode works with parallel tool calling. When multiple tools are called simultaneously, each tool call's arguments will conform to its respective schema: ```python Python theme={null} response = client.chat.completions.create( model="zai-glm-4.7", messages=messages, tools=tools, # Each tool can have strict: true parallel_tool_calls=True, ) ``` ## Multi-turn Tool Calling Most real-world workflows require more than one tool invocation. Multi-turn tool calling lets a model call a tool, incorporate its output, and then, within the same conversation, decide whether it needs to call the tool (or another tool) again to finish the task. It works as follows: 1. After every tool call you append the tool response to `messages`, then ask the model to continue. 2. The model itself decides when enough information has been gathered to produce a final answer. 3. Continue calling `client.chat.completions.create()` until you get a message without `tool_calls`. The example below demonstrates multi-turn tool calling as an extension of the calculator example above. Before continuing, make sure you’ve completed Steps 1–3 from the calculator setup section. ```python theme={null} messages = [ { "role": "system", "content": ( "You are a helpful assistant with a calculator tool. " "Use it whenever math is required." ), }, {"role": "user", "content": "First, multiply 15 by 7. Then take that result, add 20, and divide the total by 2. What's the final number?"}, ] # Register every callable tool once available_functions = { "calculate": calculate, } while True: resp = client.chat.completions.create( model="gpt-oss-120b", messages=messages, tools=tools, ) msg = resp.choices[0].message # If the assistant didn’t ask for a tool, we’re done if not msg.tool_calls: print("Assistant:", msg.content) break # Save the assistant turn exactly as returned messages.append(msg.model_dump()) # Run the requested tool call = msg.tool_calls[0] fname = call.function.name if fname not in available_functions: raise ValueError(f"Unknown tool requested: {fname!r}") args_dict = json.loads(call.function.arguments) # assumes JSON object output = available_functions[fname](**args_dict) # Feed the tool result back messages.append({ "role": "tool", "tool_call_id": call.id, "content": json.dumps(output), }) ``` ## Parallel Tool Calling Parallel tool calling allows models to call multiple tools simultaneously for reduced latency and faster responses. For example, if a user asks "Is Toronto warmer than Montreal?", the model needs to check the weather in both cities. Rather than making two separate requests, parallel tool calling enables the model to request both operations at once, reducing latency and improving efficiency. Parallel tool calling is most beneficial when: * A single query requires multiple independent data points (e.g., comparing weather in different cities) * Multiple tools need to be invoked that don't have dependencies on each other * You want to reduce the number of API calls and overall response time ### Enable Parallel Tool Calling You can explicitly control this behavior using the `parallel_tool_calls` parameter: ```python highlight={5} theme={null} response = client.chat.completions.create( model="zai-glm-4.7", messages=messages, tools=tools, parallel_tool_calls=True, # Enable parallel calling (default) ) ``` To disable parallel tool calling and force sequential execution: ```python highlight={5} theme={null} response = client.chat.completions.create( model="zai-glm-4.7", messages=messages, tools=tools, parallel_tool_calls=False, # Disable parallel calling ) ``` ### Example: Weather Comparison Let's walk through a complete example that demonstrates parallel tool calling by comparing weather in two cities. First, we'll create a simple weather function and define the tool in our schema: ```python theme={null} import os import json from cerebras.cloud.sdk import Cerebras client = Cerebras( api_key=os.environ.get("CEREBRAS_API_KEY"), ) def get_weather(location): """ Dummy function that returns mock weather data. In production, this would call a real weather API. """ weather_data = { "location": location, "temperature": 22, "condition": "sunny", "humidity": 45, } return json.dumps(weather_data) tools = [ { "type": "function", "function": { "name": "get_weather", "strict": True, "description": "Get temperature for a given location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "City and country e.g. Toronto, Canada" } }, "required": ["location"], "additionalProperties": False } } } ] ``` Now we'll send a query that requires checking weather in two different cities: ```python theme={null} messages = [ { "role": "system", "content": "You are a helpful Cerebras Assistant." }, { "role": "user", "content": "Is Toronto warmer than Montreal?" } ] response = client.chat.completions.create( model="zai-glm-4.7", messages=messages, tools=tools, parallel_tool_calls=True, ) ``` When parallel tool calling is enabled, the model's response may contain multiple tool calls in the `tool_calls` array. We need to iterate through all of them: ```python theme={null} choice = response.choices[0].message if choice.tool_calls: # Add the assistant message with tool_calls first messages.append(choice) # Process all tool calls for tool_call in choice.tool_calls: function_call = tool_call.function print(f"Model executing function '{function_call.name}' with arguments {function_call.arguments}") # Parse arguments and execute the function arguments = json.loads(function_call.arguments) result = get_weather(arguments["location"]) print(f"Weather data sent to model: {result}") # Append each tool result to messages messages.append({ "role": "tool", "content": result, "tool_call_id": tool_call.id }) # Get final response after all tool calls are processed final_response = client.chat.completions.create( model="zai-glm-4.7", messages=messages, ) if final_response: print("Final model output:", final_response.choices[0].message.content) else: print("No final response received") else: print("No tool calls in response") ``` # Account & Billing Source: https://inference-docs.cerebras.ai/console/account-billing Manage team members, billing, and organization settings in the Cloud Console. ## Members The Members page is only visible to organization admins. Invite new users and manage access to your organization. When you invite a user, assign one of the following roles: | Role | Description | | ---------------------- | -------------------------------------------------------- | | **Organization Admin** | Can manage all members, settings, and projects | | **Project Admin** | Can manage API keys and members within assigned projects | | **Project Member** | Can view and use API keys within assigned projects | For a full breakdown of what each role can do, see [Roles & Permissions](/console/projects#roles--permissions). ### Invite Users Click **New Invite** on the Members page. Enter one or more email addresses, select a role, and click **Invite Users**. Each person will receive an email with a link to accept the invite. ### Manage Invitations The Invites section lists pending and rejected invitations from the last 30 days. You can **resend** an expired or missed invitation, or **revoke** one before it's accepted. ## Billing ### Check Your Credit Balance Your remaining credits are shown on the **Overview** tab. To add credits, click **Add Credits** or go to the **Pay as you go** tab. Your full credit grant history is also available there. ### Set Up Auto-Recharge Auto-recharge is off by default. Enable it on the **Pay as you go** tab when you add credits or by clicking **Enable**. You can configure the threshold that triggers a recharge and the amount you're topped off to. ### Manage Subscriptions The **Subscriptions** tab lets you manage inference plans on a per-model basis. Each model offers multiple tiers at different monthly rates. If a tier shows **Temporarily Sold Out**, check back later or [contact us](https://www.cerebras.ai/contact) for availability. ### Update Payment Method or Billing Address Use the **Payment** and **Billing address** tabs to manage these. ## Settings The **Settings** page lets you view your **Organization ID**, update your **Organization name**, and see your **Organization type**. To change your organization type, contact support. # API Keys Source: https://inference-docs.cerebras.ai/console/api-keys Authenticate to the Cerebras API with API keys. API keys are static secrets you generate in the Cloud Console and include in every request for authentication. ## Create a Key 1. Open the [Cloud Console](https://cloud.cerebras.ai) and go to **API Keys** in the left nav. 2. On paid accounts, select a project first. The key inherits that project's rate limits and usage tracking. 3. Click **Generate API Key**, give it a descriptive name, and confirm. 4. The key is created and appears in the list. The list shows a truncated preview, but you can copy the full key at any time by clicking the copy icon next to it. ## Use the Key Set the `CEREBRAS_API_KEY` environment variable, or pass the key in the `Authorization: Bearer ` header on direct HTTP requests. See the [Authentication guide](/api-reference/authentication) for SDK initialization and shell setup. ## Rotate or Delete a Key API keys do not expire. Store them in a secrets manager and rotate them periodically. To rotate a key, create a new one, update your application to use it, then delete the old one. To delete a key, click the delete icon next to it on the **API Keys** page. Deletion is immediate and permanent — any application still using the key will receive `401 Unauthorized` errors. # Cloud Console Source: https://inference-docs.cerebras.ai/console/overview Your dashboard for managing Cerebras Inference projects, API keys, usage, billing, and team access.
Cloud Console
Your dashboard for managing Cerebras Inference projects, API keys, usage, billing, and team access.
Open the Console Sign up or log in to get started.
Features
Organize workloads, isolate environments, and control access with project-level rate limits and membership. Interactively test models and prompts without writing any code. Create, rotate, and manage org-level and project-level API keys. View your current rate limits at the org and project level. Track requests, token usage, and trends across models and projects. Monitor HTTP status code trends and inspect individual requests for debugging and auditing. Invite team members and assign org and project roles. Manage credits, payment methods, subscriptions, and invoices. Update org name, view org ID, and other organization-level configuration.
# Playground Source: https://inference-docs.cerebras.ai/console/playground Interactively test models and prompts in the Cloud Console without writing any code. The Playground lets you experiment with Cerebras models directly in the browser. Use it to evaluate models, iterate on prompts, test tool/function calling, and tune parameters, then export a working request to code when you're ready. Your playground and API requests are never used to train models. ## Message Roles | Role | Purpose | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **System** | Sets the behavior and context for the model. Use this to define a persona, provide background information, or constrain responses. | | **User** | Represents input from the human turn. This is the prompt the model responds to. | | **Assistant** | Represents a prior model response. Insert assistant messages to simulate a multi-turn conversation or prime the model toward a particular style or format. | Use the **Add** button to add a new User or Assistant message to the conversation without running inference. Use it to build multi-turn conversations manually. Click **Run** to send the full conversation to the model. After each response, the Playground displays token usage, inference time, speed (tokens per second), and round trip time in the upper-right corner of the response. ## Configuration Select a model from the dropdown at the top right. See the [Models overview](/models/overview) to learn more about the available options. | Parameter | What it controls | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | **Temperature** | Randomness of the output. Lower values produce more focused, predictable responses; higher values produce more varied responses. | | **Max Completion Tokens** | Maximum number of tokens the model will generate in a single response. | | **Top P** | Nucleus sampling threshold. Limits the model to sampling from the top portion of the probability distribution. | | **Format** | Output format. `text` (default), `json_object`, or `json_schema`. | | **Functions** | Define tool/function schemas the model can call. See [Tool Calling](/capabilities/tool-use) for the full reference. | | **Reasoning Effort** | Controls how much effort the model spends reasoning before responding. Only available for select models. | | **Seed** | Set an integer to produce deterministic outputs across requests with the same input. | | **Stream** | Stream tokens as they're generated rather than returning the full response at once. | | **Stop Sequence** | A string that causes the model to stop generating when produced. Useful for structured outputs. | ## View Code Once you have a prompt and parameters you're happy with, click **View Code** to get a ready-to-run code snippet for your application. # Projects Source: https://inference-docs.cerebras.ai/console/projects Organize workloads, isolate environments, control costs, and manage access using Projects in the Cerebras Cloud console. Projects are workspaces within your organization that group API keys, rate limits, and members. Use them to isolate environments, segment usage analytics, and control who can access what. Every organization starts with a **Default Project** that contains all existing API keys, members, and quotas. If you don't create additional projects, nothing about your setup changes. ## Concepts & Hierarchy **Organization** — Your top-level account. Billing, global rate-limit ceilings, and project management live here. **Project** — A workspace within an organization. Every API key belongs to exactly one project, and all keys in a project share that project's rate limits. ## Roles & Permissions In addition to organization roles, each project has its own set of roles. Roles are assigned from the [Members page](/console/account-billing#members). **Limited** - applies only to projects the user is assigned to. | Action | Org Admin | Project Admin | Project Member | | ------------------------- | --------------------- | --------------------- | --------------------- | | Create / archive projects | | | | | Set project rate limits | | | | | View all projects in org | | Limited | Limited | | Create API keys | | Limited | | | Delete API keys | | Limited | | | View API keys | | Limited | Limited | | Use API keys | | Limited | Limited | | View usage & logs | | Limited | Limited | | Manage org members | | | | | Manage project members | | Limited | | **Key behaviors:** * Members are invited at the organization level, then assigned to one or more projects. * Org Admins are automatically added to all projects with write access. * If a user is only assigned to one project and that project is archived, their organization access will be revoked. An Org Admin can restore it by adding them to another project. ## Navigate Projects in the Console Use the **Organization / Project dropdown** in the top-left of the console to switch context: * **Select a specific project** — all pages (Playground, API Keys, Limits, Analytics, Members, Audit Logs, Settings) filter to that project. * **Select "All projects"** — org-level view. Available to Org Admins only. Screenshot of cloud console navigation bar showing the org dropdown and the project dropdown ## Create a Project Only Org Admins can create projects. 1. In the console, open the **Projects** tab or select it from the Projects dropdown. 2. Click **Create project**. 3. Enter a project name. 4. Click **Save**. The project is created under your organization with no custom quotas, so it inherits org-level limits by default. ### Recommended Project Structures Separate projects for each deployment stage: ``` dev, test, prod ``` Separate projects for each surface or tool: ``` frontend, internal-tools ``` Separate projects per team: ``` team-console, team-platform ``` Try to keep the number of projects in your organization to a minimum. Each project adds overhead for monitoring and rate limits management. ## Configure Rate Limits Cerebras uses a two-level quota model. Every request must satisfy both levels to proceed. | Level | Description | | ----------------------- | ---------------------------------------------------------------------- | | **Org-level quota** | Hard limit for the entire organization (e.g., 1M TPM on a given model) | | **Project-level quota** | Per-project limit; cannot exceed the org-level quota | By default, projects have no explicit quotas and inherit org-level limits. For example: | Project | Configured limit | | ------- | ------------------------------------------ | | `prod` | 700,000 TPM | | `test` | 600,000 TPM | | `dev` | None (inherits org limit of 1,000,000 TPM) | If `prod` and `test` together attempt 1,300,000 TPM, some requests are throttled at the organization level — even though each project is individually within its own configured limit. ### Set Project-Level Rate Limits Only Org Admins can set project rate limits. 1. From the **All Projects** view, open the **Projects** tab. 2. Identify the project and click **Project Settings**. 3. For each model, set the desired rate limits. 4. Click **Save**. **We recommend:** * Starting with conservative limits for new projects * Monitoring 429 responses and usage graphs per project * Adjusting as you gain confidence in your workload patterns ## Manage Service Tiers If service tiers are enabled for your organization, use the `service_tier` request parameter to prioritize traffic across projects — `flex` for lower-priority projects and `priority` for production. See the [Service Tier guide](/capabilities/service-tiers) for details. ## Archive Projects Archived projects cannot be restored. Only Org Admins can archive projects. You cannot archive your last remaining project. Archiving invalidates all API keys in the project, preserves usage logs for auditing, and revokes member access to the project. ## FAQs They've been migrated into a Default Project under your organization, with the same quotas and members. Behavior is unchanged unless you configure additional projects or project-level limits. No. Using only the Default Project is fully supported. Create additional projects only if you need isolated rate limits, access control, or separate analytics by use case. Projects can be archived but not permanently deleted. Archiving disables all API keys in the project and preserves logs and usage history. No. Each API key is scoped to exactly one project. To move a workload, create a new key in the target project and update your application to use it. Projects don't create separate billing accounts. All usage accrued within a project is billed through your top-level organization. Projects provide logical segmentation for quotas, keys, and usage, but billing is always aggregated and invoiced at the organization level. You may be hitting the project-level limit for that project, or the org-level ceiling across all projects. Check the **Limits** page for both the project and the org, then review the project's analytics. # Usage & Monitoring Source: https://inference-docs.cerebras.ai/console/usage-monitoring Track API usage, inspect request logs, and view rate limits across your organization and projects. ## Analytics The **Analytics** page has three tabs: **Usage**, **Cached-Usage**, and **Cost**. All dates and timestamps are displayed in UTC. Track request volume and token consumption over a selected date range. Toggle **Show quotas** to overlay your rate limit thresholds and see how close you are to your limits. Use **Download Report** to export the data as a CSV. See how your input tokens break down between cache hits and fresh processing. A high cache hit rate means your prompts share a consistent prefix and your [prompt caching](/capabilities/prompt-caching) setup is working effectively. Use **Download Report** to export the data as a CSV. Review your spending for a selected calendar month, broken down by model and token type (input/output). Cost data may be delayed by up to 10 minutes. Requests made under an active monthly subscription are excluded from usage-based billing. ### Tips **Monitor quota headroom** — Enable **Show quotas** on the Usage tab to see how close you are to your rate limits. If you're consistently near the ceiling, consider distributing traffic across projects or [requesting a limit increase](https://www.cerebras.ai/contact). **Optimize caching** — If cache hits are low on the Cached-Usage tab, review whether your prompts have a stable, shared prefix. Effective caching reduces Time to First Token (TTFT) for long-context workloads. **Track costs by model** — Filter the Cost tab by model to compare spend. This helps when deciding whether a smaller, faster model is sufficient for a given use case. **Debug usage spikes** — Narrow the date range to isolate when a spike started, then cross-reference with Logs to identify the source. ## Logs The **Logs** page has two tabs: **Request Logs** and **Audit Logs**. All dates and timestamps are displayed in UTC. Inspect individual API calls by filtering on model, API key, date range, or HTTP status code. The status code chart at the top gives you a quick visual of error rates over time. Use **Download Report** to export the current view as a CSV. When contacting support about a failed request, include the **Request ID** from the log entry. Audit logs are only visible to admin users. Review actions taken in the console, such as adding members, managing billing, and changing organization settings. Click **View Payload Details** to see the full payload for any entry. ## Limits The **Limits** page displays a table of models available to your organization or project, along with their associated rate limits. The table shows each model's name, context length, limit type (requests or tokens), and quota by minute and day. Hourly limits are shown in the org-level view only. If you need higher limits, [contact us](https://www.cerebras.ai/contact) or reach out to your account representative. The values shown on the Limits page are **specific to your plan and project**. Your limits may differ from examples in the documentation. ### Per-Org vs. Per-Project Limits The limits shown depend on your current console context: | Console context | Limits shown | | ----------------------------- | ------------------------------------- | | **All Projects** view | Org-level limits | | **Specific project** selected | Project-level limits for that project | Cerebras uses a two-level quota model — requests are checked against both the project-level limit and the org-level ceiling. See [Projects](/console/projects) for more on how the two levels interact. # Inference Cookbook Source: https://inference-docs.cerebras.ai/cookbook

Inference Cookbook

Practical guides and examples to help you build with Cerebras

Latest Cookbooks

All Cookbooks

# Academic Research Agent Source: https://inference-docs.cerebras.ai/cookbook/agents/academic-research-agent Generate arXiv search queries, analyze academic papers, download and process PDFs, and synthesize research insights with a conversational AI research assistant powered by PydanticAI + Cerebras + Unstructured. This cookbook demonstrates how to build a conversational agent that: * Generates diverse arXiv search queries * Searches and analyzes academic papers * Downloads and processes PDFs with Unstructured * Performs deep analysis and synthesizes research insights and saves them as reports ## What You'll Learn 1. **PydanticAI Agent Architecture** - Building conversational agents with tools 2. **Cerebras Integration** - Using Cerebras LLMs with PydanticAI 3. **Pydantic Schemas** - Type-safe structured outputs from LLMs 4. **Unstructured** - High-quality PDF text extraction 5. **Tool Design** - Creating effective agent tools with RunContext ## Setup ### Install Dependencies ```python theme={null} %pip install -q pydantic-ai cerebras-cloud-sdk python-dotenv requests feedparser unstructured-client pydantic ``` ### Load API Keys Get API keys to get started with super fast inference, and Unstructured's powerful document procesing: * **Cerebras**: [https://cloud.cerebras.ai](https://cloud.cerebras.ai?utm_source=3pi_academic-research-agent\&utm_campaign=docs) (free tier available) * **Unstructured**: [https://unstructured.io](https://unstructured.io?utm_source=cerebras\&utm_campaign=academic-research-agent) (free tier available) Next, we suggest to add the secrets in the Google Collab Password service, or via a .env file, if you cloned the repository. ```bash theme={null} CEREBRAS_API_KEY=your-key-here UNSTRUCTURED_API_KEY=your-key-here ``` ```python theme={null} import os from dotenv import load_dotenv load_dotenv() required = ["CEREBRAS_API_KEY", "UNSTRUCTURED_API_KEY"] missing = [k for k in required if not os.getenv(k)] if missing: raise RuntimeError( f"Missing API keys: {', '.join(missing)}. Add them to .env file." ) print("✅ API keys loaded") ``` ## Part 1: Pydantic Schemas We are using Pydantic models for type safety. Pydantic is a production grade typing framework, that helps to create reliable LLM responses. These schemas: * Guide the LLM on expected output structure * Validate responses automatically * Provide type hints throughout the codebase ```python theme={null} from typing import List, Optional, Dict, Any from pydantic import BaseModel, Field class ArxivQueries(BaseModel): """Structured output for arXiv query generation""" queries: List[str] = Field(description="List of diverse search queries") reasoning: str = Field(description="Why these queries were chosen") class AbstractAnalysis(BaseModel): """Analysis of paper abstracts""" key_themes: List[str] = Field(description="Main themes across papers") top_papers_for_deep_analysis: List[str] = Field( description="arXiv IDs of most relevant papers" ) reasoning: str = Field(description="Why these papers were selected") class PaperAnalysis(BaseModel): """Deep analysis of a single paper""" arxiv_id: str methods: str = Field(description="Methods and architectures used") contributions: str = Field(description="Novel contributions") limitations: Optional[str] = Field(default=None) class ResearchDirection(BaseModel): """A future research direction""" direction: str = Field(description="The research direction") rationale: str = Field(description="Why this is important") class ResearchOutput(BaseModel): """Final comprehensive research output""" research_landscape_summary: str = Field( description="Overview of the research landscape" ) key_innovations: List[str] = Field( description="Major innovations identified" ) future_research_directions: List[ResearchDirection] = Field( description="Suggested future research directions" ) papers_analyzed: int = Field(description="Total papers analyzed") queries_used: List[str] = Field(description="Search queries used") ``` ### Example: Using Schemas for Type Safety Here's how schemas validate LLM outputs: ```python theme={null} # Example: Creating a validated ArxivQueries object example_queries: ArxivQueries = ArxivQueries( queries=["vision language models", "multimodal reasoning"], reasoning="These queries cover both architecture and capability aspects" ) print(f"Queries: {example_queries.queries}") print(f"Reasoning: {example_queries.reasoning}") # Example: Creating a validated ResearchOutput example_output: ResearchOutput = ResearchOutput( research_landscape_summary="The field is rapidly evolving...", key_innovations=["Cross-modal attention", "Chain-of-thought prompting"], future_research_directions=[ ResearchDirection(direction="Video reasoning", rationale="Temporal understanding is key") ], papers_analyzed=10, queries_used=["vision language models"] ) print(f"\nPapers analyzed: {example_output.papers_analyzed}") print(f"Innovations: {example_output.key_innovations}") ``` ## Part 2: Dependencies & Configuration The agent uses **dependency injection** via PydanticAI's `RunContext`. This allows tools to access shared resources like API clients and caches. ```python theme={null} from cerebras.cloud.sdk import AsyncCerebras from unstructured_client import UnstructuredClient class ResearchDeps(BaseModel): """Dependencies for the research agent""" cerebras_client: Any unstructured_client: Any papers_cache: Dict[str, Dict[str, Any]] = Field(default_factory=dict) start_year: int = 2020 max_papers_per_query: int = 15 max_papers_for_deep_analysis: int = 3 fulltext_excerpt_chars: int = 12000 model_config = {"arbitrary_types_allowed": True} def create_research_deps( start_year: int = 2020, max_papers_for_deep_analysis: int = 3 ) -> ResearchDeps: """Create research dependencies with API clients""" return ResearchDeps( cerebras_client=AsyncCerebras( api_key=os.getenv("CEREBRAS_API_KEY"), default_headers={"X-Cerebras-3rd-Party-Integration": "academic-research-agent"} ), unstructured_client=UnstructuredClient( api_key_auth=os.getenv("UNSTRUCTURED_API_KEY") ), start_year=start_year, max_papers_for_deep_analysis=max_papers_for_deep_analysis ) ``` ## Part 3: Cerebras in Strict Mode **Important**: Cerebras requires all tools to have the same `strict` parameter value. PydanticAI may generate tools with mixed values, which causes errors. We proactively avoid this with a `prepare_tools` hook that normalizes all tools to `strict=False`: ```python theme={null} from dataclasses import replace from pydantic_ai.tools import ToolDefinition async def set_consistent_strict_param( ctx: Any, tool_defs: List[ToolDefinition] ) -> List[ToolDefinition]: """ Enforce consistent strict=False for all tools. This addresses the error: "Tools with mixed values for 'strict' are not allowed" """ return [replace(tool_def, strict=False) for tool_def in tool_defs] ``` ## Part 4: Create the Agent Now we instantiate the PydanticAI agent with: * Cerebras `gpt-oss-120b` model * `ResearchDeps` for dependency injection * `prepare_tools` hook for strict mode * System prompt defining the agent's role ```python theme={null} from pydantic_ai import Agent, RunContext agent = Agent( 'cerebras:gpt-oss-120b', deps_type=ResearchDeps, prepare_tools=set_consistent_strict_param, system_prompt="""You are an expert academic research assistant specializing in literature reviews. You help researchers by: 1. Generating effective arXiv search queries 2. Searching and analyzing academic papers 3. Identifying key themes and innovations 4. Suggesting future research directions You have access to tools for each step of the research process. Use them strategically to conduct comprehensive literature reviews. When asked to research a topic: 1. First generate diverse search queries 2. Search arXiv with those queries 3. Analyze abstracts to identify most relevant papers 4. Download and analyze full papers 5. Synthesize findings into a comprehensive report Be thorough, cite specific papers, and provide actionable insights.""" ) ``` ## Part 5: Define the 7 Research Tools In PydanticAI, each tool is decorated with `@agent.tool` and receives `RunContext[ResearchDeps]` for dependency access. ### Tool 1: Generate arXiv Search Queries ```python theme={null} import json @agent.tool async def generate_arxiv_queries( ctx: RunContext[ResearchDeps], topic: str, num_queries: int = 5 ) -> str: """ Generate diverse arXiv search queries for a research topic. Args: topic: The research topic to generate queries for num_queries: Number of queries to generate (default: 5) Returns: JSON string with queries and reasoning """ print(f"\n🔍 Generating {num_queries} search queries for: {topic}") prompt = f"""Generate {num_queries} diverse arXiv search queries for researching: "{topic}" Make queries: - Specific and targeted - Cover different aspects/angles - Use relevant technical terms - Suitable for arXiv API search Return JSON: {{ "queries": ["query1", "query2", ...], "reasoning": "why these queries cover the topic well" }}""" response = await ctx.deps.cerebras_client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": prompt}], response_format={"type": "json_object"}, temperature=1.0, max_completion_tokens=12000 ) content = response.choices[0].message.content data = json.loads(content) # Validate with Pydantic schema result: ArxivQueries = ArxivQueries(**data) print(f"✓ Generated {len(result.queries)} queries") for i, q in enumerate(result.queries, 1): print(f" {i}. {q}") return json.dumps(data) ``` ### Tool 2: Search arXiv Papers ```python theme={null} import asyncio import requests import feedparser @agent.tool async def search_arxiv_papers( ctx: RunContext[ResearchDeps], queries: List[str] ) -> str: """ Search arXiv with multiple queries and cache results. Args: queries: List of search query strings Returns: Summary of papers found """ print(f"\n📚 Searching arXiv with {len(queries)} queries...") all_papers: Dict[str, Dict[str, Any]] = {} for query in queries: search_url = "http://export.arxiv.org/api/query" params = { "search_query": f"all:{query}", "start": 0, "max_results": ctx.deps.max_papers_per_query, "sortBy": "relevance", "sortOrder": "descending" } try: response = requests.get(search_url, params=params, timeout=30) feed = feedparser.parse(response.content) for entry in feed.entries: # Skip entries without required fields if not hasattr(entry, 'id') or not hasattr(entry, 'published'): continue if not hasattr(entry, 'title') or not hasattr(entry, 'summary'): continue arxiv_id = entry.id.split("/abs/")[-1] if arxiv_id not in all_papers: try: year = int(entry.published[:4]) except (ValueError, TypeError): continue if year >= ctx.deps.start_year: authors = [] if hasattr(entry, 'authors'): authors = [author.name for author in entry.authors if hasattr(author, 'name')] all_papers[arxiv_id] = { "arxiv_id": arxiv_id, "title": entry.title, "authors": authors, "year": year, "abstract": entry.summary, "link": getattr(entry, 'link', f"https://arxiv.org/abs/{arxiv_id}") } except Exception as e: print(f" ⚠️ Query failed: {query[:50]}... ({str(e)[:50]})") continue await asyncio.sleep(1) # Rate limiting # Cache papers in dependencies ctx.deps.papers_cache.update(all_papers) summary = f"Found {len(all_papers)} unique papers from {ctx.deps.start_year} onwards\n\n" summary += "Top papers:\n" for i, (arxiv_id, paper) in enumerate(list(all_papers.items())[:10], 1): summary += f"{i}. [{paper['year']}] {arxiv_id} — {paper['title'][:80]}...\n" print(f"✓ Found {len(all_papers)} papers") return summary ``` ### Tool 3: Analyze Paper Abstracts ```python theme={null} @agent.tool async def analyze_paper_abstracts( ctx: RunContext[ResearchDeps], topic: str, max_papers: int = 20 ) -> str: """ Analyze paper abstracts to identify key themes and select papers for deep analysis. Args: topic: The research topic max_papers: Maximum papers to analyze (default: 20) Returns: JSON string with analysis results """ print(f"\n📊 Analyzing abstracts for: {topic}") papers = list(ctx.deps.papers_cache.values())[:max_papers] if not papers: return json.dumps({ "key_themes": [], "top_papers_for_deep_analysis": [], "reasoning": "No papers in cache. Run search_arxiv_papers first." }) abstracts_text = "\n\n---\n\n".join([ f"Paper {i+1} (arXiv:{p['arxiv_id']})\nTitle: {p['title']}\nAbstract: {p['abstract']}" for i, p in enumerate(papers) ]) prompt = f"""Analyze these {len(papers)} paper abstracts for research on: "{topic}" {abstracts_text} Identify: 1. Key themes across papers 2. Top {ctx.deps.max_papers_for_deep_analysis} most relevant papers for deep analysis (by arXiv ID) 3. Reasoning for selections Return JSON: {{ "key_themes": ["theme1", "theme2", ...], "top_papers_for_deep_analysis": ["arxiv_id1", "arxiv_id2", ...], "reasoning": "explanation" }}""" response = await ctx.deps.cerebras_client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": prompt}], response_format={"type": "json_object"}, temperature=1.0, max_completion_tokens=12000 ) content = response.choices[0].message.content data = json.loads(content) # Validate with Pydantic schema result: AbstractAnalysis = AbstractAnalysis(**data) print(f"✓ Identified {len(result.key_themes)} key themes") print(f"✓ Selected {len(result.top_papers_for_deep_analysis)} papers for deep analysis") return json.dumps(data) ``` ### Tool 4: Download and Process PDF Next, we create a tool that downloads PDFs from arXiv and uses Unstructured's `hi_res` partitioning strategy to detect document layout and extract structured elements like tables, images, and text. You can also swap this out for VLM partitioning, add chunking, enrichment (like table descriptions or NER), and embedding nodes to your workflow. Check out this [notebook](https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_API_On_Demand_Jobs_Walkthrough.ipynb) for a hands-on tutorial. ```python theme={null} from unstructured_client.models.operations import CreateJobRequest, DownloadJobOutputRequest from unstructured_client.models.shared import BodyCreateJob, InputFiles import asyncio import time import json @agent.tool async def download_and_process_pdf( ctx: RunContext[ResearchDeps], arxiv_id: str ) -> str: """ Download and extract text from an arXiv paper PDF using Unstructured. Args: arxiv_id: The arXiv ID (e.g., "2301.12345") Returns: Extracted text excerpt """ print(f"\n📄 Processing PDF: {arxiv_id}") # Check cache first if arxiv_id in ctx.deps.papers_cache and "fulltext" in ctx.deps.papers_cache[arxiv_id]: print(f"✓ Using cached fulltext") return ctx.deps.papers_cache[arxiv_id]["fulltext"] try: # Download PDF from arXiv pdf_url = f"https://arxiv.org/pdf/{arxiv_id}.pdf" response = requests.get(pdf_url, timeout=120) response.raise_for_status() # Define the hi-res partitioning workflow node # Create the job job_response = ctx.deps.unstructured_client.jobs.create_job( request=CreateJobRequest( body_create_job=BodyCreateJob( request_data=json.dumps({ "template_id": "hi_res_partition" # Use a template }), input_files=[ InputFiles( content=response.content, file_name=f"{arxiv_id}.pdf", content_type="application/pdf" ) ] ) ) ) job_id = job_response.job_information.id file_id = job_response.job_information.input_file_ids[0] print(f" Job ID: {job_id}") # Poll job status until complete while True: status_response = ctx.deps.unstructured_client.jobs.get_job( request={"job_id": job_id} ) job = status_response.job_information if job.status == "SCHEDULED": print(" Job is scheduled, polling again in 10 seconds...") await asyncio.sleep(10) elif job.status == "IN_PROGRESS": print(" Job is in progress, polling again in 10 seconds...") await asyncio.sleep(10) elif job.status == "COMPLETED": print(" ✓ Job completed") break elif job.status in ["FAILED", "STOPPED"]: raise Exception(f"Job {job.status.lower()}") else: await asyncio.sleep(10) # Download job output output_response = ctx.deps.unstructured_client.jobs.download_job_output( request=DownloadJobOutputRequest( job_id=job_id, file_id=file_id ) ) # Extract text from elements text_parts: List[str] = [] for element in output_response.any: if isinstance(element, dict): text = element.get("text", "") elif hasattr(element, "text"): text = element.text else: text = "" if text: text_parts.append(text) fulltext = "\n".join(text_parts) # Limit length to stay within context window excerpt = fulltext[:ctx.deps.fulltext_excerpt_chars] # Cache for reuse if arxiv_id in ctx.deps.papers_cache: ctx.deps.papers_cache[arxiv_id]["fulltext"] = excerpt print(f"✓ Extracted {len(fulltext):,} chars (using {len(excerpt):,} char excerpt)") return excerpt except Exception as e: error_msg = f"Failed to process {arxiv_id}: {str(e)}" print(f"⚠️ {error_msg}") return error_msg ``` ### Tool 5: Deep Analyze Papers ```python theme={null} @agent.tool async def deep_analyze_papers( ctx: RunContext[ResearchDeps], topic: str, arxiv_ids: List[str] ) -> str: """ Perform deep analysis of papers using their full text. Args: topic: The research topic arxiv_ids: List of arXiv IDs to analyze Returns: JSON string with deep analysis results """ print(f"\n🔬 Deep analyzing {len(arxiv_ids)} papers...") analyses: List[PaperAnalysis] = [] for arxiv_id in arxiv_ids: # Get fulltext (from cache or download) if arxiv_id in ctx.deps.papers_cache and "fulltext" in ctx.deps.papers_cache[arxiv_id]: fulltext = ctx.deps.papers_cache[arxiv_id]["fulltext"] else: fulltext = await download_and_process_pdf(ctx, arxiv_id) if "Failed to process" in fulltext: continue prompt = f"""Analyze this paper in the context of research on: "{topic}" Paper ID: {arxiv_id} Full text excerpt: {fulltext[:8000]} Extract: 1. Methods and architectures used 2. Novel contributions 3. Limitations (if mentioned) Return JSON: {{ "arxiv_id": "{arxiv_id}", "methods": "description", "contributions": "description", "limitations": "description or null" }}""" response = await ctx.deps.cerebras_client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": prompt}], response_format={"type": "json_object"}, temperature=1.0, max_completion_tokens=12000 ) content = response.choices[0].message.content data = json.loads(content) # Validate with Pydantic schema analysis: PaperAnalysis = PaperAnalysis(**data) analyses.append(analysis) print(f" ✓ Analyzed {arxiv_id}") result = { "papers": [a.model_dump() for a in analyses], "count": len(analyses) } print(f"✓ Completed deep analysis of {len(analyses)} papers") return json.dumps(result) ``` ### Tool 6: Synthesize Research Findings ```python theme={null} @agent.tool async def synthesize_research_findings( ctx: RunContext[ResearchDeps], topic: str, deep_analysis_json: str, queries_used: List[str] ) -> str: """ Synthesize all research findings into a comprehensive report. Args: topic: The research topic deep_analysis_json: JSON string from deep_analyze_papers queries_used: List of search queries that were used Returns: JSON string with final research output """ print(f"\n🎯 Synthesizing research findings...") deep_analysis = json.loads(deep_analysis_json) # Safely get papers list with fallback papers_list = deep_analysis.get('papers', []) papers_count = deep_analysis.get('count', len(papers_list)) if not papers_list: # If no papers key, the JSON might be a single paper analysis or different format # Try to handle it gracefully print("⚠️ No 'papers' key found in deep_analysis_json, attempting to parse as single paper") if 'arxiv_id' in deep_analysis: # It's a single paper analysis papers_list = [deep_analysis] papers_count = 1 else: # Return a minimal synthesis return json.dumps({ "research_landscape_summary": "Unable to synthesize - no paper analysis data available.", "key_innovations": [], "future_research_directions": [], "papers_analyzed": 0, "queries_used": queries_used }) papers_text = "\n\n".join([ f"Paper {i+1} ({p.get('arxiv_id', 'unknown')}):\n" f"Methods: {p.get('methods', 'Not specified')}\n" f"Contributions: {p.get('contributions', 'Not specified')}\n" f"Limitations: {p.get('limitations', 'Not specified')}" for i, p in enumerate(papers_list) ]) prompt = f"""Synthesize research findings on: "{topic}" Deep analysis of {papers_count} papers: {papers_text} Create a comprehensive research summary with: 1. Research landscape overview (2-3 paragraphs) 2. Key innovations (3-5 items) 3. Future research directions (3-5 items with rationale) Return JSON: {{ "research_landscape_summary": "overview text", "key_innovations": ["innovation1", "innovation2", ...], "future_research_directions": [ {{"direction": "direction1", "rationale": "why"}}, ... ] }}""" response = await ctx.deps.cerebras_client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": prompt}], response_format={"type": "json_object"}, temperature=1.0, max_completion_tokens=12000 ) content = response.choices[0].message.content data = json.loads(content) # Add metadata data["papers_analyzed"] = papers_count data["queries_used"] = queries_used # Validate with Pydantic schema result: ResearchOutput = ResearchOutput(**data) print(f"✓ Synthesis complete!") return json.dumps(data) ``` ### Tool 7: Save Research Report ```python theme={null} from datetime import datetime from pathlib import Path @agent.tool def save_research_report( ctx: RunContext[ResearchDeps], topic: str, research_output_json: str ) -> str: """ Save the research report to a file. Args: topic: The research topic research_output_json: JSON string from synthesize_research_findings Returns: Path to saved file """ try: print(f"\n💾 Saving research report...") output = json.loads(research_output_json) # Create output directory output_dir = Path("research_exports") output_dir.mkdir(exist_ok=True) # Generate filename timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") filename = f"research_analysis_{timestamp}.txt" filepath = output_dir / filename # Safely get values with defaults papers_analyzed = output.get('papers_analyzed', 'N/A') research_landscape_summary = output.get('research_landscape_summary', 'No summary available.') key_innovations = output.get('key_innovations', []) future_research_directions = output.get('future_research_directions', []) queries_used = output.get('queries_used', []) # Format report report = f""" {'=' * 80} ACADEMIC RESEARCH ANALYSIS {'=' * 80} Topic: {topic} Date: {datetime.now().strftime("%Y-%m-%d %H:%M:%S")} Papers Analyzed: {papers_analyzed} {'=' * 80} RESEARCH LANDSCAPE {'=' * 80} {research_landscape_summary} {'=' * 80} KEY INNOVATIONS {'=' * 80} """ for i, innovation in enumerate(key_innovations, 1): report += f"{i}. {innovation}\n" report += f"\n{'=' * 80}\nFUTURE RESEARCH DIRECTIONS\n{'=' * 80}\n\n" for i, direction in enumerate(future_research_directions, 1): if isinstance(direction, dict): report += f"{i}. {direction.get('direction', 'Unknown')}\n" report += f" Rationale: {direction.get('rationale', 'Not specified')}\n\n" else: report += f"{i}. {direction}\n\n" report += f"{'=' * 80}\nSEARCH QUERIES USED\n{'=' * 80}\n\n" for i, query in enumerate(queries_used, 1): report += f"{i}. {query}\n" report += f"\n{'=' * 80}\n" # Save filepath.write_text(report) print(f"✓ Report saved: {filepath}") return str(filepath) except Exception as e: error_msg = f"Failed to save report: {str(e)}" print(f"⚠️ {error_msg}") return error_msg ``` ## Part 6: Conversational Interface This function handles the conversation with the agent, including extracting the response from PydanticAI's message structure: ```python theme={null} async def chat_with_agent(research_question: str, deps: ResearchDeps) -> str: """ Have a conversation with the research agent. Args: research_question: The research question or instruction deps: Research dependencies Returns: Agent's response text """ print(f"\n📋 Your request: {research_question}\n") result = await agent.run(research_question, deps=deps) # Extract response from PydanticAI result # The result contains new_messages() with TextPart and ThinkingPart objects new_msgs = result.new_messages() if new_msgs: last_msg = new_msgs[-1] if hasattr(last_msg, 'parts'): # Extract only TextPart content, skip ThinkingPart text_parts: List[str] = [] for part in last_msg.parts: if hasattr(part, 'content') and 'TextPart' in str(type(part)): text_parts.append(part.content) response = ' '.join(text_parts) if text_parts else str(last_msg) else: response = str(last_msg) else: response = str(result) print("💬 AGENT RESPONSE") print(f"\n{response}\n") return response ``` ## Part 7: Run the Agent! ### Instantiate Our Previously Created Dependencies ```python theme={null} deps = create_research_deps( start_year=2023, max_papers_for_deep_analysis=1 ) print(f" Start year: {deps.start_year}") print(f" Max papers for deep analysis: {deps.max_papers_for_deep_analysis}") print(f" Fulltext excerpt: {deps.fulltext_excerpt_chars:,} chars") ``` ### Example 1: Full Research Workflow The agent will autonomously: 1. Generate search queries 2. Search arXiv 3. Analyze abstracts 4. Download and process PDFs 5. Perform deep analysis 6. Synthesize findings 7. Save the report ```python theme={null} research_question = """ Please conduct a comprehensive literature review on "vision-language models for multimodal reasoning". Follow these steps: 1. Generate 3 diverse arXiv search queries 2. Search arXiv with those queries 3. Analyze the abstracts to identify key themes 4. Select the top 1 most relevant paper 5. Download and analyze that paper in depth 6. Synthesize the findings into a comprehensive report 7. Save the report to a file Provide a summary of your findings at the end. """ response = await chat_with_agent(research_question, deps) ``` ### Example 2: Quick Abstract-Only Analysis The agent adapts to simpler requests: ```python theme={null} quick_question = """ What are the key themes in recent papers about "persuasive natural language generation"? Just analyze abstracts, don't download full papers. """ response = await chat_with_agent(quick_question, deps) ``` ### Example 3: Follow-up Questions The agent can answer follow-up questions: ```python theme={null} followup = "What were the most innovative methods you found in those papers?" response = await chat_with_agent(followup, deps) ``` ## Part 8: Inspect Results ### View Cached Papers ```python theme={null} print(f"Papers in cache: {len(deps.papers_cache)}") print("\nCached papers:") for i, (arxiv_id, paper) in enumerate(list(deps.papers_cache.items())[:5], 1): print(f"{i}. {arxiv_id} — {paper['title'][:60]}...") if 'fulltext' in paper: print(f" ✓ Full text cached ({len(paper['fulltext']):,} chars)") ``` ### View Saved Reports ```python theme={null} export_dir = Path("research_exports") if export_dir.exists(): reports = sorted(export_dir.glob("*.txt"), key=lambda p: p.stat().st_mtime, reverse=True) print(f"Saved reports ({len(reports)}):") for report in reports[:5]: size = report.stat().st_size print(f" • {report.name} ({size:,} bytes)") else: print("No reports saved yet") ``` ## Summary ### What We Built A **conversational academic research agent** with: * **tools** for a complete research workflow * **PydanticAI** for agent orchestration and tool management * **Cerebras** `gpt-oss-120b` for fast, high-quality reasoning * **Unstructured** for PDF text extraction * **Pydantic schemas** for type-safe structured outputs ### Key Patterns 1. **Cerebras Strict Mode**: Use `prepare_tools` hook to normalize all tools to `strict=False` 2. **Dependency Injection**: Use `RunContext[ResearchDeps]` to share API clients and caches 3. **Schema Validation**: Validate all LLM outputs with Pydantic models 4. **Error Resilience**: Tools return error messages instead of raising exceptions 5. **Caching**: Cache papers and full text to avoid redundant API calls ### Next Steps * Add semantic search with vector embeddings, rather than different API calls to arxiv's API * Add a citation graph analysis * Add multi-source search (PubMed, Semantic Scholar) ### Resources * [Cerebras Inference Docs](https://inference-docs.cerebras.ai) * [PydanticAI Docs](https://ai.pydantic.dev) * [Unstructured Docs](https://docs.unstructured.io) * [arXiv API](https://info.arxiv.org/help/api/basics.html) ### Acknowledgement Thank you team from Pydantic AI and Unstructured for incredibly helpful inputs during the creation of this cookbook. Also a shoutout to my colleagues Zhenwei Gao, Ryan Loney and Sarah Chieng for great feedback on initial versions. # Automate User Research with LangChain Source: https://inference-docs.cerebras.ai/cookbook/agents/automate-user-research Learn how to build an AI-powered user research system that can automatically generate user personas, conduct interviews, and synthesize insights using LangGraph's multi-agent workflow in under 60 seconds.