AI Gateway

Control, monitor, and optimize AI API usage

AI Gateway provides a unified interface for AI providers with caching, rate limiting, analytics, and cost controls. Use the API to create gateways and manage AI traffic.

Overview

Access the AI Gateway API:

import Cloudflare from 'cloudflare';

const client = new Cloudflare({
  apiToken: process.env.CLOUDFLARE_API_TOKEN,
});

// Access AI Gateway resources
const aiGateway = client.aiGateway;

Gateways

Manage AI Gateway instances.

Create a gateway

Create a new AI Gateway.

const gateway = await client.aiGateway.create({
  account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
  id: 'my-gateway',
  cache_invalidate_on_update: true,
  cache_ttl: 3600,
  collect_logs: true,
  rate_limiting_interval: 60,
  rate_limiting_limit: 100,
  rate_limiting_technique: 'sliding',
});

account_id · string · required

Your Cloudflare account ID

id · string · required

Gateway identifier (used in API calls)

cache_ttl · number

Cache time-to-live in seconds (null to disable caching)

cache_invalidate_on_update · boolean · required

Invalidate cache when gateway is updated

collect_logs · boolean · required

Enable request/response logging

rate_limiting_interval · number

Rate limit window in seconds (null to disable)

rate_limiting_limit · number

Maximum requests per interval (null to disable)

rate_limiting_technique · string · required

Rate limiting algorithm: ‘fixed’ or ‘sliding’

id · string

The gateway ID

account_id · string

Your account ID

created_at · string

ISO 8601 timestamp when the gateway was created

modified_at · string

ISO 8601 timestamp when the gateway was last modified

List gateways

Retrieve all AI Gateways in your account.

for await (const gateway of client.aiGateway.list({
  account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
})) {
  console.log(gateway);
}

account_id · string · required

Your Cloudflare account ID

search · string

Filter gateways by ID

page · number

Page number for pagination

per_page · number

Number of gateways per page

Get a gateway

Retrieve details about a specific gateway.

const gateway = await client.aiGateway.get(
  'my-gateway',
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

gateway_id · string · required

The gateway ID

account_id · string · required

Your Cloudflare account ID

Update a gateway

Update gateway configuration.

const gateway = await client.aiGateway.update(
  'my-gateway',
  {
    account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
    cache_ttl: 7200,
    rate_limiting_limit: 200,
    rate_limiting_interval: 60,
    rate_limiting_technique: 'sliding',
    cache_invalidate_on_update: true,
    collect_logs: true,
  }
);

gateway_id · string · required

The gateway ID to update

account_id · string · required

Your Cloudflare account ID

Delete a gateway

Delete an AI Gateway.

await client.aiGateway.delete(
  'my-gateway',
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

gateway_id · string · required

The gateway ID to delete

account_id · string · required

Your Cloudflare account ID

Logs

Manage and query AI Gateway request logs.

List logs

Retrieve logs for a gateway.

for await (const log of client.aiGateway.logs.list(
  'my-gateway',
  {
    account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
    start_date: '2024-01-01',
    end_date: '2024-01-31',
  }
)) {
  console.log(log);
}

gateway_id · string · required

The gateway ID

account_id · string · required

Your Cloudflare account ID

start_date · string

Start date for logs (ISO 8601)

end_date · string

End date for logs (ISO 8601)

page · number

Page number for pagination

per_page · number

Number of logs per page

Get a log entry

Retrieve a specific log entry.

const log = await client.aiGateway.logs.get(
  'my-gateway',
  logId,
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

id · string

The log entry ID

timestamp · string

ISO 8601 timestamp of the request

provider · string

AI provider (e.g., ‘openai’, ‘anthropic’)

model · string

Model used for the request

status_code · number

HTTP status code

cached · boolean

Whether the response was served from cache

duration_ms · number

Request duration in milliseconds

Get log request

Retrieve the request payload for a log entry.

const request = await client.aiGateway.logs.request(
  'my-gateway',
  logId,
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

Get log response

Retrieve the response payload for a log entry.

const response = await client.aiGateway.logs.response(
  'my-gateway',
  logId,
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

Delete logs

Delete logs for a gateway.

await client.aiGateway.logs.delete(
  'my-gateway',
  {
    account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
    before: '2024-01-01',
  }
);

before · string

Delete logs before this date (ISO 8601)

Datasets

Manage datasets for AI model evaluation.

Create a dataset

const dataset = await client.aiGateway.datasets.create(
  'my-gateway',
  {
    account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
    name: 'test-dataset',
    data: [
      { input: 'Hello', expected_output: 'Hi there!' },
      { input: 'Goodbye', expected_output: 'See you later!' },
    ],
  }
);

gateway_id · string · required

The gateway ID

account_id · string · required

Your Cloudflare account ID

name · string · required

Dataset name

data · array · required

Array of test cases with input and expected output

List datasets

for await (const dataset of client.aiGateway.datasets.list(
  'my-gateway',
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
)) {
  console.log(dataset);
}

Get a dataset

const dataset = await client.aiGateway.datasets.get(
  'my-gateway',
  datasetId,
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

Update a dataset

const dataset = await client.aiGateway.datasets.update(
  'my-gateway',
  datasetId,
  {
    account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
    name: 'updated-dataset',
  }
);

Delete a dataset

await client.aiGateway.datasets.delete(
  'my-gateway',
  datasetId,
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

Evaluations

Run model evaluations using datasets.

Create an evaluation

const evaluation = await client.aiGateway.evaluations.create(
  'my-gateway',
  {
    account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
    dataset_id: datasetId,
    evaluation_type: 'exact_match',
  }
);

gateway_id · string · required

The gateway ID

account_id · string · required

Your Cloudflare account ID

dataset_id · string · required

Dataset to evaluate against

evaluation_type · string · required

Evaluation metric type

List evaluations

for await (const evaluation of client.aiGateway.evaluations.list(
  'my-gateway',
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
)) {
  console.log(evaluation);
}

Get an evaluation

const evaluation = await client.aiGateway.evaluations.get(
  'my-gateway',
  evaluationId,
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

id · string

Evaluation ID

status · string

Evaluation status: ‘pending’, ‘running’, or ‘complete’

score · number

Evaluation score (0-1)

results · array

Detailed results per test case

Using AI Gateway

With OpenAI SDK

import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: 'https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/openai',
});

const completion = await openai.chat.completions.create({
  model: 'gpt-3.5-turbo',
  messages: [{ role: 'user', content: 'Hello!' }],
});

With Workers AI

const response = await fetch(
  'https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/workers-ai/@cf/meta/llama-2-7b-chat-int8',
  {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${CLOUDFLARE_API_TOKEN}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      prompt: 'Tell me a joke',
    }),
  }
);

Best practices

1. Caching: Enable caching for repeated queries to reduce costs and latency
2. Rate limiting: Set appropriate rate limits to control costs
3. Logging: Enable logging for debugging and analytics
4. Monitoring: Regularly review logs and analytics for usage patterns
5. Security: Use authentication to protect your gateway from unauthorized access
6. Cost control: Set up alerts for spending thresholds