From your first key
to your first response.

A practical guide to the supported text inference API. Base URL: https://speedinfer.com/v1.

1. Prepare your account

Create an account, explicitly create an API key in the console, and fund that key if required. Keep the secret in an environment variable. Your sign-in token is not an inference API key. Credits are currently per key. The API checks the key’s permissions and balance.

2. Inspect readiness and models

curl -i https://speedinfer.com/ready
curl https://speedinfer.com/v1/models \
  -H "Authorization: Bearer $SPEEDINFER_API_KEY"
curl https://speedinfer.com/v1/usage \
  -H "Authorization: Bearer $SPEEDINFER_API_KEY"

Use an exact model ID from the catalog. Input and output rates are denominated in USD per million tokens. Hardware and availability depend on the connected worker.

3. Send a text request

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://speedinfer.com/v1",
    api_key=os.environ["SPEEDINFER_API_KEY"],
)
model = client.models.list().data[0].id
response = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Explain continuous batching."}],
    max_tokens=128,
)
print(response.choices[0].message.content)
print(response.usage)

4. Stream the response

stream = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Say hello."}],
    max_tokens=128,
    stream=True,
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="", flush=True)

SSE includes an authoritative worker usage event with empty choices before [DONE]. A stream interrupted after HTTP headers uses an SSE error event and does not send a successful termination marker. Do not treat HTTP 200 alone as completion.

Billing and concurrency

Before contacting the worker, a durable reservation holds the configured input context limit × input price plus the requested output limit × output price. This is a conservative availability check, not the final bill. Successful requests settle using the worker’s actual token counts and return unused credit. Concurrent requests cannot spend the same held balance. Failed, disconnected or unmetered streams release the hold. A gateway crash can leave a hold pending operator reconciliation.

For example, a 32,768-token context at $0.20/M input and 128 output tokens at $0.60/M reserves $0.0066304. An actual 12-input / 5-output response costs $0.0000054. Your model’s rates may differ. No fixed latency or throughput is promised.

Supported surface

The gateway exposes OpenAI-compatible text completions and SSE chat streaming. One completion per request. Default output limits are 128 for chat and 16 for legacy completion, maximum 32,768 subject to model context. Worker runtime configurations (such as FP8 precision, TensorRT kernels, and execution concurrency) are managed at the connected worker layer rather than through per-request API flags.

Error handling

Use x-request-id for support. Retry with care: inference requests do not provide client idempotency guarantees. Never include API secrets in support messages. API timing headers measure gateway response setup; streaming TTFT and end-to-end latency are separate metrics.

GPU go-live checklist

Start a compatible worker with the same served model ID, configure its private URL and server-side API key, migrate the database, verify Redis, test non-stream and stream usage, and benchmark with your actual model and GPU. Inspect GPU memory, concurrent request behaviour and recovery under load. A successful test verifies endpoint contracts; real-world GPU performance benchmarks reflect deployed acceleration hardware.

Product Architecture · Leadership & Corporate Backing · Billing and refunds · Data handling · Support