SpeedInfer.
IMPLEMENTED SYSTEM ARCHITECTURE

One control plane.
Connected inference workers.

SpeedInfer is the API gateway and control plane. It authenticates requests, applies rate limits, resolves models, selects a healthy compatible worker, streams supported responses, and meters reported token usage.

01Developer App
02SpeedInfer API Gateway
03Authentication / Rate Limits
04Model Registry & Router
05Inference Orchestration
06Connected Worker
07Inference Runtime
08Response / Usage Metering
CONTROL PLANE / DATA PLANE

Clear responsibility at each layer.

Control Plane — SpeedInfer

The gateway process owns the public API and coordination logic.

  • API authentication and scoped keys
  • RPM and TPM rate limits
  • Model registry and compatible-worker routing
  • Worker health and circuit-breaker state
  • Request forwarding and supported streaming
  • Usage reservations, settlement, and reporting

Data Plane — Connected workers

Workers load models and execute inference outside the gateway.

  • OpenAI-compatible workers
  • NVIDIA GPU workers
  • Triton HTTP v2 model endpoints
  • TensorRT-LLM through Triton or a compatible frontend
  • CUDA and GPU metadata reported through worker configuration
NVIDIA EXECUTION PATH

Supported through connected NVIDIA GPU workers.

SpeedInfer
NVIDIA GPU Worker
Triton Inference Server
TensorRT-LLM
CUDA
NVIDIA GPU
Worker selection is dynamic: a worker is routed only when it declares the requested model and reports healthy status. Direct Triton HTTP uses model readiness checks and the v2 inference protocol; streaming requires an OpenAI-compatible TensorRT-LLM frontend.
IMPLEMENTATION CONTRACT

What the gateway records and checks.

Worker registration

Each configured worker has an ID, worker type, runtime, runtime endpoint, supported models, runtime configuration, health state, and optional GPU/CUDA metadata.

Compatible routing

The registry filters unhealthy workers and workers that do not declare the requested public or base model. If none remain, the API returns a clear 503 response and releases the billing reservation.

OPERATIONS GUIDE

Configure and test a worker.

Inference workers
Registration, health, routingNVIDIA workers
GPU and CUDA metadataTriton
HTTP v2 adapterTensorRT-LLM
Triton or compatible frontend