SpeedInfer provides the control plane and OpenAI-compatible developer API. Model execution runs in connected workers, including NVIDIA GPU workers configured for CUDA, Triton Inference Server, or TensorRT-LLM.
Control plane, authentication, routing, usage, and developer API.
GPU-accelerated model execution and serving.
Server-side JSON defines the worker ID, runtime, endpoint, supported models, protocol settings, and metadata.
Triton workers use server readiness, model readiness, and the ready-model repository index.
Direct Triton workers receive HTTP v2 tensor requests. OpenAI-compatible workers use the existing chat/completions proxy.
SSE is forwarded from OpenAI-compatible TensorRT-LLM frontends. Direct Triton HTTP v2 returns a clear unsupported-streaming error.
Triton models must return configured prompt and completion token outputs. Unmetered responses fail and release the reservation.
Only healthy workers declaring the requested model are eligible. Otherwise SpeedInfer returns HTTP 503 without charging.
CUDA Quick Start Guide ↗ · Triton HTTP/REST and gRPC Protocol ↗ · TensorRT-LLM Backend ↗