NVIDIA WORKER OVERVIEW

Connect NVIDIA-backed workers to SpeedInfer

SpeedInfer remains the control plane. A connected worker owns model loading and execution. The gateway registers the worker, probes its runtime, filters by supported model, forwards inference, and settles authoritative usage.

Request path

Developer → SpeedInfer OpenAI-compatible API → authentication and rate limits → model registry and router → connected NVIDIA worker → runtime → response and usage settlement.

Environment variables

Variable	Use
`NVIDIA_WORKERS_JSON`	JSON array of NVIDIA worker registrations.
`VLLM_TIMEOUT_SECONDS`	Gateway HTTP timeout shared by connected worker requests; default 120 seconds.
`DEFAULT_MODEL`	Default public model used by the existing gateway configuration.
`VLLM_API_KEY`	Default upstream authorization for existing OpenAI-compatible workers. Worker-specific headers or `api_key` belong in server-side `runtime_config`.

Worker configuration

NVIDIA_WORKERS_JSON='[{"worker_id":"nvidia-01","worker_type":"nvidia-gpu","runtime":"triton","runtime_endpoint":"http://triton:8000","supported_models":["acme/model-1"],"runtime_config":{"model_name":"ensemble","input_name":"text_input","max_tokens_input_name":"max_tokens","output_name":"text_output","prompt_tokens_output_name":"prompt_tokens","completion_tokens_output_name":"completion_tokens"},"gpu_metadata":{"product":"set-by-operator"},"runtime_metadata":{"cuda_version":"set-by-operator"}}]'

Runtime values

cuda: an OpenAI-compatible worker executing on CUDA.
triton: direct Triton HTTP v2 health, discovery, and inference.
tensorrt-llm: Triton transport by default, or transport: openai.

Health, discovery, and availability

GET /health includes the worker runtime, supported models, non-secret metadata, circuit state, and live probe. Triton discovery uses POST /v2/repository/index with {"ready":true}. Routing still requires the worker to declare the public model in supported_models.

API example

curl https://speedinfer.com/v1/chat/completions -H "Authorization: Bearer $SPEEDINFER_API_KEY" -H "Content-Type: application/json" -d '{"model":"acme/model-1","stream":false,"messages":[{"role":"user","content":"Hello"}]}'

Verification

Check /health and locate the worker ID.
Confirm live_probe.healthy, model readiness, and discovered model names.
Call /v1/models with a SpeedInfer key.
Send a non-streaming chat request and verify usage is returned.
For an OpenAI-compatible TensorRT-LLM frontend, repeat with stream:true.

Errors

compatible_worker_unavailable: no healthy declared worker.
runtime_configuration_error: required runtime model configuration is missing.
runtime_usage_unavailable: token outputs are missing.
runtime_streaming_unavailable: direct Triton HTTP was asked to stream.

Official sources

Triton protocol · Model repository extension · TensorRT-LLM backend