Connected inference workers
Workers form the data plane. SpeedInfer registers runtime metadata in memory, routes only to a healthy worker that declares the requested model, and returns HTTP 503 when no compatible worker is available.
Registration and environment
Set NVIDIA_WORKERS_JSON to a JSON array on the gateway process. Registration runs during application startup.
NVIDIA_WORKERS_JSON='[{"worker_id":"gpu-worker-01","worker_type":"nvidia-gpu","runtime":"triton","runtime_endpoint":"http://triton:8000","supported_models":["acme/model-1"],"runtime_config":{"model_name":"model_1"},"gpu_metadata":{"product":"configured-by-operator"},"runtime_metadata":{"cuda_version":"configured-by-operator"}}]'Metadata is operator-supplied configuration, not automatic hardware discovery. Secrets in runtime_config stay server-side and are not included in health responses.
Required fields
| Field | Purpose |
|---|---|
worker_id | Stable worker identity. |
runtime | cuda, triton, or tensorrt-llm. |
runtime_endpoint | Base URL used by the selected protocol. |
supported_models | Public or base-model IDs this worker accepts. |
runtime_config | Protocol-specific model, auth, and input/output configuration. |
Health and routing
GET /health reports the registered runtime, declared models, circuit state, metadata, and a live probe for NVIDIA workers. OpenAI-compatible workers are probed through /v1/models; Triton workers use /v2/health/ready and /v2/models/{model}/ready.
The registry filters by health and supported models before least-connections or round-robin selection. Repeated failures trip the circuit breaker.
Test the connection
curl -s https://speedinfer.com/health
curl -s https://speedinfer.com/v1/models -H "Authorization: Bearer $SPEEDINFER_API_KEY"
curl https://speedinfer.com/v1/chat/completions -H "Authorization: Bearer $SPEEDINFER_API_KEY" -H "Content-Type: application/json" -d '{"model":"acme/model-1","messages":[{"role":"user","content":"Hello"}]}'Troubleshooting
compatible_worker_unavailable: confirm worker health and model declaration.runtime_configuration_error: add the required runtime configuration.runtime_usage_unavailable: configure authoritative prompt/completion token outputs.- Do not publish worker URLs, credentials, or GPU serial information in client code.