NVIDIA WORKER PATH

NVIDIA GPU workers

SpeedInfer supports connected NVIDIA GPU workers. The gateway does not require a local GPU and does not claim that one is currently connected.

Runtime choices

runtime	Transport
`cuda`	OpenAI-compatible worker API; CUDA is recorded as its execution layer.
`triton`	Triton HTTP v2 health, model readiness, and inference.
`tensorrt-llm`	Triton by default, or OpenAI-compatible HTTP with `transport: openai`.

Configuration

{"worker_id":"nvidia-01","worker_type":"nvidia-gpu","runtime":"cuda","runtime_endpoint":"http://worker:8000/v1","supported_models":["acme/model-1"],"runtime_config":{"transport":"openai","headers":{"X-Worker-Token":"server-side-secret"}},"gpu_metadata":{"product":"NVIDIA GPU"},"runtime_metadata":{"cuda_version":"12.x"}}

GPU and CUDA metadata

Set metadata from your deployment system when values are known. SpeedInfer returns configured metadata in /health; it does not inspect CUDA devices on the gateway host. Keep serial numbers and sensitive host identifiers out of this field.

Health checks

CUDA/OpenAI-compatible workers must expose GET /v1/models. Triton-based NVIDIA workers follow the Triton probes in the Triton guide.

Test

curl -s https://speedinfer.com/health
curl https://speedinfer.com/v1/chat/completions -H "Authorization: Bearer $SPEEDINFER_API_KEY" -H "Content-Type: application/json" -d '{"model":"acme/model-1","stream":false,"messages":[{"role":"user","content":"Test NVIDIA worker routing"}]}'

Troubleshooting

Verify the runtime endpoint is reachable from the gateway network.
Match the public model ID in supported_models.
Supply private auth only in server-side runtime configuration.
Inspect live_probe and circuit state in /health.