Prepare an NVIDIA worker host

CUDA runs on the worker, not in the SpeedInfer gateway. Version selection must follow the runtime container or TensorRT-LLM release you deploy.

Required foundation

• A CUDA-capable NVIDIA GPU.
• An NVIDIA driver compatible with the CUDA Toolkit/runtime version.
• A supported operating system and architecture for the selected NVIDIA release.
• A serving process reachable from the SpeedInfer gateway network.

NVIDIA’s current CUDA release notes state that running a CUDA application requires a CUDA-capable GPU and a compatible driver. Check the compatibility table for your chosen CUDA release rather than copying a version from this site.

Verify the host

Use the verification steps for your operating system from NVIDIA’s CUDA installation or quick-start guide. Confirm the driver sees the GPU and that the runtime/container version is compatible before registering the worker.

Register a CUDA worker

A cuda worker uses SpeedInfer’s existing OpenAI-compatible transport. Its runtime endpoint must serve /v1/models, /v1/chat/completions, and usage data.

{"worker_id":"cuda-01","worker_type":"nvidia-gpu","runtime":"cuda","runtime_endpoint":"http://cuda-worker:8000/v1","supported_models":["acme/model-1"],"runtime_config":{"transport":"openai"},"gpu_metadata":{"product":"operator-reported"},"runtime_metadata":{"cuda_version":"operator-reported"}}

Verification through SpeedInfer

curl -s https://speedinfer.com/health
curl https://speedinfer.com/v1/chat/completions -H "Authorization: Bearer $SPEEDINFER_API_KEY" -H "Content-Type: application/json" -d '{"model":"acme/model-1","messages":[{"role":"user","content":"CUDA worker check"}]}'

Troubleshooting

• If the driver cannot see the device, resolve that before testing SpeedInfer.
• If /health reports model readiness false, verify the worker’s /v1/models response and model ID.
• Do not publish GPU serials or worker credentials in metadata.

Official NVIDIA references

CUDA Quick Start Guide · CUDA Toolkit Release Notes and driver compatibility