Connect Triton Inference Server

SpeedInfer implements Triton server readiness, model readiness, ready-model discovery, and non-streaming HTTP v2 inference.

Start Triton

Use an NVIDIA-supported Triton container and model repository. NVIDIA’s quickstart uses the following command form; replace the image tag and repository path with values from the matching release documentation.

docker run --rm -p8000:8000 -p8001:8001 -p8002:8002 -v/full/path/to/model_repository:/models nvcr.io/nvidia/tritonserver:<xx.yy>-py3 tritonserver --model-repository=/models

SpeedInfer configuration

{"worker_id":"triton-01","worker_type":"nvidia-gpu","runtime":"triton","runtime_endpoint":"http://triton:8000","supported_models":["acme/model-1"],"runtime_config":{"model_name":"text_generation","input_name":"text_input","max_tokens_input_name":"max_tokens","output_name":"text_output","prompt_tokens_output_name":"prompt_tokens","completion_tokens_output_name":"completion_tokens"}}

The configured model accepts a BYTES prompt tensor and INT32 maximum-token tensor. It must return text and authoritative prompt/completion token counts. Tensor names are configurable because Triton model contracts vary.

Health and discovery

curl -f http://triton:8000/v2/health/ready
curl -f http://triton:8000/v2/models/text_generation/ready
curl -X POST http://triton:8000/v2/repository/index -H "Content-Type: application/json" -d '{"ready":true}'

SpeedInfer runs those checks through the adapter. The repository index is diagnostic discovery; routing still requires an explicit public model declaration.

Triton inference protocol

POST /v2/models/text_generation/infer
{"inputs":[{"name":"text_input","shape":[1,1],"datatype":"BYTES","data":[["Hello"]]},{"name":"max_tokens","shape":[1,1],"datatype":"INT32","data":[[32]]}],"outputs":[{"name":"text_output"},{"name":"prompt_tokens"},{"name":"completion_tokens"}]}

This tensor contract belongs to the configured text-generation model. Triton’s protocol defines the request envelope; your model configuration defines tensor names and shapes.

Streaming

SpeedInfer’s direct Triton HTTP v2 adapter is non-streaming. TensorRT-LLM decoupled streaming requires an appropriate Triton client path, while SpeedInfer’s public SSE path is implemented for OpenAI-compatible worker frontends. Do not set stream:true on a direct Triton worker.

Usage and errors

Usage collection is mandatory for gateway settlement. Missing token outputs produce runtime_usage_unavailable. Triton rejection produces triton_inference_failed; timeouts and repeated failures affect worker circuit health.

Official NVIDIA references

HTTP/REST and gRPC protocol · Model repository extension · Triton HTTP client reference