Serve TensorRT-LLM models

SpeedInfer supports TensorRT-LLM either through Triton’s TensorRT-LLM backend or through an OpenAI-compatible TensorRT-LLM serving frontend.

Transport A: TensorRT-LLM behind Triton

Register runtime: tensorrt-llm with transport: triton. The adapter uses the same Triton readiness, discovery, and HTTP v2 inference implementation.

{"worker_id":"trtllm-triton-01","worker_type":"nvidia-gpu","runtime":"tensorrt-llm","runtime_endpoint":"http://triton:8000","supported_models":["acme/model-1"],"runtime_config":{"transport":"triton","model_name":"ensemble","input_name":"text_input","max_tokens_input_name":"max_tokens","output_name":"text_output","prompt_tokens_output_name":"prompt_tokens","completion_tokens_output_name":"completion_tokens"},"runtime_metadata":{"backend":"tensorrt-llm"}}

NVIDIA’s TensorRT-LLM backend documentation defines parameters such as engine_dir, batching_strategy, and decoupled_mode in the Triton model configuration. Configure those in the NVIDIA model repository, not in SpeedInfer.

Transport B: OpenAI-compatible frontend

Register transport: openai when the worker exposes OpenAI-compatible models and completions. This path uses SpeedInfer’s existing SSE forwarding and requires the upstream to emit authoritative usage.

{"worker_id":"trtllm-api-01","worker_type":"nvidia-gpu","runtime":"tensorrt-llm","runtime_endpoint":"http://trtllm-frontend:8000/v1","supported_models":["acme/model-1"],"runtime_config":{"transport":"openai"},"runtime_metadata":{"backend":"tensorrt-llm"}}

Streaming

For SpeedInfer SSE, use the OpenAI-compatible transport and send stream:true. The gateway forwards chunks and requires the final authoritative usage event. NVIDIA documents decoupled streaming for the Triton TensorRT-LLM backend, but SpeedInfer does not currently expose that gRPC client path through its direct Triton adapter.

Verification

1. Verify the TensorRT-LLM engine and model repository using the NVIDIA release-matched guide.
2. Check Triton server/model readiness or the frontend’s /v1/models.
3. Confirm the SpeedInfer worker appears healthy in /health.
4. Send a non-streaming request and inspect usage.
5. For OpenAI transport, send stream:true and confirm the usage event precedes [DONE].

API example

curl https://speedinfer.com/v1/chat/completions -H "Authorization: Bearer $SPEEDINFER_API_KEY" -H "Content-Type: application/json" -d '{"model":"acme/model-1","stream":true,"messages":[{"role":"user","content":"Explain the request path"}]}'

Troubleshooting

• Keep Triton and TensorRT-LLM backend versions aligned with the NVIDIA container release.
• For Triton transport, match the configured model/ensemble and tensor names.
• For OpenAI transport, verify the public model ID and usage schema.
• If streaming returns no usage, SpeedInfer fails settlement and releases the reservation.

Official NVIDIA references

TensorRT-LLM backend · TensorRT-LLM Triton model configuration · Triton TensorRT-LLM user guide