SpeedInfer.
TENSORRT-LLM RUNTIME

TensorRT-LLM workers

TensorRT-LLM can be registered behind Triton or behind an OpenAI-compatible serving frontend. The selected transport controls how SpeedInfer probes and calls the worker.

Through Triton

Omit transport or set it to triton. Configure the Triton model or ensemble names and tensor contract.

{"worker_id":"trtllm-triton-01","worker_type":"nvidia-gpu","runtime":"tensorrt-llm","runtime_endpoint":"http://triton:8000","supported_models":["acme/model-1"],"runtime_config":{"transport":"triton","model_name":"ensemble","input_name":"text_input","max_tokens_input_name":"max_tokens","output_name":"text_output","prompt_tokens_output_name":"prompt_tokens","completion_tokens_output_name":"completion_tokens"}}

Through an OpenAI-compatible frontend

Set transport to openai. The frontend must expose /v1/models, /v1/chat/completions, and authoritative usage. This path supports the existing SSE forwarding.

{"runtime":"tensorrt-llm","runtime_endpoint":"http://trtllm-frontend:8000/v1","supported_models":["acme/model-1"],"runtime_config":{"transport":"openai"},"runtime_metadata":{"backend":"tensorrt-llm"}}

Model configuration

The public model ID belongs in supported_models. For Triton transport, runtime_config.model_name is the deployed Triton model or ensemble name.

Test through SpeedInfer

curl https://speedinfer.com/v1/chat/completions -H "Authorization: Bearer $SPEEDINFER_API_KEY" -H "Content-Type: application/json" -d '{"model":"acme/model-1","stream":false,"messages":[{"role":"user","content":"Hello"}]}'

Troubleshooting