SpeedInfer.
TRITON HTTP V2

Triton Inference Server

The adapter checks server and model readiness and sends non-streaming requests to the Triton HTTP v2 model inference endpoint.

Required configuration

{"runtime":"triton","runtime_endpoint":"http://triton:8000","supported_models":["acme/model-1"],"runtime_config":{"model_name":"text_generation","input_name":"text_input","max_tokens_input_name":"max_tokens","output_name":"text_output","prompt_tokens_output_name":"prompt_tokens","completion_tokens_output_name":"completion_tokens"}}

Expected model contract

The Triton model accepts a BYTES prompt tensor with shape [1,1] and an INT32 maximum-token tensor. It returns text plus authoritative prompt and completion token counts. Names are configurable. Chat defaults to a JSON array of role/content objects; chat_template may wrap the {messages_json} placeholder.

Health and model availability

curl -f http://triton:8000/v2/health/ready
curl -f http://triton:8000/v2/models/text_generation/ready

Direct Triton request

POST /v2/models/text_generation/infer
{"inputs":[{"name":"text_input","shape":[1,1],"datatype":"BYTES","data":[["Hello"]]},{"name":"max_tokens","shape":[1,1],"datatype":"INT32","data":[[32]]}],"outputs":[{"name":"text_output"},{"name":"prompt_tokens"},{"name":"completion_tokens"}]}

Streaming

Direct Triton HTTP v2 is implemented for non-streaming requests. SpeedInfer returns runtime_streaming_unavailable for streaming on this path. Use an OpenAI-compatible TensorRT-LLM frontend when SSE streaming is required.

Troubleshooting