SPEEDINFER PLATFORM · SOFTWARE ARCHITECTURE

Software-Defined Gateway.
Unified Model Control Plane.

SpeedInfer is the API gateway and control plane that connects developer applications to high-performance AI models. We provide unified authentication, per-token metering, model registry routing, and fine-tuning orchestration. Open-source models are served by connected model workers, while closed-source proprietary models are routed through external providers.

Launch Developer Console ↗ Explore Core Capabilities ↓

OpenAI SDK · Python Integration

from openai import OpenAI

# SpeedInfer is a drop-in software layer
client = OpenAI(
    base_url="https://speedinfer.com/v1",
    api_key=os.environ["SPEEDINFER_API_KEY"]
)

response = client.chat.completions.create(
    model="deepseek-r1",
    messages=[{"role": "user", "content": "Explain prefix caching."}],
    stream=True
)

# Low-latency streaming response
for chunk in response:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

⚙️ Unified API Gateway & Control Plane

SpeedInfer operates strictly as a software control plane. Rather than hosting monolithic compute, we build the authentication gates, token-bucket rate limiters, model routing registries, and authoritative token metering pipelines that connect client applications to model endpoints with zero vendor lock-in.

⚡ Connected NVIDIA Worker Integration

SpeedInfer integrates with connected workers running NVIDIA TensorRT-LLM, NVIDIA Triton, CUDA 12+, and PagedAttention for open-source foundation models, while securely routing proprietary closed-source models to external providers. Connected workers handle low-level continuous batching and KV-cache reuse, with measured latency and transparent per-token billing.

MODULAR ARCHITECTURE

The SpeedInfer Software Stack

A multi-layered control plane and accelerated execution pipeline with deployment-aware data handling and elastic scalability.

LAYER 01

Application & SDKs

Direct compatibility with existing codebases using OpenAI, LangChain, LlamaIndex, vLLM client, or raw HTTPS requests.

OpenAI SDK (Python & TS) RESTful /v1 API SSE Streaming

LAYER 02

Inference Gateway (Control Plane)

Handles constant-time HMAC-SHA256 authentication, pre-flight credit verification, dual token-bucket rate limiting (RPM/TPM), and smart fallback routing.

Constant-Time Auth Token-Bucket Rate Limiter Pre-Flight Hold Atomic Redis Lua

LAYER 03

Model Routing & Orchestration

Model registry routing, worker health selection, routing open-source models to connected GPU workers and proprietary closed-source models to external upstream providers.

Worker Health Selection Open-Source Worker Routing External Provider Dispatch Model Registry

LAYER 04

Execution Data Plane (Workers & Providers)

Connected NVIDIA GPU workers execute open models via CUDA, Triton, and TensorRT-LLM (continuous batching, PagedAttention, FP8), while verified external providers serve closed-source models.

NVIDIA TensorRT-LLM NVIDIA Triton Connected Workers External Providers

PRODUCT CAPABILITIES

Six Core Pillars of the Platform

Every component is engineered for production reliability, transparent developer metering, and enterprise compliance.

01 / GATEWAY ⌘

Inference Gateway

OpenAI-compatible REST and Server-Sent Events (SSE) API gateway with measured usage and connected-worker routing.

✓ OpenAI-compatible endpoints for instant drop-in integration
✓ HMAC-SHA256 constant-time auth with secret salt verification
✓ Dual token-bucket rate limiting enforcing granular RPM and TPM limits
✓ Atomic Redis Lua metering ensuring zero double-spending or held locks
✓ Intelligent provider fallback to maintain uninterrupted service availability

FastAPI Redis Lua SSE Streaming REST API

02 / REGISTRY 📚

Model Registry & Catalog

Unified model catalog indexing open-source foundation models and proprietary closed-source models with real-time health tracking.

✓ Hot dynamic model registration with zero gateway downtime
✓ State lifecycle management (Active, Degraded, Offline)
✓ Automated worker health probing via active /ready health checks
✓ Dual-source catalog: open-weights (connected workers) & closed-source (external providers)
✓ Dynamic alias routing (e.g. deepseek, llama, qwen)

Model Catalog Health Probes Dual Routing Aliasing

03 / RUNTIME ⚡

Connected Worker Execution & Runtime

Model execution running on connected NVIDIA GPU workers with continuous batching and PagedAttention, alongside verified external provider routing.

✓ Continuous iteration-level batching running on connected workers for high concurrency
✓ Prompt & KV Cache reuse on worker runtimes eliminating redundant prefill computation
✓ PagedAttention memory management preventing GPU memory fragmentation
✓ Supported architectures: DeepSeek V3/R1 (MoE & MLA), Llama 3/4, Qwen 2.5/3, Nemotron
✓ External provider integration for closed-source models (OpenAI, Anthropic)

Connected Workers KV-Cache Reuse External Providers FP8 / MoE

04 / WORKFLOWS 🎯

Model Workflows & Fine-Tuning

Full lifecycle orchestration from raw JSONL training data to fine-tuned adapters and private deployment.

✓ LoRA & QLoRA fine-tuning pipelines with MLflow parameter tracking
✓ Tiered compute pricing calculated transparently by model parameter size
✓ Automated dataset validation for JSONL schema and chat templates
✓ Adapter weight merging and automated export to vLLM-compatible Safetensors
✓ S3-compatible private storage for custom datasets and adapter weights

LoRA / QLoRA Dataset Validation Safetensors Private Endpoints

05 / TELEMETRY 📊

Observability & Audit Engine

Real-time telemetry, per-request latency profiling, token throughput accounting, and immutable audit ledgers.

✓ Real-time TTFT measurement and per-second token generation velocity
✓ End-to-end request tracing with x-request-id and timing headers
✓ Error telemetry and bounded backoff for graceful client recovery
✓ Immutable usage ledger tracking every single prompt and completion token
✓ Worker queue depth and cluster saturation metrics

TTFT Telemetry Request Tracing Immutable Ledger Audit Trail

06 / GOVERNANCE 🔒

Developer Control & Governance

Granular role-based security, spend controls, and explicit retention disclosures for each deployment.

✓ Granular API key scopes (chat:completions, storage:write, training:submit)
✓ Prepaid balance protections and customizable spend quotas
✓ Gateway default: prompt and completion content is not retained by the SpeedInfer Gateway
✓ Provider disclosure: connected-worker and external-provider retention varies by deployment
✓ GDPR & CCPA compliant architecture with on-demand data purging

Granular Scopes Retention Disclosure Spend Quotas GDPR / CCPA

STATE-OF-THE-ART ARCHITECTURES

Engineered for Modern Foundation Models

Optimized execution kernels tailored for Mixture-of-Experts (MoE), Multi-Head Latent Attention (MLA), and long-context architectures.

DeepSeek V3 & R1

Native support for Multi-Head Latent Attention (MLA) and 256-expert Mixture-of-Experts routing.

Meta Llama Family

Full support for Llama 3.1, 3.2, 3.3, and upcoming Llama 4 Scout/Maverick architectures with GQA.

Qwen 2.5 & Qwen3

Optimized bilingual and multilingual reasoning pipelines with up to 128K context window support.

NVIDIA Nemotron

High-efficiency synthetic data generation and agentic reasoning models optimized for TensorRT-LLM.

Build on SpeedInfer Today

Create an account, generate an API key, and start streaming completions in under two minutes.

Start Building ↗ View API Reference ↗

Software-Defined Gateway. Unified Model Control Plane.