Paddock Deep Dive: The Zero-Python Rust Inference Engine Outperforming vLLM and SGLang

Senior Developer Systems Breakdown

Written by Abrar Akhunji

For the past three years, the open-source LLM inference landscape has been defined by two dominant titans: vLLM (UC Berkeley) and SGLang (LMSYS). Both engines brought revolutionary breakthroughs—PagedAttention, RadixAttention, chunked prefill, and FlashInfer integrations.

Yet, despite their brilliance, both systems share a deep architectural compromise: the Python Control Plane.

While the heavy tensor algebra (GEMMs, flash attention, RMSNorm) is executed in high-performance C++ or Triton kernels, the orchestrator—the continuous batching scheduler, request queuing, tokenization worker pools, and memory pool managers—runs in Python.

Under light loads, the Python runtime overhead is hidden behind GPU compute latency. But in high-concurrency production deployments—such as multi-agent coding swarms, enterprise customer support backends, or synthetic data generation pipelines with 32 to 128 concurrent streams—the Python Global Interpreter Lock (GIL) and asyncio event-loop contention trigger catastrophic latency cliffs.

In mid-September 2026, infrastructure startup Truespar open-sourced Paddock (under MIT/Apache-2.0). Written from the ground up in 100% pure Rust with custom, handwritten C++/CUDA kernels, Paddock delivers a zero-Python, zero-PyTorch, zero-libtorch inference server designed to extract the theoretical physical maximum from NVIDIA GPUs.

Architectural Manifest

+------------------------------------------------------------------------------------+
| PADDOCK INFERENCE ENGINE ARCHITECTURAL SPECIFICATION MATRIX                        |
+--------------------------+---------------------------------------------------------+
| Primary Developer        | Truespar (Open-Sourced September 2026)                  |
| Repository License       | Dual-licensed MIT / Apache-2.0 (github.com/truespar/paddock)|
| Control Plane Language   | 100% Pure Rust (Tokio Async, Axum, Crossbeam)           |
| Execution Plane Language | Handcrafted C++20 & PTX CUDA Kernels (Zero Triton/PyTorch)|
| Dependency Footprint     | 0 Python, 0 PyTorch, 0 Docker (Single Native Binary)   |
| Hardware Requirement     | NVIDIA Driver >= 580 (Linux x64, Windows x64)           |
| Memory Management        | Direct CUDA Driver Virtual Memory APIs (cuMem*)         |
| Scheduling Paradigm      | Lock-Free Concurrent Ring Buffer Continuous Batching    |
| Prefix Caching           | Pure Rust Lock-Free Radix Tree (Zero GIL Contention)    |
| Median TTFT @ 32 Concurrency| 697 ms (vs 1.9s SGLang, 2.5s vLLM, 6.9s llama.cpp)   |
| Quantization Support     | FP16, BF16, FP8 (E4M3/E5M2), NVFP4, MXFP4, Q8_0, Q4_K  |
| Model File Support       | Safetensors, HuggingFace Checkpoints, GGUF              |
| External API Interfaces  | OpenAI (/v1/chat/completions) & Anthropic (/v1/messages)|
+--------------------------+---------------------------------------------------------+

Section 1: The Python Latency Tax in High-Concurrency Serving

To understand why Paddock achieves such radical Time-To-First-Token (TTFT) speedups, we must examine the internal request lifecycle of a Python-orchestrated inference engine versus Paddock's bare-metal Rust pipeline.

Standard Python Orchestrated Engine (vLLM / SGLang):
┌─────────────────────────────────────────────────────────────┐
│ HTTP Request Arrival (FastAPI / Uvicorn)                    │
└──────────────────────────────┬──────────────────────────────┘
                               │ GIL Contention / Event Loop serialization
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ Python Request Queue (asyncio.Queue / multiprocessing IPC)  │
└──────────────────────────────┬──────────────────────────────┘
                               │ Garbage Collector checks & Object Boxing
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ Python Scheduler Loop (Iterates batch, checks KV blocks)     │
│ Python-to-C++ pybind11 / PyTorch FFI wrapper boundary       │
└──────────────────────────────┬──────────────────────────────┘
                               │ CUDA Graph launch / Triton kernel dispatch
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ GPU Execution (SMs compute Attention + FFN)                 │
└─────────────────────────────────────────────────────────────┘

In a traditional Python serving framework:

1. Event Loop Latency: FastAPI/Uvicorn accepts incoming HTTP requests. Each connection requires async task scheduling inside Python's single-threaded event loop.
2. IPC and Serialization: In multi-process architectures (e.g. splitting the engine into a model runner and an API worker), prompts must be serialized across Unix domain sockets or IPC pipes, incurring memory copies and serialization overhead.
3. The GIL Barrier During Batch Formation: When 32 clients send concurrent requests, the scheduler must inspect active sequence lengths, compute remaining token budgets, look up prefix cache hashes, and assign paged memory blocks. In Python, this loop takes 15 to 40 milliseconds per step, completely stalling GPU kernel launches!
4. PyTorch Tensor Boxing: Every tensor dispatched to CUDA passes through PyTorch's C++ dispatch table (c10::TensorImpl), introducing dozens of CPU instruction cycles per kernel launch.

Paddock's Zero-Overhead Lock-Free Control Plane

Paddock replaces this entire software stack with a unified compiled Rust engine:

Paddock Native Architecture:
┌─────────────────────────────────────────────────────────────┐
│ High-Performance Async Network Layer (Tokio / Axum)         │
│ Non-blocking epoll/IOCP event loop; zero thread contention  │
└──────────────────────────────┬──────────────────────────────┘
                               │ Microsecond lock-free handoff
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ Crossbeam Concurrent Ring Buffer Queue                      │
│ Atomic sequence reservation; zero memory allocations        │
└──────────────────────────────┬──────────────────────────────┘
                               │ Nanosecond pointer swap
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ Pure Rust Continuous Batch Scheduler                        │
│ Traverses in-memory Radix prefix tree with zero mutex locks │
└──────────────────────────────┬──────────────────────────────┘
                               │ Direct CUDA Driver API (cuLaunchKernel)
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ Bare-Metal Handwritten CUDA Generation Kernels              │
│ Dispatched via asynchronous CUDA streams directly to SMs    │
└─────────────────────────────────────────────────────────────┘

Senior Developer Systems Breakdown

Section 2: Pure Rust Virtual Memory Management & cuMem* Paging

Managing the Key-Value (KV) cache is the single most memory-critical task in LLM serving. In long-context agentic workloads, dynamic token lengths cause severe memory fragmentation if allocations are managed through traditional cudaMalloc.

vLLM solved this in 2023 with PagedAttention, which brought virtual memory paging concepts to the KV cache. However, implementing page tables in Python introduces CPU synchronization friction.

Paddock's Low-Level Virtual Memory Engine

Paddock interfaces directly with the NVIDIA Low-Level Driver APIs (cuMemAddressReserve, cuMemCreate, and cuMemMap) from Rust:

// Conceptual Rust memory manager in Paddock using cuMem driver APIs
pub struct PagedKvCache {
    virtual_base_addr: CUdeviceptr,
    page_size_bytes: usize,
    free_pages_ring: crossbeam::queue::ArrayQueue<PhysicalPageHandle>,
    radix_tree: Arc<LockFreeRadixTree>,
}

impl PagedKvCache {
    pub fn allocate_block_for_sequence(&self, seq_id: SequenceId) -> Result<BlockIndex, GpuMemoryError> {
        // Fast O(1) lock-free physical page pop
        if let Some(page_handle) = self.free_pages_ring.pop() {
            let block_offset = self.calculate_virtual_offset(page_handle.id);
            // Map physical GPU allocation to virtual address space asynchronously
            unsafe {
                cuMemMap(self.virtual_base_addr + block_offset, self.page_size_bytes, 0, page_handle.handle, 0);
            }
            Ok(page_handle.id)
        } else {
            Err(GpuMemoryError::OutOfMemory)
        }
    }
}

1. Contiguous Virtual Address Space: Paddock reserves a single, contiguous 64-bit virtual memory window across the entire available GPU VRAM at boot time via cuMemAddressReserve.
2. Dynamic Physical Mapping: As new tokens are generated or prefix blocks are discovered, 64KB physical memory slices are mapped on-the-fly via cuMemMap on a background driver thread.
3. Lock-Free Radix Tree: Prefix cache reuse (e.g. system prompts, conversation history, agent tools) is indexed in an immutable copy-on-write Radix tree in Rust memory. Prefix lookup takes less than 1.2 microseconds, compared to 3-8 milliseconds in Python implementations.

Section 3: Custom Handwritten CUDA Kernels vs PyTorch/Triton Overheads

While many frameworks rely on OpenAI's Triton compiler to generate GPU kernels, Triton still introduces Python compilation passes, PyTorch runtime wrappers, and dynamic grid launching overheads.

Paddock takes the uncompromising systems route: its per-generation token decode and GEMV kernels are handwritten in C++20 and inline PTX assembly.

Kernel Launch Latency Comparison:
PyTorch / Triton Dispatch:
[ Python Bytecode ] ──> [ pybind11 ] ──> [ PyTorch Dispatcher ] ──> [ CUDA Runtime API ] ──> [ GPU Launch: ~12-25µs ]

Paddock Direct Driver Launch:
[ Compiled Rust Binary ] ──> [ FFI ] ──> [ CUDA Driver cuLaunchKernel ] ────────────────────> [ GPU Launch: ~1.8-3.2µs ]

Tailored Architecture Optimizations

Paddock includes distinct kernel paths optimized for specific NVIDIA microarchitectures:

• Ada Lovelace (RTX 4090, RTX 6000 Ada): Fused dequantization GEMV kernels that exploit 4th-Gen Tensor Cores and FP8 matrix multiply with dual FP32 accumulation.
• Hopper (H100, H200): Leverages Asynchronous Transaction Barriers (Arrive/Wait) and TMA (Tensor Memory Accelerator) units to stream KV cache blocks directly from HBM3 to shared memory (SRAM) without burning register files.
• Blackwell (B200, GB200): Hardware-native support for 2nd-Gen Transformer Engines, micro-tensor scaling factors, and NVFP4/MXFP4 decompression in flight.

By eliminating PyTorch kernel launch overhead, Paddock executes decode loops with a CPU kernel dispatch latency of under 3 microseconds, allowing the GPU streaming multiprocessors (SMs) to remain 100% compute-saturated.

Senior Developer Systems Breakdown

Section 4: Comprehensive Quantization & Checkpoint Interoperability

One of the greatest friction points with minimalist C++ inference engines (such as early versions of llama.cpp or ExLlama) has been fragmented weight format support. Developers frequently had to convert Safetensors checkpoints into proprietary binary shards before running inference.

Paddock solves this by implementing native zero-copy parsers for the industry's two most ubiquitous formats:

1. Hugging Face Safetensors: Direct memory-mapped (mmap) parsing in Rust. Weights are validated, aligned to GPU page boundaries, and copied directly into VRAM via DMA transfers without intermediate CPU buffer allocations.
2. GGUF Format: Native header parsing and metadata decoding, allowing developers to pull quantized community models directly from HuggingFace.

Supported Quantization Matrix

+--------------------+---------------------+--------------------+--------------------+
| Quantization Type  | Target Hardware     | Bits per Weight    | Memory Saving      |
+--------------------+---------------------+--------------------+--------------------+
| FP16 / BF16        | Universal NVIDIA    | 16 bits            | 1.0x (Baseline)    |
| FP8 (E4M3 / E5M2)  | Ada, Hopper, Black. | 8 bits             | 2.0x Compression   |
| NVFP4 / MXFP4      | Blackwell (B200)    | 4 bits             | 3.8x Compression   |
| Q8_0               | Universal NVIDIA    | 8 bits             | 1.9x Compression   |
| Q4_K_M             | Universal NVIDIA    | 4.5 bits (average) | 3.4x Compression   |
+--------------------+---------------------+--------------------+--------------------+

All quantization schemes feature fused dequantization kernels that perform on-the-fly decompression directly in GPU register space, preserving maximum memory bus bandwidth for KV cache streaming.

Section 5: Benchmark Verification & Serving Economics

In official reproducible benchmarks published by Truespar and verified by the r/LocalLLaMA community on an NVIDIA RTX PRO 6000 Ada (48GB VRAM):

+-----------------------------------+--------------------+--------------------+--------------------+
| Benchmark Scenario                | vLLM (FP8)         | SGLang (FP8)       | Paddock (FP8)      |
+-----------------------------------+--------------------+--------------------+--------------------+
| Single Stream Prefill (Tok/s)     | 4,120              | 4,280              | 4,490 (1.05x vs SGL)|
| Single Stream Decode (Tok/s)      | 112                | 118                | 126 (1.07x vs SGL) |
| 16 Concurrency Throughput (Tok/s) | 1,420              | 1,650              | 1,810 (1.10x vs SGL)|
| 32 Concurrency Throughput (Tok/s) | 2,150              | 2,490              | 2,720 (1.19x vs vLLM)|
| Median TTFT @ 32 Streams          | 2,500 ms           | 1,900 ms           | 697 ms (3.6x faster)|
| P99 TTFT Jitter @ 32 Streams      | 5,800 ms           | 4,100 ms           | 1,220 ms (Zero Jitter)|
| Peak VRAM Overhead (Process)      | 3.2 GB             | 2.8 GB             | 380 MB (Lean Binary)|
+-----------------------------------+--------------------+--------------------+--------------------+

Why These Metrics Matter to Senior Builders

1. 3.6x Faster Time-To-First-Token: In interactive agentic systems (e.g. coding agents, Copilots, and auto-completers), user perceived latency is dominated by TTFT. Waiting 2.5 seconds for a model to start generating code feels sluggish; receiving the first token in 697ms creates an immediate, snappy developer experience.
2. P99 Jitter Elimination: Python engines suffer severe tail latency (P99 TTFT > 5.8s) when background garbage collection or GIL serialization coincides with incoming request bursts. Paddock's compiled deterministic memory management suppresses P99 jitter to under 1.2s.
3. Microscopic Process Footprint: Paddock's host memory footprint is just 380 MB, leaving virtually all host RAM and GPU VRAM available for actual KV cache allocations and model weights.

Section 6: Systems Takeaway & Source Attribution

For years, the AI community believed that Python's performance tax was an acceptable price to pay for developer agility. In model training and experimental prototyping, that assumption remains valid.

In production inference serving, that assumption is dead.

Paddock demonstrates that modern AI inference is fundamentally a systems engineering problem: it is an exercise in OS virtual memory paging, low-overhead network scheduling, lock-free queue concurrency, and fine-grained GPU register management. By shedding the Python runtime and building a native Rust engine, Paddock pushes open-source LLM serving closer to bare-metal hardware efficiency than ever before.

Primary References & Technical Attribution:

• Code Repository: truespar/paddock on GitHub (Open-sourced under MIT/Apache-2.0, September 2026).
• Benchmarking Report: Truespar Research, "High-Throughput Local Inference: Comparative Benchmarks of Paddock, vLLM, SGLang, and llama.cpp on Ada Lovelace", truespar.com/blog/paddock-benchmarks.
• Community Discussion: Reddit r/LocalLLaMA, "Paddock: Native Rust/CUDA LLM Engine Open-Sourced" (September 14–15, 2026).
• API Documentation: Truespar Docs, "OpenAI and Anthropic Endpoint Emulation in Pure Rust" (truespar.com/docs/api).