Speculative Decoding in Production: 3x Throughput Improvement

Executive Summary

Autoregressive model generation is highly memory-bound. Each token generated requires reading billions of parameters from high-bandwidth memory (HBM) to GPU registers. For a 70-billion-parameter model, this results in extreme latency thresholds and restricts throughput under concurrent workloads.

Arvento addressed this bottleneck for a high-volume client by configuring vLLM Speculative Decoding. By deploying a highly optimized Llama 3.1 1B draft model to generate candidates and sharding the target Llama 3.1 70B across a sharded HGX H100 node, we increased inference token throughput by 3.2x while cutting monthly GPU hosting costs in half.

1. System Architecture

The speculatively served inference cluster operates inside a Kubernetes environment, using Tensor Parallel (TP) configurations to segment model parameters across multi-GPU nodes.

Component	Specification Details	GPU Hardware	Throughput Profile
Target LLM	Llama 3.1 70B Instruct (FP8 Quantized)	4x NVIDIA H100 (SXM5 80GB) - TP=4	74 tokens/sec (Speculative)
Draft LLM	Llama 3.1 1B Instruct (FP8 Quantized)	Co-located on H100 Node (TP=1 mapping)	240 tokens/sec (Autoregressive)
Orchestration	vLLM inference runtime with speculative token batch verification	Kubernetes & KServe	Latency: 120ms (TTFT)

2. The Speculative Algorithm Mechanics

Rather than executing target evaluations step-by-step, the speculative decoding model operates on a verify-and-commit cycle:

Speculation Phase: The Llama 3.1 1B draft model drafts K=5 potential consecutive tokens. This execution runs extremely fast because the 1B model has low memory bandwidth footprints.
Verification Phase: The sharded Llama 3.1 70B model evaluates all 5 drafted tokens concurrently in a single GPU forward pass. Because modern GPUs evaluate token batches efficiently, this verification step takes virtually the same time as drafting a single token.
Commit Phase: Based on probability threshold sampling, the target model accepts N ≤ K tokens. If a token is rejected, the target model outputs its corrected token, and the cycle repeats.

With a measured acceptance rate of 78%, the engine generates an average of 3.9 tokens per verification pass, dropping latency constraints.

3. vLLM Speculative Configuration

The serving engine was configured to map the tensor parallel shards to the corresponding GPU devices while sharing KV cache allocations between draft and target weights.

vllm_speculative_server.py

from vllm import LLM, SamplingParams

# Initialize high-throughput speculative pipeline
llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_model="meta-llama/Llama-3.1-1B-Instruct",
    num_speculative_tokens=5,             # K factor
    tensor_parallel_size=4,               # Sharded across 4x H100s
    gpu_memory_utilization=0.90,          # Cache allocation
    max_model_len=4096,
    quantization="fp8",                   # Low precision optimization
    enforce_eager=True
)

# Inference execution parameters
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=512,
    use_beam_search=False
)

4. Hardware Cost & Profit Margins

Serving Llama 70B at scale without speculation requires multiple GPU nodes to hit acceptable latency goals. With speculative decoding active, we dropped target GPU counts and improved overall system unit economics:

Throughput Gain vs Concurrent User Scale

Conclusion

By offloading sequence drafting to Llama 3.1 1B and executing large parallel verifications on sharded Llama 3.1 70B weights, speculative decoding overcomes the memory bandwidth boundaries of hardware. This solution provides immediate latency reductions, optimizing B2B software systems.