Speculative Decoding on Nova AI Engine: Up to 45.7% more output throughput on serving stacks

LLM inference is sequential by construction. The model emits a token, updates its context, and only then begins the next decoding step. Batching, KV-cache tuning, prefix caching, and scheduler configuration all improve how efficiently each step runs, but none of them reduce how many steps a response requires.

Speculative decoding does. A lightweight drafter proposes several candidate tokens ahead of time, the target model verifies them in a single forward pass, and every accepted candidate removes a sequential step from the critical path.

Turning that on is a one-line flag. Making it pay is not. The benefit depends on the drafting architecture, the target model, how many tokens are proposed, drafting overhead, request concurrency, and where the serving stack saturates. A configuration that delivers 45% on one model can deliver nothing on another, or actively lose ground.

Nova AI Engine treats this as a search problem across all of those variables at once. Here is what that produced in our latest proof of concept, on H100 under vLLM.

Headline results

Alongside the throughput gains: 31.4% shorter benchmark duration, 39.2% lower mean time per output token, and 31.1% lower median end-to-end latency at the operating points where those matter most.

We measured against a tuned baseline, not a convenient one

This is the part that determines whether any of the numbers above mean anything.

Most speculative decoding results are reported against stock serving defaults. That inflates the figure, because most of the apparent gain is ordinary tuning that should have been done regardless. We ran three configurations:

Our tuning stage is not a token gesture. On Gemma-4-31B-it it moved output throughput from 253 to 392 tokens per second before a single draft token was proposed, through FP8 KV cache, raised memory utilization, explicit batch and sequence limits, prefix caching, chunked prefill, attention backend selection, and context length matched to the workload. The full launch commands are in the appendix.

Every percentage in this post is speculative measured against that tuned configuration. The gains are incremental to serving optimization, not a restatement of it.

Gemma-4-31B-it with DFlash: +45.7%

DFlash uses a block-diffusion drafter that generates an entire candidate block in a single parallel forward pass, rather than producing draft tokens one at a time. We ran it with 15 speculative tokens.

At concurrency 16:

Across the full concurrency sweep:

The 45.7% peak comes at low concurrency, where the GPU has the most idle compute for drafting to exploit. What matters more for production is what happens after that.

This configuration reaches KV-cache saturation at around concurrency 32. Beyond that point the server is running at capacity and additional offered load queues rather than executes, which is why both the tuned and speculative curves flatten. The important result is that DFlash holds a ~21% advantage right through saturation: 392 tok/s tuned against roughly 475 tok/s speculative, sustained.

For a capacity-constrained deployment that is the number to plan around. It means 21% more tokens per GPU-hour at the point where the hardware is already fully committed.

Under heavier load

At concurrency 128, the picture gets better rather than worse:

Throughput, time to first token, time per output token, and end-to-end latency all improve simultaneously, at both mean and p99. That combination is uncommon and it is the strongest single result in this study.

Stacked on the tuning stage, the compound effect on this workload is 253 to 480 output tokens per second on identical hardware: 90% more throughput, and roughly 47% lower cost per generated token.

Qwen3-30B-A3B-Instruct-2507 with EAGLE-3: +21.9%

EAGLE-3 takes the opposite approach to DFlash, using a small autoregressive draft head conditioned on the target model's hidden states. We ran it with 3 speculative tokens.

At concurrency 32:

Across concurrency:

Strong from concurrency 16 through 64, then fading. By 512 the speculative and tuned configurations are equivalent.

The mechanism is the same one that makes speculative decoding work in the first place, running in reverse. Drafting spends compute to save sequential steps, and that trade pays only while there is idle compute available. Once the batch is large enough to saturate the GPU, verification stops being free and the drafter becomes overhead.

This is a deployment finding, not a disappointment. It says a single static serving policy is the wrong answer for this workload. The speculative configuration belongs in the concurrency range where it has a verified advantage, with the tuned non-speculative configuration handling traffic above it. Nova AI Engine identifies that boundary rather than assuming one setting applies everywhere.

Note also how differently the two models want to be configured: 3 speculative tokens for Qwen against 15 for Gemma, on the same GPU, in the same serving engine. That parameter cannot be inherited from a default or carried across models.

Not every drafter earns its place

On a Qwen reasoning workload, the n-gram drafting configurations we tested lost to the tuned non-speculative stack at every concurrency level:

An n-gram drafter with a weak acceptance profile on reasoning-style output still pays the full verification cost while proposing tokens the target model rejects. The configuration runs correctly and produces correct output. It just produces fewer tokens per GPU-hour than doing nothing at all.

We did not ship it, and we are reporting it here because the selection is the product. An optimization engine that enables every available technique is not an optimization engine. The value is in knowing which candidates clear the bar on your workload and which quietly cost you capacity while appearing to work.

Reading the whole profile, not one number

Throughput improvements do not arrive alone, and the tradeoffs decide whether a configuration should ship.

The Qwen configuration at its best throughput point also showed TTFT p99 up 32.7% and TPOT p99 up 24.1%. Some measured inter-token latency increases are an artifact of streaming granularity rather than real regressions, since a single streamed chunk now carries multiple tokens, but tail latency movements are real and need weighing against the application.

For a batch generation service, maximum tokens per second and total completion time dominate, and both Gemma and Qwen configurations are clear wins. For a real-time assistant with a time-to-first-token budget, the Gemma c128 profile qualifies comfortably while the Qwen c32 profile needs a decision about tail behavior.

Nova AI Engine optimizes against whichever objective your service actually has: maximum output throughput, minimum TTFT, minimum time per output token, lowest end-to-end latency, strongest tail performance, lowest cost per request, or highest request capacity on the existing fleet.

What a verified gain is worth

Inside the range where these configurations hold, the arithmetic is direct:

• Qwen3-30B-A3B with EAGLE-3: 1,477 to 1,800 tok/s. Approximately 18% lower cost per generated token.

• Gemma-4-31B-it with DFlash, at saturation: 392 to 475 tok/s. Approximately 17% lower cost per generated token.

• Gemma including the tuning stage: 253 to 480 tok/s. Approximately 47% lower cost per generated token.

Same GPUs, same target model, same output distribution, since verification-based speculative decoding preserves what the target model would have produced.

In practice that converts into one of three things: more traffic served on the fleet you already own, a deferred capacity expansion, or a lower infrastructure cost carried by every request.

How Nova AI Engine finds the configuration

1. Tune the non-speculative stack first. KV-cache precision, batch-token limits, sequence limits, prefix caching, chunked prefill, attention backend, memory utilization, context allocation. Without this, speculative gains are just deferred tuning gains.

2. Evaluate compatible drafting architectures. Autoregressive draft heads, block-diffusion drafters, n-gram and prompt-lookup methods. The ranking is model-specific and does not transfer.

3. Search the speculative configuration. Draft model selection, speculative token count, batch and cache limits, memory allocation. Our two models landed five times apart on token count alone.

4. Measure the complete profile. Throughput, TTFT, TPOT, ITL, end-to-end latency, tail behavior, queue depth, KV-cache pressure. A throughput win with a broken tail is not a win.

5. Sweep the real traffic range. Establish where the configuration is load-bound, where it saturates, and where the advantage begins and ends.

6. Select. Deploy inside the verified range, fall back outside it, and discard candidates that do not earn their cost.

Results are hardware-specific as well as model-specific. Compute-to-bandwidth ratio, FP8 drafter support, and available attention kernels all move where the advantage sits, which is why the configuration has to be found on the fleet you actually run.

Evaluate it on your workload

Neural Nova works with AI application companies and infrastructure providers to optimize inference on their own models, hardware, serving stacks, and traffic profiles.

Bring us one target model, a representative workload, and your current serving configuration. We will benchmark your existing stack, tune a competitive non-speculative baseline, evaluate compatible speculative approaches against it, and tell you:

• Whether speculative decoding helps this workload, and by how much

• Which drafting architecture performs best, at what token count

• The concurrency range where the advantage is real

• Which latency metrics improve and which regress

• What the result means for serving capacity and GPU cost

And if the answer is that speculative decoding does not help you, we will show you the data and say so. That was the right answer for one of the three configurations here.

Appendix: launch configurations

All runs on a single NVIDIA H100 under vLLM, tensor-parallel size 1. Tuned configurations are the output of Nova AI Engine's non-speculative tuning stage.

Qwen3-30B-A3B-Instruct-2507

Tuned

export VLLM_ATTENTION_BACKEND=FLASHINFER
export VLLM_USE_V1=1

vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \
  --host 0.0.0.0 --port 8000 \
  --trust-remote-code \
  --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8_e4m3 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 16384 \
  --enable-prefix-caching \
  --max-model-len 8192

Speculative, EAGLE-3, 3 draft tokens

vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \
  --host 0.0.0.0 --port 8000 \
  --trust-remote-code \
  --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8_e4m3 \
  --max-num-batched-tokens 16384 \
  --enable-prefix-caching \
  --max-model-len 8192 \
  --speculative-config '{"method": "eagle3", "model": "RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3", "num_speculative_tokens": 3}'

Gemma-4-31B-it

Tuned

export VLLM_ATTENTION_BACKEND=FLASHINFER

vllm serve google/gemma-4-31B-it \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 1 \
  --max-model-len 24576 \
  --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8_e4m3 \
  --max-num-seqs 128 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 16384 \
  --limit-mm-per-prompt '{"image": 0, "audio": 0}' \
  --trust-remote-code

Speculative, DFlash, 15 draft tokens

export VLLM_ATTENTION_BACKEND=FLASHINFER

vllm serve google/gemma-4-31B-it \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 1 \
  --max-model-len 24576 \
  --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8_e4m3 \
  --max-num-seqs 128 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 16384 \
  --limit-mm-per-prompt '{"image": 0, "audio": 0}' \
  --trust-remote-code \
  --attention-backend triton_attn \
  --speculative-config '{"method": "dflash", "model": "z-lab/gemma-4-31B-it-DFlash", "num_speculative_tokens": 15}'

Configuration notes. The Gemma speculative run uses the Triton attention backend, required by the DFlash path, where the tuned run uses FlashInfer. The Qwen speculative run does not pin --max-num-seqs explicitly. Both are noted for anyone reproducing these commands.

References

• EAGLE-3 in vLLM: docs.vllm.ai/projects/speculators

• DFlash: ICML 2026

• Speculator models: huggingface.co/collections/RedHatAI/speculator-models