Back to Benchmarks
Gemma-4-31B-it
A text-only H100 benchmark for Gemma’s dense instruction-tuned multimodal model.
+66.0% token/s · 40% Cost Savings over the baseline
MODEL OVERVIEW
Gemma 4 overview
google/gemma-4-31B-it is a dense instruction-tuned multimodal model with 30.7B active parameters, 60 layers, and a 5,376 hidden size. It uses sliding and global attention heads with a 262,144-token native context; image and audio inputs are disabled for this text-only benchmark.
Optimized Inference Environment
HARDWARE · 1 × NVIDIA H100-80GB
vLLM · v0.20.0 · Transformers 5.x
FEATURES · tensor parallel 1 · FlashInfer · FP8 KV cache · prefix caching · chunked prefill
Workload Profile
FRAMEWORK
vllm bench serve
DATASET
random synthetic dataset, range ratio 0.8
INPUT LENGTH
409–3,686 tokens, nominal 2,048
OUTPUT LENGTH
51–460 tokens, nominal 256
PROMPTS MEASURED
1,000 prompts, 10 warm-ups excluded
REQUEST RATE
inf
TESTED CONCURRENCY
128
Docker Container and Serving Commands
BASE DOCKER
vllm/vllm-openai:v0.20.0
BASELINE
vllm serve google/gemma-4-31B-it \
--host 0.0.0.0 \
--port 8000 \
--max-model-len 24576 \
--gpu-memory-utilization 0.92 \
--limit-mm-per-prompt '{"image": 0, "audio": 0}' \
--trust-remote-code
OPTIMIZED
export VLLM_ATTENTION_BACKEND=FLASHINFER
vllm serve google/gemma-4-31B-it \
--host 0.0.0.0 --port 8000 \
--tensor-parallel-size 1 \
--max-model-len 24576 \
--gpu-memory-utilization 0.95 \
--kv-cache-dtype fp8_e4m3 \
--max-num-seqs 128 \
--enable-prefix-caching \
--enable-chunked-prefill \
--max-num-batched-tokens 16384 \
--limit-mm-per-prompt '{"image": 0, "audio": 0}' \
--trust-remote-code
Benchmark Results
Target concurrency 128 · optimized throughput improves while per-token latency increases.
Metric
Baseline
Optimized
Output tok/s
220.3
365.6
Total tok/s
2,013.7
3,341.8
Request throughput
0.85 req/s
1.41 req/s
TTFT mean
133,009.7 ms
70,845.6 ms
TTFT p99
153,345.7 ms
81,829.6 ms
TPOT mean
34.7 ms
58.2 ms
ITL mean
34.6 ms
58.1 ms
E2E mean
142,003.6 ms
85,936.3 ms
E2E median
150,279.6 ms
89,032.0 ms
Duration
1,178.6 s
710.2 s
Optimization/Tuning Changes
KV CACHE FORMAT
FP8 e4m3 KV cache
GPU MEMORY UTILIZATION
gpu-memory-utilization 0.95
BATCHING LIMITS
max-num-seqs 128 · max-num-batched-tokens 16,384
CACHING AND PREFILL
prefix caching · chunked prefill
ATTENTION BACKEND
FlashInfer attention backend
MEDIA ENCODERS
image/audio encoders disabled to free VRAM for text-only KV cache
Cost Savings
At $3.89 per GPU-hour, optimized Gemma serving lowers the cost of this benchmark run.
THROUGHPUT GAIN
+66.0%
365.6 vs 220.3 output tok/s
ESTIMATED SAVINGS
39.7%
$3.89 / GPU-hour
Calculation: 1,178.6s × $3.89 / hour = $1.27 baseline. 710.2s × $3.89 / hour = $0.77 optimized. Estimated saving: $0.51 per run (39.7%).
On a single GPU running continuously, this Gemma optimization's 39.7% cost reduction translates to ≈ $13,500 in annual savings at $3.89/GPU-hr.