Back to Benchmarks
Gemma-4-31B-it
A text-only H100 benchmark for Gemma’s dense instruction-tuned multimodal model.
+66.0% token/s · 40% Cost Savings over the baseline
MODEL OVERVIEW
Gemma 4 overview
google/gemma-4-31B-it is a dense instruction-tuned multimodal model with 30.7B active parameters, 60 layers, and a 5,376 hidden size. It uses sliding and global attention heads with a 262,144-token native context; image and audio inputs are disabled for this text-only benchmark.
Optimized Inference Environment
HARDWARE · 1 × NVIDIA H100-80GB
vLLM · v0.20.0 · Transformers 5.x
FEATURES · tensor parallel 1 · FlashInfer · FP8 KV cache · prefix caching · chunked prefill
Workload Profile
FRAMEWORK
vllm bench serve
DATASET
random synthetic dataset, range ratio 0.8
INPUT LENGTH
409–3,686 tokens, nominal 2,048
OUTPUT LENGTH
51–460 tokens, nominal 256
PROMPTS MEASURED
1,000 prompts, 10 warm-ups excluded
REQUEST RATE
inf
TESTED CONCURRENCY
128
Baseline and Optimized Serving Configurations
BASELINE
vllm serve openai/gpt-oss-120b --host 0.0.0.0 --port 8000 --trust-remote-code --max-model-len 16384
OPTIMIZED
export VLLM_FLOAT32_MATMUL_PRECISION=medium
export VLLM_NO_USAGE_STATS=1
export VLLM_DO_NOT_TRACK=1
vllm serve openai/gpt-oss-120b --host 0.0.0.0 --port 8000 --trust-remote-code --gpu-memory-utilization 0.92 --max-num-batched-tokens 32768 --max-num-seqs 128 --watermark 0.01 --block-size 16 --attention-backend FLASH_ATTN --enable-prefix-caching --performance-mode throughput --async-scheduling --enable-chunked-prefill --max-model-len 16384
Benchmark Results
Target concurrency 128 · optimized throughput improves while per-token latency increases.
Metric
Baseline
Optimized
Output tok/s
220.3
365.6
Total tok/s
2,013.7
3,341.8
Request throughput
0.85 req/s
1.41 req/s
TTFT mean
133,009.7 ms
70,845.6 ms
TTFT p99
153,345.7 ms
81,829.6 ms
TPOT mean
34.7 ms
58.2 ms
ITL mean
34.6 ms
58.1 ms
E2E mean
142,003.6 ms
85,936.3 ms
E2E median
150,279.6 ms
89,032.0 ms
Duration
1,178.6 s
710.2 s
Optimization/Tuning Changes
KV CACHE FORMAT
FP8 e4m3 KV cache
GPU MEMORY UTILIZATION
gpu-memory-utilization 0.95
BATCHING LIMITS
max-num-seqs 128 · max-num-batched-tokens 16,384
CACHING AND PREFILL
prefix caching · chunked prefill
ATTENTION BACKEND
FlashInfer attention backend
MEDIA ENCODERS
image/audio encoders disabled to free VRAM for text-only KV cache
Cost Savings
At $3.89 per GPU-hour, optimized Gemma serving lowers the cost of this benchmark run.
THROUGHPUT GAIN
+66.0%
365.6 vs 220.3 output tok/s
ESTIMATED SAVINGS
39.7%
$3.89 / GPU-hour
Calculation: 1,178.6s × $3.89 / hour = $1.27 baseline. 710.2s × $3.89 / hour = $0.77 optimized. Estimated saving: $0.51 per run (39.7%).