Back to Benchmarks
openai/gpt-oss-120b
A single-GPU H100 benchmark for sparse MoE serving with MXFP4 expert weights.
+24.5% token/s · 20% Cost Savings over the baseline
·
MODEL OVERVIEW
GPT-OSS-120B overview
openai/gpt-oss-120b is a sparse MoE causal language model with attention sinks. MoE expert weights use MXFP4, while attention, norms, embeddings, and router remain BF16.
Optimized inference environment
HARDWARE · 1 × NVIDIA H100-80GB
PARALLELISM · tensor parallel 1
FEATURES · FlashAttention · prefix caching · chunked prefill · async scheduling
Workload Profile
FRAMEWORK
vllm bench serve
DATASET
random synthetic dataset, range ratio 0.8
INPUT LENGTH
409–3,686 tokens (nominal 2,048)
OUTPUT LENGTH
51–460 tokens (nominal 256)
PROMPTS MEASURED
200 prompts, 10 warm-ups excluded
REQUEST RATE
inf
TESTED CONCURRENCY
16
Baseline and Optimized Serving Configurations
BASELINE
vllm serve openai/gpt-oss-120b --tensor-parallel-size 1 --max-model-len 16384
OPTIMIZED
VLLM_FLOAT32_MATMUL_PRECISION=medium · VLLM_NO_USAGE_STATS=1 · VLLM_DO_NOT_TRACK=1 · vllm serve openai/gpt-oss-120b --host 0.0.0.0 --port 8000 --trust-remote-code --gpu-memory-utilization 0.92 --max-num-batched-tokens 32768 --max-num-seqs 128 --watermark 0.01 --block-size 16 --attention-backend FLASH_ATTN --enable-prefix-caching --performance-mode throughput --async-scheduling --enable-chunked-prefill --max-model-len 16384
Benchmark Results
Baseline and optimized serving results for the single-GPU benchmark configuration.
Metric
Baseline
Optimized
Output tok/s
364
453
TTFT median
14.6s
11.4s
TPOT median
169ms
127ms
ITL mean
65ms
51ms
Requests waiting (avg)
35.1
32.2
Optimization/Tuning Changes
GPU MEMORY UTILIZATION
0.92
BATCHING LIMITS
max-num-batched-tokens 32,768 · max-num-seqs 128
WATERMARK AND BLOCK SIZE
watermark 0.01 · block-size 16
ATTENTION BACKEND
FLASH_ATTN
CACHING AND PREFILL
prefix caching · chunked prefill
SCHEDULING MODE
async scheduling · performance-mode throughput
ENVIRONMENT
VLLM_FLOAT32_MATMUL_PRECISION=medium · usage tracking disabled.
COST SAVINGS
Estimated Cost Savings
At $3.89 per GPU-hour, optimized throughput lowers the effective cost of each generated token.
THROUGHPUT GAIN
+24.5%
453 vs 364 output tok/s
ESTIMATED SAVINGS
19.6%
$3.89 / GPU-hour
Calculation: At $3.89 / GPU-hour, 364 tok/s costs about $2.97 per 1M output tokens. 453 tok/s costs about $2.39. Estimated saving: $0.58 per 1M tokens (19.6%).