Back to Benchmarks
Qwen/Qwen3-235B-A22B
A full-stack benchmark for high-throughput MoE inference on Multi-GPUs.
+138.7% token/s · 58.7% Cost Savings over the baseline
MODEL OVERVIEW
Qwen3-235B-A22B Overview
Qwen3-235B-A22B is a sparse Mixture-of-Experts causal language model with 235B total parameters and 22B active per token. It uses 128 experts, with 8 active per token, a 94-layer architecture, a 4,096 hidden size, and grouped-query attention with 64 query heads and 4 KV heads. The model supports a native context length of 32,768 tokens, which can be extended up to 131,072 tokens using YaRN.
Optimized Inference Environment
A single-node vLLM deployment tuned for high-throughput serving of Qwen3-235B-A22B.
HARDWARE
8 × NVIDIA H100-80GB · single node
VLLM
v0.20.0
PARALLELISM
Tensor parallel 8 · mp distributed executor
FEATURES
Chunked prefill · prefix caching · Triton MoE backend
Workload Profile
A controlled synthetic workload designed to reflect high-concurrency serving conditions.
FRAMEWORK
vllm bench serve
DATASET
random (synthetic) · range ratio 0.8
INPUT LENGTH
409–3,686 tokens · nominal 2,048 · uniform
OUTPUT LENGTH
51–460 tokens · nominal 256 · uniform
PROMPTS MEASURED
1,000 · warm-ups excluded: 10
REQUEST RATE
inf · max concurrency is the only throttle
TESTED CONCURRENCY
512
Baseline and Optimized Serving Configurations
BASELINE
vllm serve Qwen/Qwen3-235B-A22B \
--host 0.0.0.0 --port 8000 \
--trust-remote-code \
--tensor-parallel-size 8
OPTIMIZED
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export CUDA_DEVICE_MAX_CONNECTIONS=1
vllm serve Qwen/Qwen3-235B-A22B \
--tensor-parallel-size 8 \
--gpu-memory-utilization 0.90 \
--max-num-batched-tokens 8192 \
--max-num-seqs 512 \
--enable-chunked-prefill \
--enable-prefix-caching \
--max-model-len 8192 \
--distributed-executor-backend mp \
--moe-backend triton
Benchmark Results
Baseline and optimized serving results for the multi-GPU benchmark configuration.
Metric
Baseline
Optimized
Output tok/s
759.0
1,812.0
Request throughput
2.92 req/s
6.98 req/s
TTFT mean
4,304.4 ms
3,786.8 ms
TTFT p99
16,379.8 ms
10,526.9 ms
TPOT mean
146.3 ms
53.9 ms
TPOT p99
195.1 ms
72.6 ms
ITL mean
144.3 ms
53.2 ms
ITL p99
535.7 ms
240.6 ms
E2E mean
41,766.3 ms
17,589.2 ms
E2E median
40,450.2 ms
17,168.5 ms
Duration
342.1 s
143.3 s
Optimization/Tuning Changes
Tuning changes applied to increase usable cache capacity, batch depth, and MoE execution efficiency.
max-model-len
Capped at 8,192 to convert unused context into KV cache slots.
gpu-memory-utilization = 0.90
Reserves 90% of GPU memory for serving capacity.
max-num-seqs = 512
Allows deeper request batching.
Chunked prefill + prefix caching
Enabled to improve reuse and reduce prefill pressure.
Triton MoE backend + mp distributed executor
Selected for efficient expert execution across the eight-way tensor-parallel deployment.
Environment
VLLM_WORKER_MULTIPROC_METHOD=spawn · CUDA_DEVICE_MAX_CONNECTIONS=1
Cost Savings
At the same GPU-hour rate, higher optimized throughput reduces the effective cost of each generated token.
THROUGHPUT GAIN
+138.7%
1,812 vs 759 output tok/s
ESTIMATED SAVINGS
58.1%
same GPU-hour rate
Calculation: 1 − (759 ÷ 1,812) = 58.1% estimated savings. Duration corroborates the estimate: 342.1s → 143.3s.