Back to Benchmarks
Qwen3-Next-80B-A3B-Instruct
Qwen3-Next-80B-A3B-Instruct is a sparse MoE language model with 80B total parameters and 3B active parameters per token.
+127.2% output throughput · −70.8% TTFT · −51.3% TPOT
MODEL OVERVIEW
Qwen3-Next-80B-A3B-Instruct overview
Qwen/Qwen3-Next-80B-A3B-Instruct is an instruction-tuned sparse MoE model with 80B total parameters and 3B active per token. It uses gated DeltaNet attention and 512 routed experts, selecting 10 per token alongside one shared expert. The standard checkpoint uses BF16 weights and supports long-context inference.
Optimized inference environment
HARDWARE · 8 × NVIDIA A100 80GB
PARALLELISM · tensor parallel 8 · expert parallelism enabled in optimized run
FEATURES · FlashInfer attention · prefix caching · chunked prefill · async scheduling
Workload Profile
FRAMEWORK
vllm bench serve
DATASET
GSM8K accuracy validation
INPUT LENGTH
1,024 tokens
OUTPUT LENGTH
128 tokens
PROMPTS MEASURED
200 configured requests; 10 warm-ups listed separately
REQUEST RATE
inf
TESTED CONCURRENCY
512
Serving Environment and Commands
BASE DOCKER · REFERENCE CHOICE
vllm/vllm-openai:v0.28.0
BASELINE
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct --host 0.0.0.0 --port 8000 --trust-remote-code --tensor-parallel-size 8 --max-model-len 8192
OPTIMIZED
export VLLM_LOGGING_LEVEL=WARNING; export VLLM_NO_USAGE_STATS=1; export VLLM_DO_NOT_TRACK=1; export VLLM_USE_FLASHINFER_SAMPLER=1; vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct --host 0.0.0.0 --port 8000 --trust-remote-code --dtype bfloat16 --tensor-parallel-size 8 --enable-expert-parallel --distributed-executor-backend mp --max-model-len 8192 --gpu-memory-utilization 0.92 --enable-prefix-caching --max-num-batched-tokens 32768 --max-num-seqs 256 --watermark 0.02 --enable-chunked-prefill --attention-backend FLASHINFER --performance-mode throughput --async-scheduling --disable-uvicorn-access-log
Benchmark Results
Baseline versus highlighted optimized iteration 4. Reported values are from the supplied benchmark screenshots; unreported metrics are omitted.
Metric
Baseline
Optimized
Output tok/s
1,088.1
2,472.6
TTFT reported
36,310 ms
10,594 ms
TPOT reported
98.78 ms
48.09 ms
ITL reported
85.01 ms
42.33 ms
Requests running (avg)
69.7
67.9
Test Dataset Accuracy
96%
96%
Optimization/Tuning Changes
GPU MEMORY UTILIZATION
0.92
BATCHING LIMITS
max-num-batched-tokens 32,768 · max-num-seqs 256
WATERMARK / MODEL LENGTH
watermark 0.02 · max-model-len 8192 · no explicit block-size flag
ATTENTION BACKEND
FLASHINFER · FlashInfer sampler enabled
CACHING AND PREFILL
prefix caching · chunked prefill
SCHEDULING MODE
async scheduling · performance-mode throughput
ENVIRONMENT
bfloat16 · tensor parallel 8 · expert parallel enabled (mp) · VLLM_LOGGING_LEVEL=WARNING · usage tracking disabled
BENCHMARK TAKEAWAYS
Throughput and accuracy
At fixed hardware cost, the dashboard’s separate cost-throughput pair implies 56.0% lower cost per token. Monthly savings assume continuous use at the chosen GPU rate.
THROUGHPUT GAIN
+127.2%
2,472.6 vs 1,088.1 output tok/s
COST PER TOKEN
≈56% lower
830.90988 → 1,887.4403 tok/s in cost panel