Back to Benchmarks

zai-org/GLM-5.2

A high-throughput serving benchmark for GLM-5.2 on AMD Instinct MI325X hardware.

+26.8% token/s over the baseline

MODEL OVERVIEW

GLM-5.2 overview

zai-org/GLM-5.2 is a Mixture-of-Experts causal language model that uses MLA with dynamic sparse attention. It supports a maximum context length of 65,536 tokens, and this workload is primarily prefill-dominated and compute-bound.

Optimized Inference Environment

A single-node AMD Instinct deployment tuned for high-concurrency, prefill-heavy GLM-5.2 serving.

HARDWARE

8 × AMD Instinct MI325X 256GB · single node

SOFTWARE

ROCm 7.2.3 · vLLM v0.25.1+rocm723

PARALLELISM

Tensor parallel 8 · pipeline parallel 1 · mp executor

ATTENTION + FEATURES

AITER MLA sparse decode · FA MLA prefill · FP8 KV cache · prefix caching · async scheduling · CUDA graphs

Workload Profile

FRAMEWORK

vllm bench serve

DATASET

random (synthetic) · range ratio 0.8

INPUT LENGTH

409–3,686 tokens · nominal 2,048 · uniform

OUTPUT LENGTH

51–460 tokens · nominal 256 · uniform

PROMPTS MEASURED

1,000 · warm-ups excluded: 10

REQUEST RATE

inf

TESTED CONCURRENCY

512

Baseline and Optimized Serving Configurations

BASELINE

export VLLM_ROCM_USE_AITER=1
vllm serve zai-org/GLM-5.2 \ --tensor-parallel-size 8 \ --trust-remote-code \ --gpu-memory-utilization 0.95 \ --max-model-len 65536

OPTIMIZED

export VLLM_ROCM_USE_AITER=1
export VLLM_WORKER_MULTIPROC_METHOD=spawn
vllm serve zai-org/GLM-5.2 \ --host 0.0.0.0 --port 8000 \ --tensor-parallel-size 8 \ --pipeline-parallel-size 1 \ --distributed-executor-backend mp \ --max-model-len 65536 \ --gpu-memory-utilization 0.90 \ --kv-cache-dtype fp8_e4m3 \ --max-num-seqs 256 \ --max-num-batched-tokens 16384 \ --long-prefill-token-threshold 4096 \ --enable-prefix-caching \ --async-scheduling \ --compilation-config ‘{“cudagraph_mode”: “FULL_AND_PIECEWISE”, “max_cudagraph_capture_size”: 256}’ \ --trust-remote-code

Benchmark Results

Baseline and optimized serving results for the 8× MI325X ROCm benchmark configuration.

Metric

Baseline

Optimized

Output tok/s

1,019.0

1,292.2

Duration

254.8 s

200.9 s

TTFT mean

49,744 ms

23,774 ms

TTFT median

51,841 ms

11,448 ms

TTFT p99

104,700 ms

77,224 ms

TPOT mean

269.7 ms

294.2 ms

TPOT p99

524.6 ms

744.4 ms

ITL mean

258.1 ms

272.2 ms

ITL p99

783.9 ms

1,079.5 ms

E2E median

110,457 ms

85,579 ms

Requests waiting (peak)

491

478

Requests waiting (avg)

169.5

95.7

KV cache peak

100%

Optimization/Tuning Changes

KV CACHE FORMAT

FP8 (e4m3) KV cache

GPU MEMORY UTILIZATION

0.90 for graph-capture headroom

BATCHING LIMITS

max-num-seqs 256 · max-num-batched-tokens 16,384

LONG-PREFILL TOKEN THRESHOLD

4,096

CACHING AND SCHEDULING

prefix caching · async scheduling

CUDA GRAPHS

FULL_AND_PIECEWISE CUDA graphs with capture size 256

ENVIRONMENT

VLLM_ROCM_USE_AITER=1 · VLLM_WORKER_MULTIPROC_METHOD=spawn

SPECULATIVE DECODING

Deliberately omitted: GLM-5.2’s MTP speculator deadlocks on gfx942 at TP8.