Inference Serving Configurations

Every serving configuration used in AA-AgentPerf hardware benchmarking, from June 2026 onwards. Each entry captures the model, accelerator system, precision, inference framework, parallelism topology, and full launch command.

Read the AA-AgentPerf methodology.

AA-AgentPerf

HardwarePrecisionFrameworkDate Range

Showing 20 of 108 configurations

FrameworkPrecisionAccelerators UsedParallelismSpecdecDate Added

DeepSeek V4.1 Flash — featured configurations

Vera Rubin NVL72

# DeepSeek V4.1 Flash (MXFP4) on Vera Rubin NVL72 -- 2 x prefill TP2 + 1 x decode DEP4, 8 GPUs, NVIDIA Dynamo + vLLM,
# DSpark speculative decoding (6 draft tokens). Weights: deepseek-ai/DeepSeek-V4.1-Flash @dba1be0a.
# Engine: vLLM 0.20.2rc1.dev6087+g0c1a8b357 (CentML DSv4.1-Flash branch + NVIDIA patches), ai-dynamo 1.6.0.dev20260924.
# GPU clocks: prefill node: power limit 1600 W; decode node: power limit 2300 W (default), SM clock cap 1000 MHz; HBM 4752 MHz (default).
# Each worker also sets a few build-specific tuning env flags of the patched vLLM (padding, Engram inlining); omitted here.
# prefill x2:
VLLM_MXFP8_TRITON_QUANT=1 VLLM_DSV41_FAST_CANDIDATE_TOPK=1 VLLM_DSV41_GRAPH_BOUNDED_REPLAY=1 VLLM_ENGINE_DEFER_FULL_GC=1 \
  python3 -m dynamo.vllm --model deepseek-ai/DeepSeek-V4.1-Flash --engram-config '{"cpu_offload":true,"dp_shared_memory":false,"embedding_across_dp":false,"use_thp":false}' --gpu-memory-utilization 0.92 --language-model-only --max-cudagraph-capture-size 2048 --max-model-len 1048576 --max-num-seqs 512 --speculative-config '{"draft_sample_method":"probabilistic","enable_adaptive_verification":true,"method":"dspark","num_speculative_tokens":6,"rejection_sample_method":"block"}' --tensor-parallel-size 2 --tokenizer-mode deepseek_v41 --trust-remote-code --router-mode kv --dyn-tool-call-parser deepseek_v41 --dyn-reasoning-parser deepseek_v41 --reasoning-parser deepseek_v41 --disaggregation-mode prefill --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
# decode x1:
VLLM_MXFP8_TRITON_QUANT=1 VLLM_DSV41_FAST_CANDIDATE_TOPK=1 VLLM_DSV41_GRAPH_BOUNDED_REPLAY=1 VLLM_ENGINE_DEFER_FULL_GC=1 VLLM_DSV41_MEGAMOE_SM107=1 VLLM_DSV41_MEGAMOE_PERDIE=routed VLLM_WORKER_MULTIPROC_METHOD=spawn \
  python3 -m dynamo.vllm --model deepseek-ai/DeepSeek-V4.1-Flash --engram-config '{"cpu_offload":true,"dp_shared_memory":true,"embedding_across_dp":false,"use_thp":false}' --gpu-memory-utilization 0.92 --language-model-only --max-cudagraph-capture-size 2048 --max-model-len 1048576 --max-num-seqs 512 --speculative-config '{"draft_sample_method":"probabilistic","enable_adaptive_verification":true,"method":"dspark","num_speculative_tokens":6,"rejection_sample_method":"block"}' --tensor-parallel-size 1 --tokenizer-mode deepseek_v41 --trust-remote-code --data-parallel-size 4 --enable-expert-parallel --kernel-config '{"moe_backend":"deep_gemm_mega_moe"}' --numa-bind --numa-bind-nodes 0 0 1 1 --no-swa-bounded-replay --dyn-tool-call-parser deepseek_v41 --dyn-reasoning-parser deepseek_v41 --reasoning-parser deepseek_v41 --disaggregation-mode decode --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
# Frontend (fastokens tokenizer, default routing; etcd for discovery):
DYN_TOKENIZER=fastokens python3 -m dynamo.frontend --http-port 8000

SystemVera Rubin NVL72EnginevLLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend over vLLM workers, 8 GPUs on 2 Vera Rubin NVL72 traysPrefill2 × 2 GPUs · TP2Decode1 × 4 GPUs · EP4 · DP4MoE kernelDeepGEMM MegaMoESpeculative decodingDSpark (6 draft tokens, probabilistic drafts, block rejection, adaptive verification)KV transferNIXLEngram tablesOffloaded to host memoryGPU clocksprefill node: power limit 1600 W; decode node: power limit 2300 W (default), SM clock cap 1000 MHz; HBM 4752 MHz (default)PrecisionMXFP4Accelerators used8 of 72

Submitted by NVIDIA

Configuration reference

# DeepSeek V4.1 Flash (MXFP4) on Vera Rubin NVL72 -- 1 x prefill TP2 + 1 x decode DEP4, 6 GPUs, NVIDIA Dynamo + vLLM,
# DSpark speculative decoding (6 draft tokens). Weights: deepseek-ai/DeepSeek-V4.1-Flash @dba1be0a.
# Engine: vLLM 0.20.2rc1.dev6087+g0c1a8b357 (CentML DSv4.1-Flash branch + NVIDIA patches), ai-dynamo 1.6.0.dev20260924.
# GPU clocks: prefill node: power limit 1600 W; decode node: power limit 2300 W (default), SM clock cap 1000 MHz; HBM 4752 MHz (default).
# Each worker also sets a few build-specific tuning env flags of the patched vLLM (padding, Engram inlining); omitted here.
# prefill x1:
VLLM_MXFP8_TRITON_QUANT=1 VLLM_DSV41_FAST_CANDIDATE_TOPK=1 VLLM_DSV41_GRAPH_BOUNDED_REPLAY=1 VLLM_ENGINE_DEFER_FULL_GC=1 \
  python3 -m dynamo.vllm --model deepseek-ai/DeepSeek-V4.1-Flash --engram-config '{"cpu_offload":true,"dp_shared_memory":false,"embedding_across_dp":false,"use_thp":false}' --gpu-memory-utilization 0.92 --language-model-only --max-cudagraph-capture-size 2048 --max-model-len 1048576 --max-num-seqs 512 --speculative-config '{"draft_sample_method":"probabilistic","enable_adaptive_verification":true,"method":"dspark","num_speculative_tokens":6,"rejection_sample_method":"block"}' --tensor-parallel-size 2 --tokenizer-mode deepseek_v41 --trust-remote-code --dyn-tool-call-parser deepseek_v41 --dyn-reasoning-parser deepseek_v41 --reasoning-parser deepseek_v41 --disaggregation-mode prefill --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
# decode x1:
VLLM_MXFP8_TRITON_QUANT=1 VLLM_DSV41_FAST_CANDIDATE_TOPK=1 VLLM_DSV41_GRAPH_BOUNDED_REPLAY=1 VLLM_ENGINE_DEFER_FULL_GC=1 VLLM_DSV41_MEGAMOE_SM107=1 VLLM_DSV41_MEGAMOE_PERDIE=routed VLLM_WORKER_MULTIPROC_METHOD=spawn \
  python3 -m dynamo.vllm --model deepseek-ai/DeepSeek-V4.1-Flash --engram-config '{"cpu_offload":true,"dp_shared_memory":true,"embedding_across_dp":false,"use_thp":false}' --gpu-memory-utilization 0.92 --language-model-only --max-cudagraph-capture-size 2048 --max-model-len 1048576 --max-num-seqs 512 --speculative-config '{"draft_sample_method":"probabilistic","enable_adaptive_verification":true,"method":"dspark","num_speculative_tokens":6,"rejection_sample_method":"block"}' --tensor-parallel-size 1 --tokenizer-mode deepseek_v41 --trust-remote-code --data-parallel-size 4 --enable-expert-parallel --kernel-config '{"moe_backend":"deep_gemm_mega_moe"}' --numa-bind --numa-bind-nodes 0 0 1 1 --no-swa-bounded-replay --dyn-tool-call-parser deepseek_v41 --dyn-reasoning-parser deepseek_v41 --reasoning-parser deepseek_v41 --disaggregation-mode decode --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
# Frontend (fastokens tokenizer, default routing; etcd for discovery):
DYN_TOKENIZER=fastokens python3 -m dynamo.frontend --http-port 8000

SystemVera Rubin NVL72EnginevLLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend over vLLM workers, 6 GPUs on 2 Vera Rubin NVL72 traysPrefill1 × 2 GPUs · TP2Decode1 × 4 GPUs · EP4 · DP4MoE kernelDeepGEMM MegaMoESpeculative decodingDSpark (6 draft tokens, probabilistic drafts, block rejection, adaptive verification)KV transferNIXLEngram tablesOffloaded to host memoryGPU clocksprefill node: power limit 1600 W; decode node: power limit 2300 W (default), SM clock cap 1000 MHz; HBM 4752 MHz (default)PrecisionMXFP4Accelerators used6 of 72

Submitted by NVIDIA

Configuration reference

# DeepSeek V4.1 Flash (MXFP4) on Vera Rubin NVL72 -- aggregated TP4, 4 GPUs, NVIDIA Dynamo + vLLM,
# DSpark speculative decoding (6 draft tokens). Weights: deepseek-ai/DeepSeek-V4.1-Flash @dba1be0a.
# Engine: vLLM 0.20.2rc1.dev6087+g0c1a8b357 (CentML DSv4.1-Flash branch + NVIDIA patches), ai-dynamo 1.6.0.dev20260924.
# GPU clocks: serving node: power limit 2300 W (default); HBM 4752 MHz (default).
# Each worker also sets a few build-specific tuning env flags of the patched vLLM (padding, Engram inlining); omitted here.
# Worker:
VLLM_MXFP8_TRITON_QUANT=1 VLLM_DSV41_FAST_CANDIDATE_TOPK=1 VLLM_DSV41_GRAPH_BOUNDED_REPLAY=1 VLLM_ENGINE_DEFER_FULL_GC=1 VLLM_WORKER_MULTIPROC_METHOD=spawn \
  python3 -m dynamo.vllm --model deepseek-ai/DeepSeek-V4.1-Flash --engram-config '{"cpu_offload":true,"dp_shared_memory":false,"embedding_across_dp":false,"use_thp":false}' --gpu-memory-utilization 0.92 --language-model-only --max-cudagraph-capture-size 2048 --max-model-len 1048576 --max-num-seqs 512 --speculative-config '{"draft_sample_method":"probabilistic","enable_adaptive_verification":false,"method":"dspark","num_speculative_tokens":6,"rejection_sample_method":"block"}' --tensor-parallel-size 4 --tokenizer-mode deepseek_v41 --trust-remote-code --numa-bind --numa-bind-nodes 0 0 1 1 --dyn-tool-call-parser deepseek_v41 --dyn-reasoning-parser deepseek_v41 --reasoning-parser deepseek_v41
# Frontend (fastokens tokenizer, default routing):
DYN_TOKENIZER=fastokens python3 -m dynamo.frontend --http-port 8000

SystemVera Rubin NVL72EnginevLLM (Dynamo)ServingAggregated in-flight batching - one vLLM worker behind the Dynamo frontend on 4 GPUs of one Vera Rubin NVL72 trayTensor parallelTP4Speculative decodingDSpark (6 draft tokens, probabilistic drafts, block rejection, no adaptive verification)Engram tablesOffloaded to host memoryGPU clocksserving node: power limit 2300 W (default); HBM 4752 MHz (default)PrecisionMXFP4Accelerators used4 of 72

Submitted by NVIDIA

Configuration reference

GB300 NVL72

# DeepSeek V4.1 Flash (NVFP4) on GB300 NVL72 — vLLM 0.30.0 in-flight batching, TP2 on GPUs 0-1 of a 4-GPU
# tray, DSpark speculative decoding (5 draft tokens, probabilistic drafts, block rejection, adaptive verification),
# 4 API servers, Engram tables offloaded to host memory.
# Image: vllm/vllm-openai:v0.30.0-aarch64 with Dynamo 1.5.0 added (Dynamo unused here)
# Patched over stock vLLM 0.30.0: vllm#58784, plus a local patch (NVFP4 DSpark-drafter fix: the draft experts
# keep MXFP4; same dispatch as draft PR vllm#59655).
# SM clock capped at 1200 MHz (memory clock set high, 3996 MHz; default power limit).
# VLLM_USE_BREAKABLE_CUDAGRAPH is left unset (vLLM auto-enables it).
VLLM_ENGINE_READY_TIMEOUT_S=3600 \
PYTHONUNBUFFERED=1 \
vllm serve <model-path> \
  --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}' \
  --engram-config '{"cpu_offload":true}' \
  --language-model-only \
  --max-cudagraph-capture-size 2048 \
  --api-server-count 4 \
  --max-num-seqs 512 \
  --tokenizer-mode deepseek_v41 \
  --tool-call-parser deepseek_v41 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v41 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 1048576 \
  --enable-prompt-tokens-details \
  --trust-remote-code

SystemGB300 NVL72EnginevLLM (v0.30.0 + patches)ServingAggregated in-flight batching - one vLLM server on 2 of the GB300 tray's 4 GPUsTensor parallelTP2Speculative decodingDSpark (5 draft tokens, probabilistic drafts, block rejection, adaptive verification)SM clock cap1200 MHz (memory 3996 MHz, default power limit)API servers4PrecisionNVFP4Accelerators used2 of 72

Submitted by NVIDIA

# NVIDIA Dynamo 1.5.0 + vLLM 0.30.0 disaggregated serving of DeepSeek V4.1 Flash (MXFP4) on GB300 NVL72 —
# 4 GPUs: 1x prefill worker + 1x decode worker, each TP2 on GPUs 0-1 of its own tray; DSpark speculative
# decoding on both (5 draft tokens, probabilistic drafts, block rejection, adaptive verification); NIXL (UCX)
# KV transfer. etcd and the Dynamo frontend (KV-aware routing) share a third tray that does no GPU work.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash
# Image: vllm-v0.30.0-dynamo-rel.sqsh (vLLM 0.30.0 + Dynamo 1.5.0)
# Patched over stock vLLM 0.30.0: vllm#58784, vllm#57662.

# Frontend / router and worker environment (every role; JIT cache paths omitted):
#   ETCD_ENDPOINTS=http://<node2>:2379 HF_HUB_OFFLINE=1 PYTHONUNBUFFERED=1 DSV41_DROP_FIRST_DRAFT=
#   UCX_RCACHE_MAX_UNRELEASED=1024 VLLM_ENGINE_READY_TIMEOUT_S=3600
#   DYN_TCP_RPC_HOST=<node-ip> DYN_TCP_RESPONSE_STREAM_HOST=<node-ip> DYN_EVENT_PLANE_HOST=<node-ip>
#   (<node-ip> = that node's own cluster-network IP)

# Worker environment, in addition:
#   DYN_SYSTEM_PORT=8081 (prefill) / 8091 (decode)

# etcd and the frontend (node 2, no GPUs; OpenAI-compatible on :8000, KV routing, no session-affinity TTL here):
etcd \
  --listen-client-urls http://0.0.0.0:2379 \
  --advertise-client-urls http://<node2>:2379 \
  --listen-peer-urls http://127.0.0.1:2380 \
  --initial-advertise-peer-urls http://127.0.0.1:2380 \
  --initial-cluster default=http://127.0.0.1:2380 \
  --data-dir <host-path>
python3 -m dynamo.frontend \
  --http-host 0.0.0.0 \
  --http-port 8000 \
  --router-mode kv \
  --kv-cache-block-size 128

# Prefill worker (node 0, TP2 on GPUs 0-1):
python3 -m dynamo.vllm \
  --model deepseek-ai/DeepSeek-V4.1-Flash \
  --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
  --tokenizer-mode deepseek_v41 \
  --dyn-tool-call-parser deepseek_v41 \
  --dyn-reasoning-parser deepseek_v41 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 1048576 \
  --kv-events-config '{"enable_kv_cache_events":true,"publisher":"zmq"}' \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}' \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}' \
  --disaggregation-mode prefill \
  --router-mode kv \
  --router-session-affinity-ttl-secs 3600

# Decode worker (node 1, TP2 on GPUs 0-1; no --router-* flags):
python3 -m dynamo.vllm \
  --model deepseek-ai/DeepSeek-V4.1-Flash \
  --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
  --tokenizer-mode deepseek_v41 \
  --dyn-tool-call-parser deepseek_v41 \
  --dyn-reasoning-parser deepseek_v41 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 1048576 \
  --kv-events-config '{"enable_kv_cache_events":true,"publisher":"zmq"}' \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}' \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}' \
  --disaggregation-mode decode

Host-specific paths are redacted as <host-path>.

SystemGB300 NVL72EnginevLLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend + KV-aware router over vLLM workers, GPUs 0-1 on each of 2 GB300 trays (other GPUs of the allocation idle and excluded)Prefill1 × 2 GPUs · TP2Speculative decodingDSpark (5 draft tokens, probabilistic drafts, block rejection, adaptive verification)Decode1 × 2 GPUs · TP2Speculative decodingDSpark (5 draft tokens, probabilistic drafts, block rejection, adaptive verification)KV transferNIXL (UCX)RouterKV-aware (prefill session affinity TTL 3600 s)PrecisionMXFP4Accelerators used4 of 72

Submitted by NVIDIA

# DeepSeek V4.1 Flash (MXFP4) on GB300 NVL72 — SGLang in-flight batching on one 4-GPU tray, TP4 + EP4,
# DSPARK speculative decoding (block size 5), decode CUDA graphs up to batch 64; low-latency recipe.
# The server resolves FP8 (e4m3) KV cache, page size 256, MoE backend flashinfer_mxfp4, 256 max running
# requests; parsers auto-detect to deepseek-v41 / deepseekv41. No HiCache, no cache report, no extra env vars.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash
# Image: lmsysorg/sglang:dev-cu13-dsv41@sha256:4a5d132a06a77c8331e15845f2e925adc788b00105097ad55409afa3f4fa4860
sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4.1-Flash \
  --tp 4 \
  --ep-size 4 \
  --mem-fraction-static 0.8 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 5 \
  --cuda-graph-max-bs-decode 64 \
  --reasoning-parser auto \
  --tool-call-parser auto \
  --host 0.0.0.0 \
  --port 30000 \
  --served-model-name deepseek-ai/DeepSeek-V4.1-Flash

SystemGB300 NVL72EngineSGLang (dev-dsv41)ServingAggregated in-flight batching - one SGLang server on all 4 GPUs of one GB300 trayTensor parallelTP4Expert parallelEP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)CUDA graphsDecode graphs up to batch 64PrecisionMXFP4Accelerators used4 of 72

Submitted by NVIDIA

B300 x8

# DeepSeek V4.1 Flash (MXFP4) on B300 x8 — vLLM 0.31.0, TP=2 on 2 of the node's 8 GPUs, Rust frontend.
# vLLM recipes B300 TP2 settings (FLASHINFER_MLA_SPARSE_DSV41 attention, MXFP4 indexer with sparse logits, FP8 KV cache,
# Engram CPU offload, B300 compile and CUDA graph sizes, --max-num-seqs 256, --max-num-batched-tokens 8192), plus text-only mode
# and DSpark speculative decoding (5 tokens, adaptive verification on).
# Weights: deepseek-ai/DeepSeek-V4.1-Flash @ dba1be0a40aa45a94ad051997016db3960a90277
# Image: vllm/vllm-openai:v0.31.0@sha256:c1c9f6fd5c109ba7f0546a59f5b2f15fb87f64c77782e90a27b648b42a8e67c3
# Server pinned to CPU cores on its GPUs' socket.
VLLM_ENGINE_READY_TIMEOUT_S=3600 \
VLLM_USE_RUST_FRONTEND=1 \
vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
  --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
  --host 0.0.0.0 \
  --port 30000 \
  --tensor-parallel-size 2 \
  --trust-remote-code \
  --tokenizer-mode deepseek_v41 \
  --tool-call-parser deepseek_v41 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v41 \
  --language-model-only \
  --enable-prompt-tokens-details \
  --disable-uvicorn-access-log \
  --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' \
  --kv-cache-dtype fp8 \
  --engram-config '{"cpu_offload":true}' \
  --compilation-config '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}' \
  --max-cudagraph-capture-size 8190 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 256 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}'

SystemB300 x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 2 of the node's 8 GPUs (other servers on the rest)Tensor parallelTP2Speculative decodingDSpark (bundled draft module, 5 speculative tokens, probabilistic drafting)Engram tablesHost memory (pinned, read through UVA)CPU bindingserver pinned to its GPUs' socket cores (taskset) with memory bound to that NUMA node; client on its own coresPrecisionMXFP4Accelerators used2 of 8

Submitted by Artificial Analysis

# DeepSeek V4.1 Flash (MXFP4) on B300 x8 — vLLM 0.31.0, TP=8 on all 8 GPUs with vLLM NUMA binding, Rust frontend.
# vLLM recipes B300 settings at TP8 (FLASHINFER_MLA_SPARSE_DSV41 attention, MXFP4 indexer with sparse logits, FP8 KV cache,
# Engram CPU offload, B300 compile and CUDA graph sizes, --max-num-seqs 256, --max-num-batched-tokens 8192), plus text-only mode
# and DSpark speculative decoding (5 tokens, adaptive verification on).
# Weights: deepseek-ai/DeepSeek-V4.1-Flash @ dba1be0a40aa45a94ad051997016db3960a90277
# Image: vllm/vllm-openai:v0.31.0@sha256:c1c9f6fd5c109ba7f0546a59f5b2f15fb87f64c77782e90a27b648b42a8e67c3
VLLM_ENGINE_READY_TIMEOUT_S=3600 \
VLLM_USE_RUST_FRONTEND=1 \
vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
  --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
  --host 0.0.0.0 \
  --port 30000 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --tokenizer-mode deepseek_v41 \
  --tool-call-parser deepseek_v41 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v41 \
  --language-model-only \
  --enable-prompt-tokens-details \
  --disable-uvicorn-access-log \
  --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' \
  --kv-cache-dtype fp8 \
  --engram-config '{"cpu_offload":true}' \
  --compilation-config '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}' \
  --max-cudagraph-capture-size 8190 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 256 \
  --numa-bind \
  --numa-bind-nodes 0 0 0 0 1 1 1 1 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}'

SystemB300 x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (bundled draft module, 5 speculative tokens, probabilistic drafting)Engram tablesHost memory (pinned, read through UVA)PrecisionMXFP4Accelerators used8 of 8

Submitted by Artificial Analysis

GB200 NVL72

# DeepSeek V4.1 Flash (MXFP4) on GB200 NVL72 — vLLM 0.31.0 in-flight batching on 2 of a tray's 4 GPUs, TP2,
# DSpark speculative decoding (5 draft tokens, probabilistic drafts, block rejection, adaptive verification off),
# FlashInfer sparse MLA attention with an MXFP4 indexer, FP8 KV cache, Engram tables in host memory,
# Rust frontend.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash
# Image: vllm-v0310.sqsh (vLLM 0.31.0, aarch64)
# Server pinned to the Grace CPU socket attached to its GPUs.
export CUDA_VISIBLE_DEVICES=0,1
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export VLLM_USE_RUST_FRONTEND=1
vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
  --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --trust-remote-code \
  --tokenizer-mode deepseek_v41 \
  --tool-call-parser deepseek_v41 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v41 \
  --language-model-only \
  --enable-prompt-tokens-details \
  --disable-uvicorn-access-log \
  --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' \
  --kv-cache-dtype fp8 \
  --engram-config '{"cpu_offload":true}' \
  --max-num-seqs 128 \
  --max-model-len 262144 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'

SystemGB200 NVL72EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 2 of one GB200 tray's 4 GPUs (power = those 2 GPUs only)Tensor parallelTP2Speculative decodingDSpark (5 draft tokens, probabilistic drafts, block rejection, adaptive verification off)Engram tablesHost memory (--engram-config cpu_offload)PrecisionMXFP4Accelerators used2 of 72

Submitted by NVIDIA

# DeepSeek V4.1 Flash (MXFP4) on GB200 NVL72 — SGLang in-flight batching on one 4-GPU tray, TP4 + EP4,
# DSPARK speculative decoding (block size 5), decode CUDA graphs up to batch 128, cache reporting on.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash
# Image: sglang-dsv41.sqsh (SGLang 0.0.0.dev1+gda64c5cbb, aarch64)
export SGLANG_JIT_DEEPGEMM_PRECOMPILE=0
export SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1
export SGLANG_DEFAULT_THINKING=true
export PYTORCH_CUDA_ALLOC_CONF=
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4.1-Flash \
  --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
  --trust-remote-code \
  --tp 4 \
  --ep-size 4 \
  --page-size 256 \
  --mem-fraction-static 0.8 \
  --max-running-requests 256 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 5 \
  --cuda-graph-max-bs-decode 128 \
  --enable-cache-report \
  --reasoning-parser auto \
  --tool-call-parser auto \
  --watchdog-timeout 3600 \
  --host 0.0.0.0 \
  --port 30000

SystemGB200 NVL72EngineSGLang (da64c5cbb)ServingAggregated in-flight batching - one SGLang server on all 4 GPUs of one GB200 trayTensor parallelTP4Expert parallelEP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)CUDA graphsDecode graphs up to batch 128PrecisionMXFP4Accelerators used4 of 72

Submitted by NVIDIA

B200 (SXM) x8

# DeepSeek V4.1 Flash (MXFP4) on B200 (SXM) x8 — vLLM 0.31.0, TP2 on 2 of the node's 8 GPUs, DSpark speculative decoding
# (5 tokens, probabilistic draft, block verification; adaptive verification off, as it crashes vLLM DSpark at high concurrency on
# B300/GB300). Otherwise vLLM recipes B200 TP2 settings (FP8 KV cache, Engram tables in host memory, max-num-seqs 128)
# with Rust frontend, uvicorn access log off, max-model-len 262144.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash @ dba1be0a40aa45a94ad051997016db3960a90277
# Image: vllm/vllm-openai:v0.31.0 (vLLM 0.31.0, CUDA 13.0)
# Server pinned to the GPUs' NUMA node.
docker run -d \
  --gpus '"device=<2 GPUs>"' \
  --ipc=host \
  --shm-size=32g \
  --network host \
  --ulimit memlock=-1 \
  -v <host-path>:/hub:ro \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_USE_RUST_FRONTEND=1 \
  --entrypoint vllm \
  vllm/vllm-openai:v0.31.0 \
  serve deepseek-ai/DeepSeek-V4.1-Flash \
    --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
    --host 127.0.0.1 \
    --port 30014 \
    --tensor-parallel-size 2 \
    --trust-remote-code \
    --tokenizer-mode deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v41 \
    --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' \
    --kv-cache-dtype fp8 \
    --engram-config '{"cpu_offload":true}' \
    --max-model-len 262144 \
    --enable-prompt-tokens-details \
    --disable-uvicorn-access-log \
    --max-num-seqs 128 \
    --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'

Host-specific paths are redacted as <host-path>.

SystemB200 (SXM) x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 2 of the node's 8 GPUsTensor parallelTP2Speculative decodingDSpark (bundled mtp.0-2 head, 5 tokens, block verify, adaptive verification off)Engram tablesHost-memory Engram tables (--engram-config cpu_offload, recipe B200 setting)PrecisionMXFP4Accelerators used2 of 8

Submitted by Artificial Analysis

# DeepSeek V4.1 Flash (MXFP4) on B200 (SXM) x8 — vLLM 0.31.0, TP4 on 4 of the node's 8 GPUs, DSpark speculative decoding
# (5 tokens, probabilistic draft, block verification, adaptive verification on). vLLM recipes B200 settings (FP8 KV cache,
# Engram tables in host memory, max-num-seqs 128) with Rust frontend, uvicorn access log off, max-model-len 262144.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash @ dba1be0a40aa45a94ad051997016db3960a90277
# Image: vllm/vllm-openai:v0.31.0 (vLLM 0.31.0, CUDA 13.0)
# Server pinned to the GPUs' NUMA node.
docker run -d \
  --gpus '"device=<4 GPUs>"' \
  --ipc=host \
  --shm-size=32g \
  --network host \
  --ulimit memlock=-1 \
  -v <host-path>:/hub:ro \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_USE_RUST_FRONTEND=1 \
  --entrypoint vllm \
  vllm/vllm-openai:v0.31.0 \
  serve deepseek-ai/DeepSeek-V4.1-Flash \
    --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
    --host 127.0.0.1 \
    --port 30001 \
    --tensor-parallel-size 4 \
    --trust-remote-code \
    --tokenizer-mode deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v41 \
    --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' \
    --kv-cache-dtype fp8 \
    --engram-config '{"cpu_offload":true}' \
    --max-model-len 262144 \
    --enable-prompt-tokens-details \
    --disable-uvicorn-access-log \
    --max-num-seqs 128 \
    --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}'

Host-specific paths are redacted as <host-path>.

SystemB200 (SXM) x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUsTensor parallelTP4Speculative decodingDSpark (bundled mtp.0-2 head, 5 tokens, block verify, adaptive verification on)Engram tablesHost-memory Engram tables (--engram-config cpu_offload, recipe B200 setting)PrecisionMXFP4Accelerators used4 of 8

Submitted by Artificial Analysis

H200 (SXM) x8

# DeepSeek V4.1 Flash (MXFP4) on H200 (SXM) x8 — vLLM, TP4 on 4 of the node's 8 GPUs, no speculative decoding
# (Engram offloaded to host memory, the vLLM default), Rust frontend, uvicorn access log off, max model length 262144.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash @ dba1be0a40aa45a94ad051997016db3960a90277
# Image: vllm/vllm-openai:deepseekv41-flash-0909 (vLLM 0.1.dev20904+g179dd0fa9)
# Server pinned to the GPUs' NUMA node.
docker run -d \
  --gpus '"device=<4 GPUs>"' \
  --ipc=host --shm-size=32g --network host \
  --ulimit memlock=-1 \
  -v <host-path>:/hub:ro \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_USE_RUST_FRONTEND=1 \
  --entrypoint vllm \
  vllm/vllm-openai:deepseekv41-flash-0909 \
  serve deepseek-ai/DeepSeek-V4.1-Flash \
    --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
    --host 127.0.0.1 \
    --port 30002 \
    --tensor-parallel-size 4 \
    --trust-remote-code \
    --tokenizer-mode deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v41 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 262144 \
    --enable-prompt-tokens-details \
    --disable-uvicorn-access-log

Host-specific paths are redacted as <host-path>.

SystemH200 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUsTensor parallelTP4PrecisionMXFP4Accelerators used4 of 8

Submitted by Artificial Analysis

# DeepSeek V4.1 Flash (MXFP4) on H200 (SXM) x8 — vLLM, TP4 on 4 of the node's 8 GPUs, DSpark speculative decoding
# (5 tokens, block rejection sampling, adaptive verification).
# --max-model-len 262144 and --gpu-memory-utilization 0.92: at the default 1M context the draft left no KV blocks.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash @ dba1be0a40aa45a94ad051997016db3960a90277
# Image: vllm/vllm-openai:deepseekv41-flash-0909 (vLLM 0.1.dev20904+g179dd0fa9)
docker run -d \
  --gpus '"device=<4 GPUs>"' \
  --ipc=host --shm-size=32g --network host \
  --ulimit memlock=-1 \
  -v <host-path>:/hub:ro \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  --entrypoint vllm \
  vllm/vllm-openai:deepseekv41-flash-0909 \
  serve deepseek-ai/DeepSeek-V4.1-Flash \
    --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
    --host 127.0.0.1 \
    --port 30002 \
    --tensor-parallel-size 4 \
    --trust-remote-code \
    --tokenizer-mode deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v41 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 262144 \
    --enable-prompt-tokens-details \
    --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}'

Host-specific paths are redacted as <host-path>.

SystemH200 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUsTensor parallelTP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)PrecisionMXFP4Accelerators used4 of 8

Submitted by Artificial Analysis

# DeepSeek V4.1 Flash (MXFP4) on H200 (SXM) x8 — vLLM, TP8 on all 8 GPUs, DSpark speculative decoding
# (5 tokens, block rejection sampling, adaptive verification), Rust frontend, uvicorn access log off.
# --max-model-len 262144 and --gpu-memory-utilization 0.92: at the default 1M context the draft left no KV blocks.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash @ dba1be0a40aa45a94ad051997016db3960a90277
# Image: vllm/vllm-openai:deepseekv41-flash-0909 (vLLM 0.1.dev20904+g179dd0fa9)
docker run -d \
  --gpus '"device=<8 GPUs>"' \
  --ipc=host --shm-size=32g --network host \
  --ulimit memlock=-1 \
  -v <host-path>:/hub:ro \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_USE_RUST_FRONTEND=1 \
  --entrypoint vllm \
  vllm/vllm-openai:deepseekv41-flash-0909 \
  serve deepseek-ai/DeepSeek-V4.1-Flash \
    --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
    --host 127.0.0.1 \
    --port 30000 \
    --tensor-parallel-size 8 \
    --trust-remote-code \
    --tokenizer-mode deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v41 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 262144 \
    --enable-prompt-tokens-details \
    --disable-uvicorn-access-log \
    --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}'

Host-specific paths are redacted as <host-path>.

SystemH200 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)PrecisionMXFP4Accelerators used8 of 8

Submitted by Artificial Analysis

H100 (SXM) x8

# DeepSeek V4.1 Flash (MXFP4) on H100 (SXM) x8 — vLLM (deepseekv41-flash-0909 image), TP8 on all 8 GPUs,
# no speculative decoding. Rust frontend, uvicorn access log off, max-model-len 262144.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash @ dba1be0a40aa45a94ad051997016db3960a90277
# Image: vllm/vllm-openai:deepseekv41-flash-0909 (vLLM 0.1.dev20904+g179dd0fa9)
docker run -d \
  --gpus '"device=<8 GPUs>"' \
  --ipc=host \
  --shm-size=32g \
  --network host \
  --ulimit memlock=-1 \
  -v <host-path>:/hub:ro \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_USE_RUST_FRONTEND=1 \
  --entrypoint vllm \
  vllm/vllm-openai:deepseekv41-flash-0909 \
  serve deepseek-ai/DeepSeek-V4.1-Flash \
    --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
    --host 127.0.0.1 \
    --port 30000 \
    --tensor-parallel-size 8 \
    --trust-remote-code \
    --tokenizer-mode deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v41 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 262144 \
    --enable-prompt-tokens-details \
    --disable-uvicorn-access-log

Host-specific paths are redacted as <host-path>.

SystemH100 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8PrecisionMXFP4Accelerators used8 of 8

Submitted by Artificial Analysis

# DeepSeek V4.1 Flash (MXFP4) on H100 (SXM) x8 — vLLM (deepseekv41-flash-0909 image), TP8 on all 8 GPUs,
# DSpark speculative decoding (5 tokens, probabilistic draft, block verification, adaptive verification on).
# Rust frontend, uvicorn access log off, vLLM --numa-bind, max-model-len 262144; Engram tables in host memory (vLLM default).
# --gpu-memory-utilization 0.95 and --max-num-seqs 64: at 0.9, DSpark CUDA-graph profiling leaves no KV cache blocks.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash @ dba1be0a40aa45a94ad051997016db3960a90277
# Image: vllm/vllm-openai:deepseekv41-flash-0909 (vLLM 0.1.dev20904+g179dd0fa9)
docker run -d \
  --gpus '"device=<8 GPUs>"' \
  --ipc=host \
  --shm-size=32g \
  --network host \
  --ulimit memlock=-1 \
  -v <host-path>:/hub:ro \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_USE_RUST_FRONTEND=1 \
  --entrypoint vllm \
  vllm/vllm-openai:deepseekv41-flash-0909 \
  serve deepseek-ai/DeepSeek-V4.1-Flash \
    --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
    --host 127.0.0.1 \
    --port 30000 \
    --tensor-parallel-size 8 \
    --trust-remote-code \
    --tokenizer-mode deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v41 \
    --gpu-memory-utilization 0.95 \
    --max-model-len 262144 \
    --max-num-seqs 64 \
    --enable-prompt-tokens-details \
    --disable-uvicorn-access-log \
    --numa-bind \
    --numa-bind-nodes 0 0 0 0 1 1 1 1 \
    --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}'

Host-specific paths are redacted as <host-path>.

SystemH100 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)PrecisionMXFP4Accelerators used8 of 8

Submitted by Artificial Analysis

MI355X x8

# DeepSeek V4.1 Flash (MXFP4) on MI355X x8 — vLLM ROCm nightly, TP2 on GPUs 0-1, DSpark speculative decoding
# (5 tokens, probabilistic draft, block verification; adaptive verification off, not supported on ROCm).
# vLLM recipes DeepSeek V4.1 Flash MI355X settings: AITER MoE, MXFP4 sparse indexer, block size 128, breakable CUDA
# graphs, --no-swa-bounded-replay, Engram tables in pinned host memory; Rust frontend, max-model-len 262144, max-num-seqs 256.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash @ dba1be0a40aa45a94ad051997016db3960a90277
# Image: vllm/vllm-openai-rocm:nightly-rocm100-bb87d227d4b964abb2a966cdf9194f3d376d9bbe (vLLM 0.31.1rc1.dev1+gbb87d227d, ROCm 10.0.0)
# Server pinned to the GPUs' NUMA node.
docker run -d \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add 44 \
  --group-add 992 \
  --security-opt seccomp=unconfined \
  --ipc=host \
  --shm-size=32g \
  --network host \
  --privileged \
  --ulimit memlock=-1 \
  --ulimit nofile=1048576:1048576 \
  -v <host-path>:/cache \
  -v <host-path>:/root/.cache/vllm \
  -e HF_HOME=/cache \
  -e HF_HUB_OFFLINE=1 \
  -e VLLM_ROCM_USE_AITER=1 \
  -e VLLM_ROCM_USE_AITER_MOE=1 \
  -e AITER_TRITON_LOG_LEVEL=ERROR \
  -e VLLM_USE_BREAKABLE_CUDAGRAPH=1 \
  -e VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=1 \
  -e VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \
  -e OMP_NUM_THREADS=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_USE_RUST_FRONTEND=1 \
  -e ROCR_VISIBLE_DEVICES=0,1 \
  -e HIP_VISIBLE_DEVICES=0,1 \
  --entrypoint bash \
  vllm/vllm-openai-rocm:nightly-rocm100-bb87d227d4b964abb2a966cdf9194f3d376d9bbe \
  -c "exec vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
    --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
    --host 127.0.0.1 \
    --port 30000 \
    --language-model-only \
    --tokenizer-mode deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v41 \
    --moe-backend aiter \
    --gpu-memory-utilization 0.9 \
    --no-swa-bounded-replay \
    --attention-config '{\"indexer_kv_dtype\":\"mxfp4\",\"indexer_sparse_logits\":true}' \
    --block-size 128 \
    --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":5,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":false}' \
    --max-model-len 262144 \
    --max-num-batched-tokens 16384 \
    --enable-prompt-tokens-details \
    --disable-uvicorn-access-log \
    --tensor-parallel-size 2 \
    --engram-config '{\"cpu_offload\":true}' \
    --max-num-seqs 256 \
    --max-cudagraph-capture-size 1536"

Host-specific paths are redacted as <host-path>.

SystemMI355X x8EnginevLLM (nightly 10-06 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on 2 of the node's 8 GPUs (server CPUs pinned to that NUMA node, client on the other; power = those GPUs only)Tensor parallelTP2Speculative decodingDSpark (bundled mtp.0-2 head, block size 5, full-block verification)Engram tablesHost-memory Engram tables (vLLM --engram-config cpu_offload)PrecisionMXFP4Accelerators used2 of 8

Submitted by Artificial Analysis

# DeepSeek V4.1 Flash (MXFP4) on MI355X x8 — vLLM ROCm nightly, TP2 on GPUs 0-1, DSpark speculative decoding
# (5 tokens, probabilistic draft, block verification; adaptive verification off, not supported on ROCm).
# vLLM recipes DeepSeek V4.1 Flash MI355X settings: AITER MoE, MXFP4 sparse indexer, block size 128, breakable CUDA
# graphs, --no-swa-bounded-replay, Engram tables in pinned host memory; Rust frontend, max-model-len 262144, max-num-seqs 64.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash @ dba1be0a40aa45a94ad051997016db3960a90277
# Image: vllm/vllm-openai-rocm:nightly-rocm100-bb87d227d4b964abb2a966cdf9194f3d376d9bbe (vLLM 0.31.1rc1.dev1+gbb87d227d, ROCm 10.0.0)
# Server pinned to the GPUs' NUMA node.
docker run -d \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add 44 \
  --group-add 992 \
  --security-opt seccomp=unconfined \
  --ipc=host \
  --shm-size=32g \
  --network host \
  --privileged \
  --ulimit memlock=-1 \
  --ulimit nofile=1048576:1048576 \
  -v <host-path>:/cache \
  -v <host-path>:/root/.cache/vllm \
  -e HF_HOME=/cache \
  -e HF_HUB_OFFLINE=1 \
  -e VLLM_ROCM_USE_AITER=1 \
  -e VLLM_ROCM_USE_AITER_MOE=1 \
  -e AITER_TRITON_LOG_LEVEL=ERROR \
  -e VLLM_USE_BREAKABLE_CUDAGRAPH=1 \
  -e VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=1 \
  -e VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \
  -e OMP_NUM_THREADS=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_USE_RUST_FRONTEND=1 \
  -e ROCR_VISIBLE_DEVICES=0,1 \
  -e HIP_VISIBLE_DEVICES=0,1 \
  --entrypoint bash \
  vllm/vllm-openai-rocm:nightly-rocm100-bb87d227d4b964abb2a966cdf9194f3d376d9bbe \
  -c "exec vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
    --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
    --host 127.0.0.1 \
    --port 30000 \
    --language-model-only \
    --tokenizer-mode deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v41 \
    --moe-backend aiter \
    --gpu-memory-utilization 0.9 \
    --no-swa-bounded-replay \
    --attention-config '{\"indexer_kv_dtype\":\"mxfp4\",\"indexer_sparse_logits\":true}' \
    --block-size 128 \
    --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":5,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":false}' \
    --max-model-len 262144 \
    --max-num-batched-tokens 16384 \
    --enable-prompt-tokens-details \
    --disable-uvicorn-access-log \
    --tensor-parallel-size 2 \
    --engram-config '{\"cpu_offload\":true}' \
    --max-num-seqs 64 \
    --max-cudagraph-capture-size 384"

Host-specific paths are redacted as <host-path>.

Submitted by Artificial Analysis

# DeepSeek V4.1 Flash (MXFP4) on MI355X x8 — vLLM ROCm nightly, TP4 on GPUs 0-3, DSpark speculative decoding
# (5 tokens, probabilistic draft, block verification; adaptive verification off, not supported on ROCm).
# vLLM recipes (vllm-project/recipes @ 21302fdc75) MI355X settings: mono decode layer, AITER MoE, MXFP4 sparse indexer,
# breakable CUDA graphs, INT4 quick reduce, Engram tables on GPU (the recipe's TP4 form); Rust frontend, max-model-len 262144.
# Weights: deepseek-ai/DeepSeek-V4.1-Flash @ dba1be0a40aa45a94ad051997016db3960a90277
# Image: vllm/vllm-openai-rocm:nightly-rocm100-fe4df3df0eb703862c686b256e9fe44fa94c8945 (vLLM 0.31.1rc1.dev296+gfe4df3df0, ROCm 10.0.0)
# Server pinned to the GPUs' NUMA node.
docker run -d \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add 44 \
  --group-add 992 \
  --security-opt seccomp=unconfined \
  --ipc=host \
  --shm-size=32g \
  --network host \
  --privileged \
  --ulimit memlock=-1 \
  --ulimit nofile=1048576:1048576 \
  -v <host-path>:/cache \
  -v <host-path>:/root/.cache/vllm \
  -e HF_HOME=/cache \
  -e HF_HUB_OFFLINE=1 \
  -e VLLM_ROCM_USE_AITER=1 \
  -e VLLM_ROCM_USE_AITER_MOE=1 \
  -e AITER_TRITON_LOG_LEVEL=ERROR \
  -e VLLM_USE_BREAKABLE_CUDAGRAPH=1 \
  -e VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=1 \
  -e VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \
  -e OMP_NUM_THREADS=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_USE_RUST_FRONTEND=1 \
  -e VLLM_ROCM_MONO_DECODE=1 \
  -e ROCR_VISIBLE_DEVICES=0,1,2,3 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3 \
  --entrypoint bash \
  vllm/vllm-openai-rocm:nightly-rocm100-fe4df3df0eb703862c686b256e9fe44fa94c8945 \
  -c "exec vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
    --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
    --host 127.0.0.1 \
    --port 30000 \
    --language-model-only \
    --tokenizer-mode deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v41 \
    --moe-backend aiter \
    --gpu-memory-utilization 0.9 \
    --no-swa-bounded-replay \
    --attention-config '{\"indexer_kv_dtype\":\"mxfp4\",\"indexer_sparse_logits\":true}' \
    --block-size 128 \
    --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":5,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":false}' \
    --max-model-len 262144 \
    --max-num-batched-tokens 16384 \
    --enable-prompt-tokens-details \
    --disable-uvicorn-access-log \
    --tensor-parallel-size 4 \
    --engram-config '{\"cpu_offload\":false}' \
    --max-num-seqs 64 \
    --max-cudagraph-capture-size 384"

Host-specific paths are redacted as <host-path>.

SystemMI355X x8EnginevLLM (nightly 10-12 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUs (server CPUs pinned to that NUMA node, client on the other; power = those GPUs only)Tensor parallelTP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 5, full-block verification)PrecisionMXFP4Accelerators used4 of 8

Submitted by Artificial Analysis

Inference Serving Configurations

AA-AgentPerf

DeepSeek V4.1 Flash — featured configurations

Vera Rubin NVL72

Peak throughputvLLMMXFP48 of 722 × 2 GPUs · TP2 · 1 × 4 GPUs · EP4 · DP4DSpark15 October 2026Peak throughputvLLMMXFP42 × 2 GPUs · TP2 · 1 × 4 GPUs · EP4 · DP4 · 15 October 2026

Best at 100+ tok/svLLMMXFP46 of 721 × 2 GPUs · TP2 · 1 × 4 GPUs · EP4 · DP4DSpark15 October 2026Best at 100+ tok/svLLMMXFP41 × 2 GPUs · TP2 · 1 × 4 GPUs · EP4 · DP4 · 15 October 2026

Fastest single agentvLLMMXFP44 of 72TP4DSpark15 October 2026Fastest single agentvLLMMXFP4TP4 · 15 October 2026

GB300 NVL72

Peak throughputvLLMNVFP42 of 72TP2DSpark7 October 2026Peak throughputvLLMNVFP4TP2 · 7 October 2026

Best at 100+ tok/svLLMMXFP44 of 721 × 2 GPUs · TP2 · 1 × 2 GPUs · TP2DSpark7 October 2026Best at 100+ tok/svLLMMXFP41 × 2 GPUs · TP2 · 1 × 2 GPUs · TP2 · 7 October 2026

Fastest single agentSGLangMXFP44 of 72TP4 · EP4DSpark7 October 2026Fastest single agentSGLangMXFP4TP4 · EP4 · 7 October 2026

B300 x8

Peak throughputBest at 100+ tok/svLLMMXFP42 of 8TP2DSpark7 October 2026Peak throughputBest at 100+ tok/svLLMMXFP4TP2 · 7 October 2026

Fastest single agentvLLMMXFP48 of 8TP8DSpark7 October 2026Fastest single agentvLLMMXFP4TP8 · 7 October 2026

GB200 NVL72

Peak throughputBest at 100+ tok/svLLMMXFP42 of 72TP2DSpark10 October 2026Peak throughputBest at 100+ tok/svLLMMXFP4TP2 · 10 October 2026

Fastest single agentSGLangMXFP44 of 72TP4 · EP4DSpark7 October 2026Fastest single agentSGLangMXFP4TP4 · EP4 · 7 October 2026

B200 (SXM) x8

Peak throughputBest at 100+ tok/svLLMMXFP42 of 8TP2DSpark8 October 2026Peak throughputBest at 100+ tok/svLLMMXFP4TP2 · 8 October 2026

Fastest single agentvLLMMXFP44 of 8TP4DSpark8 October 2026Fastest single agentvLLMMXFP4TP4 · 8 October 2026

H200 (SXM) x8

Peak throughputvLLMMXFP44 of 8TP4—1 October 2026Peak throughputvLLMMXFP4TP4 · 1 October 2026

Best at 100+ tok/svLLMMXFP44 of 8TP4DSpark1 October 2026Best at 100+ tok/svLLMMXFP4TP4 · 1 October 2026

Fastest single agentvLLMMXFP48 of 8TP8DSpark1 October 2026Fastest single agentvLLMMXFP4TP8 · 1 October 2026

H100 (SXM) x8

Peak throughputvLLMMXFP48 of 8TP8—1 October 2026Peak throughputvLLMMXFP4TP8 · 1 October 2026

Best at 100+ tok/sFastest single agentvLLMMXFP48 of 8TP8DSpark1 October 2026Best at 100+ tok/sFastest single agentvLLMMXFP4TP8 · 1 October 2026

MI355X x8

Peak throughputvLLMMXFP42 of 8TP2DSpark7 October 2026Peak throughputvLLMMXFP4TP2 · 7 October 2026

Best at 100+ tok/svLLMMXFP42 of 8TP2DSpark7 October 2026Best at 100+ tok/svLLMMXFP4TP2 · 7 October 2026

Fastest single agentvLLMMXFP44 of 8TP4DSpark12 October 2026Fastest single agentvLLMMXFP4TP4 · 12 October 2026