Inference Serving Configurations
Every serving configuration used in AA-AgentPerf hardware benchmarking, from June 2026 onwards. Each entry captures the model, accelerator system, precision, inference framework, parallelism topology, and full launch command.
AA-AgentPerf
Serving configurations from AA-AgentPerf benchmark runs, with the full launch command for each. Read the methodology
Showing 234 of 234 configurations
DeepSeek V4.1 Flash (max) — 39 configurations
SystemFrameworkPrecisionAccelerators UsedParallelismSpecdecDate Added
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 10-12 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on 2 of the node's 8 GPUs (server CPUs pinned to that NUMA node, client on the other; power = those GPUs only)Tensor parallelTP2Speculative decodingDSpark (bundled mtp.0-2 head, block size 5, full-block verification)Engram tablesHost-memory Engram tables (vLLM --engram-config cpu_offload)PrecisionMXFP4Accelerators used2 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 10-12 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUs (server CPUs pinned to that NUMA node, client on the other; power = those GPUs only)Tensor parallelTP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 5, full-block verification)PrecisionMXFP4Accelerators used4 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemGB200 NVL72EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on all 4 GPUs of one GB200 trayTensor parallelTP4Speculative decodingDSpark (5 draft tokens, probabilistic drafts, block rejection, adaptive verification on)Engram tablesHost memory (--engram-config cpu_offload)CPU binding--numa-bind 0 0 1 1; server on the tray's Grace CPUs, client on its own nodePrecisionMXFP4Accelerators used4 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB200 NVL72EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 2 of one GB200 tray's 4 GPUs (power = those 2 GPUs only)Tensor parallelTP2Speculative decodingDSpark (5 draft tokens, probabilistic drafts, block rejection, adaptive verification on)Engram tablesHost memory (--engram-config cpu_offload)PrecisionMXFP4Accelerators used2 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB200 NVL72EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 2 of one GB200 tray's 4 GPUs (power = those 2 GPUs only)Tensor parallelTP2Speculative decodingDSpark (5 draft tokens, probabilistic drafts, block rejection, adaptive verification off)Engram tablesHost memory (--engram-config cpu_offload)PrecisionMXFP4Accelerators used2 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemB200 (SXM) x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 2 of the node's 8 GPUsTensor parallelTP2Speculative decodingDSpark (bundled mtp.0-2 head, 5 tokens, block verify, adaptive verification off)Engram tablesHost-memory Engram tables (--engram-config cpu_offload, recipe B200 setting)PrecisionMXFP4Accelerators used2 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB200 (SXM) x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUsTensor parallelTP4Speculative decodingDSpark (bundled mtp.0-2 head, 5 tokens, block verify, adaptive verification on)Engram tablesHost-memory Engram tables (--engram-config cpu_offload, recipe B200 setting)PrecisionMXFP4Accelerators used4 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB200 (SXM) x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUsTensor parallelTP4Speculative decodingDSpark (bundled mtp.0-2 head, 5 tokens, block verify, adaptive verification off)Engram tablesHost-memory Engram tables (--engram-config cpu_offload, recipe B200 setting)PrecisionMXFP4Accelerators used4 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB200 (SXM) x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 2 of the node's 8 GPUsTensor parallelTP2Speculative decodingDSpark (bundled mtp.0-2 head, 5 tokens, block verify, adaptive verification on)Engram tablesHost-memory Engram tables (--engram-config cpu_offload, recipe B200 setting)PrecisionMXFP4Accelerators used2 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemGB300 NVL72EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on all 4 GPUs of one trayTensor parallelTP4Speculative decodingDSpark (bundled draft module, 5 speculative tokens, adaptive verification on)Engram tablesHost memory (pinned Grace memory, read through UVA)CPU bindingvLLM --numa-bind: each GPU's workers on its own Grace socketPrecisionMXFP4Accelerators used4 of 72
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemGB300 NVL72EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 2 of the tray's 4 GPUs (the others idle, not charged)Tensor parallelTP2Speculative decodingDSpark (bundled draft module, 5 speculative tokens, adaptive verification on)Engram tablesHost memory (pinned Grace memory, read through UVA)CPU bindingserver pinned to its GPUs' Grace socket (taskset), client on the other socketPrecisionMXFP4Accelerators used2 of 72
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB300 x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 2 of the node's 8 GPUs (other servers on the rest)Tensor parallelTP2Speculative decodingDSpark (bundled draft module, 5 speculative tokens, adaptive verification off)Engram tablesHost memory (pinned, read through UVA)CPU bindingserver pinned to its GPUs' socket cores (taskset) with memory bound to that NUMA node; client on its own coresPrecisionMXFP4Accelerators used2 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB300 x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUs (other servers on the rest)Tensor parallelTP4Speculative decodingDSpark (bundled draft module, 5 speculative tokens, probabilistic drafting)Engram tablesHost memory (pinned, read through UVA)CPU bindingserver pinned to its GPUs' socket cores (taskset) with memory bound to that NUMA node; client on its own coresPrecisionMXFP4Accelerators used4 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB300 x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 2 of the node's 8 GPUs (other servers on the rest)Tensor parallelTP2Speculative decodingDSpark (bundled draft module, 5 speculative tokens, probabilistic drafting)Engram tablesHost memory (pinned, read through UVA)CPU bindingserver pinned to its GPUs' socket cores (taskset) with memory bound to that NUMA node; client on its own coresPrecisionMXFP4Accelerators used2 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB300 x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (bundled draft module, 5 speculative tokens, probabilistic drafting)Engram tablesHost memory (pinned, read through UVA)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB300 x8EnginevLLM (v0.30.0)ServingAggregated in-flight batching - one vLLM server on 2 of the node's 8 GPUsTensor parallelTP2Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)PrecisionMXFP4Accelerators used2 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemGB200 NVL72EngineSGLang (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend + KV-aware router over SGLang workers, 8 GPUs on 2 GB200 trays (frontend on a 3rd tray, GPUs idle and excluded)Prefill1 × 4 GPUs · TP4 · EP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)Decode1 × 4 GPUs · TP4 · EP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)CUDA graphsDecode graphs up to batch 64KV transferMooncake (RDMA, InfiniBand mlx5_0,1,3,4)RouterKV-aware (prefill session affinity TTL 3600 s)PrecisionMXFP4Accelerators used8 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB200 NVL72EngineSGLang (da64c5cbb)ServingAggregated in-flight batching - one SGLang server on all 4 GPUs of one GB200 trayTensor parallelTP4Expert parallelEP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)CUDA graphsDecode graphs up to batch 128PrecisionMXFP4Accelerators used4 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB200 NVL72EngineSGLang (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend + KV-aware router over SGLang workers, 8 GPUs on 2 GB200 trays (frontend on a 3rd tray, GPUs idle and excluded)Prefill1 × 4 GPUs · TP4 · EP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 3)Decode1 × 4 GPUs · TP4 · EP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 3)KV transferMooncake (RDMA, InfiniBand mlx5_0,1,3,4)RouterKV-aware (prefill session affinity TTL 3600 s)PrecisionMXFP4Accelerators used8 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB200 NVL72EngineSGLang (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend + KV-aware router over SGLang workers, 8 GPUs on 2 GB200 trays (frontend on a 3rd tray, GPUs idle and excluded)Prefill1 × 4 GPUs · TP4 · EP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 4)Decode1 × 4 GPUs · TP4 · EP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 4)KV transferMooncake (RDMA, InfiniBand mlx5_0,1,3,4)RouterKV-aware (prefill session affinity TTL 3600 s)PrecisionMXFP4Accelerators used8 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EnginevLLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend + KV-aware router over vLLM workers, GPUs 0-1 on each of 2 GB300 trays (other GPUs of the allocation idle and excluded)Prefill1 × 2 GPUs · TP2Speculative decodingDSpark (5 draft tokens, probabilistic drafts, block rejection, adaptive verification)Decode1 × 2 GPUs · TP2Speculative decodingDSpark (5 draft tokens, probabilistic drafts, block rejection, adaptive verification)KV transferNIXL (UCX)RouterKV-aware (prefill session affinity TTL 3600 s)PrecisionMXFP4Accelerators used4 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EngineSGLang (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend + KV-aware router over SGLang workers, 12 GPUs on 3 GB300 trays (etcd + frontend on a 4th tray, GPUs idle and excluded)Prefill1 × 4 GPUs · TP4 · EP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)Hierarchical KV cacheHiCache (ratio 2, write-through)Decode2 × 4 GPUs · TP4 · EP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)KV transferMooncake (RDMA, InfiniBand mlx5_0-3)RouterKV-aware (prefill session affinity TTL 3600 s, conversation-aware routing)PrecisionMXFP4Accelerators used12 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EnginevLLM (v0.30.0 + patches)ServingAggregated in-flight batching - one vLLM server on 2 of the GB300 tray's 4 GPUsTensor parallelTP2Speculative decodingDSpark (5 draft tokens, probabilistic drafts, block rejection, adaptive verification)SM clock cap1200 MHz (memory 3996 MHz, default power limit)API servers4PrecisionNVFP4Accelerators used2 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EngineSGLang (dev-dsv41)ServingAggregated in-flight batching - one SGLang server on all 4 GPUs of one GB300 trayTensor parallelTP4Expert parallelEP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)CUDA graphsDecode graphs up to batch 64PrecisionMXFP4Accelerators used4 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 10-06 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on 2 of the node's 8 GPUs (server CPUs pinned to that NUMA node, client on the other; power = those GPUs only)Tensor parallelTP2Speculative decodingDSpark (bundled mtp.0-2 head, block size 5, full-block verification)Engram tablesHost-memory Engram tables (vLLM --engram-config cpu_offload)PrecisionMXFP4Accelerators used2 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 10-06 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on 2 of the node's 8 GPUs (server CPUs pinned to that NUMA node, client on the other; power = those GPUs only)Tensor parallelTP2Speculative decodingDSpark (bundled mtp.0-2 head, block size 5, full-block verification)Engram tablesHost-memory Engram tables (vLLM --engram-config cpu_offload)PrecisionMXFP4Accelerators used2 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 10-06 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on 2 of the node's 8 GPUs (server CPUs pinned to that NUMA node, client on the other; power = those GPUs only)Tensor parallelTP2Speculative decodingDSpark (bundled mtp.0-2 head, block size 5, full-block verification)Engram tablesHost-memory Engram tables (vLLM --engram-config cpu_offload)PrecisionMXFP4Accelerators used2 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH100 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUsTensor parallelTP4PrecisionMXFP4Accelerators used4 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH100 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH100 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH100 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUsTensor parallelTP4PrecisionMXFP4Accelerators used4 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUsTensor parallelTP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)PrecisionMXFP4Accelerators used4 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EnginevLLM (deepseekv41-flash-0909)ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUsTensor parallelTP4Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)PrecisionMXFP4Accelerators used4 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EngineSGLang (dev-dsv41-mi35x (ROCm 7.2, 09-10))Serving4 SGLang TP2 replicas (--dp 4) on one node, client conversation->replica affinityTensor parallelTP2Data parallelDP4 (replicas)Expert parallelEP2Speculative decodingDSpark (bundled 3-stage draft, block size 5)Engram tablesHost-memory Engram tables (SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1)KV pool cap--max-total-tokens 3145728 per replica (int32 compressed-KV offsets on this image)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB300 x8EngineSGLang (nightly (09-23))ServingTwo SGLang TP2 replicas (--dp 2) on 4 of a DGX B300's 8 GPUs, client conversation->replica affinityData parallelDP2 (replicas)Tensor parallelTP2Expert parallelEP2Speculative decodingDSpark (bundled mtp.0-2 head, block size 5)Engram tablesHost-memory Engram tables (SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1, cookbook opt-in)PrecisionNVFP4Accelerators used4 of 8
Submitted by Artificial Analysis
DeepSeek V4 Pro 0813 (max) — 26 configurations
SystemFrameworkPrecisionAccelerators UsedParallelismSpecdecDate Added
No launch command recorded for this configuration.
SystemB300 x8EngineSGLang (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend + KV-aware router (conversation affinity) over SGLang workers, 16 GPUs on 2 B300 nodesPrefill1 × 8 GPUs · TP8 · DP8 · EP8 (MegaMoE W4A4-MXFP4)Decode1 × 8 GPUs · TP8 · DP8 · EP8 (MegaMoE W4A4-MXFP4)Speculative decodingDSpark (bundled draft, block size 6)KV transferMooncake over 8 x IB NDRPrecisionMXFP4Accelerators used16 of 8
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemB300 x8EngineSGLang (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend + KV-aware router (conversation affinity) over SGLang workers, 16 GPUs on 2 B300 nodesPrefill1 × 8 GPUs · TP8 · DP8MoETP MoE (flashinfer_mxfp4)Decode1 × 8 GPUs · TP8 · DP8 · EP8 (MegaMoE W4A4-MXFP4)Speculative decodingDSpark (bundled draft, block size 6)KV transferMooncake over 8 x IB NDRPrecisionMXFP4Accelerators used16 of 8
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemMI355X x8EngineSGLang (0.5.21 dev (ROCm 7.2, 10-07))ServingAggregated in-flight batching - one SGLang server on all 8 GPUs of the nodeTensor parallelTP8PrefillPrefill context parallel 8 (interleave)Speculative decodingDSpark (bundled head, block size 6)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EngineSGLang (0.5.21 dev (ROCm 7.2, 10-07))ServingAggregated in-flight batching - one SGLang server on all 8 GPUs of the nodeTensor parallelTP8PrefillPrefill context parallel 8 (interleave)Speculative decodingDSpark (bundled head, block size 6)Hierarchical KV cacheHiCache host DRAM tier (ratio 1.5, write-through, direct IO)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB200 (SXM) x8EngineSGLang (0.5.21)ServingAggregated in-flight batching - one SGLang server on all 8 GPUs of the nodeData parallelDP8 attentionTensor parallelTP8Expert parallelEP8 (MegaMoE)Speculative decodingDSpark (bundled draft, block size 6)Hierarchical KV cacheHost DRAM KV tier, ratio 8, write_throughPrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB200 (SXM) x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (bundled draft head, 7 tokens)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB200 (SXM) x8EngineSGLang (0.5.21)ServingAggregated in-flight batching - one SGLang server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (bundled draft, block size 6)Hierarchical KV cacheHost DRAM KV tier, ratio 2.75, write_throughPrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB300 x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Expert parallelEP8Speculative decodingDSpark (bundled draft module, 7 speculative tokens, probabilistic drafting)PrecisionFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB300 x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (bundled draft module, 7 speculative tokens, probabilistic drafting)PrecisionFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemGB300 NVL72EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on all 4 GPUs of one trayTensor parallelTP4Expert parallelEP4Speculative decodingDSpark (bundled draft module, 7 speculative tokens, probabilistic drafting)CPU bindingvLLM --numa-bind: each GPU's workers on its own Grace socketPrecisionFP4Accelerators used4 of 72
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemGB300 NVL72EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on all 4 GPUs of one trayTensor parallelTP4Speculative decodingDSpark (bundled draft module, 7 speculative tokens, probabilistic drafting)CPU bindingvLLM --numa-bind: each GPU's workers on its own Grace socketPrecisionFP4Accelerators used4 of 72
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemGB300 NVL72EngineSGLang (0.5.21)ServingAggregated in-flight batching - one SGLang server on all 4 GPUs of one trayTensor parallelTP4Speculative decodingDSpark (bundled draft module, block size from the checkpoint)PrecisionFP4Accelerators used4 of 72
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EngineSGLang (0.5.21 dev (ROCm 7.2.4, 10-06))ServingAggregated in-flight batching - one SGLang server on all 8 GPUs of the nodeTensor parallelTP8Data parallelDP8 attentionSpeculative decodingDSpark (bundled head, block size 6)Hierarchical KV cacheHiCache host DRAM tier (ratio 1.5, write-through, direct IO)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EngineSGLang (0.5.21 dev (ROCm 7.2.4, 10-06))ServingAggregated in-flight batching - one SGLang server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (bundled head, block size 6)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EngineSGLang (0.5.19)ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingDSpark (draft head bundled in the checkpoint, block size 6)Max running requests32PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EngineSGLang (0.5.19)ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingDSpark (draft head bundled in the checkpoint, block size 6)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EngineSGLang (0.5.19 dev (ROCm 7.2, 09-14))ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingDSpark (bundled head, block size 6)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EngineSGLang (0.5.19 dev (ROCm 7.2, 09-14))ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingDSpark (bundled head, block size 6)Hierarchical KV cacheHiCache host DRAM tier (write-through, direct IO)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EngineSGLang (0.5.19 dev (ROCm 7.2, 09-14))ServingAggregated in-flight batching — one SGLang server, 8 GPUs, DP attention with client conversation->rank affinityTensor parallelTP8Data parallelDP8 attentionSpeculative decodingDSpark (bundled head, block size 6)Hierarchical KV cacheHiCache host DRAM tier (write-through, direct IO)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
# NVIDIA Dynamo (ff18f7210d99) + TensorRT-LLM (gb300_PatchPr-3d27c24b) # disaggregated serving of DeepSeek-V4-Pro (NVFP4 + DSpark draft) on GB300 NVL72 — 44 serving GPUs # across 11 nodes: 3x context workers (TP4 + MoE EP4, attention-DP, 4 GPUs each) # + 1x generation worker (TP32 + MoE EP32, attention-DP, DSpark draft length 7, 32 GPUs each) # + Dynamo frontend with conversation-affinity KV routing (fastokens tokenizer, deepseek_v4 tool-call and # reasoning parsers, KV events publication disabled). # The frontend + etcd ran on an additional allocation node with no serving GPUs (node 11 of 12). # Container image: <host-path>/dynamo_ff18f7210d99_trtllm_gb300_PatchPr-3d27c24b_patchpr.sqsh # sha256 789607ba70f7eb4287872838f3e031f46b58dcb2857b0ac1d48a6e052b72568c # Worker environment (every context and generation worker; per-allocation endpoints and host bind-mount paths redacted): # DYN_SERVED_MODEL_NAME=DeepSeek-V4-Pro TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 # TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0 # TLLM_METRICS_ALL_RANKS=1 HF_HUB_OFFLINE=1 TLLM_DISAGG_GEN_PRIORITIZE_FIRST_TOKEN=1 # TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 TRTLLM_SERVE_ENABLE_MSGSPEC=1 # TRTLLM_DISABLE_KV_CACHE_RATIO_UPDATE=1 TRTLLM_MHC_ENABLE_FUSED_HC=1 # TRTLLM_MOE_A2A_DISABLE_CFT_COUNTED_WRITES=1 TRTLLM_MLA_EXTRA_OVERLAP=1 # TRTLLM_FUSED_FP8_QUANT_PACK=1 MIMALLOC_PURGE_DELAY=0 TLLM_NUMA_AWARE_WORKER_AFFINITY=0 # TRTLLM_DSV4_MEM_OPTS=1 TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=1 # PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_SCALE_LAUNCH_QUEUES=4x # ENROOT_ALLOW_DEV=yes VIRTUAL_ENV=/opt/dynamo/venv # PATH=/opt/dynamo/venv/bin:/usr/local/bin/etcd:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/mpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # LD_PRELOAD=/opt/dynamo/libstdc++.so.6:/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/libnixl.so # NIXL_PLUGIN_DIR=/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/plugins # UCX_TLS=cuda_ipc,cuda_copy,sm,self,tcp DYN_REQUEST_PLANE=tcp DYN_KV_BLOCK_SIZE=128 # DYN_TOOL_CALL_PARSER=deepseek_v4 DYN_REASONING_PARSER=deepseek_v4 DYN_ENGINE_CONV_AFFINITY=1 # DYN_DEFAULT_THINKING_MODE=disabled DYN_HEALTH_CHECK_ENABLED=false # Context workers additionally (weight-loading peak-memory workaround): # TRT_LLM_DISABLE_LOAD_WEIGHTS_IN_PARALLEL=1 TRTLLM_HF_WEIGHT_LOAD_THREADS=1 # TRTLLM_LOCAL_WEIGHT_LOAD_CONCURRENCY=2 # Dynamo frontend / router (OpenAI-compatible on :8333): # ROUTER_MODE=kv DYN_KV_BLOCK_SIZE=128 ETCD_LEASE_TTL=120 DYN_ROUTER_QUEUE_THRESHOLD=None # DYN_TOKENIZER_CACHE=1 DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TOKENIZER=fastokens # DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=14400 python3 -m dynamo.frontend --http-port 8333 --router-mode kv --no-router-kv-events --kv-cache-block-size 128 --enforce-disagg --request-plane tcp --event-plane zmq --active-decode-blocks-threshold None --active-prefill-tokens-threshold None --active-prefill-tokens-threshold-frac None # Bring-up is driven by the vendor launcher in the AgentPerf submission: # one sbatch over the # 12-node allocation that starts etcd + frontend, then the workers below via `srun --overlap` per role: OUTPUT_DIR=<host-path> ./run.sh # ---------------- gen_config.yaml ---------------- tensor_parallel_size: 32 moe_expert_parallel_size: 32 enable_attention_dp: true enable_lm_head_tp_in_adp: true pipeline_parallel_size: 1 max_batch_size: 8 max_num_tokens: 448 max_seq_len: 147456 print_iter_log: true enable_autotuner: false kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.7 dtype: fp8 host_cache_size: 0 avg_seq_len: 39040 moe_config: backend: MEGAMOE_CUTEDSL use_low_precision_moe_combine: true load_balancer: <host-path>/eplb/moe_load_balancer_gen_ep32_slots384_dspark.yaml cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 stream_interval: 100 num_postprocess_workers: 8 cuda_graph_config: enable_padding: true batch_sizes: - 1 - 2 - 4 - 8 - 16 - 32 - 64 speculative_config: decoding_type: DSpark speculative_model: <host-path>/dsv4pro-nvfp4-dspark max_draft_len: 7 block_size: 7 nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core sparse_attention_config: algorithm: deepseek_v4 enable_heuristic_topk: true return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false # ---------------- ctx_config.yaml ---------------- max_batch_size: 128 max_num_tokens: 8192 max_seq_len: 147456 tensor_parallel_size: 4 moe_expert_parallel_size: 4 enable_attention_dp: true pipeline_parallel_size: 1 enable_chunked_prefill: true print_iter_log: true cuda_graph_config: null enable_autotuner: true disable_overlap_scheduler: false moe_config: backend: MEGAMOE_CUTEDSL kv_cache_config: enable_block_reuse: true free_gpu_memory_fraction: 0.4 dtype: fp8 host_cache_size: 107374182400 pool_ratio: - 0.45 - 0.3 - 0.25 cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 scheduler_config: capacity_scheduler_policy: MAX_UTILIZATION attention_dp_config: enable_kv_cache_aware_routing: false kv_cache_routing_conversation_affinity: true kv_cache_routing_max_sessions: 65536 return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core
Host-specific paths are redacted as <host-path>.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend + conversation-affinity KV router over TensorRT-LLM workers, 44 GPUs across 11 nodes (frontend on a separate CPU-only-role node, excluded from GPU count and power)Prefill3 × 4 GPUs · TP4 · EP4 · Enabled (TP4)Decode1 × 32 GPUs · TP32 · EP32 · Enabled (TP32)Speculative decodingDSpark (draft length 7)PrecisionNVFP4Accelerators used44 of 72
Submitted by NVIDIA
# NVIDIA Dynamo (ff18f7210d99) + TensorRT-LLM (gb300_PatchPr-3d27c24b) # disaggregated serving of DeepSeek-V4-Pro (NVFP4 + DSpark draft) on GB300 NVL72 — 36 serving GPUs # across 9 nodes: 1x context worker (TP4 + MoE EP4, attention-DP, 4 GPUs each) # + 1x generation worker (TP32 + MoE EP32, attention-DP, DSpark draft length 6, 32 GPUs each) # + Dynamo frontend with conversation-affinity KV routing (fastokens tokenizer, deepseek_v4 tool-call and # reasoning parsers, KV events publication disabled). # The frontend + etcd ran on an additional allocation node with no serving GPUs (node 9 of 10). # Container image: <host-path>/dynamo_ff18f7210d99_trtllm_gb300_PatchPr-3d27c24b_patchpr.sqsh # sha256 789607ba70f7eb4287872838f3e031f46b58dcb2857b0ac1d48a6e052b72568c # Worker environment (every context and generation worker; per-allocation endpoints and host bind-mount paths redacted): # DYN_SERVED_MODEL_NAME=DeepSeek-V4-Pro TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 # TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0 # TLLM_METRICS_ALL_RANKS=1 HF_HUB_OFFLINE=1 TLLM_DISAGG_GEN_PRIORITIZE_FIRST_TOKEN=1 # TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 TRTLLM_SERVE_ENABLE_MSGSPEC=1 # TRTLLM_DISABLE_KV_CACHE_RATIO_UPDATE=1 TRTLLM_MHC_ENABLE_FUSED_HC=1 # TRTLLM_MOE_A2A_DISABLE_CFT_COUNTED_WRITES=1 TRTLLM_MLA_EXTRA_OVERLAP=1 # TRTLLM_FUSED_FP8_QUANT_PACK=1 MIMALLOC_PURGE_DELAY=0 TLLM_NUMA_AWARE_WORKER_AFFINITY=0 # TRTLLM_DSV4_MEM_OPTS=1 TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=1 # PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_SCALE_LAUNCH_QUEUES=4x # ENROOT_ALLOW_DEV=yes VIRTUAL_ENV=/opt/dynamo/venv # PATH=/opt/dynamo/venv/bin:/usr/local/bin/etcd:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/mpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # LD_PRELOAD=/opt/dynamo/libstdc++.so.6:/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/libnixl.so # NIXL_PLUGIN_DIR=/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/plugins # UCX_TLS=cuda_ipc,cuda_copy,sm,self,tcp DYN_REQUEST_PLANE=tcp DYN_KV_BLOCK_SIZE=128 # DYN_TOOL_CALL_PARSER=deepseek_v4 DYN_REASONING_PARSER=deepseek_v4 DYN_ENGINE_CONV_AFFINITY=1 # DYN_DEFAULT_THINKING_MODE=disabled DYN_HEALTH_CHECK_ENABLED=false # Context workers additionally (weight-loading peak-memory workaround): # TRT_LLM_DISABLE_LOAD_WEIGHTS_IN_PARALLEL=1 TRTLLM_HF_WEIGHT_LOAD_THREADS=1 # TRTLLM_LOCAL_WEIGHT_LOAD_CONCURRENCY=2 # Dynamo frontend / router (OpenAI-compatible on :8333): # ROUTER_MODE=kv DYN_KV_BLOCK_SIZE=128 ETCD_LEASE_TTL=120 DYN_ROUTER_QUEUE_THRESHOLD=None # DYN_TOKENIZER_CACHE=1 DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TOKENIZER=fastokens # DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=14400 python3 -m dynamo.frontend --http-port 8333 --router-mode kv --no-router-kv-events --kv-cache-block-size 128 --enforce-disagg --request-plane tcp --event-plane zmq --active-decode-blocks-threshold None --active-prefill-tokens-threshold None --active-prefill-tokens-threshold-frac None # Bring-up is driven by the vendor launcher in the AgentPerf submission: # one sbatch over the # 10-node allocation that starts etcd + frontend, then the workers below via `srun --overlap` per role: OUTPUT_DIR=<host-path> ./run.sh # ---------------- gen_config.yaml ---------------- tensor_parallel_size: 32 moe_expert_parallel_size: 32 enable_attention_dp: true enable_lm_head_tp_in_adp: true pipeline_parallel_size: 1 max_batch_size: 1 max_num_tokens: 64 max_seq_len: 147456 print_iter_log: true enable_autotuner: false kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.7 dtype: fp8 host_cache_size: 0 avg_seq_len: 39040 moe_config: backend: MEGAMOE_CUTEDSL use_low_precision_moe_combine: true cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 stream_interval: 100 num_postprocess_workers: 8 cuda_graph_config: enable_padding: true batch_sizes: - 1 - 2 - 4 - 8 - 16 - 32 - 64 speculative_config: decoding_type: DSpark speculative_model: <host-path>/dsv4pro-nvfp4-dspark max_draft_len: 6 block_size: 6 nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core sparse_attention_config: algorithm: deepseek_v4 enable_heuristic_topk: true return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false # ---------------- ctx_config.yaml ---------------- max_batch_size: 128 max_num_tokens: 8192 max_seq_len: 147456 tensor_parallel_size: 4 moe_expert_parallel_size: 4 enable_attention_dp: true pipeline_parallel_size: 1 enable_chunked_prefill: true print_iter_log: true cuda_graph_config: null enable_autotuner: true disable_overlap_scheduler: false moe_config: backend: MEGAMOE_CUTEDSL kv_cache_config: enable_block_reuse: true free_gpu_memory_fraction: 0.4 dtype: fp8 host_cache_size: 107374182400 pool_ratio: - 0.45 - 0.3 - 0.25 cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 scheduler_config: capacity_scheduler_policy: MAX_UTILIZATION attention_dp_config: enable_kv_cache_aware_routing: false kv_cache_routing_conversation_affinity: true kv_cache_routing_max_sessions: 65536 return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core
Host-specific paths are redacted as <host-path>.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend + conversation-affinity KV router over TensorRT-LLM workers, 36 GPUs across 9 nodes (frontend on a separate CPU-only-role node, excluded from GPU count and power)Prefill1 × 4 GPUs · TP4 · EP4 · Enabled (TP4)Decode1 × 32 GPUs · TP32 · EP32 · Enabled (TP32)Speculative decodingDSpark (draft length 6)PrecisionNVFP4Accelerators used36 of 72
Submitted by NVIDIA
# NVIDIA Dynamo (ff18f7210d99) + TensorRT-LLM (gb300_PatchPr-3d27c24b) # disaggregated serving of DeepSeek-V4-Pro (NVFP4 + DSpark draft) on GB300 NVL72 — 36 serving GPUs # across 9 nodes: 1x context worker (TP4 + MoE EP4, attention-DP, 4 GPUs each) # + 1x generation worker (TP32 + MoE EP32, attention-DP, DSpark draft length 6, 32 GPUs each) # + Dynamo frontend with conversation-affinity KV routing (fastokens tokenizer, deepseek_v4 tool-call and # reasoning parsers, KV events publication disabled). # The frontend + etcd ran on an additional allocation node with no serving GPUs (node 9 of 10). # Container image: <host-path>/dynamo_ff18f7210d99_trtllm_gb300_PatchPr-3d27c24b_patchpr.sqsh # sha256 789607ba70f7eb4287872838f3e031f46b58dcb2857b0ac1d48a6e052b72568c # Worker environment (every context and generation worker; per-allocation endpoints and host bind-mount paths redacted): # DYN_SERVED_MODEL_NAME=DeepSeek-V4-Pro TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 # TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0 # TLLM_METRICS_ALL_RANKS=1 HF_HUB_OFFLINE=1 TLLM_DISAGG_GEN_PRIORITIZE_FIRST_TOKEN=1 # TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 TRTLLM_SERVE_ENABLE_MSGSPEC=1 # TRTLLM_DISABLE_KV_CACHE_RATIO_UPDATE=1 TRTLLM_MHC_ENABLE_FUSED_HC=1 # TRTLLM_MOE_A2A_DISABLE_CFT_COUNTED_WRITES=1 TRTLLM_MLA_EXTRA_OVERLAP=1 # TRTLLM_FUSED_FP8_QUANT_PACK=1 MIMALLOC_PURGE_DELAY=0 TLLM_NUMA_AWARE_WORKER_AFFINITY=0 # TRTLLM_DSV4_MEM_OPTS=1 TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=1 # PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_SCALE_LAUNCH_QUEUES=4x # ENROOT_ALLOW_DEV=yes VIRTUAL_ENV=/opt/dynamo/venv # PATH=/opt/dynamo/venv/bin:/usr/local/bin/etcd:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/mpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # LD_PRELOAD=/opt/dynamo/libstdc++.so.6:/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/libnixl.so # NIXL_PLUGIN_DIR=/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/plugins # UCX_TLS=cuda_ipc,cuda_copy,sm,self,tcp DYN_REQUEST_PLANE=tcp DYN_KV_BLOCK_SIZE=128 # DYN_TOOL_CALL_PARSER=deepseek_v4 DYN_REASONING_PARSER=deepseek_v4 DYN_ENGINE_CONV_AFFINITY=1 # DYN_DEFAULT_THINKING_MODE=disabled DYN_HEALTH_CHECK_ENABLED=false # Context workers additionally (weight-loading peak-memory workaround): # TRT_LLM_DISABLE_LOAD_WEIGHTS_IN_PARALLEL=1 TRTLLM_HF_WEIGHT_LOAD_THREADS=1 # TRTLLM_LOCAL_WEIGHT_LOAD_CONCURRENCY=2 # Dynamo frontend / router (OpenAI-compatible on :8333): # ROUTER_MODE=kv DYN_KV_BLOCK_SIZE=128 ETCD_LEASE_TTL=120 DYN_ROUTER_QUEUE_THRESHOLD=None # DYN_TOKENIZER_CACHE=1 DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TOKENIZER=fastokens # DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=14400 python3 -m dynamo.frontend --http-port 8333 --router-mode kv --no-router-kv-events --kv-cache-block-size 128 --enforce-disagg --request-plane tcp --event-plane zmq --active-decode-blocks-threshold None --active-prefill-tokens-threshold None --active-prefill-tokens-threshold-frac None # Bring-up is driven by the vendor launcher in the AgentPerf submission: # one sbatch over the # 10-node allocation that starts etcd + frontend, then the workers below via `srun --overlap` per role: OUTPUT_DIR=<host-path> ./run.sh # ---------------- gen_config.yaml ---------------- tensor_parallel_size: 32 moe_expert_parallel_size: 32 enable_attention_dp: true enable_lm_head_tp_in_adp: true pipeline_parallel_size: 1 max_batch_size: 2 max_num_tokens: 64 max_seq_len: 147456 print_iter_log: true enable_autotuner: false kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.7 dtype: fp8 host_cache_size: 0 avg_seq_len: 39040 moe_config: backend: MEGAMOE_CUTEDSL use_low_precision_moe_combine: true load_balancer: <host-path>/eplb/moe_load_balancer_gen_ep32_slots384_dspark.yaml cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 stream_interval: 100 num_postprocess_workers: 8 cuda_graph_config: enable_padding: true batch_sizes: - 1 - 2 - 4 - 8 - 16 - 32 - 64 speculative_config: decoding_type: DSpark speculative_model: <host-path>/dsv4pro-nvfp4-dspark max_draft_len: 6 block_size: 6 nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core sparse_attention_config: algorithm: deepseek_v4 enable_heuristic_topk: true return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false # ---------------- ctx_config.yaml ---------------- max_batch_size: 128 max_num_tokens: 8192 max_seq_len: 147456 tensor_parallel_size: 4 moe_expert_parallel_size: 4 enable_attention_dp: true pipeline_parallel_size: 1 enable_chunked_prefill: true print_iter_log: true cuda_graph_config: null enable_autotuner: true disable_overlap_scheduler: false moe_config: backend: MEGAMOE_CUTEDSL kv_cache_config: enable_block_reuse: true free_gpu_memory_fraction: 0.4 dtype: fp8 host_cache_size: 107374182400 pool_ratio: - 0.45 - 0.3 - 0.25 cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 scheduler_config: capacity_scheduler_policy: MAX_UTILIZATION attention_dp_config: enable_kv_cache_aware_routing: false kv_cache_routing_conversation_affinity: true kv_cache_routing_max_sessions: 65536 return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core
Host-specific paths are redacted as <host-path>.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend + conversation-affinity KV router over TensorRT-LLM workers, 36 GPUs across 9 nodes (frontend on a separate CPU-only-role node, excluded from GPU count and power)Prefill1 × 4 GPUs · TP4 · EP4 · Enabled (TP4)Decode1 × 32 GPUs · TP32 · EP32 · Enabled (TP32)Speculative decodingDSpark (draft length 6)PrecisionNVFP4Accelerators used36 of 72
Submitted by NVIDIA
# NVIDIA Dynamo (ff18f7210d99) + TensorRT-LLM (gb300_PatchPr-3d27c24b) # disaggregated serving of DeepSeek-V4-Pro (NVFP4 + DSpark draft) on GB300 NVL72 — 64 serving GPUs # across 16 nodes: 8x context workers (TP4 + MoE EP4, attention-DP, 4 GPUs each) # + 1x generation worker (TP32 + MoE EP32, attention-DP, DSpark draft length 5, 32 GPUs each) # + Dynamo frontend with conversation-affinity KV routing (fastokens tokenizer, deepseek_v4 tool-call and # reasoning parsers, KV events publication disabled). # Container image: <host-path>/dynamo_ff18f7210d99_trtllm_gb300_PatchPr-3d27c24b_patchpr.sqsh # sha256 789607ba70f7eb4287872838f3e031f46b58dcb2857b0ac1d48a6e052b72568c # Worker environment (every context and generation worker; per-allocation endpoints and host bind-mount paths redacted): # DYN_SERVED_MODEL_NAME=DeepSeek-V4-Pro TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 # TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0 # TLLM_METRICS_ALL_RANKS=1 HF_HUB_OFFLINE=1 TLLM_DISAGG_GEN_PRIORITIZE_FIRST_TOKEN=1 # TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 TRTLLM_SERVE_ENABLE_MSGSPEC=1 # TRTLLM_DISABLE_KV_CACHE_RATIO_UPDATE=1 TRTLLM_MHC_ENABLE_FUSED_HC=1 # TRTLLM_MOE_A2A_DISABLE_CFT_COUNTED_WRITES=1 TRTLLM_MLA_EXTRA_OVERLAP=1 # TRTLLM_FUSED_FP8_QUANT_PACK=1 MIMALLOC_PURGE_DELAY=0 TLLM_NUMA_AWARE_WORKER_AFFINITY=0 # TRTLLM_DSV4_MEM_OPTS=1 TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=1 # PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_SCALE_LAUNCH_QUEUES=4x # ENROOT_ALLOW_DEV=yes VIRTUAL_ENV=/opt/dynamo/venv # PATH=/opt/dynamo/venv/bin:/usr/local/bin/etcd:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/mpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # LD_PRELOAD=/opt/dynamo/libstdc++.so.6:/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/libnixl.so # NIXL_PLUGIN_DIR=/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/plugins # UCX_TLS=cuda_ipc,cuda_copy,sm,self,tcp DYN_REQUEST_PLANE=tcp DYN_KV_BLOCK_SIZE=128 # DYN_TOOL_CALL_PARSER=deepseek_v4 DYN_REASONING_PARSER=deepseek_v4 DYN_ENGINE_CONV_AFFINITY=1 # DYN_DEFAULT_THINKING_MODE=disabled DYN_HEALTH_CHECK_ENABLED=false # Context workers additionally (weight-loading peak-memory workaround): # TRT_LLM_DISABLE_LOAD_WEIGHTS_IN_PARALLEL=1 TRTLLM_HF_WEIGHT_LOAD_THREADS=1 # TRTLLM_LOCAL_WEIGHT_LOAD_CONCURRENCY=2 # Dynamo frontend / router (OpenAI-compatible on :8333): # ROUTER_MODE=kv DYN_KV_BLOCK_SIZE=128 ETCD_LEASE_TTL=120 DYN_ROUTER_QUEUE_THRESHOLD=None # DYN_TOKENIZER_CACHE=1 DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TOKENIZER=fastokens # DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=14400 python3 -m dynamo.frontend --http-port 8333 --router-mode kv --no-router-kv-events --kv-cache-block-size 128 --enforce-disagg --request-plane tcp --event-plane zmq --active-decode-blocks-threshold None --active-prefill-tokens-threshold None --active-prefill-tokens-threshold-frac None # Bring-up is driven by the vendor launcher in the AgentPerf submission: # one sbatch over the # 16-node allocation that starts etcd + frontend, then the workers below via `srun --overlap` per role: OUTPUT_DIR=<host-path> ./run.sh # ---------------- gen_config.yaml ---------------- tensor_parallel_size: 32 moe_expert_parallel_size: 32 enable_attention_dp: true enable_lm_head_tp_in_adp: true pipeline_parallel_size: 1 max_batch_size: 32 max_num_tokens: 224 max_seq_len: 147456 print_iter_log: true enable_autotuner: false kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.7 dtype: fp8 host_cache_size: 0 avg_seq_len: 39040 moe_config: backend: MEGAMOE_CUTEDSL use_low_precision_moe_combine: true load_balancer: <host-path>/eplb/moe_load_balancer_gen_ep32_slots384_dspark.yaml cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 stream_interval: 100 num_postprocess_workers: 8 cuda_graph_config: enable_padding: true batch_sizes: - 1 - 2 - 4 - 8 - 16 - 32 - 64 speculative_config: decoding_type: DSpark speculative_model: <host-path>/dsv4pro-nvfp4-dspark max_draft_len: 5 block_size: 5 nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core sparse_attention_config: algorithm: deepseek_v4 enable_heuristic_topk: true return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false # ---------------- ctx_config.yaml ---------------- max_batch_size: 128 max_num_tokens: 8192 max_seq_len: 147456 tensor_parallel_size: 4 moe_expert_parallel_size: 4 enable_attention_dp: true pipeline_parallel_size: 1 enable_chunked_prefill: true print_iter_log: true cuda_graph_config: null enable_autotuner: true disable_overlap_scheduler: false moe_config: backend: MEGAMOE_CUTEDSL kv_cache_config: enable_block_reuse: true free_gpu_memory_fraction: 0.4 dtype: fp8 host_cache_size: 107374182400 pool_ratio: - 0.45 - 0.3 - 0.25 cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 scheduler_config: capacity_scheduler_policy: MAX_UTILIZATION attention_dp_config: enable_kv_cache_aware_routing: false kv_cache_routing_conversation_affinity: true kv_cache_routing_max_sessions: 65536 return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core
Host-specific paths are redacted as <host-path>.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend + conversation-affinity KV router over TensorRT-LLM workers, 64 GPUs across 16 nodesPrefill8 × 4 GPUs · TP4 · EP4 · Enabled (TP4)Decode1 × 32 GPUs · TP32 · EP32 · Enabled (TP32)Speculative decodingDSpark (draft length 5)PrecisionNVFP4Accelerators used64 of 72
Submitted by NVIDIA
# NVIDIA Dynamo (ff18f7210d99) + TensorRT-LLM (gb300_PatchPr-3d27c24b) # disaggregated serving of DeepSeek-V4-Pro (NVFP4 + DSpark draft) on GB300 NVL72 — 40 serving GPUs # across 10 nodes: 2x context workers (TP4 + MoE EP4, attention-DP, 4 GPUs each) # + 1x generation worker (TP32 + MoE EP32, attention-DP, DSpark draft length 7, 32 GPUs each) # + Dynamo frontend with conversation-affinity KV routing (fastokens tokenizer, deepseek_v4 tool-call and # reasoning parsers, KV events publication disabled). # Container image: <host-path>/dynamo_ff18f7210d99_trtllm_gb300_PatchPr-3d27c24b_patchpr.sqsh # sha256 789607ba70f7eb4287872838f3e031f46b58dcb2857b0ac1d48a6e052b72568c # Worker environment (every context and generation worker; per-allocation endpoints and host bind-mount paths redacted): # DYN_SERVED_MODEL_NAME=DeepSeek-V4-Pro TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 # TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0 # TLLM_METRICS_ALL_RANKS=1 HF_HUB_OFFLINE=1 TLLM_DISAGG_GEN_PRIORITIZE_FIRST_TOKEN=1 # TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 TRTLLM_SERVE_ENABLE_MSGSPEC=1 # TRTLLM_DISABLE_KV_CACHE_RATIO_UPDATE=1 TRTLLM_MHC_ENABLE_FUSED_HC=1 # TRTLLM_MOE_A2A_DISABLE_CFT_COUNTED_WRITES=1 TRTLLM_MLA_EXTRA_OVERLAP=1 # TRTLLM_FUSED_FP8_QUANT_PACK=1 MIMALLOC_PURGE_DELAY=0 TLLM_NUMA_AWARE_WORKER_AFFINITY=0 # TRTLLM_DSV4_MEM_OPTS=1 TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=1 # PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_SCALE_LAUNCH_QUEUES=4x # ENROOT_ALLOW_DEV=yes VIRTUAL_ENV=/opt/dynamo/venv # PATH=/opt/dynamo/venv/bin:/usr/local/bin/etcd:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/mpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # LD_PRELOAD=/opt/dynamo/libstdc++.so.6:/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/libnixl.so # NIXL_PLUGIN_DIR=/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/plugins # UCX_TLS=cuda_ipc,cuda_copy,sm,self,tcp DYN_REQUEST_PLANE=tcp DYN_KV_BLOCK_SIZE=128 # DYN_TOOL_CALL_PARSER=deepseek_v4 DYN_REASONING_PARSER=deepseek_v4 DYN_ENGINE_CONV_AFFINITY=1 # DYN_DEFAULT_THINKING_MODE=disabled DYN_HEALTH_CHECK_ENABLED=false # Context workers additionally (weight-loading peak-memory workaround): # TRT_LLM_DISABLE_LOAD_WEIGHTS_IN_PARALLEL=1 TRTLLM_HF_WEIGHT_LOAD_THREADS=1 # TRTLLM_LOCAL_WEIGHT_LOAD_CONCURRENCY=2 # Dynamo frontend / router (OpenAI-compatible on :8333): # ROUTER_MODE=kv DYN_KV_BLOCK_SIZE=128 ETCD_LEASE_TTL=120 DYN_ROUTER_QUEUE_THRESHOLD=None # DYN_TOKENIZER_CACHE=1 DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TOKENIZER=fastokens # DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=14400 python3 -m dynamo.frontend --http-port 8333 --router-mode kv --no-router-kv-events --kv-cache-block-size 128 --enforce-disagg --request-plane tcp --event-plane zmq --active-decode-blocks-threshold None --active-prefill-tokens-threshold None --active-prefill-tokens-threshold-frac None # Bring-up is driven by the vendor launcher in the AgentPerf submission: # one sbatch over the # 10-node allocation that starts etcd + frontend, then the workers below via `srun --overlap` per role: OUTPUT_DIR=<host-path> ./run.sh # ---------------- gen_config.yaml ---------------- tensor_parallel_size: 32 moe_expert_parallel_size: 32 enable_attention_dp: true enable_lm_head_tp_in_adp: true pipeline_parallel_size: 1 max_batch_size: 4 max_num_tokens: 448 max_seq_len: 147456 print_iter_log: true enable_autotuner: false kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.7 dtype: fp8 host_cache_size: 0 avg_seq_len: 39040 moe_config: backend: MEGAMOE_CUTEDSL use_low_precision_moe_combine: true load_balancer: <host-path>/eplb/moe_load_balancer_gen_ep32_slots384_dspark.yaml cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 stream_interval: 100 num_postprocess_workers: 8 cuda_graph_config: enable_padding: true batch_sizes: - 1 - 2 - 4 - 8 - 16 - 32 - 64 speculative_config: decoding_type: DSpark speculative_model: <host-path>/dsv4pro-nvfp4-dspark max_draft_len: 7 block_size: 7 nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core sparse_attention_config: algorithm: deepseek_v4 enable_heuristic_topk: true return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false # ---------------- ctx_config.yaml ---------------- max_batch_size: 128 max_num_tokens: 8192 max_seq_len: 147456 tensor_parallel_size: 4 moe_expert_parallel_size: 4 enable_attention_dp: true pipeline_parallel_size: 1 enable_chunked_prefill: true print_iter_log: true cuda_graph_config: null enable_autotuner: true disable_overlap_scheduler: false moe_config: backend: MEGAMOE_CUTEDSL kv_cache_config: enable_block_reuse: true free_gpu_memory_fraction: 0.4 dtype: fp8 host_cache_size: 107374182400 pool_ratio: - 0.45 - 0.3 - 0.25 cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 scheduler_config: capacity_scheduler_policy: MAX_UTILIZATION attention_dp_config: enable_kv_cache_aware_routing: false kv_cache_routing_conversation_affinity: true kv_cache_routing_max_sessions: 65536 return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core
Host-specific paths are redacted as <host-path>.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend + conversation-affinity KV router over TensorRT-LLM workers, 40 GPUs across 10 nodesPrefill2 × 4 GPUs · TP4 · EP4 · Enabled (TP4)Decode1 × 32 GPUs · TP32 · EP32 · Enabled (TP32)Speculative decodingDSpark (draft length 7)PrecisionNVFP4Accelerators used40 of 72
Submitted by NVIDIA
# NVIDIA Dynamo (ff18f7210d99) + TensorRT-LLM (gb300_PatchPr-3d27c24b) # disaggregated serving of DeepSeek-V4-Pro (NVFP4 + DSpark draft) on GB300 NVL72 — 52 serving GPUs # across 13 nodes: 5x context workers (TP4 + MoE EP4, attention-DP, 4 GPUs each) # + 1x generation worker (TP32 + MoE EP32, attention-DP, DSpark draft length 6, 32 GPUs each) # + Dynamo frontend with conversation-affinity KV routing (fastokens tokenizer, deepseek_v4 tool-call and # reasoning parsers, KV events publication disabled). # The frontend + etcd ran on an additional allocation node with no serving GPUs (node 13 of 14). # Container image: <host-path>/dynamo_ff18f7210d99_trtllm_gb300_PatchPr-3d27c24b_patchpr.sqsh # sha256 789607ba70f7eb4287872838f3e031f46b58dcb2857b0ac1d48a6e052b72568c # Worker environment (every context and generation worker; per-allocation endpoints and host bind-mount paths redacted): # DYN_SERVED_MODEL_NAME=DeepSeek-V4-Pro TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 # TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0 # TLLM_METRICS_ALL_RANKS=1 HF_HUB_OFFLINE=1 TLLM_DISAGG_GEN_PRIORITIZE_FIRST_TOKEN=1 # TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 TRTLLM_SERVE_ENABLE_MSGSPEC=1 # TRTLLM_DISABLE_KV_CACHE_RATIO_UPDATE=1 TRTLLM_MHC_ENABLE_FUSED_HC=1 # TRTLLM_MOE_A2A_DISABLE_CFT_COUNTED_WRITES=1 TRTLLM_MLA_EXTRA_OVERLAP=1 # TRTLLM_FUSED_FP8_QUANT_PACK=1 MIMALLOC_PURGE_DELAY=0 TLLM_NUMA_AWARE_WORKER_AFFINITY=0 # TRTLLM_DSV4_MEM_OPTS=1 TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=1 # PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_SCALE_LAUNCH_QUEUES=4x # ENROOT_ALLOW_DEV=yes VIRTUAL_ENV=/opt/dynamo/venv # PATH=/opt/dynamo/venv/bin:/usr/local/bin/etcd:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/mpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # LD_PRELOAD=/opt/dynamo/libstdc++.so.6:/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/libnixl.so # NIXL_PLUGIN_DIR=/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/plugins # UCX_TLS=cuda_ipc,cuda_copy,sm,self,tcp DYN_REQUEST_PLANE=tcp DYN_KV_BLOCK_SIZE=128 # DYN_TOOL_CALL_PARSER=deepseek_v4 DYN_REASONING_PARSER=deepseek_v4 DYN_ENGINE_CONV_AFFINITY=1 # DYN_DEFAULT_THINKING_MODE=disabled DYN_HEALTH_CHECK_ENABLED=false # Context workers additionally (weight-loading peak-memory workaround): # TRT_LLM_DISABLE_LOAD_WEIGHTS_IN_PARALLEL=1 TRTLLM_HF_WEIGHT_LOAD_THREADS=1 # TRTLLM_LOCAL_WEIGHT_LOAD_CONCURRENCY=2 # Dynamo frontend / router (OpenAI-compatible on :8333): # ROUTER_MODE=kv DYN_KV_BLOCK_SIZE=128 ETCD_LEASE_TTL=120 DYN_ROUTER_QUEUE_THRESHOLD=None # DYN_TOKENIZER_CACHE=1 DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TOKENIZER=fastokens # DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=14400 python3 -m dynamo.frontend --http-port 8333 --router-mode kv --no-router-kv-events --kv-cache-block-size 128 --enforce-disagg --request-plane tcp --event-plane zmq --active-decode-blocks-threshold None --active-prefill-tokens-threshold None --active-prefill-tokens-threshold-frac None # Bring-up is driven by the vendor launcher in the AgentPerf submission: # one sbatch over the # 14-node allocation that starts etcd + frontend, then the workers below via `srun --overlap` per role: OUTPUT_DIR=<host-path> ./run.sh # ---------------- gen_config.yaml ---------------- tensor_parallel_size: 32 moe_expert_parallel_size: 32 enable_attention_dp: true enable_lm_head_tp_in_adp: true pipeline_parallel_size: 1 max_batch_size: 16 max_num_tokens: 128 max_seq_len: 147456 print_iter_log: true enable_autotuner: false kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.7 dtype: fp8 host_cache_size: 0 avg_seq_len: 39040 moe_config: backend: MEGAMOE_CUTEDSL use_low_precision_moe_combine: true load_balancer: <host-path>/eplb/moe_load_balancer_gen_ep32_slots384_dspark.yaml cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 stream_interval: 100 num_postprocess_workers: 8 cuda_graph_config: enable_padding: true batch_sizes: - 1 - 2 - 4 - 8 - 16 - 32 - 64 speculative_config: decoding_type: DSpark speculative_model: <host-path>/dsv4pro-nvfp4-dspark max_draft_len: 6 block_size: 6 nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core sparse_attention_config: algorithm: deepseek_v4 enable_heuristic_topk: true return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false # ---------------- ctx_config.yaml ---------------- max_batch_size: 128 max_num_tokens: 8192 max_seq_len: 147456 tensor_parallel_size: 4 moe_expert_parallel_size: 4 enable_attention_dp: true pipeline_parallel_size: 1 enable_chunked_prefill: true print_iter_log: true cuda_graph_config: null enable_autotuner: true disable_overlap_scheduler: false moe_config: backend: MEGAMOE_CUTEDSL kv_cache_config: enable_block_reuse: true free_gpu_memory_fraction: 0.4 dtype: fp8 host_cache_size: 107374182400 pool_ratio: - 0.45 - 0.3 - 0.25 cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 scheduler_config: capacity_scheduler_policy: MAX_UTILIZATION attention_dp_config: enable_kv_cache_aware_routing: false kv_cache_routing_conversation_affinity: true kv_cache_routing_max_sessions: 65536 return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core
Host-specific paths are redacted as <host-path>.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend + conversation-affinity KV router over TensorRT-LLM workers, 52 GPUs across 13 nodes (frontend on a separate CPU-only-role node, excluded from GPU count and power)Prefill5 × 4 GPUs · TP4 · EP4 · Enabled (TP4)Decode1 × 32 GPUs · TP32 · EP32 · Enabled (TP32)Speculative decodingDSpark (draft length 6)PrecisionNVFP4Accelerators used52 of 72
Submitted by NVIDIA
# NVIDIA Dynamo (ff18f7210d99) + TensorRT-LLM (gb300_PatchPr-3d27c24b) # disaggregated serving of DeepSeek-V4-Pro (NVFP4 + DSpark draft) on GB300 NVL72 — 40 serving GPUs # across 10 nodes: 6x context workers (TP4 + MoE EP4, attention-DP, 4 GPUs each) # + 1x generation worker (TP16 + MoE EP16, attention-DP, DSpark draft length 4, 16 GPUs each) # + Dynamo frontend with conversation-affinity KV routing (fastokens tokenizer, deepseek_v4 tool-call and # reasoning parsers, KV events publication disabled). # Container image: <host-path>/dynamo_ff18f7210d99_trtllm_gb300_PatchPr-3d27c24b_patchpr.sqsh # sha256 789607ba70f7eb4287872838f3e031f46b58dcb2857b0ac1d48a6e052b72568c # Worker environment (every context and generation worker; per-allocation endpoints and host bind-mount paths redacted): # DYN_SERVED_MODEL_NAME=DeepSeek-V4-Pro TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 # TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0 # TLLM_METRICS_ALL_RANKS=1 HF_HUB_OFFLINE=1 TLLM_DISAGG_GEN_PRIORITIZE_FIRST_TOKEN=1 # TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 TRTLLM_SERVE_ENABLE_MSGSPEC=1 # TRTLLM_DISABLE_KV_CACHE_RATIO_UPDATE=1 TRTLLM_MHC_ENABLE_FUSED_HC=1 # TRTLLM_MOE_A2A_DISABLE_CFT_COUNTED_WRITES=1 TRTLLM_MLA_EXTRA_OVERLAP=1 # TRTLLM_FUSED_FP8_QUANT_PACK=1 MIMALLOC_PURGE_DELAY=0 TLLM_NUMA_AWARE_WORKER_AFFINITY=0 # TRTLLM_DSV4_MEM_OPTS=1 TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=1 # PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_SCALE_LAUNCH_QUEUES=4x # ENROOT_ALLOW_DEV=yes VIRTUAL_ENV=/opt/dynamo/venv # PATH=/opt/dynamo/venv/bin:/usr/local/bin/etcd:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/mpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # LD_PRELOAD=/opt/dynamo/libstdc++.so.6:/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/libnixl.so # NIXL_PLUGIN_DIR=/usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/nixl/plugins # UCX_TLS=cuda_ipc,cuda_copy,sm,self,tcp DYN_REQUEST_PLANE=tcp DYN_KV_BLOCK_SIZE=128 # DYN_TOOL_CALL_PARSER=deepseek_v4 DYN_REASONING_PARSER=deepseek_v4 DYN_ENGINE_CONV_AFFINITY=1 # DYN_DEFAULT_THINKING_MODE=disabled DYN_HEALTH_CHECK_ENABLED=false # Context workers additionally (weight-loading peak-memory workaround): # TRT_LLM_DISABLE_LOAD_WEIGHTS_IN_PARALLEL=1 TRTLLM_HF_WEIGHT_LOAD_THREADS=1 # TRTLLM_LOCAL_WEIGHT_LOAD_CONCURRENCY=2 # Dynamo frontend / router (OpenAI-compatible on :8333): # ROUTER_MODE=kv DYN_KV_BLOCK_SIZE=128 ETCD_LEASE_TTL=120 DYN_ROUTER_QUEUE_THRESHOLD=None # DYN_TOKENIZER_CACHE=1 DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TOKENIZER=fastokens # DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=14400 python3 -m dynamo.frontend --http-port 8333 --router-mode kv --no-router-kv-events --kv-cache-block-size 128 --enforce-disagg --request-plane tcp --event-plane zmq --active-decode-blocks-threshold None --active-prefill-tokens-threshold None --active-prefill-tokens-threshold-frac None # Bring-up is driven by the vendor launcher in the AgentPerf submission: # one sbatch over the # 10-node allocation that starts etcd + frontend, then the workers below via `srun --overlap` per role: OUTPUT_DIR=<host-path> ./run.sh # ---------------- gen_config.yaml ---------------- tensor_parallel_size: 16 moe_expert_parallel_size: 16 enable_attention_dp: true enable_lm_head_tp_in_adp: true pipeline_parallel_size: 1 max_batch_size: 64 max_num_tokens: 384 max_seq_len: 147456 print_iter_log: true enable_autotuner: false kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.7 dtype: fp8 host_cache_size: 0 avg_seq_len: 39040 moe_config: backend: MEGAMOE_CUTEDSL use_low_precision_moe_combine: true load_balancer: <host-path>/eplb/moe_load_balancer_gen_ep16_slots384_dspark.yaml cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 stream_interval: 100 num_postprocess_workers: 8 cuda_graph_config: enable_padding: true batch_sizes: - 1 - 2 - 4 - 8 - 16 - 32 - 64 speculative_config: decoding_type: DSpark speculative_model: <host-path>/dsv4pro-nvfp4-dspark max_draft_len: 4 block_size: 4 nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core sparse_attention_config: algorithm: deepseek_v4 enable_heuristic_topk: true return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false # ---------------- ctx_config.yaml ---------------- max_batch_size: 128 max_num_tokens: 8192 max_seq_len: 147456 tensor_parallel_size: 4 moe_expert_parallel_size: 4 enable_attention_dp: true pipeline_parallel_size: 1 enable_chunked_prefill: true print_iter_log: true cuda_graph_config: null enable_autotuner: true disable_overlap_scheduler: false moe_config: backend: MEGAMOE_CUTEDSL kv_cache_config: enable_block_reuse: true free_gpu_memory_fraction: 0.4 dtype: fp8 host_cache_size: 107374182400 pool_ratio: - 0.45 - 0.3 - 0.25 cache_transceiver_config: max_tokens_in_buffer: 147456 backend: NIXL transceiver_runtime: PYTHON kv_transfer_timeout_ms: 600000 scheduler_config: capacity_scheduler_policy: MAX_UTILIZATION attention_dp_config: enable_kv_cache_aware_routing: false kv_cache_routing_conversation_affinity: true kv_cache_routing_max_sessions: 65536 return_perf_metrics: false enable_iter_perf_stats: false enable_iter_req_stats: false nvfp4_gemm_config: allowed_backends: - cutlass - cublaslt - cutedsl - cuda_core
Host-specific paths are redacted as <host-path>.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend + conversation-affinity KV router over TensorRT-LLM workers, 40 GPUs across 10 nodesPrefill6 × 4 GPUs · TP4 · EP4 · Enabled (TP4)Decode1 × 16 GPUs · TP16 · EP16 · Enabled (TP16)Speculative decodingDSpark (draft length 4)PrecisionNVFP4Accelerators used40 of 72
Submitted by NVIDIA
Kimi K3 (max) — 27 configurations
SystemFrameworkPrecisionAccelerators UsedParallelismSpecdecDate Added
No launch command recorded for this configuration.
SystemB300 x8EngineSGLang (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend + KV-aware router (conversation affinity) over SGLang workers, 16 GPUs on 2 B300 nodesPrefill1 × 8 GPUs · TP8 · DCP8Decode1 × 8 GPUs · TP8 · DCP8Speculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, block size 7)KV transferMooncake over 8 x IB NDRPrecisionMXFP4Accelerators used16 of 8
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemB300 x8EngineSGLang (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend + KV-aware router (conversation affinity) over SGLang workers, 16 GPUs on 2 B300 nodesPrefill1 × 8 GPUs · TP8 · DCP8Decode1 × 8 GPUs · TP8 · DCP8Speculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, block size 4)KV transferMooncake over 8 x IB NDRPrecisionMXFP4Accelerators used16 of 8
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemB300 x8EngineSGLang (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend + KV-aware router (conversation affinity) over SGLang workers, 24 GPUs on 3 B300 nodesPrefill2 × 8 GPUs · TP8 · DCP8Decode1 × 8 GPUs · TP8 · DCP8Speculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, block size 4)KV transferMooncake over 8 x IB NDRPrecisionMXFP4Accelerators used24 of 8
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemB300 x8EngineSGLang (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend + KV-aware router (conversation affinity) over SGLang workers, 32 GPUs on 4 B300 nodesPrefill3 × 8 GPUs · TP8 · DCP8Decode1 × 8 GPUs · TP8 · DCP8Speculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, block size 4)KV transferMooncake over 8 x IB NDRPrecisionMXFP4Accelerators used32 of 8
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend (conversation-affinity routing) over TensorRT-LLM workers, 32 GPUs on 8 GB300 traysPrefill1 × 8 GPUs · TP8 · EP8 · DisabledDecode3 × 8 GPUs · TP1 · Helix CP8 · EP8 · DisabledSpeculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, depth 7)KV transferNIXL (UCX)PrecisionNVFP4Accelerators used32 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend (conversation-affinity routing) over TensorRT-LLM workers, 32 GPUs on 8 GB300 traysPrefill1 × 8 GPUs · TP8 · EP8 · DisabledDecode3 × 8 GPUs · TP1 · Helix CP8 · EP8 · DisabledSpeculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, depth 5)KV transferNIXL (UCX)PrecisionNVFP4Accelerators used32 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend (conversation-affinity routing) over TensorRT-LLM workers, 32 GPUs on 8 GB300 traysPrefill1 × 8 GPUs · TP8 · EP8 · DisabledDecode3 × 8 GPUs · TP1 · Helix CP8 · EP8 · DisabledSpeculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, depth 7)KV transferNIXL (UCX)PrecisionNVFP4Accelerators used32 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend (conversation-affinity routing) over TensorRT-LLM workers, 48 GPUs on 12 GB300 traysPrefill16 × 1 GPU · TP1 · EnabledPrefill groupDWDP16 group of one-GPU workers (TRT-LLM dwdp_config, 56 experts/worker)Decode2 × 16 GPUs · TP1 · Helix CP16 · EP16 · DisabledSpeculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, depth 5)KV transferNIXL (UCX)PrecisionNVFP4Accelerators used48 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend (conversation-affinity routing) over TensorRT-LLM workers, 32 GPUs on 8 GB300 traysPrefill16 × 1 GPU · TP1 · EnabledPrefill groupDWDP16 group of one-GPU workers (TRT-LLM dwdp_config, 56 experts/worker)Decode1 × 16 GPUs · TP16 · EP16 · EnabledSpeculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, depth 5)KV transferNIXL (UCX)PrecisionNVFP4Accelerators used32 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend (conversation-affinity routing) over TensorRT-LLM workers, 32 GPUs on 8 GB300 traysPrefill1 × 8 GPUs · TP8 · EP8 · DisabledDecode3 × 8 GPUs · TP1 · Helix CP8 · EP8 · DisabledSpeculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, depth 7)KV transferNIXL (UCX)PrecisionNVFP4Accelerators used32 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend (conversation-affinity routing) over TensorRT-LLM workers, 32 GPUs on 8 GB300 traysPrefill1 × 8 GPUs · TP8 · EP8 · DisabledDecode3 × 8 GPUs · TP1 · Helix CP8 · EP8 · DisabledSpeculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, depth 7)KV transferNIXL (UCX)PrecisionNVFP4Accelerators used32 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend (conversation-affinity routing) over TensorRT-LLM workers, 48 GPUs on 12 GB300 traysPrefill16 × 1 GPU · TP1 · EnabledPrefill groupDWDP16 group of one-GPU workers (TRT-LLM dwdp_config, 56 experts/worker)Decode2 × 16 GPUs · TP1 · Helix CP16 · EP16 · DisabledSpeculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, depth 5)KV transferNIXL (UCX)PrecisionNVFP4Accelerators used48 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend (conversation-affinity routing) over TensorRT-LLM workers, 32 GPUs on 8 GB300 traysPrefill16 × 1 GPU · TP1 · EnabledPrefill groupDWDP16 group of one-GPU workers (TRT-LLM dwdp_config, 56 experts/worker)Decode1 × 16 GPUs · TP16 · EP16 · EnabledSpeculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, depth 7)KV transferNIXL (UCX)PrecisionNVFP4Accelerators used32 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated - Dynamo frontend (conversation-affinity routing) over TensorRT-LLM workers, 48 GPUs on 12 GB300 traysPrefill16 × 1 GPU · TP1 · EnabledPrefill groupDWDP16 group of one-GPU workers (TRT-LLM dwdp_config, 56 experts/worker)Decode2 × 16 GPUs · TP16 · EP16 · EnabledSpeculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, depth 5)KV transferNIXL (UCX)PrecisionNVFP4Accelerators used48 of 72
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 10-12 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (Inferact/Kimi-K3-DSpark draft, 7 speculative tokens, probabilistic draft, block rejection)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 10-12 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8DecodeDCP8 (a2a)Speculative decodingDSpark (Inferact/Kimi-K3-DSpark draft, 4 speculative tokens, probabilistic draft, block rejection)KV offloadvLLM SimpleCPUOffloadConnector, 150 GB pinned DRAM per rank (recipe: size to the host)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 10-12 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8DecodeDCP8 (a2a)Speculative decodingDSpark (Inferact/Kimi-K3-DSpark draft, 4 speculative tokens, probabilistic draft, block rejection)KV offloadvLLM SimpleCPUOffloadConnector, 150 GB pinned DRAM per rank (recipe: size to the host)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB300 x8EngineSGLang (0.5.21)ServingAggregated in-flight batching - one SGLang server on all 8 GPUs of the nodeTensor parallelTP8DecodeDCP8Speculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, block size 7)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 09-29 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (Inferact/Kimi-K3-DSpark draft, 3 speculative tokens, probabilistic draft, block rejection)KV offloadvLLM SimpleCPUOffloadConnector, 150 GB pinned DRAM per rankPrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 09-29 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingDSpark (Inferact/Kimi-K3-DSpark draft, 7 speculative tokens, probabilistic draft, block rejection)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 09-29 (ROCm 10))ServingAggregated in-flight batching — one vLLM server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingDSpark (Inferact/Kimi-K3-DSpark draft, 5 speculative tokens, probabilistic draft, block rejection)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 09-29 (ROCm 10))ServingAggregated in-flight batching — one vLLM server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingDSpark (Inferact/Kimi-K3-DSpark draft, 7 speculative tokens, probabilistic draft, block rejection)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 09-29 (ROCm 10))ServingAggregated in-flight batching — one vLLM server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingDSpark (Inferact/Kimi-K3-DSpark draft, 7 speculative tokens, probabilistic draft, block rejection)KV offloadvLLM SimpleCPUOffloadConnector, 150 GB pinned DRAM per rankPrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 09-29 (ROCm 10))ServingAggregated in-flight batching — one vLLM server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingDSpark (Inferact/Kimi-K3-DSpark draft, 3 speculative tokens, probabilistic draft, block rejection)KV offloadvLLM SimpleCPUOffloadConnector, 150 GB pinned DRAM per rankPrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EngineSGLang (0.5.19 dev (ROCm 7.2, 09-16))ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one nodeTensor parallelTP8DecodeDCP8Speculative decodingDSpark (RadixArk/Kimi-K3-DSpark draft, block size 7)Mamba state memory ratio7.07PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
# NVIDIA Dynamo 1.5.0.dev20260816 (SGLang d59c1ddf) disaggregated serving of
# Kimi-K3 (MXFP4) on GB300 NVL72 — 16 GPUs across 4 nodes:
# 1x prefill worker + 1x decode worker at 8 GPUs each, behind a Dynamo
# frontend (fastokens tokenizer, kimi_k3 tool-call and reasoning parsers).
#
# Launched by the vendor launcher,
# which renders the worker YAML below and submits it through bench_sflow.
# Container: sglang-d59c1ddf-dynamo-1.5.0.dev20260816.sqsh
# Prefill worker:
--context-length 262144
--decode-log-interval 1
--disaggregation-transfer-backend mooncake
--enable-hierarchical-cache
--expert-parallel-size 8
--hicache-io-backend direct
--hicache-ratio 1
--hicache-write-policy write_back
--host 0.0.0.0
--kv-cache-dtype fp8_e4m3
--mamba-full-memory-ratio 0.14
--max-running-requests 256
--mem-fraction-static 0.85
--moe-a2a-backend megamoe
--moe-runner-backend deep_gemm
--pipeline-parallel-size 1
--reasoning-parser kimi_k3
--speculative-algorithm DSPARK
--speculative-draft-model-path <host-path>/Kimi-K3-DSpark/56ce616ad7486f0e96cbb51ef23ed5a1bce1d92d
--speculative-dspark-block-size 7
--tensor-parallel-size 8
--tool-call-parser kimi_k3
--trust-remote-code
# Prefill worker environment:
MC_FORCE_MNNVL=1
NCCL_CUMEM_ENABLE=1
NCCL_MNNVL_ENABLE=1
SGLANG_DG_CACHE_DIR=/tmp/k3-deep-gemm-${USER}-${STAMP}
SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
SGLANG_MAMBA_SSM_DTYPE=bfloat16
SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=2048
UCX_NET_DEVICES=all
UCX_TLS=rc,cuda_copy
# Decode worker:
--context-length 262144
--decode-log-interval 1
--disaggregation-transfer-backend mooncake
--enable-linear-replayssm-spec
--enable-symm-mem
--host 0.0.0.0
--kv-cache-dtype fp8_e4m3
--mamba-full-memory-ratio 0.5
--max-running-requests 256
--mem-fraction-static 0.85
--pipeline-parallel-size 1
--reasoning-parser kimi_k3
--speculative-algorithm DSPARK
--speculative-draft-model-path <host-path>/Kimi-K3-DSpark/56ce616ad7486f0e96cbb51ef23ed5a1bce1d92d
--speculative-dspark-block-size 7
--tensor-parallel-size 8
--tool-call-parser kimi_k3
--trust-remote-code
# Decode worker environment:
DYNAMO_SGLANG_EXTRA_ARGS=--enable-linear-replayssm-spec
MC_FORCE_MNNVL=1
NCCL_CUMEM_ENABLE=1
NCCL_MNNVL_ENABLE=1
SGLANG_DG_CACHE_DIR=/tmp/k3-deep-gemm-${USER}-${STAMP}
SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
SGLANG_MAMBA_SSM_DTYPE=bfloat16
SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True
UCX_NET_DEVICES=all
UCX_TLS=rc,cuda_copy
Host-specific paths are redacted as <host-path>.
SystemGB300 NVL72EngineSGLang (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend with round-robin routing over SGLang workers, DSPARK speculative decoding, 16 GPUs across 4 nodesPrefill1 × 8 GPUsDecode1 × 8 GPUs · TP8Tensor / expert parallelTP8 / EP8Hierarchical KV cacheHiCache ratio 1 (Mooncake, write-back)Speculative decodingDSpark7 + linear ReplaySSMKV transferMooncakeRequest routinground-robinPrecisionMXFP4Accelerators used16 of 72
# NVIDIA Dynamo 1.5.0.dev20260816 (SGLang d59c1ddf) disaggregated serving of
# Kimi-K3 (MXFP4) on GB300 NVL72 — 16 GPUs across 4 nodes:
# 1x prefill worker + 1x decode worker at 8 GPUs each, behind a Dynamo
# frontend (fastokens tokenizer, kimi_k3 tool-call and reasoning parsers).
#
# Launched by the vendor launcher,
# which renders the worker YAML below and submits it through bench_sflow.
# Container: sglang-d59c1ddf-dynamo-1.5.0.dev20260816.sqsh
# Prefill worker:
--context-length 262144
--decode-log-interval 1
--disaggregation-transfer-backend mooncake
--enable-hierarchical-cache
--expert-parallel-size 8
--hicache-io-backend direct
--hicache-ratio 8
--hicache-write-policy write_back
--host 0.0.0.0
--kv-cache-dtype fp8_e4m3
--mamba-full-memory-ratio 0.14
--max-running-requests 256
--mem-fraction-static 0.85
--moe-a2a-backend megamoe
--moe-runner-backend deep_gemm
--pipeline-parallel-size 1
--reasoning-parser kimi_k3
--speculative-algorithm DSPARK
--speculative-draft-model-path <host-path>/Kimi-K3-DSpark/56ce616ad7486f0e96cbb51ef23ed5a1bce1d92d
--speculative-dspark-block-size 7
--tensor-parallel-size 8
--tool-call-parser kimi_k3
--trust-remote-code
# Prefill worker environment:
MC_FORCE_MNNVL=1
NCCL_CUMEM_ENABLE=1
NCCL_MNNVL_ENABLE=1
SGLANG_DG_CACHE_DIR=/tmp/k3-deep-gemm-${USER}-${STAMP}
SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
SGLANG_MAMBA_SSM_DTYPE=bfloat16
SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=2048
UCX_NET_DEVICES=all
UCX_TLS=rc,cuda_copy
# Decode worker:
--context-length 262144
--decode-log-interval 1
--disaggregation-transfer-backend mooncake
--enable-linear-replayssm-spec
--enable-symm-mem
--host 0.0.0.0
--kv-cache-dtype fp8_e4m3
--mamba-full-memory-ratio 0.5
--max-running-requests 256
--mem-fraction-static 0.85
--pipeline-parallel-size 1
--reasoning-parser kimi_k3
--speculative-algorithm DSPARK
--speculative-draft-model-path <host-path>/Kimi-K3-DSpark/56ce616ad7486f0e96cbb51ef23ed5a1bce1d92d
--speculative-dspark-block-size 7
--tensor-parallel-size 8
--tool-call-parser kimi_k3
--trust-remote-code
# Decode worker environment:
DYNAMO_SGLANG_EXTRA_ARGS=--enable-linear-replayssm-spec
MC_FORCE_MNNVL=1
NCCL_CUMEM_ENABLE=1
NCCL_MNNVL_ENABLE=1
SGLANG_DG_CACHE_DIR=/tmp/k3-deep-gemm-${USER}-${STAMP}
SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
SGLANG_MAMBA_SSM_DTYPE=bfloat16
SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True
UCX_NET_DEVICES=all
UCX_TLS=rc,cuda_copy
Host-specific paths are redacted as <host-path>.
SystemGB300 NVL72EngineSGLang (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend with round-robin routing over SGLang workers, DSPARK speculative decoding, 16 GPUs across 4 nodesPrefill1 × 8 GPUsDecode1 × 8 GPUs · TP8Tensor / expert parallelTP8 / EP8Hierarchical KV cacheHiCache ratio 8 (Mooncake, write-back)Speculative decodingDSpark7 + linear ReplaySSMKV transferMooncakeRequest routinground-robinPrecisionMXFP4Accelerators used16 of 72
GLM-5.2 & 5.3 (max) — 26 configurations
SystemFrameworkPrecisionAccelerators UsedParallelismSpecdecDate Added
No launch command recorded for this configuration.
SystemB300 x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUs (other 4 idle; power = those 4 GPUs only)Tensor parallelTP4Speculative decodingMTP (bundled nextn layer, 5 speculative tokens)PrecisionNVFP4Accelerators used4 of 8
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemB300 x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUs (other 4 idle; power = those 4 GPUs only)Tensor parallelTP4Expert parallelEP4Speculative decodingMTP (bundled nextn layer, 5 speculative tokens)PrecisionNVFP4Accelerators used4 of 8
Submitted by NVIDIA
No launch command recorded for this configuration.
SystemB200 (SXM) x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Expert parallelEP8Speculative decodingMTP (bundled nextn layer, 5 speculative tokens)PrecisionNVFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB200 (SXM) x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingMTP (bundled nextn layer, 5 speculative tokens)PrecisionNVFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 10-06 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on 4 of the node's 8 GPUs (server CPUs pinned to that NUMA node, client on the other; power = those GPUs only)Tensor parallelTP4Speculative decodingMTP (bundled nextn layer, 5 speculative tokens)PrecisionMXFP4Accelerators used4 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB300 x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingMTP (bundled nextn layer, 5 speculative tokens)PrecisionNVFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemB300 x8EnginevLLM (v0.31.0)ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Expert parallelEP8Speculative decodingMTP (bundled nextn layer, 5 speculative tokens)PrecisionNVFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EnginevLLM (nightly 10-06 (ROCm 10))ServingAggregated in-flight batching - one vLLM server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingMTP (bundled nextn layer, 5 speculative tokens)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EngineSGLang (0.5.20 dev (ROCm 7.2, 09-23))ServingAggregated in-flight batching - one SGLang server on all 8 GPUs of the nodeTensor parallelTP8Speculative decodingEAGLE / MTP (topk 1, 6 draft tokens, 5 steps)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EngineSGLang (0.5.20 dev (ROCm 7.2, 09-23))Serving2 SGLang TP4 replicas (--dp 2) on one node, client conversation->replica affinityTensor parallelTP4Data parallelDP2 (replicas)Speculative decodingEAGLE / MTP (topk 1, 4 draft tokens, 3 steps)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EngineSGLang (0.5.19)ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one node, DP attention with client conversation->rank affinityTensor parallelTP8Data parallelDP8 attentionExpert parallelEP8 (DeepEP all-to-all)Speculative decodingEAGLE / MTP (topk 1, 2 draft tokens, 1 step)KV cacheFP8 e4m3 (DSA flashmla_sparse_q8 prefill, flashmla_kv decode)Hierarchical KV cacheHiCache host DRAM tier (ratio 4, write-through)PrecisionFP8Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EngineSGLang (0.5.19)ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingEAGLE / MTP (topk 1, 6 draft tokens, 5 steps)KV cacheFP8 e4m3 (DSA flashmla_sparse_q8 prefill, flashmla_kv decode)PrecisionNVFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EngineSGLang (0.5.19)ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingEAGLE / MTP (topk 1, 6 draft tokens, 5 steps)KV cacheFP8 e4m3 (DSA flashmla_sparse_q8 prefill, flashmla_kv decode)PrecisionFP8Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EngineSGLang (0.5.19)ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one node, DP attention with client conversation->rank affinityTensor parallelTP8Data parallelDP8 attentionExpert parallelEP8 (DeepEP all-to-all)Speculative decodingEAGLE / MTP (topk 1, 2 draft tokens, 1 step)KV cacheFP8 e4m3 (DSA flashmla_sparse_q8 prefill, flashmla_kv decode)PrecisionFP8Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EngineSGLang (0.5.19)ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one node, DP attention with client conversation->rank affinityTensor parallelTP8Data parallelDP8 attentionSpeculative decodingEAGLE / MTP (topk 1, 2 draft tokens, 1 step)KV cacheFP8 e4m3 (DSA flashmla_sparse_q8 prefill, flashmla_kv decode)PrecisionNVFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemH200 (SXM) x8EngineSGLang (0.5.19)ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingEAGLE / MTP (topk 1, 6 draft tokens, 5 steps)KV cacheBF16 (SGLang's SM90 default for DSA)PrecisionFP8Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EngineSGLang (0.5.20 dev (ROCm 7.2, 09-20))ServingTwo SGLang TP4 replicas (--dp 2) on one node, client conversation->replica affinityTensor parallelTP4Data parallelDP2 (replicas)Speculative decodingEAGLE / MTP (topk 1, 4 draft tokens, 3 steps)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
No launch command recorded for this configuration.
SystemMI355X x8EngineSGLang (0.5.20 dev (ROCm 7.2, 09-20))ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingEAGLE / MTP (topk 1, 6 draft tokens, 5 steps)PrecisionMXFP4Accelerators used8 of 8
Submitted by Artificial Analysis
# NVIDIA Dynamo (81eee4bb) + TensorRT-LLM (92d37d84) disaggregated serving of
# GLM-5.2 (NVFP4) on GB300 NVL72 — 12 GPUs across 3 nodes:
# 1x context worker (TP4 + MoE EP4, 4 GPUs) + 1x generation worker
# (TP8 + MoE EP8, MTP depth 3; P01 decode recipe: MoE backend TRTLLM, autotuner on,
# CuTe DSL top-k) + Dynamo frontend with KV-cache-aware routing
# (fastokens tokenizer, glm47 tool-call parser, glm45 reasoning parser).
# Worker environment (every worker; container paths and per-allocation endpoints omitted):
# DYN_TRTLLM_ENABLE_ATTENTION_DP=1 DYN_TRTLLM_BATCHED_EGRESS=1
# DYN_HEALTH_CHECK_ENABLED=false DYN_ENGINE_CONV_AFFINITY=1 DYN_SYSTEM_PORT=8081
# DYN_UCX_TLS=^ib DYN_TCP_REQUEST_TIMEOUT=30 DYN_TOKENIZER=fastokens DYN_KV_BLOCK_SIZE=128
# TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1
# TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0
# TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 HF_HUB_OFFLINE=1
# TRTLLM_KV_CACHE_TRANSFER_TIMEOUT_SEC=480 TRTLLM_REQUEST_KV_CACHE_CONCURRENT=1
# TLLM_PREFIX_TOKEN_CACHE=1 TRTLLM_SERVE_ENABLE_MSGSPEC=1
# TRTLLM_KVCACHE_SEND_MAX_CONCURRENCY_NUM=2 PYTHONUNBUFFERED=1 DYN_PUBLISH_KV_EVENTS=0
# DYN_TOOL_CALL_PARSER=glm47 DYN_REASONING_PARSER=glm45 SERVED_MODEL_NAME=GLM-5.2
# Dynamo frontend / router environment (OpenAI-compatible on :8333):
# ROUTER_MODE=kv DYN_KV_BLOCK_SIZE=128 DEBUG_ROUTER=0 PYTHONUNBUFFERED=1
# DYN_ROUTER_QUEUE_THRESHOLD=None DYN_ROUTER_TEMPERATURE=0
# DYN_ROUTER_TRACK_PREFILL_TOKENS=1 DYN_TOKENIZER_CACHE=1
# DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TOKENIZER=fastokens DYN_TCP_REQUEST_TIMEOUT=30
# DYN_LOG=warn DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=2400
# Bring-up: the vendor launcher renders and launches the
# worker commands with the configs below (image sha256
# f7b3c6170331df17cde8c2477c1c612930d7d6a077070bb78207e53c89497a17):
OUTPUT_DIR=<shared-output-directory> ./run.sh
# ---------------- gen_config.yaml ----------------
tensor_parallel_size: 8
moe_expert_parallel_size: 8
enable_attention_dp: false
enable_lm_head_tp_in_adp: false
pipeline_parallel_size: 1
max_batch_size: 8
max_num_tokens: 48
max_seq_len: 131072
print_iter_log: true
enable_autotuner: true
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.7
dtype: fp8
avg_seq_len: 36020
moe_config:
backend: TRTLLM
use_low_precision_moe_combine: true
cache_transceiver_config:
max_tokens_in_buffer: 524288
backend: UCX
transceiver_runtime: CPP
stream_interval: 100
num_postprocess_workers: 4
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
- 8
allreduce_strategy: MNNVL
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
sparse_attention_config:
algorithm: dsa
use_cute_dsl_paged_mqa_logits: true
use_cute_dsl_topk: true
index_share_for_mtp_iteration: true
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false
# ---------------- ctx_config.yaml ----------------
max_batch_size: 128
max_num_tokens: 8192
max_seq_len: 131072
tensor_parallel_size: 4
moe_expert_parallel_size: 4
enable_attention_dp: true
pipeline_parallel_size: 1
enable_chunked_prefill: true
print_iter_log: true
cuda_graph_config: null
enable_autotuner: true
disable_overlap_scheduler: true
moe_config:
backend: TRTLLM
kv_cache_config:
enable_block_reuse: true
free_gpu_memory_fraction: 0.7
dtype: fp8
host_cache_size: 100000000000
event_buffer_max_size: 0
avg_seq_len: 39040
cache_transceiver_config:
max_tokens_in_buffer: 524288
backend: UCX
transceiver_runtime: CPP
scheduler_config:
capacity_scheduler_policy: MAX_UTILIZATION
attention_dp_config:
enable_kv_cache_aware_routing: false
kv_cache_routing_conversation_affinity: true
kv_cache_routing_max_sessions: 65536
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
# ---------------- server_config.yaml (topology; hosts/ports omitted) ----------------
backend: pytorch
context_servers:
num_instances: 1
router:
type: kv_cache_aware
tokens_per_block: 128
backfill_block_hashes_on_finish: true
generation_servers:
num_instances: 1
router:
type: kv_cache_aware
tokens_per_block: 128
backfill_block_hashes_on_finish: true
gen_strip_message_history: true
gen_tokids_ctxbytes: trueSystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend + KV-cache-aware router over TensorRT-LLM workers, 12 GPUs across 3 nodesPrefill1 × 4 GPUs · TP4 · EP4Decode1 × 8 GPUs · TP8 · EP8Speculative decodingMTP (depth 3)PrecisionNVFP4Accelerators used12 of 72
Submitted by NVIDIA
# SGLang 0.5.17 (lmsysorg/sglang:v0.5.17-cu130, digest 16aba8925507e631e1dc1e23d95d026533602591775f6a8db68b74ee99746155) # serving GLM-5.2 (NVFP4) — GB300 2x4: TP8 across trays, EAGLE(k1,d6,s5), maxreq 16 — low-latency. # 2 nodes x 4 GPUs (--nnodes 2). NVIDIA baseline flags verbatim. python3 -m sglang.launch_server \ --config ifb_config.yaml \ --model-path nvidia/GLM-5.2-NVFP4 \ --served-model-name GLM-5.2 \ --host 0.0.0.0 --port 20800 \ --enable-cache-report # ---------------- ifb_config.yaml (verbatim, verified against the running server) ---------------- chunked-prefill-size: 8192 mem-fraction-static: 0.85 quantization: modelopt_fp4 reasoning-parser: glm45 tensor-parallel-size: 8 tool-call-parser: glm47 cuda-graph-max-bs: 16 max-prefill-tokens: 8192 max-running-requests: 16 speculative-algorithm: EAGLE speculative-eagle-topk: 1 speculative-num-draft-tokens: 6 speculative-num-steps: 5
SystemGB300 NVL72EngineSGLang (0.5.17)Tensor parallelTP8Speculative decodingEAGLE (topk 1, 6 draft tokens, 5 steps)PrecisionNVFP4Accelerators used8 of 72
Submitted by NVIDIA
# SGLang 0.5.17 (lmsysorg/sglang:v0.5.17-cu130, digest 16aba8925507e631e1dc1e23d95d026533602591775f6a8db68b74ee99746155) # serving GLM-5.2 (NVFP4) on B300 x8 — TP8, EAGLE speculative decode (topk 1, 6 draft tokens, 5 steps), max-running-requests 32. # Single node, 8 GPUs, aggregated in-flight batching. NVIDIA baseline config # glm-5-2-nvfp4-sglang-ifb-tp8-eagle5-maxreq32-bia-v1 (bench_sflow). python3 -m sglang.launch_server \ --config ifb_config.yaml \ --model-path nvidia/GLM-5.2-NVFP4 \ --served-model-name GLM-5.2 \ --host 0.0.0.0 --port 20800 \ --enable-cache-report # ---------------- ifb_config.yaml (verbatim, verified against the running server) ---------------- chunked-prefill-size: 8192 mem-fraction-static: 0.85 quantization: modelopt_fp4 reasoning-parser: glm45 tensor-parallel-size: 8 tool-call-parser: glm47 cuda-graph-max-bs: 16 max-prefill-tokens: 8192 max-running-requests: 32 speculative-algorithm: EAGLE speculative-eagle-topk: 1 speculative-num-draft-tokens: 6 speculative-num-steps: 5
SystemB300 x8EngineSGLang (0.5.17)ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingEAGLE (topk 1, 6 draft tokens, 5 steps)PrecisionNVFP4Accelerators used8 of 8
Submitted by NVIDIA
# SGLang 0.5.17 (lmsysorg/sglang:v0.5.17-cu130, digest 16aba8925507e631e1dc1e23d95d026533602591775f6a8db68b74ee99746155) # serving GLM-5.2 (NVFP4) on B300 x8 — TP8, EAGLE speculative decode (topk 1, 6 draft tokens, 5 steps), max-running-requests 16 — low-latency. # Single node, 8 GPUs, aggregated in-flight batching. NVIDIA baseline config # glm-5-2-sglang-ifb-tp8-eagle-low-latency-b300-v1 (bench_sflow). python3 -m sglang.launch_server \ --config ifb_config.yaml \ --model-path nvidia/GLM-5.2-NVFP4 \ --served-model-name GLM-5.2 \ --host 0.0.0.0 --port 20800 \ --enable-cache-report # ---------------- ifb_config.yaml (verbatim, verified against the running server) ---------------- chunked-prefill-size: 8192 mem-fraction-static: 0.85 quantization: modelopt_fp4 reasoning-parser: glm45 tensor-parallel-size: 8 tool-call-parser: glm47 cuda-graph-max-bs: 16 max-prefill-tokens: 8192 max-running-requests: 16 speculative-algorithm: EAGLE speculative-eagle-topk: 1 speculative-num-draft-tokens: 6 speculative-num-steps: 5
SystemB300 x8EngineSGLang (0.5.17)ServingAggregated in-flight batching — one SGLang server, 8 GPUs on one nodeTensor parallelTP8Speculative decodingEAGLE (topk 1, 6 draft tokens, 5 steps)PrecisionNVFP4Accelerators used8 of 8
Submitted by NVIDIA
# NVIDIA Dynamo (81eee4bb) + TensorRT-LLM (92d37d84) disaggregated serving of
# GLM-5.2 (NVFP4) on GB300 NVL72 — 56 GPUs across 14 nodes:
# 6x context workers (TP4 + MoE EP4, 4 GPUs each) + 1x generation worker
# (TP32 + MoE EP32, MTP depth 5) + Dynamo frontend with KV-cache-aware routing
# (fastokens tokenizer, glm47 tool-call parser, glm45 reasoning parser).
# Worker environment (every worker; container paths and per-allocation endpoints omitted):
# DYN_TRTLLM_ENABLE_ATTENTION_DP=1 DYN_TRTLLM_BATCHED_EGRESS=1
# DYN_HEALTH_CHECK_ENABLED=false DYN_ENGINE_CONV_AFFINITY=1 DYN_SYSTEM_PORT=8081
# DYN_UCX_TLS=^ib DYN_TCP_REQUEST_TIMEOUT=30 DYN_TOKENIZER=fastokens DYN_KV_BLOCK_SIZE=128
# TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1
# TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0
# TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 HF_HUB_OFFLINE=1
# TRTLLM_KV_CACHE_TRANSFER_TIMEOUT_SEC=480 TRTLLM_REQUEST_KV_CACHE_CONCURRENT=1
# TLLM_PREFIX_TOKEN_CACHE=1 TRTLLM_SERVE_ENABLE_MSGSPEC=1
# TRTLLM_KVCACHE_SEND_MAX_CONCURRENCY_NUM=2 PYTHONUNBUFFERED=1 DYN_PUBLISH_KV_EVENTS=0
# DYN_TOOL_CALL_PARSER=glm47 DYN_REASONING_PARSER=glm45 SERVED_MODEL_NAME=GLM-5.2
# Dynamo frontend / router environment (OpenAI-compatible on :8333):
# ROUTER_MODE=kv DYN_KV_BLOCK_SIZE=128 DEBUG_ROUTER=0 PYTHONUNBUFFERED=1
# DYN_ROUTER_QUEUE_THRESHOLD=None DYN_ROUTER_TEMPERATURE=0
# DYN_ROUTER_TRACK_PREFILL_TOKENS=1 DYN_TOKENIZER_CACHE=1
# DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TOKENIZER=fastokens DYN_TCP_REQUEST_TIMEOUT=30
# DYN_LOG=warn DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=2400
# Bring-up is driven by NVIDIA's submitter, which renders and launches the worker
# commands with the configs below.
# ---------------- gen_config.yaml ----------------
tensor_parallel_size: 32
moe_expert_parallel_size: 32
enable_attention_dp: true
enable_lm_head_tp_in_adp: false
pipeline_parallel_size: 1
max_batch_size: 8
max_num_tokens: 48
max_seq_len: 131072
print_iter_log: true
enable_autotuner: false
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.7
dtype: fp8
avg_seq_len: 36020
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
cache_transceiver_config:
max_tokens_in_buffer: 524288
backend: UCX
transceiver_runtime: CPP
stream_interval: 100
num_postprocess_workers: 4
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
- 8
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 5
sparse_attention_config:
algorithm: dsa
use_cute_dsl_paged_mqa_logits: true
index_share_for_mtp_iteration: true
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false
# ---------------- ctx_config.yaml ----------------
max_batch_size: 128
max_num_tokens: 8192
max_seq_len: 131072
tensor_parallel_size: 4
moe_expert_parallel_size: 4
enable_attention_dp: true
pipeline_parallel_size: 1
enable_chunked_prefill: true
print_iter_log: true
cuda_graph_config: null
enable_autotuner: true
disable_overlap_scheduler: true
moe_config:
backend: TRTLLM
kv_cache_config:
enable_block_reuse: true
free_gpu_memory_fraction: 0.7
dtype: fp8
host_cache_size: 100000000000
event_buffer_max_size: 0
avg_seq_len: 39040
cache_transceiver_config:
max_tokens_in_buffer: 524288
backend: UCX
transceiver_runtime: CPP
scheduler_config:
capacity_scheduler_policy: MAX_UTILIZATION
attention_dp_config:
enable_kv_cache_aware_routing: false
kv_cache_routing_conversation_affinity: true
kv_cache_routing_max_sessions: 65536
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 5
# ---------------- server_config.yaml (topology; hosts/ports omitted) ----------------
backend: pytorch
context_servers:
num_instances: 6
router:
type: kv_cache_aware
tokens_per_block: 128
backfill_block_hashes_on_finish: true
generation_servers:
num_instances: 1
router:
type: kv_cache_aware
tokens_per_block: 128
backfill_block_hashes_on_finish: true
gen_strip_message_history: true
gen_tokids_ctxbytes: trueSystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend + KV-cache-aware router over TensorRT-LLM workers, 56 GPUs across 14 nodesPrefill6 × 4 GPUs · TP4 · EP4Decode1 × 32 GPUs · TP32 · EP32 · Enabled (TP32)Speculative decodingMTP (depth 5)PrecisionNVFP4Accelerators used56 of 72
Submitted by NVIDIA
# NVIDIA Dynamo (81eee4bb) + TensorRT-LLM (92d37d84) disaggregated serving of
# GLM-5.2 (NVFP4) on GB300 NVL72 — 48 GPUs across 12 nodes:
# 4x context workers (TP4 + MoE EP4, 4 GPUs each) + 1x generation worker
# (TP32 + MoE EP32, MTP depth 5) + Dynamo frontend with KV-cache-aware routing
# (fastokens tokenizer, glm47 tool-call parser, glm45 reasoning parser).
# Worker environment (every worker; container paths and per-allocation endpoints omitted):
# DYN_TRTLLM_ENABLE_ATTENTION_DP=1 DYN_TRTLLM_BATCHED_EGRESS=1
# DYN_HEALTH_CHECK_ENABLED=false DYN_ENGINE_CONV_AFFINITY=1 DYN_SYSTEM_PORT=8081
# DYN_UCX_TLS=^ib DYN_TCP_REQUEST_TIMEOUT=30 DYN_TOKENIZER=fastokens DYN_KV_BLOCK_SIZE=128
# TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1
# TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0
# TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 HF_HUB_OFFLINE=1
# TRTLLM_KV_CACHE_TRANSFER_TIMEOUT_SEC=480 TRTLLM_REQUEST_KV_CACHE_CONCURRENT=1
# TLLM_PREFIX_TOKEN_CACHE=1 TRTLLM_SERVE_ENABLE_MSGSPEC=1
# TRTLLM_KVCACHE_SEND_MAX_CONCURRENCY_NUM=2 PYTHONUNBUFFERED=1 DYN_PUBLISH_KV_EVENTS=0
# DYN_TOOL_CALL_PARSER=glm47 DYN_REASONING_PARSER=glm45 SERVED_MODEL_NAME=GLM-5.2
# Dynamo frontend / router environment (OpenAI-compatible on :8333):
# ROUTER_MODE=kv DYN_KV_BLOCK_SIZE=128 DEBUG_ROUTER=0 PYTHONUNBUFFERED=1
# DYN_ROUTER_QUEUE_THRESHOLD=None DYN_ROUTER_TEMPERATURE=0
# DYN_ROUTER_TRACK_PREFILL_TOKENS=1 DYN_TOKENIZER_CACHE=1
# DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TOKENIZER=fastokens DYN_TCP_REQUEST_TIMEOUT=30
# DYN_LOG=warn DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=2400
# Bring-up is driven by NVIDIA's submitter, which renders and launches the worker
# commands with the configs below.
# ---------------- gen_config.yaml ----------------
tensor_parallel_size: 32
moe_expert_parallel_size: 32
enable_attention_dp: true
enable_lm_head_tp_in_adp: false
pipeline_parallel_size: 1
max_batch_size: 4
max_num_tokens: 24
max_seq_len: 131072
print_iter_log: true
enable_autotuner: false
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.7
dtype: fp8
avg_seq_len: 36020
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
cache_transceiver_config:
max_tokens_in_buffer: 524288
backend: UCX
transceiver_runtime: CPP
stream_interval: 100
num_postprocess_workers: 4
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 5
sparse_attention_config:
algorithm: dsa
use_cute_dsl_paged_mqa_logits: true
index_share_for_mtp_iteration: true
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false
# ---------------- ctx_config.yaml ----------------
max_batch_size: 128
max_num_tokens: 8192
max_seq_len: 131072
tensor_parallel_size: 4
moe_expert_parallel_size: 4
enable_attention_dp: true
pipeline_parallel_size: 1
enable_chunked_prefill: true
print_iter_log: true
cuda_graph_config: null
enable_autotuner: true
disable_overlap_scheduler: true
moe_config:
backend: TRTLLM
kv_cache_config:
enable_block_reuse: true
free_gpu_memory_fraction: 0.7
dtype: fp8
host_cache_size: 100000000000
event_buffer_max_size: 0
avg_seq_len: 39040
cache_transceiver_config:
max_tokens_in_buffer: 524288
backend: UCX
transceiver_runtime: CPP
scheduler_config:
capacity_scheduler_policy: MAX_UTILIZATION
attention_dp_config:
enable_kv_cache_aware_routing: false
kv_cache_routing_conversation_affinity: true
kv_cache_routing_max_sessions: 65536
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 5
# ---------------- server_config.yaml (topology; hosts/ports omitted) ----------------
backend: pytorch
context_servers:
num_instances: 4
router:
type: kv_cache_aware
tokens_per_block: 128
backfill_block_hashes_on_finish: true
generation_servers:
num_instances: 1
router:
type: kv_cache_aware
tokens_per_block: 128
backfill_block_hashes_on_finish: true
gen_strip_message_history: true
gen_tokids_ctxbytes: trueSystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend + KV-cache-aware router over TensorRT-LLM workers, 48 GPUs across 12 nodesPrefill4 × 4 GPUs · TP4 · EP4Decode1 × 32 GPUs · TP32 · EP32 · Enabled (TP32)Speculative decodingMTP (depth 5)PrecisionNVFP4Accelerators used48 of 72
Submitted by NVIDIA
# NVIDIA Dynamo (81eee4bb) + TensorRT-LLM (92d37d84) disaggregated serving of
# GLM-5.2 (NVFP4) on GB300 NVL72 — 64 GPUs across 16 nodes:
# 12x context workers (TP4 + MoE EP4, 4 GPUs each) + 1x generation worker
# (TP16 + MoE EP16, MTP depth 2) + Dynamo frontend with KV-cache-aware routing
# (fastokens tokenizer, glm47 tool-call parser, glm45 reasoning parser).
# Worker environment (every worker; container paths and per-allocation endpoints omitted):
# DYN_TRTLLM_ENABLE_ATTENTION_DP=1 DYN_TRTLLM_BATCHED_EGRESS=1
# DYN_HEALTH_CHECK_ENABLED=false DYN_ENGINE_CONV_AFFINITY=1 DYN_SYSTEM_PORT=8081
# DYN_UCX_TLS=^ib DYN_TCP_REQUEST_TIMEOUT=30 DYN_TOKENIZER=fastokens DYN_KV_BLOCK_SIZE=128
# TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1
# TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0
# TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 HF_HUB_OFFLINE=1
# TRTLLM_KV_CACHE_TRANSFER_TIMEOUT_SEC=480 TRTLLM_REQUEST_KV_CACHE_CONCURRENT=1
# TLLM_PREFIX_TOKEN_CACHE=1 TRTLLM_SERVE_ENABLE_MSGSPEC=1
# TRTLLM_KVCACHE_SEND_MAX_CONCURRENCY_NUM=2 TRTLLM_ENABLE_KVCACHE_RECEIVE_PARALLEL=1
# TRTLLM_KVCACHE_RECV_BUFFER_COUNT=2 PYTHONUNBUFFERED=1 DYN_PUBLISH_KV_EVENTS=0
# DYN_TOOL_CALL_PARSER=glm47 DYN_REASONING_PARSER=glm45 SERVED_MODEL_NAME=GLM-5.2
# Dynamo frontend / router environment (OpenAI-compatible on :8333):
# ROUTER_MODE=kv DYN_KV_BLOCK_SIZE=128 DEBUG_ROUTER=0 PYTHONUNBUFFERED=1
# DYN_ROUTER_QUEUE_THRESHOLD=None DYN_ROUTER_TEMPERATURE=0
# DYN_ROUTER_TRACK_PREFILL_TOKENS=1 DYN_TOKENIZER_CACHE=1
# DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TOKENIZER=fastokens DYN_TCP_REQUEST_TIMEOUT=30
# DYN_LOG=warn DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=2400
# Bring-up is driven by NVIDIA's submitter, which renders and launches the worker
# commands with the configs below.
# ---------------- gen_config.yaml ----------------
tensor_parallel_size: 16
moe_expert_parallel_size: 16
enable_attention_dp: true
enable_lm_head_tp_in_adp: false
pipeline_parallel_size: 1
max_batch_size: 48
max_num_tokens: 288
max_seq_len: 131072
print_iter_log: true
enable_autotuner: false
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.7
dtype: fp8
avg_seq_len: 36020
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
cache_transceiver_config:
max_tokens_in_buffer: 524288
backend: UCX
transceiver_runtime: CPP
stream_interval: 100
num_postprocess_workers: 4
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
- 8
- 16
- 32
- 48
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 2
sparse_attention_config:
algorithm: dsa
use_cute_dsl_paged_mqa_logits: true
index_share_for_mtp_iteration: true
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false
# ---------------- ctx_config.yaml ----------------
max_batch_size: 128
max_num_tokens: 8192
max_seq_len: 131072
tensor_parallel_size: 4
moe_expert_parallel_size: 4
enable_attention_dp: true
pipeline_parallel_size: 1
enable_chunked_prefill: true
print_iter_log: true
cuda_graph_config: null
enable_autotuner: true
disable_overlap_scheduler: true
moe_config:
backend: TRTLLM
kv_cache_config:
enable_block_reuse: true
free_gpu_memory_fraction: 0.7
dtype: fp8
host_cache_size: 100000000000
event_buffer_max_size: 0
avg_seq_len: 39040
cache_transceiver_config:
max_tokens_in_buffer: 524288
backend: UCX
transceiver_runtime: CPP
scheduler_config:
capacity_scheduler_policy: MAX_UTILIZATION
attention_dp_config:
enable_kv_cache_aware_routing: false
kv_cache_routing_conversation_affinity: true
kv_cache_routing_max_sessions: 65536
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 2
# ---------------- server_config.yaml (topology; hosts/ports omitted) ----------------
backend: pytorch
context_servers:
num_instances: 12
router:
type: kv_cache_aware
tokens_per_block: 128
backfill_block_hashes_on_finish: true
generation_servers:
num_instances: 1
router:
type: kv_cache_aware
tokens_per_block: 128
backfill_block_hashes_on_finish: true
gen_strip_message_history: true
gen_tokids_ctxbytes: trueSystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend + KV-cache-aware router over TensorRT-LLM workers, 64 GPUs across 16 nodesPrefill12 × 4 GPUs · TP4 · EP4Decode1 × 16 GPUs · TP16 · EP16 · Enabled (TP16)Speculative decodingMTP (depth 2)PrecisionNVFP4Accelerators used64 of 72
Submitted by NVIDIA
# NVIDIA Dynamo (81eee4bb) + TensorRT-LLM (92d37d84) disaggregated serving of
# GLM-5.2 (NVFP4) on GB300 NVL72 — 48 GPUs across 12 nodes:
# 8x context workers (TP4 + MoE EP4, 4 GPUs each) + 1x generation worker
# (TP16 + MoE EP16, MTP depth 3) + Dynamo frontend with KV-cache-aware routing
# (fastokens tokenizer, glm47 tool-call parser, glm45 reasoning parser).
# Worker environment (every worker; container paths and per-allocation endpoints omitted):
# DYN_TRTLLM_ENABLE_ATTENTION_DP=1 DYN_TRTLLM_BATCHED_EGRESS=1
# DYN_HEALTH_CHECK_ENABLED=false DYN_ENGINE_CONV_AFFINITY=1 DYN_SYSTEM_PORT=8081
# DYN_UCX_TLS=^ib DYN_TCP_REQUEST_TIMEOUT=30 DYN_TOKENIZER=fastokens DYN_KV_BLOCK_SIZE=128
# TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1
# TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0
# TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 HF_HUB_OFFLINE=1
# TRTLLM_KV_CACHE_TRANSFER_TIMEOUT_SEC=480 TRTLLM_REQUEST_KV_CACHE_CONCURRENT=1
# TLLM_PREFIX_TOKEN_CACHE=1 TRTLLM_SERVE_ENABLE_MSGSPEC=1
# TRTLLM_KVCACHE_SEND_MAX_CONCURRENCY_NUM=2 TRTLLM_ENABLE_KVCACHE_RECEIVE_PARALLEL=1
# TRTLLM_KVCACHE_RECV_BUFFER_COUNT=2 PYTHONUNBUFFERED=1 DYN_PUBLISH_KV_EVENTS=0
# DYN_TOOL_CALL_PARSER=glm47 DYN_REASONING_PARSER=glm45 SERVED_MODEL_NAME=GLM-5.2
# Dynamo frontend / router environment (OpenAI-compatible on :8333):
# ROUTER_MODE=kv DYN_KV_BLOCK_SIZE=128 DEBUG_ROUTER=0 PYTHONUNBUFFERED=1
# DYN_ROUTER_QUEUE_THRESHOLD=None DYN_ROUTER_TEMPERATURE=0
# DYN_ROUTER_TRACK_PREFILL_TOKENS=1 DYN_TOKENIZER_CACHE=1
# DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TOKENIZER=fastokens DYN_TCP_REQUEST_TIMEOUT=30
# DYN_LOG=warn DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=2400
# Bring-up is driven by NVIDIA's submitter, which renders and launches the worker
# commands with the configs below.
# ---------------- gen_config.yaml ----------------
tensor_parallel_size: 16
moe_expert_parallel_size: 16
enable_attention_dp: true
enable_lm_head_tp_in_adp: false
pipeline_parallel_size: 1
max_batch_size: 32
max_num_tokens: 192
max_seq_len: 131072
print_iter_log: true
enable_autotuner: false
kv_cache_config:
enable_block_reuse: false
free_gpu_memory_fraction: 0.7
dtype: fp8
avg_seq_len: 36020
moe_config:
backend: CUTEDSL
use_low_precision_moe_combine: true
cache_transceiver_config:
max_tokens_in_buffer: 524288
backend: UCX
transceiver_runtime: CPP
stream_interval: 100
num_postprocess_workers: 4
cuda_graph_config:
enable_padding: true
batch_sizes:
- 1
- 2
- 4
- 8
- 16
- 32
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
sparse_attention_config:
algorithm: dsa
use_cute_dsl_paged_mqa_logits: true
index_share_for_mtp_iteration: true
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false
# ---------------- ctx_config.yaml ----------------
max_batch_size: 128
max_num_tokens: 8192
max_seq_len: 131072
tensor_parallel_size: 4
moe_expert_parallel_size: 4
enable_attention_dp: true
pipeline_parallel_size: 1
enable_chunked_prefill: true
print_iter_log: true
cuda_graph_config: null
enable_autotuner: true
disable_overlap_scheduler: true
moe_config:
backend: TRTLLM
kv_cache_config:
enable_block_reuse: true
free_gpu_memory_fraction: 0.7
dtype: fp8
host_cache_size: 100000000000
event_buffer_max_size: 0
avg_seq_len: 39040
cache_transceiver_config:
max_tokens_in_buffer: 524288
backend: UCX
transceiver_runtime: CPP
scheduler_config:
capacity_scheduler_policy: MAX_UTILIZATION
attention_dp_config:
enable_kv_cache_aware_routing: false
kv_cache_routing_conversation_affinity: true
kv_cache_routing_max_sessions: 65536
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
# ---------------- server_config.yaml (topology; hosts/ports omitted) ----------------
backend: pytorch
context_servers:
num_instances: 8
router:
type: kv_cache_aware
tokens_per_block: 128
backfill_block_hashes_on_finish: true
generation_servers:
num_instances: 1
router:
type: kv_cache_aware
tokens_per_block: 128
backfill_block_hashes_on_finish: true
gen_strip_message_history: true
gen_tokids_ctxbytes: trueSystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated — Dynamo frontend + KV-cache-aware router over TensorRT-LLM workers, 48 GPUs across 12 nodesPrefill8 × 4 GPUs · TP4 · EP4Decode1 × 16 GPUs · TP16 · EP16 · Enabled (TP16)Speculative decodingMTP (depth 3)PrecisionNVFP4Accelerators used48 of 72
Submitted by NVIDIA
DeepSeek V4 Pro (max) — 0 configurations
SystemFrameworkPrecisionAccelerators UsedParallelismSpecdecDate Added
GLM-5.3 (max) — 0 configurations
SystemFrameworkPrecisionAccelerators UsedParallelismSpecdecDate Added
gpt-oss-120b (high) — 0 configurations
SystemFrameworkPrecisionAccelerators UsedParallelismSpecdecDate Added