Datacenter Inference Systems

Inference Serving Configurations

Every serving configuration used in AA-AgentPerf hardware benchmarking, from June 2026 onwards. Each entry captures the model, accelerator system, precision, inference framework, parallelism topology, and full launch command.

Read the AA-AgentPerf methodology.

AA-AgentPerf

Serving configurations from AA-AgentPerf benchmark runs, with the full launch command for each. Read the methodology

Showing 15 of 15 configurations

Kimi K3 (max) — 6 configurations

# Kimi K3 (MXFP4) on B300 x8 — SGLang low-latency profile, TP=8, no speculative decoding.
docker run --gpus all --shm-size 32g --network host --ipc=host \
  lmsysorg/sglang:kimi-k3 sglang serve \
    --trust-remote-code --model-path moonshotai/Kimi-K3 --tp-size 8 \
    --disable-custom-all-reduce --mem-fraction-static 0.85 \
    --reasoning-parser kimi_k3 --tool-call-parser kimi_k3 --max-running-requests 256 \
    --enable-cache-report --enable-metrics --enable-symm-mem
SystemB300 x8EngineSGLang (kimi-k3)Tensor parallelTP8Speculative decodingNonePrecisionMXFP4Accelerators used8 of 8

Submitted by Artificial Analysis

# Kimi K3 (MXFP4) on B300 x8 — SGLang low-latency profile, TP=8, DSPARK speculative decoding.
docker run --gpus all --shm-size 32g --network host --ipc=host \
  lmsysorg/sglang:kimi-k3 sglang serve \
    --trust-remote-code --model-path moonshotai/Kimi-K3 --tp-size 8 \
    --disable-custom-all-reduce --mem-fraction-static 0.85 \
    --reasoning-parser kimi_k3 --tool-call-parser kimi_k3 --max-running-requests 256 \
    --enable-cache-report --enable-metrics --enable-symm-mem \
    --speculative-algorithm DSPARK --speculative-draft-model-path RadixArk/Kimi-K3-DSpark \
    --speculative-dspark-block-size 7 --enable-linear-replayssm-spec
SystemB300 x8EngineSGLang (kimi-k3)Tensor parallelTP8Speculative decodingDSPARK (Kimi-K3-DSpark draft, block size 7)PrecisionMXFP4Accelerators used8 of 8

Submitted by Artificial Analysis

# Kimi K3 (MXFP4) on B300 x8 — SGLang balanced profile, TP=8 / DCP=8, no speculative decoding.
docker run --gpus all --shm-size 32g --network host --ipc=host \
  lmsysorg/sglang:kimi-k3 sglang serve \
    --trust-remote-code --model-path moonshotai/Kimi-K3 --tp-size 8 \
    --disable-custom-all-reduce --mem-fraction-static 0.85 \
    --reasoning-parser kimi_k3 --tool-call-parser kimi_k3 --max-running-requests 256 \
    --enable-cache-report --enable-metrics --dcp-size 8
SystemB300 x8EngineSGLang (kimi-k3)Tensor parallelTP8DecodeDCP8Speculative decodingNonePrecisionMXFP4Accelerators used8 of 8

Submitted by Artificial Analysis

# Kimi K3 (MXFP4) on GB300 NVL72 — SGLang, TP=8 across 2 trays (8 GPUs), DSPARK speculative decoding. One launch per node.
sglang serve \
  --trust-remote-code \
  --max-running-requests 256 \
  --model-path /models/Kimi-K3 \
  --tp-size 8 \
  --nnodes 2 \
  --node-rank <0..1> \
  --dist-init-addr <head-ip>:20000 \
  --mem-fraction-static 0.85 \
  --context-length 262144 \
  --reasoning-parser kimi_k3 \
  --tool-call-parser kimi_k3 \
  --mamba-full-memory-ratio 0.22 \
  --host 0.0.0.0 \
  --port 30000 \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path /models/Kimi-K3-DSpark \
  --speculative-dspark-block-size 7 \
  --enable-linear-replayssm-spec
SystemGB300 NVL72EngineSGLang (kimi-k3)Tensor parallelTP8 (2 trays × 4 GPUs)Speculative decodingDSPARK (Kimi-K3-DSpark draft, block size 7)Mamba state memory ratio0.22PrecisionMXFP4Accelerators used8 of 72

Submitted by Artificial Analysis

# Kimi K3 (MXFP4) on GB300 NVL72 — SGLang, TP=8 / DCP=8 across 2 trays (8 GPUs), no speculative decoding. One launch per node.
sglang serve \
  --trust-remote-code \
  --max-running-requests 256 \
  --model-path /models/Kimi-K3 \
  --tp-size 8 \
  --nnodes 2 \
  --node-rank <0..1> \
  --dist-init-addr <head-ip>:20000 \
  --dcp-size 8 \
  --mem-fraction-static 0.85 \
  --context-length 262144 \
  --reasoning-parser kimi_k3 \
  --tool-call-parser kimi_k3 \
  --mamba-full-memory-ratio 1.89 \
  --host 0.0.0.0 \
  --port 30000
SystemGB300 NVL72EngineSGLang (kimi-k3)Tensor parallelTP8 (2 trays × 4 GPUs)DecodeDCP8Speculative decodingNoneMamba state memory ratio1.89PrecisionMXFP4Accelerators used8 of 72

Submitted by Artificial Analysis

# Kimi K3 (MXFP4) on GB300 NVL72 — SGLang, TP=8 across 2 trays (8 GPUs), no speculative decoding. One launch per node.
sglang serve \
  --trust-remote-code \
  --max-running-requests 256 \
  --model-path /models/Kimi-K3 \
  --tp-size 8 \
  --nnodes 2 \
  --node-rank <0..1> \
  --dist-init-addr <head-ip>:20000 \
  --mem-fraction-static 0.85 \
  --context-length 262144 \
  --reasoning-parser kimi_k3 \
  --tool-call-parser kimi_k3 \
  --mamba-full-memory-ratio 0.24 \
  --host 0.0.0.0 \
  --port 30000
SystemGB300 NVL72EngineSGLang (kimi-k3)Tensor parallelTP8 (2 trays × 4 GPUs)Speculative decodingNoneMamba state memory ratio0.24PrecisionMXFP4Accelerators used8 of 72

Submitted by Artificial Analysis

DeepSeek V4 Pro (max) — 6 configurations

# TensorRT-LLM (feat/deepseek_v4 @ ace71873) + NVIDIA Dynamo (@ 5a43d4ad) disaggregated
# serving on GB300 NVL72 — 32 GPUs (8 compute trays x 4):
# 6x context workers (TP4, attention-DP -> DP4 + MoE EP4, 4 GPUs each) + 1x generation
# worker (8 GPUs over 2 trays, attention-DP8 + MoE EP8) + Dynamo frontend with KV-aware
# conversation-affinity routing.

# Worker environment (every worker; CUDA_VISIBLE_DEVICES=0,1,2,3 per tray):
#   TLLM_LOG_LEVEL=INFO TRTLLM_SERVE_ENABLE_ORJSON=1 TRTLLM_SERVER_DISABLE_GC=1
#   TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0
#   TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 HF_HUB_OFFLINE=1
#   TRTLLM_KV_CACHE_TRANSFER_TIMEOUT_SEC=480 TLLM_DISAGG_GEN_PRIORITIZE_FIRST_TOKEN=1
#   DYN_TOKENIZER=fastokens DYN_TCP_REQUEST_TIMEOUT=30 DYN_UCX_TLS=^ib
#   DYN_HEALTH_CHECK_ENABLED=false DYN_ENGINE_CONV_AFFINITY=1 DYN_KV_BLOCK_SIZE=128
#   DYN_TOOL_CALL_PARSER=deepseek_v4 DYN_REASONING_PARSER=deepseek_v4
#   (context workers additionally: TRTLLM_SKIP_KV_CACHE_ESTIMATION=1)

# Generation worker (8 GPUs, MPI across 2 trays):
trtllm-llmapi-launch python3 -m dynamo.trtllm \
  --model-path <model-path> --served-model-name <model-path> \
  --extra-engine-args gen_config.yaml --disaggregation-mode decode \
  --request-plane tcp --event-plane zmq

# Context workers (x6, one per tray):
trtllm-llmapi-launch python3 -m dynamo.trtllm \
  --model-path <model-path> --served-model-name <model-path> \
  --extra-engine-args ctx_config.yaml --disaggregation-mode prefill \
  --request-plane tcp --event-plane zmq

# Dynamo frontend (router env: DYN_ROUTER_TEMPERATURE=0 DYN_ROUTER_QUEUE_THRESHOLD=None
#   DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=300 DYN_TOKENIZER=fastokens DYN_TOKENIZER_CACHE=1
#   DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TCP_REQUEST_TIMEOUT=30):
python3 -m dynamo.frontend --http-port 8333 --router-mode kv --no-router-kv-events \
  --kv-cache-block-size 128 --enforce-disagg --router-reset-states \
  --request-plane tcp --event-plane zmq --dyn-tool-call-parser deepseek_v4

# ---------------- gen_config.yaml ----------------
tensor_parallel_size: 8
moe_expert_parallel_size: 8
enable_attention_dp: true
enable_lm_head_tp_in_adp: true
pipeline_parallel_size: 1
max_batch_size: 288
max_num_tokens: 1152
max_seq_len: 131072
print_iter_log: true
enable_autotuner: false
kv_cache_config:
  enable_block_reuse: false
  free_gpu_memory_fraction: 0.8
  dtype: fp8
  host_cache_size: 42949672960
  avg_seq_len: 39040
moe_config:
  backend: MEGAMOE_DEEPGEMM
  use_low_precision_moe_combine: true
cache_transceiver_config:
  max_tokens_in_buffer: 131072
  backend: NIXL
  transceiver_runtime: PYTHON
stream_interval: 100
num_postprocess_workers: 8
cuda_graph_config:
  enable_padding: true
  batch_sizes:
  - 1
  - 2
  - 4
  - 8
  - 16
  - 32
  - 64
  - 128
  - 192
  - 256
  - 272
  - 288
speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 3
  allow_advanced_sampling: true
sparse_attention_config:
  algorithm: deepseek_v4
  enable_heuristic_topk: true
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false

# ---------------- ctx_config.yaml ----------------
max_batch_size: 128
max_num_tokens: 8192
max_seq_len: 131072
tensor_parallel_size: 4
moe_expert_parallel_size: 4
enable_attention_dp: true
pipeline_parallel_size: 1
enable_chunked_prefill: true
print_iter_log: true
cuda_graph_config: null
enable_autotuner: true
disable_overlap_scheduler: true
moe_config:
  backend: TRTLLM
kv_cache_config:
  enable_block_reuse: true
  free_gpu_memory_fraction: 0.6
  dtype: fp8
  host_cache_size: 171798691840
  event_buffer_max_size: 0
  pool_ratio:
  - 0.3
  - 0.45
  - 0.25
cache_transceiver_config:
  max_tokens_in_buffer: 131072
  backend: NIXL
  transceiver_runtime: PYTHON
scheduler_config:
  capacity_scheduler_policy: MAX_UTILIZATION
attention_dp_config:
  enable_kv_cache_aware_routing: false
  kv_cache_routing_conversation_affinity: true
  kv_cache_routing_max_sessions: 65536
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false
speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 3

# ---------------- server_config.yaml (topology; hosts/ports omitted) ----------------
context_servers:
  num_instances: 6
  router:
    type: kv_cache_aware
    tokens_per_block: 128
generation_servers:
  num_instances: 1
  router:
    type: kv_cache_aware
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated (TensorRT-LLM + Dynamo, KV-aware conversation-affinity routing)Prefill6 × 4 GPUs (1 tray each) · TP4 · DP4 · EP4Decode1 × 8 GPUs (2 trays) · DP8 · EP8Speculative decodingMTP, 3 draft tokensPrecisionNVFP4Accelerators used32 of 72

Submitted by NVIDIA

# TensorRT-LLM (feat/deepseek_v4 @ ace71873) + NVIDIA Dynamo (@ 5a43d4ad) disaggregated
# serving on GB300 NVL72 — 36 GPUs (9 compute trays x 4):
# 5x context workers (TP4, attention-DP -> DP4 + MoE EP4, 4 GPUs each) + 1x generation
# worker (16 GPUs over 4 trays, attention-DP16 + MoE EP16) + Dynamo frontend with KV-aware
# conversation-affinity routing.

# Worker environment (every worker; CUDA_VISIBLE_DEVICES=0,1,2,3 per tray):
#   TLLM_LOG_LEVEL=INFO TRTLLM_SERVE_ENABLE_ORJSON=1 TRTLLM_SERVER_DISABLE_GC=1
#   TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0
#   TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10 HF_HUB_OFFLINE=1
#   TRTLLM_KV_CACHE_TRANSFER_TIMEOUT_SEC=480 TLLM_DISAGG_GEN_PRIORITIZE_FIRST_TOKEN=1
#   DYN_TOKENIZER=fastokens DYN_TCP_REQUEST_TIMEOUT=30 DYN_UCX_TLS=^ib
#   DYN_HEALTH_CHECK_ENABLED=false DYN_ENGINE_CONV_AFFINITY=1 DYN_KV_BLOCK_SIZE=128
#   DYN_TOOL_CALL_PARSER=deepseek_v4 DYN_REASONING_PARSER=deepseek_v4
#   (context workers additionally: TRTLLM_SKIP_KV_CACHE_ESTIMATION=1)

# Generation worker (16 GPUs, MPI across 4 trays):
trtllm-llmapi-launch python3 -m dynamo.trtllm \
  --model-path <model-path> --served-model-name <model-path> \
  --extra-engine-args gen_config.yaml --disaggregation-mode decode \
  --request-plane tcp --event-plane zmq

# Context workers (x5, one per tray):
trtllm-llmapi-launch python3 -m dynamo.trtllm \
  --model-path <model-path> --served-model-name <model-path> \
  --extra-engine-args ctx_config.yaml --disaggregation-mode prefill \
  --request-plane tcp --event-plane zmq

# Dynamo frontend (router env: DYN_ROUTER_TEMPERATURE=0 DYN_ROUTER_QUEUE_THRESHOLD=None
#   DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=300 DYN_TOKENIZER=fastokens DYN_TOKENIZER_CACHE=1
#   DYN_TOKENIZER_CACHE_BYTES=8000000000 DYN_TCP_REQUEST_TIMEOUT=30
#   LD_PRELOAD=libjemalloc.so.2):
python3 -m dynamo.frontend --http-port 8333 --router-mode kv --no-router-kv-events \
  --kv-cache-block-size 128 --enforce-disagg --router-reset-states \
  --request-plane tcp --event-plane zmq --dyn-tool-call-parser deepseek_v4

# ---------------- gen_config.yaml ----------------
tensor_parallel_size: 16
moe_expert_parallel_size: 16
enable_attention_dp: true
enable_lm_head_tp_in_adp: true
pipeline_parallel_size: 1
max_batch_size: 44
max_num_tokens: 1024
max_seq_len: 131072
print_iter_log: true
enable_autotuner: false
kv_cache_config:
  enable_block_reuse: false
  free_gpu_memory_fraction: 0.8
  dtype: fp8
  host_cache_size: 42949672960
  avg_seq_len: 39040
moe_config:
  backend: MEGAMOE_DEEPGEMM
  use_low_precision_moe_combine: true
cache_transceiver_config:
  max_tokens_in_buffer: 131072
  backend: NIXL
  transceiver_runtime: PYTHON
stream_interval: 100
num_postprocess_workers: 8
cuda_graph_config:
  enable_padding: true
  batch_sizes:
  - 1
  - 2
  - 4
  - 8
  - 16
  - 32
  - 36
  - 40
  - 44
speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 3
  allow_advanced_sampling: true
sparse_attention_config:
  algorithm: deepseek_v4
  enable_heuristic_topk: true
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false

# ---------------- ctx_config.yaml ----------------
max_batch_size: 128
max_num_tokens: 8192
max_seq_len: 131072
tensor_parallel_size: 4
moe_expert_parallel_size: 4
enable_attention_dp: true
pipeline_parallel_size: 1
enable_chunked_prefill: true
print_iter_log: true
cuda_graph_config: null
enable_autotuner: true
disable_overlap_scheduler: true
moe_config:
  backend: TRTLLM
kv_cache_config:
  enable_block_reuse: true
  free_gpu_memory_fraction: 0.6
  dtype: fp8
  host_cache_size: 171798691840
  event_buffer_max_size: 0
  pool_ratio:
  - 0.3
  - 0.45
  - 0.25
cache_transceiver_config:
  max_tokens_in_buffer: 131072
  backend: NIXL
  transceiver_runtime: PYTHON
scheduler_config:
  capacity_scheduler_policy: MAX_UTILIZATION
attention_dp_config:
  enable_kv_cache_aware_routing: false
  kv_cache_routing_conversation_affinity: true
  kv_cache_routing_max_sessions: 65536
return_perf_metrics: false
perf_metrics_max_requests: 200000
enable_iter_perf_stats: false
enable_iter_req_stats: false
speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 3

# ---------------- server_config.yaml (topology; hosts/ports omitted) ----------------
context_servers:
  num_instances: 5
  router:
    type: kv_cache_aware
    tokens_per_block: 128
generation_servers:
  num_instances: 1
  router:
    type: kv_cache_aware
SystemGB300 NVL72EngineTRT-LLM (Dynamo)ServingPrefill/decode disaggregated (TensorRT-LLM + Dynamo, KV-aware conversation-affinity routing)Prefill5 × 4 GPUs (1 tray each) · TP4 · DP4 · EP4Decode1 × 16 GPUs (4 trays) · DP16 · EP16Speculative decodingMTP, 3 draft tokensPrecisionNVFP4Accelerators used36 of 72

Submitted by NVIDIA

# TensorRT-LLM 1.3.0rc15.post1 disaggregated serving on a single 8x B300 node:
# 1x context worker (TP4 + EP4, attention-DP, GPUs 4-7) + 1x generation worker (TP4 + EP4,
# attention-DP, GPUs 0-3), colocated, + disagg orchestrator with conversation-affinity routing.
# SLURM allocation: --exclusive --cpus-per-task=28 (224 CPUs).

# Worker environment:
#   TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1
#   TRTLLM_ENABLE_PDL=1 NCCL_GRAPH_MIXING_SUPPORT=0 TLLM_ADP_ROUTER_MATCH_RATE_THRESHOLD=0.10
#   TLLM_ADP_ROUTER_DECISION_LOG=1 TLLM_ADP_ROUTER_INCLUDE_TRANSFER_LOAD=1
#   TLLM_ADP_ROUTER_RANDOMIZE_TIEBREAK=1 TRTLLM_DISABLE_KV_CACHE_RATIO_UPDATE=1
#   TRTLLM_MHC_ENABLE_FUSED_HC=0 UCX_TLS=rc,sm,self,cuda_copy,cuda_ipc
#   HF_HUB_OFFLINE=1 TRTLLM_KV_CACHE_TRANSFER_TIMEOUT_SEC=480

# Generation worker (CUDA_VISIBLE_DEVICES=0,1,2,3):
trtllm-serve <model-path> --config gen_config.yaml \
  --custom_tokenizer deepseek_v4 --tool_parser deepseek_v4

# Context worker (CUDA_VISIBLE_DEVICES=4,5,6,7):
trtllm-serve <model-path> --config ctx_config.yaml \
  --custom_tokenizer deepseek_v4 --tool_parser deepseek_v4

# Orchestrator:
trtllm-serve disaggregated -c server_config.yaml -t 7200 -r 7200

# ---------------- gen_config.yaml ----------------
backend: pytorch
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
enable_attention_dp: true
enable_lm_head_tp_in_adp: true
max_batch_size: 64
max_num_tokens: 1024
max_seq_len: 131072
enable_autotuner: false
print_iter_log: true
stream_interval: 100
num_postprocess_workers: 4
moe_config:
  backend: TRTLLM
  use_low_precision_moe_combine: true
kv_cache_config:
  enable_block_reuse: false
  free_gpu_memory_fraction: 0.8
  dtype: fp8
  host_cache_size: 0
  iteration_stats_interval: 1
  use_kv_cache_manager_v2: false
cache_transceiver_config:
  max_tokens_in_buffer: 131072
  backend: NIXL
  transceiver_runtime: PYTHON
cuda_graph_config:
  enable_padding: true
  batch_sizes:
  - 1
  - 2
  - 4
  - 8
  - 16
  - 32
  - 64
speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 3
return_perf_metrics: false
perf_metrics_max_requests: 100000
enable_iter_perf_stats: false
enable_iter_req_stats: false

# ---------------- ctx_config.yaml ----------------
backend: pytorch
tensor_parallel_size: 4
moe_expert_parallel_size: 4
pipeline_parallel_size: 1
enable_attention_dp: true
max_batch_size: 128
max_num_tokens: 8192
max_seq_len: 131072
enable_chunked_prefill: true
disable_overlap_scheduler: true
enable_autotuner: true
print_iter_log: true
cuda_graph_config: null
moe_config:
  backend: TRTLLM
kv_cache_config:
  enable_block_reuse: true
  free_gpu_memory_fraction: 0.8
  dtype: fp8
  host_cache_size: 214748364800
  event_buffer_max_size: 16384
  iteration_stats_interval: 1
  use_kv_cache_manager_v2: false
cache_transceiver_config:
  max_tokens_in_buffer: 131072
  backend: NIXL
  transceiver_runtime: PYTHON
scheduler_config:
  capacity_scheduler_policy: MAX_UTILIZATION
attention_dp_config:
  enable_kv_cache_aware_routing: true
  kv_cache_routing_load_balance_weight: 0.5
  kv_cache_routing_cold_start_warmup: true
  kv_cache_routing_match_rate_threshold: 0.1
speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 3
return_perf_metrics: false
perf_metrics_max_requests: 100000
enable_iter_perf_stats: false
enable_iter_req_stats: false

# ---------------- server_config.yaml (topology; hosts/ports omitted) ----------------
backend: pytorch
context_servers:
  num_instances: 1
  router:
    type: conversation
generation_servers:
  num_instances: 1
  router:
    type: conversation
SystemB300 x8EngineTRT-LLM (1.3.0rc15.post1)ServingPrefill/decode disaggregated, colocated on one 8× B300 node (TensorRT-LLM disagg orchestrator, conversation-affinity routing)Prefill1 × 4 GPUs (GPUs 4–7) · TP4 · DP4 · EP4Decode1 × 4 GPUs (GPUs 0–3) · TP4 · DP4 · EP4Speculative decodingMTP, 3 draft tokensPrecisionNVFP4Accelerators used8 of 8

Submitted by NVIDIA

docker run --gpus all --shm-size 32g --network host \
  -v path/to/huggingface:/root/.cache/huggingface \
  lmsysorg/sglang:deepseek-v4-hopper \
  python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V4-Pro \
    --tp 8 --context-length 393216 --mem-fraction-static 0.88 \
    --speculative-algorithm EAGLE --speculative-num-steps 1 \
    --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
    --moe-runner-backend marlin --host 0.0.0.0 --port 30000
SystemH200 (SXM) x8EngineSGLang (0.5.11 DSv4)Tensor parallelTP8Speculative decodingEAGLE (1 step, 2 draft tokens)PrecisionNVFP4Accelerators used8 of 8

Submitted by Artificial Analysis

docker run -d \
  --device=/dev/kfd --device=/dev/dri \
  --shm-size 32g --ipc=host --network=host --privileged \
  -v <host-path>:/model \
  -e SGLANG_DSV4_FP4_EXPERTS=True \
  -e SGLANG_FORCE_TRITON_MOE_FP8=0 \
  -e SGLANG_USE_AITER=1 \
  -e SGLANG_USE_ROCM700A=1 \
  -e SGLANG_HACK_FLASHMLA_BACKEND=triton \
  -e SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1 \
  -e SGLANG_TOPK_TRANSFORM_512_TORCH=0 \
  -e SGLANG_OPT_USE_FUSED_COMPRESS=true \
  -e SGLANG_OPT_USE_OLD_COMPRESSOR=false \
  -e SGLANG_OPT_USE_AITER_MHC_PRE=true \
  -e SGLANG_OPT_USE_AITER_MHC_POST=true \
  -e SGLANG_OPT_USE_FUSED_STORE_CACHE=true \
  -e SGLANG_OPT_USE_OVERLAP_STORE_CACHE=false \
  -e SGLANG_OPT_USE_FUSED_PAGED_COMPRESS=true \
  -e SGLANG_OPT_FUSE_WQA_WKV=true \
  -e SGLANG_OPT_USE_TRITON_SWA_PREPARE=true \
  -e SGLANG_OPT_USE_FUSED_HASH_TOPK=true \
  -e SGLANG_OPT_DPSK_V4_RADIX=1 \
  -e SGLANG_OPT_USE_TILELANG_INDEXER=true \
  -e AITER_BF16_FP8_MOE_BOUND=1 \
  -e SGLANG_OPT_USE_MULTI_STREAM_OVERLAP=0 \
  rocm/sgl-dev:rocm720-mi35x-f96ac98-20260527-DSv4 \
  python3 -m sglang.launch_server \
    --model-path /model \
    --served-model-name deepseek-ai/DeepSeek-V4-Pro \
    --tp 8 \
    --port 30000 \
    --host 0.0.0.0 \
    --trust-remote-code \
    --attention-backend compressed \
    --max-running-requests 256 \
    --page-size 256 \
    --context-length 32768 \
    --mem-fraction-static 0.90 \
    --swa-full-tokens-ratio 0.15 \
    --chunked-prefill-size 8192 \
    --disable-shared-experts-fusion \
    --tool-call-parser deepseekv4 \
    --reasoning-parser deepseek-v4

Host-specific paths are redacted as <host-path>.

SystemMI355X x8EngineSGLang (ROCm 7.2 DSv4 dev)Tensor parallelTP8Data parallelDP1Expert parallelEP1PrecisionFP4Accelerators used8 of 8

Submitted by Artificial Analysis

docker run -d \
  --device=/dev/kfd --device=/dev/dri \
  --shm-size 32g --ipc=host --network=host --privileged \
  -v <host-path>:/model \
  -e SGLANG_DSV4_FP4_EXPERTS=True \
  -e SGLANG_FORCE_TRITON_MOE_FP8=0 \
  -e SGLANG_USE_AITER=1 \
  -e SGLANG_USE_ROCM700A=1 \
  -e SGLANG_HACK_FLASHMLA_BACKEND=triton \
  -e SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1 \
  -e SGLANG_TOPK_TRANSFORM_512_TORCH=0 \
  -e SGLANG_OPT_USE_FUSED_COMPRESS=true \
  -e SGLANG_OPT_USE_OLD_COMPRESSOR=false \
  -e SGLANG_OPT_USE_AITER_MHC_PRE=true \
  -e SGLANG_OPT_USE_AITER_MHC_POST=true \
  -e SGLANG_OPT_USE_FUSED_STORE_CACHE=true \
  -e SGLANG_OPT_USE_OVERLAP_STORE_CACHE=false \
  -e SGLANG_OPT_USE_FUSED_PAGED_COMPRESS=true \
  -e SGLANG_OPT_FUSE_WQA_WKV=true \
  -e SGLANG_OPT_USE_TRITON_SWA_PREPARE=true \
  -e SGLANG_OPT_USE_FUSED_HASH_TOPK=true \
  -e SGLANG_OPT_DPSK_V4_RADIX=1 \
  -e SGLANG_OPT_USE_TILELANG_INDEXER=true \
  -e AITER_BF16_FP8_MOE_BOUND=1 \
  -e SGLANG_OPT_USE_MULTI_STREAM_OVERLAP=0 \
  rocm/sgl-dev:rocm720-mi35x-3b59630-20260526-DSv4 \
  python3 -m sglang.launch_server \
    --model-path /model \
    --served-model-name deepseek-ai/DeepSeek-V4-Pro \
    --tp 8 \
    --port 30000 \
    --host 0.0.0.0 \
    --trust-remote-code \
    --attention-backend compressed \
    --max-running-requests 256 \
    --page-size 256 \
    --chunked-prefill-size 8192 \
    --disable-shared-experts-fusion \
    --tool-call-parser deepseekv4 \
    --reasoning-parser deepseek-v4

Host-specific paths are redacted as <host-path>.

SystemMI355X x8EngineSGLang (ROCm 7.2 DSv4 dev)Tensor parallelTP8Data parallelDP1Expert parallelEP1PrecisionFP4Accelerators used8 of 8

Submitted by Artificial Analysis

gpt-oss-120b (high) — 3 configurations

#!/bin/bash
# gpt-oss-120b (FP4) on B200 — TensorRT-LLM 1.3.0rc12.post1 + NVIDIA Dynamo, disaggregated:
# 1x prefill worker (GPU0, TP1) + 3x independent decode workers (GPU1..3, TP1 each),
# Dynamo frontend (:8000) with KV-aware routing across decode replicas.
# EAGLE3 speculative decoding (3 draft tokens) on both roles.
set -u
export DYN_REQUEST_PLANE=tcp
export DYN_EVENT_PLANE=zmq
export UCX_TLS=cuda_ipc,cuda_copy,tcp,self
export DYN_REASONING_PARSER=gpt_oss
export DYN_TOOL_CALL_PARSER=harmony
export ETCD_ENDPOINTS=http://localhost:2379
MODEL_PATH=openai/gpt-oss-120b
SERVED=gpt-oss-120b

etcd --data-dir /tmp/etcd --listen-client-urls http://0.0.0.0:2379 \
  --advertise-client-urls http://localhost:2379 &
sleep 4

# Prefill worker (GPU0, TP1)
CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.trtllm \
  --model-path "$MODEL_PATH" --served-model-name "$SERVED" \
  --disaggregation-mode prefill --extra-engine-args /work/nsc_prefill_engine.yaml \
  --request-plane tcp --event-plane zmq &

# 3x decode workers (GPU1..3, TP1 each)
for g in 1 2 3; do
  CUDA_VISIBLE_DEVICES=$g python3 -m dynamo.trtllm \
    --model-path "$MODEL_PATH" --served-model-name "$SERVED" \
    --disaggregation-mode decode --extra-engine-args /work/nsc_decode_engine.yaml \
    --request-plane tcp --event-plane zmq &
done

# Dynamo frontend (KV router, :8000)
python3 -m dynamo.frontend --http-port 8000 --request-plane tcp --event-plane zmq \
  --router-mode kv --no-router-kv-events --router-ttl-secs 2400.0 &
SystemB200 (SXM) x8EngineTRT-LLM (Dynamo)Prefill1 × 1 GPU · TP1Decode3 × 1 GPU · TP1Speculative decodingEAGLE3 (3 draft tokens)PrecisionFP4Accelerators used4 of 8

Submitted by NVIDIA

#!/bin/bash
# gpt-oss-120b (FP4) on B200 — TensorRT-LLM 1.3.0rc12.post1 + NVIDIA Dynamo, disaggregated:
# 1x prefill worker (GPU0, TP1) + 1x decode worker (GPU1..4, TP4 via
# mpirun trtllm-llmapi-launch per the NVIDIA recipe), Dynamo frontend (:8000)
# with KV-aware routing. EAGLE3 speculative decoding (3 draft tokens) on both roles.
set -u
export DYN_REQUEST_PLANE=tcp
export DYN_EVENT_PLANE=zmq
export UCX_TLS=cuda_ipc,cuda_copy,tcp,self
export DYN_REASONING_PARSER=gpt_oss
export DYN_TOOL_CALL_PARSER=harmony
export ETCD_ENDPOINTS=http://localhost:2379
MODEL_PATH=openai/gpt-oss-120b
SERVED=gpt-oss-120b

etcd --data-dir /tmp/etcd --listen-client-urls http://0.0.0.0:2379 \
  --advertise-client-urls http://localhost:2379 &
sleep 4

# Prefill worker (GPU0, TP1)
CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.trtllm \
  --model-path "$MODEL_PATH" --served-model-name "$SERVED" \
  --disaggregation-mode prefill --extra-engine-args /work/nsc_prefill_engine.yaml \
  --request-plane tcp --event-plane zmq &

# Decode worker (GPU1..4, TP4 via mpirun)
CUDA_VISIBLE_DEVICES=1,2,3,4 mpirun --allow-run-as-root -n 4 trtllm-llmapi-launch \
  python3 -m dynamo.trtllm \
  --model-path "$MODEL_PATH" --served-model-name "$SERVED" \
  --disaggregation-mode decode --extra-engine-args /work/nsc_decode_engine_tp4.yaml \
  --request-plane tcp --event-plane zmq &

# Dynamo frontend (KV router, :8000)
python3 -m dynamo.frontend --http-port 8000 --request-plane tcp --event-plane zmq \
  --router-mode kv --no-router-kv-events --router-ttl-secs 2400.0 &
SystemB200 (SXM) x8EngineTRT-LLM (Dynamo)Prefill1 × 1 GPU · TP1Decode1 × 4 GPUs · TP4Speculative decodingEAGLE3 (3 draft tokens)PrecisionFP4Accelerators used5 of 8

Submitted by NVIDIA

# gpt-oss-120b (FP4) on B200 — TensorRT-LLM 1.3.0rc12.post1, single GPU (TP1),
# Eagle3-v3 speculative decoding (3 draft tokens), stream_interval 10.
docker run -d --gpus '"device=0"' --ipc=host --shm-size=64g -p 8000:8000 \
  -v <host-path>/hf-cache:/root/.cache/huggingface \
  -v <host-path>/extra_llm_api_options.yaml:/etc/trtllm/extra_llm_api_options.yaml:ro \
  -e HF_HOME=/root/.cache/huggingface -e TRTLLM_ENABLE_PDL=1 -e OPAL_PREFIX=/opt/hpcx/ompi \
  -e OMPI_MCA_coll_hcoll_enable=0 -e UCC_CL_BASIC_TLS='^sharp' -e UCC_EC_CUDA_EXEC_NUM_THREADS=256 \
  <trtllm-image> \
  trtllm-serve openai/gpt-oss-120b --host 0.0.0.0 --port 8000 \
    --extra_llm_api_options /etc/trtllm/extra_llm_api_options.yaml

# ---------------- extra_llm_api_options.yaml ----------------
tensor_parallel_size: 1
moe_expert_parallel_size: 1
pipeline_parallel_size: 1
enable_attention_dp: false
cuda_graph_config:
  enable_padding: true
  max_batch_size: 256
kv_cache_config:
  dtype: fp8
  enable_block_reuse: true
  free_gpu_memory_fraction: 0.9
moe_config:
  backend: TRTLLM
max_seq_len: 131072
max_batch_size: 256
max_num_tokens: 20000
num_postprocess_workers: 4
stream_interval: 10
print_iter_log: true
enable_chunked_prefill: true
trust_remote_code: true
speculative_config:
  decoding_type: Eagle3
  max_draft_len: 3
  speculative_model: nvidia/gpt-oss-120b-Eagle3-v3
  eagle3_one_model: true
  eagle3_layers_to_capture: [23, 29, 35]
  allow_advanced_sampling: true

Host-specific paths are redacted as <host-path>.

SystemB200 (SXM) x8EngineTRT-LLM (1.3.0rc12.post1 (@ 7a20c8))Tensor parallelTP1Data parallelDP1Expert parallelEP1Speculative decodingEAGLE3-v3 (3 draft tokens)PrecisionFP4Accelerators used1 of 8

Submitted by NVIDIA

AA-SLT

Coming soon. In the meantime, AA-SLT serving configurations are published in the System & Benchmark Specifications table.