Build a standalone, production grade system that trains a 4,...

Prompt

Build a standalone, production grade system that trains a 4,960,153,600 parameter language model from scratch and runs long sequence generation on one high bandwidth datacenter accelerator (SM103 architecture, 288 GB unified HBM, 8.0 TB/s bandwidth, dense microscaling tensor cores). It converges inside 250 million total token presentations (0.05 tokens per parameter), natively supports a 1,048,576 token context and sustains generation of 131,072 tokens. Done means all eleven files below are emitted complete in this answer, every function, method, class, tensor op and Triton kernel fully implemented with real mathematics, runnable as written. 1 ARCHITECTURE 32 layer hybrid decoder, exact 3:1 periodic interleave: 24 Gated Delta Rule Matrix Memory layers, 8 Grouped Query Attention layers. Three recurrent blocks then one attention block per cycle, so attention sits at layer indices 3, 7, 11, 15, 19, 23, 27, 31. Hidden dim 3584. Feed forward: gated SwiGLU, intermediate dim 11008 (3.072 times hidden); intermediate = elementwise product of SiLU(gate projection) and the up projection, then down projection to 3584. RMSNorm with learnable scale and stabilizer 0.000001. Activation projections keep hypersphere geometry, holding intermediate state vectors at fixed unit radius sqrt(3584) = 59.866 to stabilize optimization. Vocabulary 65,536. Embedding weights tied between input lookup table and output language modeling head. Recurrent Matrix Memory blocks (24 layers): associative state [Batch, Heads=28, KeyDim=128, ValueDim=128]. Inputs project to query, key, value, write gate, erase gate and data dependent decay vectors, head dim 128 across 28 heads. Input conditioned delta rule matrix scan, S_prev being the previous state: S_t = gamma_t ⊙ S_prev + beta_t ⊙ (v_t − S_prev k_t) k_t^T y_t = S_t q_t beta_t = sigmoid(proj_beta(x_t)), write gate alpha_t = sigmoid(proj_alpha(x_t)), decay scalar gamma_t = exp(−softplus(proj_gamma(x_t))). Chunk parallel training: uniform chunks of C=128 tokens; intra chunk feedback solved by lower triangular inversion (I − A)^{−1} with A_{i,j} = beta_j · (k_j^T k_i) · prod_{m=j+1}^{i} gamma_m for i > j; intra chunk outputs and terminal state matrices from fused matmuls; inter chunk state propagated by associative prefix scan in shared GPU SRAM. Step mode: constant O(1) update, about 550 MB total across the 24 recurrent layers at any sequence position. Spectral dynamics: decay parameterization regularized across a broad eigenvalue spectrum to sustain gradient propagation along flat curvature directions. Grouped Query Attention blocks (8 layers): 28 query heads, 4 key value heads (7:1), head dim 128. Extended rotary embeddings, base frequency 5,000,000 with progressive wavelength scaling over 1,048,576 contiguous positions. Sliding window buffer of 8,192 tokens plus persistent anchor key value states for bounded cache: 67.1 MB in 8 bit across the 8,192 token window, 8.59 GB when all 1,048,576 positions are kept in 8 bit. Drafting head: dual depth projection on final hidden states producing K=2 draft tokens, evaluated concurrently through speculative tree verification masks during decoding. 2 MICROSCALING PRECISION AND KERNELS Microscaling 4 bit matmul for linear projections: E2M1 values, FP8 E4M3 block scales per tile of 32 elements. Activations pass orthogonal Random Hadamard Transform rotations via fast butterfly transforms before quantization, dispersing magnitude peaks across channels and preserving fidelity. Backward passes use stochastic rounding adding uniform noise U(0, 1) before mantissa truncation. Auxiliary microscaled 8 bit path with delayed scaling factors and dynamic exponent mantissa formats. 16 bit brain float and 32 bit single precision stay in use for recurrent state accumulation, associative prefix scans, normalization reductions, rotary rotations, softmax and final loss. Five fused Triton kernels: 1 Fused chunked cross entropy: vocabulary projections in blocks of 2,048 tokens over 65,536 classes directly from final hidden activations, online log sum exp, gradients written straight into hidden states, peak memory below 500 MB so the 131 GB unchunked logit tensor stays unallocated. 2 Chunk parallel associative scan: intra chunk lower triangular delta interactions over C=128 blocks plus inter chunk state propagation by shared memory scan reductions. 3 Matrix orthogonalization: in place 5th order Newton Schulz iterations normalizing momentum matrices to exact unit spectral norm; inspects dimensions R and C, prescales by 1/sqrt(||M||_F · ||M||_inf + 0.0000001), operates on X X^T when R < C. 4 Block scaled microscopic GEMM: contractions over 4 bit tensors with packed scale vectors. 5 Fused RMSNorm: row wise variance reduction and affine scaling in one memory pass. 3 OPTIMIZATION Partitioned dual optimizer over the 250 million token budget. Polar decomposition matrix orthogonalized momentum optimizer on every 2D internal weight matrix (query, key, value, output projections; feed forward gate, up, down projections; recurrent input, gating, state transition matrices). Momentum accumulator beta=0.95. Each step the momentum tensor undergoes polar decomposition by 5th order Newton Schulz over five iterations: X_0 = M / sqrt(||M||_F · ||M||_inf + 0.0000001) X_{k+1} = X_k (3.4445 I − 4.7750 X_k^T X_k + 2.0315 (X_k^T X_k)^2) The resulting orthogonal matrix updates weights with a unit spectral step scale times muP learning rate multipliers. Decoupled weight decay adaptive moment optimizer for all 1D parameters (norm scales, biases, scalar gates) and the tied embedding table: first moment 0.9, second moment 0.95, epsilon 0.00000001, weight decay 0.1. Maximal Update Parameterization: linear projection learning rates scale as eta_base · (d_base / d_in), d_base = 256; attention output projection as eta_base · (1 / d_model); hidden linear weights initialize Gaussian with std 1/sqrt(d_in), output head with 1/d_model, embeddings with std 1.0; final logits divided by sqrt(d_model)/16.0 to stabilize softmax temperature. Annealed spectral conditioning: soft orthogonality penalty on internal projection matrices adding lambda_spectral · ||W^T W − I||_F^2, intensity 0.05 over the first 10 percent of tokens (25 million), then cosine decay to 0 across the next 15 percent (reaching 0 at 62.5 million tokens), expanding representation rank early and leaving later adaptation fluid. Warmup stable decay schedule over 250 million tokens: linear warmup 0 to peak over the first 2 percent (5 million); peak held over 70 percent (175 million) for bulk parameter travel; cosine warmdown to 1 percent of peak over the final 28 percent (70 million). Terminal shrinkage averaging: across the final 5 percent of steps (12.5 million tokens) keep an exponential moving average of terminal iteration weights, balancing late trajectory progress with variance reduction, and emit it as the final checkpoint. 4 CURRICULUM AND PIPELINE Phase 1, bulk ingestion: 225 million tokens (90 percent) in packed sequences of length 2,048 to 8,192, documents concatenated contiguously into fixed length buffers with document boundary index arrays and sequence boundary reset indicators, maximizing arithmetic intensity and tensor core saturation and building foundational representations and factual associations. Phase 2, progressive context scaling: remaining 25 million tokens (10 percent) over four tiers. Tier A length 16,384 over 10 million tokens; Tier B 65,536 over 7.5 million; Tier C 131,072 over 5 million; Tier D 1,048,576 over 2.5 million. This adapts matrix memory state transition dynamics, aligns long range decay variables and stabilizes rotary interpolation at extreme depth. Token scaled supervised masking: loss restricted to assistant target completion positions, user instruction tokens masked out, loss normalized by active target token count per micro batch. 5 INFERENCE AND GENERATION Ingestion and state compression: prompt tokens flow through the 24 recurrent layers by chunk parallel associative scan, collapsing 1,048,576 tokens into a fixed size state [Batch, 28, 128, 128] per layer, recurrent memory invariant to prompt length at about 550 MB total. The 8 attention layers hold key value states in a paged, block allocated cache with 4 bit or 8 bit quantization, an 8,192 token sliding window and anchor tokens, the aggregate cache staying below 10 GB across the full context. Decoding loop: O(1) recurrent state updates and circular buffer insertions in attention blocks, per step bandwidth governed by streaming compact weights and active cache slices over the 8.0 TB/s bus. Speculative verification: the dual depth draft head proposes K=2 candidates ahead of the stream; the backbone validates them in one parallel forward pass with a causal tree attention mask, commits matching tokens, reaches up to 1.8 times baseline throughput and preserves the exact output distribution. Long horizon stability: semantic coherence, structural consistency and apt termination across all 131,072 generated tokens via dynamic frequency and presence penalties and bounded entropy monitoring. 6 FILES 1 model_config.py: ModelConfig dataclass with explicit integer attributes layers=32, hidden_dim=3584, ffn_dim=11008, num_query_heads=28, num_kv_heads=4, head_dim=128, vocab_size=65536, recurrent_layers=24, attention_layers=8, recurrent_heads=28, recurrent_key_dim=128, recurrent_value_dim=128, max_context_len=1048576, max_output_len=131072, sliding_window_size=8192, rope_base_theta=5000000.0, rms_norm_eps=0.000001, hypersphere_radius=59.866. Periodic layer typing: recurrent where (layer_index + 1) % 4 != 0, attention where (layer_index + 1) % 4 == 0. Plus precision configuration, muP multipliers (d_base=256) and optimization schedules. 2 custom_kernels.py: Triton JIT kernels implementing the five kernels of section 2, named _chunk_associative_scan_kernel (triangular inversion, block parallel scan, state transformations and outputs in SRAM), _fused_chunked_cross_entropy_kernel (tiles of 2,048 tokens by 4,096 vocabulary classes, online log sum exp, scalar loss, in place hidden gradients), _newton_schulz_kernel (aspect ratio transposition handled), _microscaled_fp4_gemm_kernel, _fused_rmsnorm_kernel (sum of squares, reciprocal square root, affine scaling in one pass). PyTorch autograd wrappers: chunk_associative_scan(), fused_chunked_cross_entropy(), newton_schulz_orthogonalize(), microscaled_fp4_gemm(), fused_rmsnorm(). 3 recurrent_layers.py: GatedDeltaRuleMatrixMemoryBlock(nn.Module). Projects hidden dim 3584 into query, key, value, alpha write gate, beta erase gate and gamma decay, head dim 128 across 28 heads. Pretraining forward partitions into C=128 chunks, calls chunk_associative_scan(), returns [Batch, SeqLen, 3584]. Stateful step forward takes single step inputs and previous state [Batch, 28, 128, 128], applies rank 1 delta rule updates with decay gating, returns updated state and step output. RMSNorm on output projection before residual addition. 4 attention_layers.py: GroupedQueryAttentionBlock(nn.Module), 28 query and 4 key value heads of dim 128; rotary embeddings of section 1 up to position 1,048,576; causal attention over the 8,192 token sliding window with persistent anchor tokens; paged 4 bit and 8 bit quantized key value cache buffers appending new keys and values into indexed memory pools; output projection and RMSNorm. 5 hybrid_model.py: HybridLanguageModel(nn.Module). Tied embedding [65536, 3584]; 32 decoder layers in the periodic 3:1 pattern (24 recurrent, 8 attention), each pairing its sequence operator with a SwiGLU feed forward network and pre operator RMSNorm; final RMSNorm before the output head; auxiliary multi token prediction draft head of two linear layers with intermediate SwiGLU mapping final hidden states to speculative candidate logits. Forward supports training mode (hidden representations for chunked loss) and generation mode (accepting and returning recurrent states and paged cache blocks). 6 spectral_optimizer.py: DualSpectralOptimizer splitting parameters into 2D matrices and 1D vectors plus embeddings and applying the two optimizers of section 3 (2D: momentum beta=0.95 with the Newton Schulz kernel normalizing steps to unit spectral norm; 1D and embeddings: adaptive moment 0.9, 0.95, decay 0.1), enforcing muP multipliers eta_base · (256 / d_in) and annealed spectral conditioning with soft orthogonality gradients decaying smoothly to 0. 7 scheduler.py: WarmupStableDecayScheduler computing the step dependent learning rate multiplier of section 3, and TerminalShrinkageAverager keeping the exponential moving average buffer across the final 5 percent of steps and consolidating it into deployable weights. 8 dataset_pipeline.py: PackedSequenceDataLoader streaming pre tokenized binary datasets of 250 million tokens, assembling packed sequences of target length 2,048 to 1,048,576 by concatenating documents with sequence boundary indicator tensors, generating attention block masks and response only loss masks (user instruction tokens weight 0, normalization over active response tokens), dispatching progressive length batches per curriculum phase. 9 train_engine.py: TrainingEngine handling accelerator allocation, model instantiation, optimizer setup and precision casting; tracking Model FLOPs Utilization against SM103 peak compute and logging tokens per second; running Phase 1 and Phase 2 of section 4; invoking fused chunked cross entropy for gradients directly from hidden states; selective activation recomputation and gradient accumulation; terminal shrinkage averaging at completion and export of final weights. 10 generation_engine.py: StreamingGenerationEngine ingesting prompts up to 1,048,576 tokens by chunk parallel recurrent processing and seeding paged key value buffers; running the autoregressive loop for 131,072 consecutive tokens; speculative decoding with the draft head, parallel validation through the main decoder with tree masks, commitment of accepted tokens; paged cache eviction and in place recurrent state updates; dynamic temperature, frequency and presence penalties for coherence. 11 main.py: command line entry point with a train command starting pretraining on the 250 million token dataset and a generate command loading trained weights, ingesting long prompts and streaming up to 131,072 tokens; parses configuration, verifies accelerator capability and triggers workflows. 7 DELIVERY Deliver pure, raw executable source code and valid configuration syntax, file by file, each complete and self contained, with concrete mathematics and tensor logic in every function, method, class and kernel. Expose open, direct local execution interfaces using full accelerator compute and memory bandwidth. Use generic, descriptive technical identifiers for every framework, module, algorithm and architecture. Where a detail is open, pick a sound value, note it in a brief comment and continue to full completion.