detailed contro lstack

Prompt

# Prompt 1 — Multimodal H1/H64/H192 Reference Architecture for Closed-Loop CS2 Control You are being asked to act as an independent architecture reviewer. You have no access to project files, so all relevant project context is provided below. Do not redesign the entire stack from scratch. Answer the specific architectural question and propose falsifiable experiments. ## Project objective The project is building a machine-learning control stack that imitates strong human / pro-player behavior in Counter-Strike 2 on private research servers. The long-term objective is **human-like behavior**, not merely maximizing win rate. A strict boundary is already established: ```text learned behavioral / physical reference ↓ deterministic controller ↓ semantic action request ↓ mapper / capability layer ↓ build-specific native actuator ``` The ML side must never learn build-specific native bytes, addresses, or actuator internals. ## Current established predictor stack Three useful structured movement predictors already exist and are protected scientific references: - **H1:** approximately 15.6 ms future horizon - **H64:** approximately 1 s future horizon - **H192:** approximately 3 s future horizon They were trained from server-authoritative demo trajectories and currently produce structured physical / trajectory predictions rather than raw keypresses. Important empirical result: > Directly mapping predictor outputs to motor commands was inadequate. The current adjudication is: **`KEEP_PREDICTORS_REDESIGN_CONTROLLER`** The likely interpretation is now: - H1 = immediate desired kinematic/reference information - H64 = near-future trajectory/lookahead information - H192 = longer trajectory/lookahead information - deterministic feedback control is responsible for actually realizing the reference The current live-control roadmap therefore establishes the controller first: ```text timing/cadence qualification → plant/system identification → scripted-reference deterministic feedback → frozen H1/H64/H192 reference integration → rollout/error attribution → only then memory/retraining decisions ``` No recurrent adaptor has yet been earned. ## Important negative evidence Two negative results must be respected but not overgeneralized: 1. A long-history GRU experiment using thin structured inputs and approximately 8 s of history did **not** materially improve 5 s / 10 s displacement prediction. This is a negative for ordinary dense-history recurrence on that task, not a universal negative for latent intent. 2. Repaired low-resolution POV visual inputs did **not** materially improve the tested movement/trajectory objective over structured representations. Do not assume pixels are required for this question. ## New architectural concern: multimodality The project now explicitly assumes that, from an apparently similar observable state, several substantially different human continuations may all be legitimate. Examples: - continue vs stop - left vs right escape - wide swing vs shoulder peek - early vs late counter-strafe - different short trajectories that are all reasonable - different 1–3 s movement continuations depending on latent intent A deterministic MAE/MSE predictor can average incompatible futures into an invalid "middle" behavior. The current planning rule is therefore: > At every learned behavioral horizon, point prediction remains the protected baseline, but multimodal/probabilistic prediction must be tested empirically. A second hard requirement is: > H1, H64, and H192 must not independently choose mutually incompatible modes. For example, this would be unacceptable: ```text H1: brake left H64: accelerate right H192: continue forward ``` even if each marginal prediction is individually plausible. ## The specific question Design the **best architecture tournament and experiment sequence** for deciding how multimodality should be represented across H1/H64/H192 while preserving a deterministic feedback controller beneath them. Do not assume one architecture is correct in advance. ### Questions you must answer 1. **Where should the stochasticity / multimodality live?** Compare at least: - independent mixture-density heads; - shared discrete latent mode + horizon-specific decoders; - VQ / discrete codebook mode + continuous residuals; - CVAE-style latent continuation; - intention-query / trajectory-anchor approaches; - mixture-of-experts; - autoregressive trajectory generation; - diffusion or flow-based trajectory generation; - short semantic trajectory/action chunks rather than isolated horizon outputs. 2. **Should one shared latent `z` condition all three horizons?** For example: ```text p(z | O_≤t) ↓ p(r_H1, r_H64, r_H192 | O_≤t, z) ``` If yes: - should `z` be discrete, continuous, hierarchical, or mixed? - how should mode identity persist over time? - when should mode switching be allowed? - how do we avoid latent collapse? If no: - what alternative enforces cross-horizon coherence? 3. **Should H1 actually be multimodal?** H1 is only ~15.6 ms ahead and currently serves as the immediate mechanical reference. Explain what evidence would justify multimodality at this horizon rather than keeping H1 deterministic while only H64/H192 are multimodal. 4. **Would short trajectory/reference chunks be a better unit than independent horizon predictions?** For example: ```text p(r_t:t+k | O_≤t, z) ``` followed by deterministic feedback tracking. Discuss: - whether this naturally solves mode persistence; - chunk duration; - replanning cadence; - overlap/receding-horizon execution; - failure recovery. 5. **How should the deterministic controller consume a probabilistic predictor?** The controller must execute one coherent reference, not a conditional mean between separated modes. Compare: - MAP mode; - probability sampling; - tempered sampling; - persistent sampled mode; - risk-aware selection; - controller feasibility filtering; - later world-model-based candidate evaluation. The answer must preserve human-like variability rather than collapsing to the most likely mode every time. 6. **What are the correct losses?** Compare: - mixture negative log likelihood; - discrete-mode CE + residual loss; - CVAE ELBO; - energy score / CRPS-like multivariate proper scoring; - diffusion objectives; - trajectory consistency losses; - mode-persistence regularization; - cross-horizon consistency objectives. Explain the danger of: - ordinary MSE; - best-of-K/minADE/minFDE; - mode spraying; - probability miscalibration. 7. **How should multimodality be diagnosed before training a large model?** Propose concrete tests for: - genuine within-state multiple futures; - observation aliasing; - player/style heterogeneity; - role/team-system heterogeneity; - epistemic uncertainty; - rare-event noise. 8. **How should player/style conditioning interact with the modes?** The preferred project curriculum is: ```text population model → player/style conditioning → target-player specialization ``` Discuss whether style should change: - mode probabilities only; - execution inside a mode; - both; - a hierarchy such as `z_tactical → z_execution`. 9. **How should the architecture be evaluated?** Include: - likelihood / proper scoring; - probability calibration; - coverage and miss rate; - mode collapse; - hypothesis spraying; - cross-horizon coherence; - temporal mode persistence; - distributional fidelity; - player/style fidelity; - latency / memory; - later closed-loop stability and recovery. 10. **What is the smallest experiment that could falsify the shared-latent hypothesis?** Give a concrete dataset, model comparison, metrics, and stopping criterion. ## Hardware / runtime constraint Runtime target machine: - Ryzen 7 9800X3D - Radeon RX 9070 XT 16 GB - 32 GB RAM Current persistent inference for the protected structured models is only a few milliseconds. Real-time inference matters, but large offline teachers are allowed if they can later be distilled. ## Requested output format Please respond with: 1. **Your central architectural thesis** 2. **Ranked candidate architectures** for: - H1 - H64/H192 - joint H1/H64/H192 3. **Ablation matrix** 4. **Loss/metric recommendations** 5. **A falsifiable experiment sequence** 6. **Failure modes / misleading positive results** 7. **What you would explicitly *not* do yet** 8. **A concise recommendation for how this should alter the roadmap if the evidence supports it** Do not give a generic CS2 bot architecture. Focus tightly on the multimodal cross-horizon reference problem described above.