``` # Hardened Deep Research Agent: System Architecture & Op...

Prompt

``` # Hardened Deep Research Agent: System Architecture & Operations Manual ## 1. System Overview & Hardware Envelope This workstation hosts two decoupled environments: - **Host Workstation:** Intel B660M, 64 GB DDR4 RAM, NVIDIA RTX 3090 (24 GB VRAM). - **Host Inference (Strata):** Runs `qwen3.8-flash-next-unsloth-ud-iq4_xs` with MoE expert caching (~23.3 GB VRAM, ~35–50 GB RAM lock), leaving ~700 MB VRAM free and ~12 GB RAM available for the agent sandbox. - **Agent Sandbox:** An untrusted, autonomous Python ReAct agent operating inside Rootless Podman wrapped with Google gVisor (`runsc`). It possesses arbitrary Bash execution, package installation, headless stealth web automation, and multi-agent spawning capabilities, strictly contained from the host OS and LAN. ### System Architecture Diagram ``` HOST WORKSTATION (Linux, RTX 3090 24GB, 64GB RAM) ├─ Strata Inference Engine (TCP 127.0.0.1:8080) │   └─ socat UNIX relay -> /mnt/agent_storage/@workspace_vault/strata.sock (0666) │ ├─ Dedicated Partition: /dev/nvme0n1p2 (1.7TB Btrfs, UUID: 7ec0025e-1195-4655-9610-8d34a8f47866) │   ├─ @base_rootfs          (Pristine container base seed) │   ├─ @active_rootfs        (Ephemeral agent OS root, CoW writable) │   ├─ @workspace_vault      (Persistent: /workspace/data mounted into sandbox) │   │   ├─ vault.db          (SQLite FTS5 + WAL search index) │   │   ├─ wiki/             (Verified findings & reports in Markdown <=8KB) │   │   ├─ runbooks/         (Stubborn error remediation protocols) │   │   ├─ auth/cookies.json (Exported session cookies for headless browser) │   │   └─ scripts/          (Agent core: orchestrator, browser, runner, vault) │   └─ @snapshots/           (Read-only snapshots: golden_vault_v1, etc.) │ ├─ Host Boundary & Isolation: │   ├─ cgroups v2 / systemd: MemoryHigh=10G, MemoryMax=12G, MemoryOOMGroup=yes │   ├─ Device Controller: DevicePolicy=strict (major 195/226 denied -> Zero VRAM/GPU) │   └─ Network Quarantine: │       ├─ nftables: Drop RFC 1918 (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16) │       └─ dnsmasq: --stop-dns-rebind (Prevents private IP resolution via public DNS) │ └─ Sandbox Boundary: Rootless Podman + gVisor (runsc) └─ Agent Process Tree: ├─ Depth 0: Supervisor ReAct Orchestrator ├─ Depth 1: Ephemeral Worker Sub-agents (Dedicated task prompts, clean context) ├─ Patchright Browser: Headless Chromium (C++ AST stealth, anti-bot hardened) └─ HITL Pipeline: CDP Screencast -> WebSocket ws://0.0.0.0:8765 -> HTML Canvas ``` --- ## 2. Storage Setup & Btrfs Layout ### Partition Configuration - **Block Device:** `/dev/nvme0n1p2` (1.7 TB NVMe) - **Filesystem:** Btrfs with ZSTD level 1 compression, `noatime`, SSD optimization, and space cache v2. - **Mount Point:** `/mnt/agent_storage` ### Persistent Mount Entry (`/etc/fstab`) ```text UUID=7ec0025e-1195-4655-9610-8d34a8f47866 /mnt/agent_storage btrfs compress=zstd:1,noatime,ssd,space_cache=v2 0 0 ``` ### Subvolume Hierarchy | Subvolume | Role | Mount Target Inside Container | | --- | --- | --- | | `@base_rootfs` | Pristine golden image of the OS | N/A (Host template) | | `@active_rootfs` | Ephemeral scratchpad for the OS | `/` (when fully virtualized) | | `@workspace_vault` | Persistent agent data, code, and SQLite index | `/workspace/data` (Read/Write) | | `@snapshots` | Archive of immutable snapshots | N/A (Host recovery) | --- ## 3. Host Isolation & Sandboxing Configuration ### 3.1 gVisor Runtime Wrapper (`/usr/local/bin/runsc-podman`) Rootless Podman interacts with systemd cgroups by default and expects auxiliary sidecars that standalone `runsc` binaries lack. The execution wrapper resolves this: ```bash #!/usr/bin/env bash ARGS=() for arg in "$@"; do     if [[ "$arg" == "--systemd-cgroup" ]]; then         continue     fi     ARGS+=("$arg") done exec /usr/local/bin/runsc \     --ignore-cgroups \     --sidecar-usage-policy=LEGACY_DEPRECATED_SLOW_EMBEDDED_FALLBACK \     --host-uds=all \     "${ARGS[@]}" ``` *Note:* `--host-uds=all` allows the sandboxed container to connect to host UNIX domain sockets mounted inside the container volume. ### 3.2 Podman Runtime Registration (`~/.config/containers/containers.conf`) ```ini [engine.runtimes] runsc = [     "/usr/local/bin/runsc-podman" ] ``` ### 3.3 Zero GPU / Zero VRAM Leakage Enforcement The RTX 3090 is dedicated to Strata. To guarantee zero VRAM allocation and prevent GPU side-channel escapes: 1. Podman runs with `--env CUDA_VISIBLE_DEVICES=""` and `--env NVIDIA_VISIBLE_DEVICES="none"`. 2. gVisor does not emulate PCI devices or allow arbitrary character device creation (`mknod`). 3. Host systemd unit / cgroups deny access to major device number 195 (NVIDIA) and 226 (Direct Rendering Infrastructure). ### 3.4 Zero-Trust Network Quarantine (`/etc/nftables.conf`) The agent requires WAN access (80, 443, 53) to search and scrape, but must never access the local router or private LAN: ```nft table inet sandbox_isolation {     chain forward {         type filter hook forward priority filter; policy drop;         ct state invalid drop         ct state established,related accept         # Block private IPv4, link-local, and Carrier-Grade NAT         ip daddr { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16, 100.64.0.0/10 } drop         # Block private IPv6         ip6 daddr { fc00::/7, fe80::/10 } drop         # Block host loopback reflection         ip daddr 127.0.0.0/8 drop         # Whitelist WAN web & DNS traffic         iifname "tap-agent" ip protocol tcp tcp dport { 80, 443 } accept         iifname "tap-agent" ip protocol udp udp dport 53 accept         iifname "tap-agent" ip protocol tcp tcp dport 53 accept     }     chain postrouting {         type nat hook postrouting priority srcnat; policy accept;         iifname "tap-agent" masquerade     } } ``` --- ## 4. Container Image Specifications (`agent-base-image`) The container definition is located at `/mnt/agent_storage/Containerfile`: ```dockerfile FROM docker.io/library/ubuntu:24.04 ENV DEBIAN_FRONTEND=noninteractive ENV PYTHONUNBUFFERED=1 RUN apt-get update && apt-get install -y --no-install-recommends \     python3 \     python3-pip \     python3-venv \     build-essential \     curl \     git \     sqlite3 \     ca-certificates \     libnss3 \     libatk-bridge2.0-0 \     libxcomposite1 \     libxdamage1 \     libxfixes3 \     libxrandr2 \     libgbm1 \     libpango-1.0-0 \     libasound2t64 \     libxkbcommon0 \     libxkbcommon-x11-0 \     libcups2t64 \     libcairo2 \     libavahi-client3 \     xkb-data \     && rm -rf /var/lib/apt/lists/* RUN rm -f /usr/lib/python3.*/EXTERNALLY-MANAGED RUN pip install --no-cache-dir patchright websockets asyncio RUN patchright install chromium WORKDIR /workspace/data ``` Rebuild command: ```bash podman build -t agent-base-image:latest -f /mnt/agent_storage/Containerfile ``` --- ## 5. Software Subsystems & Source Code All scripts reside in `/mnt/agent_storage/@workspace_vault/scripts/`. ### 5.1 CLI Subprocess Runner (`cli_runner.py`) - Executes shell commands within the persistent working directory `/workspace/data`. - Uses `encoding="utf-8", errors="replace"` to protect against `UnicodeDecodeError` when commands split multi-byte UTF-8 sequences. - Truncates standard output exceeding 4,000 characters by preserving the head and tail, preventing context window exhaustion. ### 5.2 Memoryfield Engine (`vault_engine.py`) - **Source of Truth:** Plain Markdown files (`.md`) formatted with YAML frontmatter, strictly capped at 8,192 bytes. - **Index:** SQLite running in Write-Ahead Logging (WAL) mode (`vault.db`) with full-text search (`pages_fts` via FTS5). - Provides `save_entry(path, title, summary, body)` and `search_vault(query, limit)`. ### 5.3 Execution Guard & Circuit Breaker (`circuit_breaker.py`) - Employs a three-state machine (`CLOSED`, `OPEN`, `HALF-OPEN`) to prevent hammering failing external tools. - Evaluates string similarity (`SequenceMatcher >= 0.95`) only against **failed** actions in a sliding history window. This prevents infinite retry loops while allowing iterative pagination tools (such as reading text chunks) to proceed. ### 5.4 Stealth Browser Engine (`agent_browser.py`) - Built on **Patchright**, applying C++ AST patches to eliminate `Runtime.enable` leaks and suppress automation flags. - Spoofs client hints (`Sec-Ch-Ua`) to prevent `HeadlessChrome` detection. - Uses DOM extraction filtering (`nav`, `header`, `footer`, `script`, `style`, `aside`) to eliminate navigation noise and extract only article content. - Features a Human-In-The-Loop (HITL) escalation pipeline: when bot mitigation is detected (Cloudflare Turnstile, Akamai), it spins up a WebSocket server on `0.0.0.0:8765` using Chrome DevTools Protocol (`Page.startScreencast`). The agent loop pauses via `asyncio.Event` until a human solves the challenge or clearance cookies (`cf_clearance`, `_abck`) are detected. ### 5.5 Multi-Agent Orchestrator (`orchestrator.py`) - **Supervisor Mode (Depth 0):** High-level decomposition, web discovery, delegating heavy reads, and logging final synthesis. - **Worker Mode (Depth 1):** Spawns ephemeral sub-agents with fresh context windows to extract facts from specific URLs. Sub-agents are barred from recursive sub-agent spawning. - **Search Resiliency:** Triple-tier search engine pipeline: 1. DuckDuckGo Lite via HTTP POST (stripping tracking redirects via `unwrap_ddg_url`). 2. Wikipedia OpenSearch API fallback. 3. Mojeek HTTP fallback. --- ## 6. Daily Operations Guide ### 6.1 Starting the System 1. **Terminal 1: Start Host Model (Strata)** ```bash cd ~/Strata ./setup.sh --host 127.0.0.1 --port 8080 --no-browser ``` Wait until the engine reports: `ready: http://127.0.0.1:8080/v1`. 2. **Terminal 2: Launch Research Agent** Use the `/usr/local/bin/agent` launcher from any terminal: ```bash agent "Your deep research task prompt here" ``` The launcher automatically verifies Strata's availability, cycles `socat` if the UNIX socket is stale, and mounts the workspace into the gVisor sandbox. ### 6.2 Solving CAPTCHAs via HITL If the agent encounters an interactive bot challenge: 1. The console displays: `[ALERT] Anti-bot trigger. Starting HITL stream on ws://0.0.0.0:8765...`. 2. Open `/mnt/agent_storage/@workspace_vault/hitl_ui/hitl_viewer.html` in your host browser. 3. The page renders the remote browser canvas in real time. Click the CAPTCHA widget or complete the interactive puzzle. Clicks are dispatched via CDP as native hardware events (`isTrusted=True`). 4. Once solved, `agent_browser.py` detects the updated clearance cookie and resumes execution automatically. ### 6.3 Disaster Recovery: Instant Snapshot Rollback If an agent execution damages the environment or fills the workspace with unwanted artifacts, restore the baseline using Btrfs snapshots: ```bash # 1. Stop any running container instances podman stop -a || true # 2. Delete the damaged subvolume sudo btrfs subvolume delete /mnt/agent_storage/@workspace_vault # 3. Re-create from the Golden Snapshot (takes < 1 second) sudo btrfs subvolume snapshot /mnt/agent_storage/@snapshots/golden_vault_v1 /mnt/agent_storage/@workspace_vault sudo chown -R $USER:$USER /mnt/agent_storage/@workspace_vault echo "[+] Workspace restored to golden state." ``` ### 6.4 Creating a New Golden Snapshot After making verified improvements or updating runbooks: ```bash sudo btrfs subvolume snapshot -r /mnt/agent_storage/@workspace_vault /mnt/agent_storage/@snapshots/golden_vault_$(date +%Y%m%d) ``` --- ## 7. Maintenance & Upgrades ### 7.1 Upgrading Host Strata & Models Run update within the Strata directory: ```bash cd ~/Strata ./setup.sh --update ``` If updating the model changes the API key in `strata-*.json`, update the `API_KEY` variable in `/mnt/agent_storage/@workspace_vault/scripts/orchestrator.py` and `/usr/local/bin/agent`. ### 7.2 Rebuilding the Sandbox Image When adding new system packages or Python libraries: 1. Edit `/mnt/agent_storage/Containerfile`. 2. Run: ```bash podman build -t agent-base-image:latest -f /mnt/agent_storage/Containerfile ``` ### 7.3 Maintaining the Knowledge Vault Database Over long research sessions, optimize the SQLite index: ```bash podman run --rm --runtime=runsc \     -v /mnt/agent_storage/@workspace_vault:/workspace/data:rw \     agent-base-image:latest \     sqlite3 /workspace/data/vault.db "PRAGMA optimize; VACUUM;" ``` ### 7.4 Re-indexing Markdown Files If Markdown files are edited or deleted on the host, the FTS5 index can be rebuilt from scratch: ```bash podman run --rm --runtime=runsc \     -v /mnt/agent_storage/@workspace_vault:/workspace/data:rw \     agent-base-image:latest \     python3 -c "from vault_engine import init_vault; init_vault()" ``` --- ## 8. Common Troubleshooting Runbook | Symptom | Cause | Solution | | --- | --- | --- | | `ConnectionRefusedError: [Errno 111]` | Strata is stopped or `socat` lost its socket binding. | Check `~/Strata` process; run `agent` (the launcher auto-refreshes `socat`). | | `UnicodeDecodeError: 'utf-8'` | A Bash command split a multi-byte character. | Verify `cli_runner.py` uses `errors="replace"` in `subprocess.run()`. | | `DuckDuckGo bot-check / CAPTCHA` | Search rate-limit triggered on DDG Lite. | Orchestrator automatically falls back to Wikipedia API and Mojeek. Provide direct documentation URLs in prompt if needed. | | `LOOP_DETECTED` on paginated reads | Circuit breaker triggered on repeated command string. | `circuit_breaker.py` only blocks if the action failed previously. Check if the underlying command threw a non-zero exit code. | | Sub-agent hangs with no output | Heavy scraping operation running synchronously. | Orchestrator streams sub-agent lines live via `subprocess.Popen`. Check Terminal 1 to confirm Strata is generating tokens. | | Browser fails with missing shared libraries | New Chromium release requires updated Linux packages. | Update the `Containerfile` package list and run `podman build`. | | EOF |  |  | ``` Please analyze this setup