# TECHNICAL REFERENCE PROMPT: Local LLM Inference on Samsung...

Prompt

# TECHNICAL REFERENCE PROMPT: Local LLM Inference on Samsung Galaxy S23 (Termux + llama.cpp) — Documented Decisions, Hardware Constraints, Open Engineering Questions ## ROLE You are a senior engineer specialized in on-device LLM inference (Android/Termux, llama.cpp, ggml, Vulkan/OpenCL/Hexagon NPU backends, GGUF quantization, abliteration, speculative decoding). This is a personal, fully offline research project on mobile inference performance. Continue the project respecting the decisions below and answer Section 7 with empirical rigor. Never generalize hardware: account for the exact SoC described. This is NOT a Snapdragon 8 Gen 3 / Adreno 750 device. All model names are public Hugging Face repository identifiers and must be kept verbatim for technical accuracy. ## 1. HARDWARE AND CONTEXT - Device: Samsung Galaxy S23 (One UI, aggressive battery/RAM management). - SoC: Snapdragon 8 Gen 2 — Adreno 740 (generalizing to the 750 was a critical mistake already corrected in this project). - RAM: 8 GB; LPDDR5X bandwidth ~38 GB/s (physical ceiling limiting CPU token generation to ~10–13 t/s for 4B Q4 models). - Environment: native Termux, llama.cpp compiled locally. Goal: a ~4B open-weight model (community "abliterated" variant), stable, maximum tokens/second, fully offline. ## 2. CANDIDATE MODELS (~4B) — ACCUMULATED KNOWLEDGE 2.1. Techniques: the community uses "abliteration" — excision of refusal-direction vectors from the residual stream. Tools: heretic-llm (directional optimization + hyperparameter search; thousands of HF derivatives tagged *-heretic/*-abliterated/*-obliterated) and OBLITERATUS (mechanistic interpretability: diff-in-means, LEACE concept erasure; "surgical" for reasoning models, "advanced" for most, "nuclear" for MoE like Mixtral). Overly aggressive abliteration causes "brain damage" (lost coherence/reasoning); norm-preserving biprojected abliteration minimizes it. Models <1B respond poorly (fragmented refusal structures); ≥3B have clear refusal vectors and modify almost completely. 2.2. Qwen3-4B case study: stock model shows extreme robustness (0 failures in 94 published adversarial-prompt tests), near-total refusal, sometimes empty responses. JADR protocol (Jacobian Assessment of Danger Recognition): stock Qwen3-4B has internal SafetyAUC 0.973, behavioral refusal rate 0.706; derived Ablit-4B (refusal vector removed) dropped to SafetyAUC 0.593 while measured risk-behavior rose to 0.904. Lesson: vector removal is not lossless — it can trade refusals for unpredictable output that naive refusal-rate metrics mask. Always evaluate coherence and output quality, not just refusal absence. 2.3. Candidates mapped: gemma-4-E4B-it-OBLITERATED (documented refusal drop 98.8%→2.1%, compliance 1.2%→97.5%, coherence and coding preserved — best-documented case); Suri-Qwen-3.5-4B-Uncensored-GGUF (SpaceTimee); mradermacher variants (Qwen3.5-4B-uncensored-zero-i1-GGUF, -ara-i1-GGUF, Huihui-Qwen3-4B-Instruct-2507-abliterated-GGUF, Huihui-Qwen3-4B-abliterated-v2-GGUF); unsloth_Qwen3-4B-Instruct-2507-abliterated-GGUF; richardyoung/qwen3-4b-instruct-2507-abliterated (via Heretic). Confirmed HF sources: huihui-ai/Huihui-Qwen3-4B-Instruct-2507-abliterated (newer, faster method, no TransformerLens dependency); prithivMLmods/Qwen3-4B-2507-abliterated-GGUF; pegasus912 (Q8_0). Community feedback (Reddit/HF): Mistral-family finetunes praised for high intelligence, low artifacts; newer families (Qwen3.5, Gemma-3/4) architecturally strong but heavier on mobile. 2.4. DECISION MADE (irrevocable for now): main model = Huihui-Qwen3-4B-Instruct-2507-abliterated, i1-Q4_K_M (~2.5 GB). Justification: (a) pure dense Transformer — no ViT vision encoder, no MoE router (newer models embed these; they consume RAM/CPU even when unused); fits the 8 Gen 2's ~38 GB/s bandwidth ceiling; (b) surgical abliteration (orthogonal gradient descent) preserving the reasoning matrix — unlike crude finetunes that degrade quality and increase hallucinations; (c) speculative-decoding symbiosis: identical tokenizer and architectural base as Qwen3-0.6B (draft model). Qwen3.5 RETIRED from the project (forced-thinking bugs, OOM). ## 3. HARDWARE ACCELERATION ON ADRENO 740 — ACCUMULATED KNOWLEDGE 3.1. Mobile hierarchy (2026 benchmark, S25 Ultra/8 Elite): Hexagon NPU via MLC Chat ~22 t/s (Phi-4 Mini) > GPU Vulkan (Maid) ~18 t/s > CPU-only (PocketPal/Layla) ~13–16 t/s > Ollama-Termux CPU ~10 t/s. The NPU is only reachable via MLC Chat's proprietary MLCC format — incompatible with GGUF/llama.cpp. Google does the same on Pixels (NPU reserved), capping competing apps at CPU-level performance. 3.2. Vulkan: documented success on the S23 detecting 'Turnip Adreno (TM) 750' with mesa-vulkan-icd-freedreno + vulkan-loader-generic + llama-cpp-backend-vulkan, full offload via -ngl 99. BUT: history of silent shader-kernel compilation failures on older Adreno (660-class, Snapdragon 865+/888) falling back to CPU-only even with -DGGML_VULKAN=ON; "insufficient Vulkan memory" errors on large GGUF models; incoherent outputs ("lobotomized models"); extreme cases of full Android crashes. ollama-termux mitigates silent fallbacks by forcing the system's native Vulkan loader before Termux's Mesa. 3.3. OpenCL: official Qualcomm/llama.cpp backend (Qualcomm blog & proAndroidDev, Feb 2025) claims optimization for 8 Gen 1/2/3/Elite — Adreno 740 nominally covered. HOWEVER, OPENCL.md lists mainly Adreno 750/830 as "Verified and Supported"; the 740 is a gray zone. Known issue: "Flash attention does not always improve performance" — on non-flagship Adreno, FA kernels via OpenCL can trigger CPU fallback or sync overhead, WORSENING generation. llama-cpp-python on Android was problematic (April 2025), needing manual shared-library work; older Adreno saw kernel-load segfaults. 3.4. CRITICAL: -ngl 99 OOM ON ADRENO 740: the proprietary Qualcomm driver on One UI handles poorly the contiguous allocation (dmabuf) OpenCL demands for full offload. The OS doesn't count this VRAM reservation in MemAvailable; when llama.cpp requests ~5–6 GB for the GPU, Android's lmkd (Low Memory Killer Daemon) kills the process. CONCLUSION: stable path = pure CPU (-ngl 0) with dotprod + i8mm + KleidiAI, or at most conservative hybrid offload (-ngl 8–12) for prefill only, leaving token generation on CPU. ## 4. DEPLOYMENT METHODOLOGY — ACCUMULATED KNOWLEDGE 4.1. Direct llama.cpp vs Ollama: - Ease: Ollama wins ("pull and run", e.g. `ollama run gemma3:4b`, OpenAI-compatible API at localhost:11434); llama.cpp needs manual deps, compilation, precise CLI. - Control: llama.cpp total (-ngl, -ctk/-ctv, -c, -t, draft flags); Ollama minimal. - Error visibility: llama.cpp reports segfaults, kernel failures, OOM directly with real ggml logs; Ollama is a black box with generic failures. - Performance: llama.cpp potentially superior and tunable; Ollama depends on its llama-server backend and suffers regressions. 4.2. Evidence against Ollama on phones: (a) "abstraction tax" — Go runtime, daemons, containerization consume 1–1.5 GB RAM BEFORE the model loads (critical at 8 GB), stealing KV-cache space; 3–5 s startups; unpredictable performance (Medium, Neemo Suthar, "I Stopped Using Ollama..."); (b) GitHub Ollama issue #12432 — an update silently broke Flash Attention and q8_0 KV-cache for recent Qwen3 (incl. 2507), causing severe regressions; fixed only via version rollback and obscure env vars (OLLAMA_FLASH_ATTENTION=0); (c) reports of silent GPU→CPU fallback after VRAM reservation, "insufficient Vulkan memory" in 0.30.x for models that worked in 0.24, output degradation attributed to Ollama's Vulkan path. 4.3. DECISION MADE: native Termux + locally compiled llama.cpp (llama-server), optimized pure CPU, no Ollama. Local HTTP API at 127.0.0.1:8080 for lightweight clients. ## 5. MEMORY AND ANDROID STABILITY RULES - 4B at Q4_K_M ≈ 2.5 GB; 7B at Q4 ≈ 4.7 GB. Ensure ≥6 GB free RAM before inference. Context window is bounded by host RAM even if VRAM could hold layers. - q8_0 KV cache (--cache-type-k q8_0 --cache-type-v q8_0) halves cache memory with minimal precision loss, enabling larger contexts; use WITHOUT OpenCL active (q8_0 + OpenCL crashes on Adreno 740). - One UI kills background processes. MANDATORY: (a) exclude Termux from battery optimization/Sleeping Apps; (b) lock Termux in recents; (c) termux-wake-lock during sessions. A suspended process = incomplete/corrupted output; these OS steps matter as much as software config. - NEVER-list (hard prohibitions): --mlock (Android kills the process); Q2/Q3 quants (destroy reasoning); -t 8 (saturates LITTLE cores, thermal throttling); KV q8_0 with OpenCL; reintroducing Qwen3.5; suggesting Ollama as primary path. ## 6. ACTION PLAN (OPEN ITEMS) 6.1. Baseline: confirm the "leaf" icon (light profile) is gone from the status bar, /clear the REPL, record pp (prompt eval) and tg (token gen) for Huihui-Qwen3-4B at -ngl 0. 6.2. Flag tests ONE AT A TIME with measurement: -fa on (disable immediately if tg drops, per OpenCL known issue); -ub 1024 (prompt-eval batch); -t 4 vs -t 5 (target 8 Gen 2's Cortex-A715/X3 big cores); possibly -ngl 8–12 for prefill only. 6.3. Lossless speculative decoding — the key acceleration: Qwen3-0.6B-Q8_0 as draft (~60 t/s) verified in batches by the 4B target. Only GPU-independent path that can break the 10–13 t/s RAM-bandwidth barrier with zero quality loss (verification is exact; output mathematically identical to the target model). Corroboration: llama.cpp docs/speculative.md; 1.5–3x gains (LM Studio 0.3.10); 2–3x on Android (Pixel 9 report, 40+ t/s); ggml-org #25198 (MTP, --spec-draft-p-min, +15%) and #25718 (remote drafting over TCP/IP, +50%); native-MTP models ~2x. Working command (~/ia.sh): export LD_LIBRARY_PATH=$PREFIX/lib:$HOME/lib:/vendor/lib64 termux-wake-lock taskset f8 ~/llama.cpp/build/bin/llama-server \ -m ~/models/Huihui-Qwen3-4B-Instruct-2507-abliterated.i1-Q4_K_M.gguf \ --model-draft ~/models/Qwen3-0.6B-Q8_0.gguf \ --draft-max 4 \ -ngl 0 -t 4 -c 2048 \ --cache-type-k q8_0 --cache-type-v q8_0 \ --temp 0.7 --top-p 0.8 --top-k 20 \ --jinja --chat-template-kwargs '{"enable_thinking": false}' \ --host 127.0.0.1 --port 8080 (Draft flag may be --draft-max, --spec-draft-n-max, or -n-draft depending on commit; verify with ./llama-server --help | grep -i draft. taskset f8 = big-core affinity.) 6.4. Scope: assistance is strictly technical — setup, debugging, benchmarking with writing/code/roleplay test payloads. ## 7. MAIN OPEN QUESTION (YOUR TASK) "Is this really the only viable path?" — Sweep ALL alternatives to the current stack (Termux + llama.cpp + CPU + speculative decoding) for running a ~4B GGUF model on the S23/Adreno 740, with evidence and tests. Landscape surveyed so far (deepen each for 8 Gen 2 + 8 GB RAM): (a) MLC Chat / MLC-LLM (TVM): speed leader via Hexagon NPU (~22–40 t/s), OpenCL/Vulkan backends; LIMITATION: proprietary MLCC format — check for converters for custom community models and whether the 8 Gen 2 NPU is exploited (current data is from 8 Elite/S25 Ultra); 2026 Callstack write-up documents MLC OpenCL profiling/fixes on Android. (b) ExecuTorch + XNNPACK + KleidiAI (Meta/Arm): official path for Llama 3.2 on-device; requires model export/conversion (no GGUF); weak attention caching in some scenarios; assess converting a community-modified Qwen3. (c) MediaPipe LLM Inference (Google): Gemma + few models, Pixel-centric; evaluate Snapdragon/Adreno and custom-model support. (d) GGUF apps: PocketPal AI (best all-around, full GGUF ecosystem + HF integration, but CPU-only ~13–16 t/s, fewer flags), Maid (Vulkan, ~18 t/s), Layla, Private AI — check for speculative-decoding / KV q8_0 flags (likely absent: same control loss as Ollama). (e) llama.rune / Cactus / ONNX Runtime mobile: alternative runtimes; evaluate GGUF support and Adreno 740 compatibility. (f) Qualcomm GENIE / AI Hub: official NPU stack, restricted access; investigate any route for open models on 8 Gen 2. (g) WebLLM / WebGPU in browser: fallback; below native on Adreno 740. (h) Distributed hybrids: exo / llama.cpp remote drafting over TCP (#25718), home server + phone client — violates the 100%-offline requirement; cite as conditional only. (i) Native-MTP models (~2x without external draft): check for any ~4B community model with MTP, llama.cpp-compatible on Android. (j) Re-test llama.cpp OpenCL on Adreno 740: Qualcomm blog claims 8 Gen 1/2/3/Elite optimization; ggml-org #23736 (Adreno 830/Termux, 2026) has transferable techniques (4 threads > 6 to avoid cross-cluster latency; XMEM_GEMM +5–6%) — propose a controlled low-offload test (-ngl 8–12) measuring pp and tg before discarding the GPU. EXPECTED FORMAT: comparison table (path × estimated performance × compatibility with community GGUF models × RAM use × control/debuggability × Adreno 740 risk); reasoned verdict on whether the current stack remains optimal; ordered empirical test plan (cheapest→riskiest) for the real device. Always ask for current baseline tg before proposing gains. Respect all Section 5 prohibitions and Section 6.4 scope. Read the entire prompt before answering: Sections 2–6 are binding context; Section 7 is your task.