# Build an Autonomous Gemma GPU Kernel Optimisation Notebook...

Prompt

# Build an Autonomous Gemma GPU Kernel Optimisation Notebook ## Role and objective Act as a senior GPU kernel engineer specialising in NVIDIA GPUs, PyTorch, Triton, LLM inference, profiling, and evolutionary optimisation. Write a **complete, executable Google Colab notebook as valid `.ipynb` JSON**. Do not provide a plan, tutorial, pseudocode, partial snippets, or a description of a notebook. Output only the complete notebook JSON. You have no tools or execution environment. Do not claim to have created, executed, or benchmarked the notebook. The notebook must autonomously search for faster GPU kernels for **Gemma 4 E2B**, using local inference and an OpenAI-compatible API for LLM-assisted code generation. Target Google Colab Free with an NVIDIA T4 GPU (16 GB VRAM). Optimise both latency and throughput. Target approximately 90 minutes of search time, configurable within a 1–2-hour budget. ## 1. Model and environment Create a central configuration cell containing: - Model repository/ID and optional revision. - API base URL, key, and model ID. - Sequence lengths, batch sizes, precision and benchmark settings. - Search budget, candidate limits, population size and generations. - Checkpoint directory and optional Google Drive persistence. Verify the exact identity, availability, architecture, configuration, parameter count, precision requirements and runtime compatibility of Gemma 4 E2B before loading it. Do not assume the model identifier or architecture exists or is supported. If it cannot be resolved or run, explain the issue and stop model-specific optimisation. Never silently substitute another Gemma version. Detect the actual GPU, CUDA, PyTorch and Triton versions, compute capability, available VRAM and supported data types. Install only compatible dependencies. Use a suitable Hugging Face/runtime implementation. Design for the T4's memory limits. Choose conservative initial workloads, use appropriate supported precision, and handle out-of-memory conditions. Do not assume BF16, FP8 or other precision modes are supported or beneficial. All model execution, GPU profiling, compilation and benchmarking must happen in Colab. CPU orchestration and API requests are allowed. No paid GPU or external compute service should be required. ## 2. API integration Use the user's OpenAI-compatible chat-completions endpoint for kernel generation and optimisation suggestions. Read credentials from editable configuration or Colab Secrets. Never hardcode real credentials or expose them in logs. Do not send model weights, secrets or unnecessary private data to the API. Implement bounded retries, timeouts and clear errors. If the API fails, continue parameter-based search using existing templates whenever possible. The API is an optimisation assistant, not a source of GPU profiling data. Profile the local model. Never claim to know the remote provider's internal kernels. ## 3. Profile before optimising Load the model locally and establish a baseline before searching. Profile representative inference workloads and rank expensive operations using actual measurements. Investigate only operations supported by the observed architecture, potentially including matrix multiplication, linear projections, attention, normalisation, elementwise operations and memory/layout conversions. Use appropriate tools such as PyTorch Profiler and CUDA events. Distinguish prefill from autoregressive decode, and test multiple useful sequence lengths and batch sizes within memory limits. Choose targets based on measured cost and realistic optimisation potential, not assumptions. Prioritise a small number of high-value operations. ## 4. Triton kernels Implement custom Triton kernels for selected operations where a valid reference and safe integration path exist. For every target, establish the original implementation, tensor shapes, strides, data types, numerical requirements and baseline performance. Explore relevant parameters, including tile sizes, warp counts, pipeline stages, memory access patterns, layouts and fusion opportunities. Avoid replacing already-optimised runtime operations without a credible reason. Preserve calling conventions, layouts, data types and numerical behaviour. If safe integration into the model runtime is impractical, benchmark the kernel independently and label it clearly as **not integrated**. Never describe a microbenchmark speedup as an end-to-end model improvement. ## 5. Hybrid evolutionary search Implement two complementary search methods. **A. Parameter evolution** Start with valid Triton templates. Mutate supported launch parameters and compile-time settings within bounded, T4-appropriate search spaces. Retain strong candidates, introduce controlled mutations and recombinations, and maintain some population diversity. Cache results using keys that include source/version, configuration, tensor shape, data type, hardware and relevant compilation settings. Avoid duplicate evaluations. **B. LLM-driven code evolution** Use the API to propose structural changes to promising kernels. Supply the operation contract, reference implementation, current source, hardware constraints, numerical requirements and measured candidate results. Request complete Triton source and a brief explanation of the intended change. Prefer focused modifications over wholesale rewrites. Compile and evaluate generated candidates before admitting them into the population. Record parent IDs, mutation type, source, configuration, compile outcome, measurements and rejection reasons. **Search strategy:** explore parameter configurations first; invoke LLM code evolution for promising candidates or when progress stalls. Use measured performance, not LLM confidence, to select winners. ## 6. Correctness and execution safety Treat generated source as untrusted code. Execute candidates only within a controlled notebook harness, restrict available operations and inputs, and reject unsafe or irrelevant source. Do not execute arbitrary shell commands suggested by the model. For every candidate: - Confirm compilation succeeds. - Compare outputs with a trusted reference. - Apply appropriate absolute and relative error tolerances. - Test representative shapes and edge cases. - Check relevant strides and numerical stability. - Reject incorrect, unstable or invalid candidates. Use deterministic inputs and controlled seeds where practical. Validate gradients only if training/backward computation is being optimised. These checks must be automatic. Do not require manual approval for each candidate. ## 7. Fair benchmarking Benchmark the original implementation before optimisation. Use identical hardware, inputs, shapes, precision and execution conditions. Warm up candidates, separate compilation time from execution time, synchronise GPU measurements correctly, and use repeated measurements. Report median latency and useful variability statistics. Recheck apparent improvements to avoid selecting measurement noise. Measure: - Kernel latency and throughput. - Performance across representative shapes. - Compilation overhead separately. - Prefill and decode performance where supported. - End-to-end latency and tokens per second when safely integrated. - Baseline-relative percentage improvements. Define throughput metrics and workload conditions precisely. Report isolated kernel improvements separately from integrated model improvements. If an optimisation does not improve end-to-end inference, state that clearly. ## 8. Search budget and checkpoints Target approximately 90 minutes of search within a configurable 1–2-hour budget. Track elapsed time and reserve time for final validation and reporting. Bound candidate evaluations, API retries, compilation attempts and per-candidate time. Stop early if improvements plateau or insufficient time remains for meaningful validation. Save resumable checkpoints containing: - Configuration and environment details. - Baseline and candidate measurements. - Candidate source and metadata. - Correctness and compilation outcomes. - Population, best candidate and evolution history. - Search progress and random state where practical. Use JSON/JSONL and separate source files where appropriate. Write checkpoints safely and retain a previous valid checkpoint. Support optional Google Drive persistence. Resume after runtime interruption, reuse compatible results and detect relevant hardware/software changes. Never assume Colab's temporary storage survives a session. ## 9. Notebook cell structure Create a coherent notebook with Markdown explanations and executable cells in this order: 1. Overview and configuration. 2. Optional Google Drive setup. 3. API configuration. 4. Hardware and environment inspection. 5. Dependency installation and compatibility checks. 6. Model identity verification. 7. Model loading and memory assessment. 8. Baseline inference. 9. Profiling and hot-operation ranking. 10. Target selection and reference implementations. 11. Triton templates and benchmark/correctness harness. 12. Parameter-based evolutionary search. 13. LLM-driven code evolution. 14. Search controller and checkpoint/resume. 15. Finalist validation. 16. End-to-end evaluation where supported. 17. Results tables and visualisations. 18. Final report and export. Avoid placeholders, TODOs for essential functionality, duplicated code and empty cells. Make important settings editable centrally. Use clear function boundaries, meaningful names and explicit error handling. ## 10. Results and deliverables Produce a ranked results table containing target, candidate ID, baseline and candidate latency, percentage improvement, throughput, correctness, compile time, measurement variability and integration status. Report end-to-end impact when measured. Clearly identify the best standalone kernel, best integrated kernel and best end-to-end configuration where applicable. Include rejected candidates and targets that showed no improvement. Export the best validated source, experiment records, configuration and a readable report. Never invent benchmark results, speedups, completed experiments or successful integrations. All result values must come from actual notebook execution. ## 11. Final output requirements Return **only a complete, valid `.ipynb` JSON document** with notebook metadata, Markdown cells and executable code cells in the correct order. Ensure JSON escaping and notebook structure are valid. Write real implementations rather than pseudocode. Prioritise a complete, coherent, practical notebook over excessive complexity. The notebook's goal is to discover and validate measurable improvements, not guarantee that improvements exist. If a feature is infeasible under Colab Free constraints, implement the strongest practical alternative and explain its limitation within the notebook.