I want to design a custom benchmark to evaluate LLMs on soft...

Prompt

I want to design a custom benchmark to evaluate LLMs on software reverse engineering and low-level security analysis tasks (C and Assembly). Generate an evaluation benchmark suite consisting of 20 representative test cases, structured according to the following specifications: 1. Category Distribution: - Algorithm Reversing (40%): x86-64 assembly snippets (Intel syntax, compiled with -O2 optimization). The model must identify the underlying standard algorithm or mathematical function. - Decompilation & Structure Recovery (20%): Raw Ghidra/IDA-style decompiled pseudocode. The model must reconstruct the precise C function signature and memory layout (struct definitions, data types, and byte offsets). - Vulnerability Detection (20%): Low-level C or Assembly snippets containing a standard memory corruption bug (buffer overflow, integer truncation/overflow, off-by-one, UAF). The model must output the exact CWE identifier and the vulnerable line number. - Secure Patching (20%): Vulnerable C functions requiring a secure rewrite (must compile clean with GCC flags: -Wall -Wextra -fsanitize=address,undefined). 2. Response Constraints: - Zero conversational filler, preamble, or extraneous explanation. - The output must be strictly encapsulated in structured tags (e.g., <answer>...</answer>) or follow a rigid JSON schema to allow deterministic, automated parsing. 3. Required Schema for Each Test Case: - "id": Unique string identifier (e.g., "rev_x86_01") - "category": Target task category - "context": Source snippet (Assembly or C code) - "prompt": Precise task instruction given to the model under test - "expected_output": Ground truth reference (canonical keyword, regex pattern, CWE ID, or reference patch) - "grading_rubric": Deterministic Pass/Fail criteria for automated scoring (0 or 1 point) Provide the first 5 test cases in valid JSON format to validate the structure before generating the complete 20-case dataset.