{"id":"gen-1786175628-MuHpfjROywzryjP4wfLJ","choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":"下面是一个完整的「EvalScope × 代码能力评测」实战页面，包含基准介绍、快速开始代码、可在浏览器中真实运行的 Python 判分沙箱（模拟 HumanEval 式 pass@1 评测流程）、pass@k 计算器等：\n\n```html\n\n\n\n\n\nEvalScope · 代码能力评测实战\n\n\n\n\n\n\n\n\n

\n EvalScope · 代码生成能力评测\n

用 EvalScope
系统化测试代码能力

从 HumanEval 到 LiveCodeBench：配置任务 → 模型采样 → 沙箱执行测试用例 → 计算 pass@k，一站式复现主流代码基准评测流程。

\n 🚀 快速开始\n \n 🧪 在线实验\n

evalscope — bash

\n\n\n

Overview

代码评测怎么跑？

EvalScope 是 ModelScope 开源的模型评测框架。针对代码生成任务，它把「生成 → 执行 → 判分」封装成一条命令，核心环节如下：

STEP 01

启动模型服务

vLLM / SGLang 部署 OpenAI 兼容接口，或直接接入 API 模型

STEP 02

配置评测任务

选择数据集、生成参数（低温度）、limit 小样本冒烟

STEP 03

采样生成代码

批量推理收集补全结果，可多次采样用于 pass@k

STEP 04

沙箱执行判分

在隔离环境中运行单元测试，逐题判定通过与否

STEP 05

指标与报告

汇总 pass@1 / pass@k，输出表格报告与判分详情

🎯

无偏 pass@k 估计

采用 HumanEval 论文提出的无偏估计器：n 次采样中 c 次通过，则 pass@k = 1 − C(n−c,k)/C(n,k)。

🔒

执行隔离

模型生成的代码不可信——测试用例在受控环境中执行，避免污染评测机，保证结果可复现。

🧊

低温度采样

代码任务对确定性要求高，实践中 temperature 常取 0~0.2；pass@k>1 时才需要多样性采样。

\n\n\n

Benchmarks

主流代码评测基准

EvalScope 内置多种代码生成基准，并支持自定义数据集接入私有题库。

📜

HumanEval

OpenAI 发布的 164 道手写函数补全题，代码评测的\"开山基准\"，以 pass@k 为指标。

164 题pass@1函数补全

🐍

MBPP

约 1000 道入门级 Python 编程题，覆盖基础语法与简单算法，每题附带 3 个测试用例。

~1000 题基础算法

🛡️

LiveCodeBench

持续滚动收录 LeetCode / AtCoder / Codeforces 竞赛真题，可按时间窗选择子集，有效缓解数据污染。

防污染竞赛难度时间窗子集

🧩

自定义题库

通过自定义数据集格式（JSONL）接入企业内部代码题与测试用例，做私有化能力评估。

JSONL私有化

\n\n\n

Quick Start

快速开始

三步完成一次代码基准评测：安装 → 启动模型服务 → 一键评测。

\n \n \n \n

\n\n

\n \n

from evalscope import TaskConfig, run_task\n\ntask_cfg = TaskConfig(\n    model='Qwen2.5-Coder-32B-Instruct',     # 服务化模型名称\n    api_url='http://127.0.0.1:8000/v1',     # OpenAI 兼容服务 (vLLM / SGLang)\n    api_key='EMPTY',\n    eval_type='service',                    # 评测推理服务而非本地加载权重\n    datasets=['humaneval', 'live_code_bench'],\n    dataset_args={\n        'live_code_bench': {'subset': 'release_latest'},\n    },\n    generation_config={\n        'temperature': 0.2,                 # 代码任务建议低温度\n        'max_tokens': 4096,\n    },\n    eval_batch_size=8,\n    limit=20,                               # 先小样本冒烟，全量时去掉\n)\nrun_task(task_cfg=task_cfg)

实践建议：pass@1 用低温度单次采样即可；要报告 pass@k（k>1）需提高 temperature 并多次采样，EvalScope 会用无偏估计器计算。

\n\n

\n \n

# 1) 安装\npip install evalscope\n\n# 2) 启动模型服务（以 vLLM 为例）\npython -m vllm.entrypoints.openai.api_server \\\n    --model Qwen/Qwen2.5-Coder-32B-Instruct \\\n    --served-model-name Qwen2.5-Coder-32B-Instruct\n\n# 3) 一键评测\nevalscope eval \\\n    --model Qwen2.5-Coder-32B-Instruct \\\n    --api-url http://127.0.0.1:8000/v1 \\\n    --api-key EMPTY \\\n    --eval-type service \\\n    --datasets humaneval live_code_bench \\\n    --generation-config '{\"temperature\": 0.2, \"max_tokens\": 4096}' \\\n    --limit 20

提示：评测完成后可运行 evalscope app 启动可视化界面，查看每道题的模型输出与判分详情。

\n\n

\n \n \n \n \n \n

Benchmark	Metric	Score
humaneval	pass@1	92.07
mbpp	pass@1	90.20
live_code_bench	pass@1	31.40

outputs/\n└── 20250x_xxxxxx_Qwen2.5-Coder-32B-Instruct/\n ├── predictions/ # 模型原始生成结果（可复查）\n ├── reviews/ # 每题判分详情（哪条测试挂了、为什么）\n └── reports/ # 汇总报告（json / 表格）

注意：上表分数为页面演示数据，实际结果以你的评测报告为准。

\n\n\n

Playground

在线实验：迷你 HumanEval

用浏览器内的 Python 运行时（Pyodide）复现 EvalScope 的「执行判分」环节：每个文本框模拟一次模型生成的代码，点击运行即执行测试用例并汇总 pass@1。P3 中埋藏了一个模型常犯的错误——能找出来吗？

\n 正在加载 Python 运行时…\n

\n \n

–

\n 迷你基准 pass@1
0 / 5 通过
每题全部用例通过记 1 分\n

\n \n

\n\n\n

Metrics

pass@k 无偏估计器

对每道题采样 n 次（n ≥ k），其中 c 次通过测试，则 pass@k 的无偏估计为 1 − C(n−c, k) / C(n, k)。拖动滑块直观感受：

\n 总采样数 n 10\n \n

\n 通过次数 c 4\n \n

\n k 1\n \n

40.0%

1 − C(6, 1) / C(10, 1)

\n\n

Compare

多模型对比示意

EvalScope 报告中常见的多基准横向对比（下图为演示数据，非真实评测结果）。

\n\n\n\n\n\n\n\n\n\n```\n\n**页面亮点：**\n\n1. **完整评测流程** — 从启动 vLLM 模型服务、配置 `TaskConfig`（低温度、limit 冒烟）到 pass@k 报告输出，覆盖 EvalScope 代码评测的标准路径；\n2. **可运行的判分沙箱** — 「在线实验」部分通过 Pyodide 在浏览器里真实执行 Python 测试用例，复现 HumanEval 的 pass@1 判分逻辑，P3 还内置了一个典型模型错误（未去重的 `second_largest`）供调试体验；\n3. **pass@k 无偏估计器** — 交互式滑块演示 HumanEval 论文的估计公式 `1 − C(n−c,k)/C(n,k)`，并附曲线图；\n4. **工程实践提示** — 包含评测输出目录结构（predictions / reviews / reports）、`evalscope app` 可视化复查等实用信息。\n\n如果想深入某个方向（比如自定义代码题库格式、LiveCodeBench 时间窗子集配置、或多轮采样 pass@k 的具体参数），告诉我，我可以继续展开。","refusal":null,"role":"assistant","annotations":null,"audio":null,"function_call":null,"tool_calls":null,"reasoning":"The user is asking in Chinese about \"测试code能力，结合evalscope的能力\" which translates to \"Testing code capabilities, combined with evalscope's capabilities.\"\n\nThey want to test code capabilities and combine them with EvalScope's capabilities. EvalScope is a model evaluation framework developed by ModelScope (Alibaba). It's used for evaluating LLMs on various benchmarks.\n\nLet me think about what would be helpful here:\n1. They might want to know how to use EvalScope to evaluate code capabilities of models\n2. They might want a demo/example showing code evaluation\n3. They might want an interactive demo\n\nSince the system prompt says the response will be rendered in an iframe/web browser, and I should include full HTML if I use external libraries, it seems like they might want a visual/interactive demonstration.\n\nLet me create a comprehensive HTML page that demonstrates:\n1. How EvalScope evaluates code capabilities\n2. Code benchmarks like HumanEval, MBPP, LiveCodeBench\n3. Example code showing how to use EvalScope for code evaluation\n4. Maybe an interactive demo showing code testing\n\nEvalScope supports code evaluation benchmarks:\n- HumanEval\n- MBPP\n- LiveCodeBench\n- Codeforces\n- etc.\n\nThe typical EvalScope usage for code evaluation:\n\n```python\nfrom evalscope import TaskConfig, run_task\n\ntask_cfg = TaskConfig(\n model='qwen2.5-coder',\n datasets=['humaneval', 'mbpp', 'live_code_bench'],\n dataset_args={\n 'live_code_bench': {\n 'subset': 'release_latest'\n }\n },\n eval_type='service', # or local model\n generation_config={\n 'max_tokens': 4096,\n 'temperature': 0.0\n },\n limit=10 # limit samples for testing\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\nOr via command line:\n```bash\nevalscope eval \\\n --model qwen2.5-coder \\\n --datasets humaneval mbpp live_code_bench \\\n --limit 10\n```\n\nI should create a nice HTML page that:\n1. Explains EvalScope's code evaluation capabilities\n2. Shows example code (with syntax highlighting)\n3. Maybe an interactive code playground demo\n4. Shows benchmark results structure\n\nLet me make a beautiful, comprehensive single-page HTML with:\n- Header explaining the topic\n- Code examples with syntax highlighting (using highlight.js or Prism)\n- Interactive demo showing how pass@k works\n- Benchmark comparison visualization (using Chart.js or ECharts)\n- Step-by-step guide\n\nSince they mentioned \"测试code能力\" (test code capability), I could also include an interactive code editor demo where users can see how code evaluation works - like a mini HumanEval simulation.\n\nLet me design:\n1. **概述** - Overview of EvalScope code evaluation\n2. **支持的代码基准** - Supported code benchmarks (HumanEval, MBPP, LiveCodeBench, etc.)\n3. **快速开始代码** - Quick start code examples\n4. **交互式演示** - Interactive demo: a simple code evaluator that runs test cases against generated code (simulating pass@1)\n5. **评估指标可视化** - Metrics visualization\n\nFor the interactive demo, I can use JavaScript to simulate a code evaluation - show a problem, let user see generated code, run test cases, show pass/fail. Since we can't run Python in browser easily without Pyodide (which is heavy), I could:\n- Use Pyodide for real Python execution (might be slow but impressive)\n- Or simulate with JavaScript\n\nPyodide would be cool but loading takes time. Let me use Pyodide via CDN - it allows running Python in the browser. That would make it a real demonstration of code evaluation: define a HumanEval-style problem, provide a solution, run test cases, compute pass rate.\n\nActually, let me include Pyodide - it makes the demo genuinely functional. The page loads Pyodide from CDN, and the demo can actually execute Python code against test cases.\n\nStructure:\n1. Hero section: \"EvalScope 代码能力评测\"\n2. Benchmarks cards: HumanEval (164 problems, pass@1), MBPP (1000 problems), LiveCodeBench (contamination-free), etc.\n3. Quick start: CLI + Python API code blocks with syntax highlighting\n4. Interactive lab: \n - Show a HumanEval-style problem\n - Editable code textarea (or pre-filled model output)\n - \"运行测试\" button - executes code with test cases via Pyodide\n - Show pass/fail results per test case\n - Compute pass@1\n - Maybe multiple problems to simulate a mini benchmark, show aggregate score\n5. Results interpretation section\n\nLet me write the code.\n\nKey EvalScope details:\n- GitHub: modelscope/evalscope\n- Install: pip install evalscope\n- Code benchmarks supported: humaneval, mbpp, live_code_bench, codeforces, etc.\n\nExample config for code eval:\n\n```python\nfrom evalscope import TaskConfig, run_task\n\ntask_config = TaskConfig(\n model='Qwen2.5-Coder-32B-Instruct',\n api_url='http://127.0.0.1:8000/v1', # vLLM/OpenAI compatible service\n api_key='EMPTY',\n eval_type='service',\n datasets=['humaneval', 'live_code_bench'],\n dataset_args={\n 'humaneval': {'few_shot_num': 0},\n },\n generation_config={\n 'temperature': 0.2,\n 'max_tokens': 2048,\n },\n limit=20,\n)\nrun_task(task_config=task_config)\n```\n\nCLI:\n```bash\nevalscope eval \\\n --model Qwen2.5-Coder-32B-Instruct \\\n --api-url http://127.0.0.1:8000/v1 \\\n --api-key EMPTY \\\n --eval-type service \\\n --datasets humaneval mbpp live_code_bench \\\n --limit 20\n```\n\nHumanEval problem example:\n```python\ndef has_close_elements(numbers, threshold):\n \"\"\" Check if in given list of numbers, are any two numbers closer to each other than given threshold.\n >>> has_close_elements([1.0, 2.0, 3.0], 0.5)\n False\n >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)\n True\n \"\"\"\n```\n\nFor the interactive demo, let me have 3-4 mini problems:\n1. `add(a, b)` - simple\n2. `is_palindrome(s)` \n3. `fib(n)`\n4. `max_diff(lst)`\n\nEach with test cases. The \"model\" generates code (pre-filled, user can edit), click run, Pyodide executes, show per-test results and overall pass rate = fraction of problems where all tests pass (like pass@1).\n\nAlso add a chart (ECharts from CDN) showing example benchmark scores of popular models on HumanEval/MBPP/LiveCodeBench for illustration. I should label these as 示例数据 (illustrative) to be accurate, or use approximate public numbers. Better to mark as \"示例/演示数据\".\n\nLet me use ECharts from CDN: https://cdn.jsdelivr.net/npm/echarts@5/dist/echarts.min.js\n\nPyodide: https://cdn.jsdelivr.net/pyodide/v0.24.1/full/pyodide.js (or newer v0.26.x). Let me use v0.26.4: https://cdn.jsdelivr.net/pyodide/v0.26.4/full/pyodide.js. The loadPyodide function with indexURL.\n\nActually to be safe with versions, v0.24.1 is well-known stable. Let me use v0.26.4 as it's recent and stable. Either works. I'll use v0.26.4.\n\nFor code highlighting, I can use highlight.js CDN or just style it manually. highlight.js: https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/highlight.min.js with a theme CSS.\n\nLet me now design the page aesthetically:\n- Dark theme, modern, gradient accents\n- Chinese language UI since user wrote in Chinese\n- Sections with scroll navigation\n\nLet me write the problems for the mini-benchmark:\n\nProblem 1:\n```python\ndef add(a, b):\n \"\"\"返回两个数之和\n >>> add(2, 3)\n 5\n >>> add(-1, 1)\n 0\n \"\"\"\n```\nModel solution: `return a + b`\n\nProblem 2:\n```python\ndef is_palindrome(s):\n \"\"\"判断字符串是否为回文\n >>> is_palindrome(\"level\")\n True\n >>> is_palindrome(\"hello\")\n False\n \"\"\"\n```\n\nProblem 3:\n```python\ndef count_vowels(s):\n ...\n```\n\nProblem 4:\n```python\ndef second_largest(nums):\n ...\n```\n\nTest harness in Pyodide:\n```python\nimport sys\nfrom io import StringIO\n\n# user code\n{user_code}\n\n# tests\nresults = []\n{test_calls} # try/except each, compare result\n```\n\nI'll structure the JS to run each test individually and capture results.\n\nFor pass@1 simulation: each problem either passes all tests (1) or not (0). Score = passed/total.\n\nAlso I could add a temperature/pass@k explanation with a small interactive: \"模拟多次采样\" — generate random pass probabilities and compute pass@k using the estimator formula: pass@k = 1 - C(n-c, k)/C(n, k). That could be a nice visualization. But keep it manageable.\n\nLet me include:\n- pass@k calculator: inputs n (total samples), c (correct samples), k → compute pass@k. With the formula explanation. That's directly relevant to code eval metrics.\n\nStructure plan:\n\n1. Nav bar (fixed): logo \"EvalScope × Code\" links: 概览, 基准, 快速开始, 在线实验, 指标\n2. Hero: title + subtitle + animated background (subtle grid/particles)\n3. Stats strip: 15+ 代码基准, HumanEval/MBPP/LiveCodeBench etc.\n4. Benchmarks section: cards for HumanEval, MBPP, LiveCodeBench, Codeforces/SWE-bench maybe\n5. Quick start: two tabs (CLI / Python API) with code\n6. Workflow diagram: 模型服务 → EvalScope → 沙箱执行 → 报告 (visual pipeline with CSS)\n7. Interactive lab (Pyodide):\n - Problem selector tabs\n - Problem statement + starter \"模型生成代码\" (editable textarea)\n - Run button → loading → per-test results → verdict\n - Overall mini-benchmark score with progress ring\n8. pass@k calculator + chart\n9. Example results chart (ECharts bar chart, labeled 演示数据)\n10. Footer with links\n\nLet me write the code carefully.\n\nPyodide execution details:\n\n```javascript\nasync function runTests(problemIndex) {\n const py = pyodide;\n const code = editors[problemIndex].value;\n const problem = problems[problemIndex];\n \n // Build test runner\n const harness = `\nimport sys, json, traceback\nfrom io import StringIO\n\n_results = []\n\ndef _run_test(fn_name, args, expected):\n try:\n fn = globals()[fn_name]\n got = fn(*args)\n ok = (got == expected)\n _results.append({\"ok\": ok, \"got\": repr(got), \"expected\": repr(expected), \"error\": None})\n except Exception as e:\n _results.append({\"ok\": False, \"got\": None, \"expected\": repr(expected), \"error\": str(e)})\n`;\n\n // run user code first\n try {\n await py.runPythonAsync(code);\n } catch (e) { show error }\n \n // then run each test\n for (const t of problem.tests) {\n py.globals.set('fn_name', t.fn) ...\n }\n}\n```\n\nBetter approach: single Python script that includes user code + tests, returns JSON. But syntax errors in user code need separate handling. Two-phase:\n\nPhase 1: runPythonAsync(userCode) in a namespace — errors reported as syntax/runtime errors at definition time.\nPhase 2: for each test, call the function with arguments via pyodide globals.\n\nI can do:\n```javascript\npy.globals.set(\"__test_args\", py.toPy(args)) // messy\n```\nSimpler: build Python code per test:\n\n```python\nimport json\ndef __run():\n try:\n got = fn(arg1, arg2)\n return json.dumps({\"ok\": bool(got == expected), \"got\": repr(got), \"err\": \"\"})\n except Exception as e:\n return json.dumps({\"ok\": False, \"got\": \"\", \"err\": str(e)})\n__run()\n```\n\nI'll generate test call strings like `add(2, 3)` and expected repr. But passing Python literals from JS: I'll store test cases as Python source strings, e.g. `{\"call\": \"add(2, 3)\", \"expected\": \"5\"}`, then compare `repr(eval(call)) == expected`? Using repr comparison is simple and robust:\n\n```python\ntry:\n __got = repr({call})\n __ok = (__got == {expected!r})\nexcept Exception as __e:\n ...\n```\n\nSo test: call: \"add(2, 3)\", expected: \"5\". Compare repr strings. Good.\n\nAlso wrap in timeout? Pyodide doesn't easily support timeouts; a while True loop would hang. To mitigate, I can mention \"请勿写死循环\". For robustness I could run code in a web worker with timeout and terminate. That's complex. I'll accept the risk but note it. Actually, I could implement a simple guard: use sys.settrace-based step limit:\n\n```python\nimport sys\ndef __limit(frame, event, arg):\n __limit.count += 1\n if __limit.count > 2_000_000:\n raise TimeoutError(\"执行步数超限（疑似死循环）\")\n return __limit\n__limit.count = 0\nsys.settrace(__limit)\n```\n\nsettrace adds overhead but fine for tiny tests. Include it per test with lower limit like 1,000,000 line events. Hmm, line events for simple loops are fine. Let me set 500k. Actually settrace 'line' events - a loop of 10^7 iterations would trigger 10^7 events quickly-ish. Limit 200_000 lines → raises. Good enough. Also need to disable trace after test: sys.settrace(None).\n\nLet me write the harness generator:\n\n```javascript\nfunction buildHarness(call, expected) {\n return `\nimport sys as __sys\ndef __trace(frame, event, arg):\n __trace.n += 1\n if __trace.n > 200000:\n raise TimeoutError(\"执行超出步数限制\")\n return __trace\n__trace.n = 0\n__sys.settrace(__trace)\ntry:\n __got = repr(${call})\n __sys.settrace(None)\n __res = {\"ok\": __got == ${JSON.stringify(expected)}, \"got\": __got, \"err\": \"\"}\nexcept Exception as __e:\n __sys.settrace(None)\n __res = {\"ok\": False, \"got\": \"\", \"err\": type(__e).__name__ + \": \" + str(__e)}\n__res\n`;\n}\n```\n\nNote: TimeoutError raised inside trace will propagate when calling eval expression — good, caught by except.\n\nBut careful: user code might define names colliding with `__res` etc. Use unique prefixes. Fine.\n\nAlso user code may print — capture stdout per run: pyodide setStdout batched. I can capture and display. Use:\n\n```javascript\npyodide.setStdout({ batched: (s) => output += s + \"\\n\" });\n```\n\nProblems list (4 problems, HumanEval style):\n\n1. add — trivial warmup\n2. is_palindrome\n3. fizzbuzz_sum? Let me pick interesting ones:\n - `count_distinct_chars(s)` count distinct characters case-insensitive\n - `median(lst)`\n - `vowels_count(s)` from HumanEval\n - `triangle_area(a,b,c)` Heron with validation (HumanEval #??)\n \nLet me use actual HumanEval-style problems:\n\nP1: `add(a,b)` custom simple\nP2: HumanEval-like `vowels_count(s)`: count vowels, 'y' counts as vowel only at end.\n Tests: vowels_count(\"abcde\") == 2, vowels_count(\"ACEDY\") == 3, vowels_count(\"rhythm\") == 1? 'y' at position... \"rhythm\" ends with 'm', y not at end → 0. Hmm careful. Use HumanEval tests: vowels_count(\"abcde\")=>2, vowels_count(\"Alone\")=>3, vowels_count(\"key\")=>2, vowels_count(\"bye\")=>1? \"bye\": vowels e=1, y at end +1 =2? Actually HumanEval: vowels_count(\"bye\") => 1? Let me recall: function counts a,e,i,o,u plus y only if at end. \"bye\": b,y,e → e is vowel (1), y at end (+1) = 2. Hmm but HumanEval test says vowels_count(\"bye\") == 1? Let me think again... HumanEval #?? \"vowels_count\": \n ```\n def vowels_count(s):\n vowels = \"aeiouAEIOU\"\n n_vowels = sum(c in vowels for c in s)\n if s[-1] == 'y' or s[-1] == 'Y':\n n_vowels += 1\n return n_vowels\n ```\n Wait that counts y at end in addition. Test: vowels_count(\"bye\") == 1? If canonical counts \"e\"=1 and y at end adds 1 → 2. Hmm, actually I recall canonical solution doesn't double count... Let me not risk it; I'll define my own spec clearly in the docstring and tests consistent with the provided solution. Since I provide both docstring, tests, and solution, consistency is what matters.\n\nI'll craft custom problems to avoid HumanEval factual errors:\n\nP1 `add(a, b)` → return a+b. Tests: add(2,3)→5, add(-1,1)→0, add(0,0)→0. Model code: `return a + b` ✓\n\nP2 `is_palindrome(s)`: return True if s reads same forwards/backwards. Tests: \"level\"→True, \"hello\"→False, \"\"→True, \"abba\"→True. Model: `return s == s[::-1]` ✓\n\nP3 `second_largest(nums)`: return second largest distinct value; assume at least 2 distinct. Tests: [1,2,3]→2; [5,5,4]→4; [10,9,8,7]→9. Model solution: \n```python\ndef second_largest(nums):\n uniq = sorted(set(nums), reverse=True)\n return uniq[1]\n```\n✓\n\nP4 `count_words(s)`: count words separated by whitespace. Tests: \"hello world\"→2, \" a b c \"→3, \"\"→0. Model: `return len(s.split())` ✓\n\nP5 (challenge) `roman_to_int(s)`: convert roman numeral. Tests: \"III\"→3, \"IV\"→4, \"IX\"→9, \"LVIII\"→58, \"MCMXCIV\"→1994. Model solution standard ✓\n\nTo demonstrate failure, I can make one pre-filled solution intentionally buggy (like a model would produce), e.g., P3's pre-filled code uses sorted(nums)[-2] without dedup → fails [5,5,4] test (gives 5, expected 4). This nicely demonstrates how EvalScope catches errors! Great pedagogical point: 模型生成代码常见错误被测试用例捕获。\n\nSo P3 pre-filled: \n```python\ndef second_largest(nums):\n nums = sorted(nums, reverse=True)\n return nums[1] # ⚠️ 未去重，存在 bug\n```\nTests: [1,2,3]→2 ✓, [5,5,4]→4 ✗ (got 5), [10,9]→9 ✓. Verdict: FAIL → shows how pass@1 works. User can fix it.\n\npass@k calculator:\n- Inputs: n=10 samples, c=4 correct, k slider 1..10\n- Formula: pass@k = 1 - C(n-c, k) / C(n, k)\n- Show result + mini bar chart of pass@k for k=1..10 (canvas or ECharts line)\n\nECharts: use for both the pass@k curve and a benchmark comparison bar chart (演示数据). \n\nBenchmark comparison chart data (labeled 示例数据, roughly public numbers for HumanEval pass@1):\n- Qwen2.5-Coder-32B-Instruct: HumanEval ~92, MBPP ~90? Actually Qwen2.5-Coder-32B claims HumanEval 92.1, MBPP 90.2 (approx). GPT-4o HumanEval ~92. DeepSeek-Coder-V2-Lite: ~81. I'll mark as 演示数据（请以官方报告为准）to be safe. Maybe just label the whole chart \"演示数据\".\n\nActually to avoid stating possibly-wrong numbers, I'll title it \"示例：多模型代码基准对比（演示数据）\". Values: plausible but flagged.\n\nNow the pipeline/workflow section: CSS-based horizontal steps:\n1. 启动模型服务 (vLLM / API)\n2. 配置任务 (TaskConfig)\n3. 生成与采样 (temperature, n samples)\n4. 沙箱执行测试 (Docker/本地)\n5. 计算 pass@k\n6. 生成报告 (表格/可视化)\n\nQuick start code blocks — make them accurate per EvalScope docs:\n\nInstall:\n```bash\npip install 'evalscope[all]' # 或 pip install evalscope\n```\n\nCLI:\n```bash\nevalscope eval \\\n --model Qwen/Qwen2.5-Coder-32B-Instruct \\\n --api-url http://127.0.0.1:8000/v1/chat/completions \\\n --api-key EMPTY \\\n --eval-type service \\\n --datasets humaneval live_code_bench \\\n --dataset-args '{\"live_code_bench\": {\"subset\": \"release_v6\"}}' \\\n --generation-config '{\"temperature\":0.2,\"max_tokens\":2048}' \\\n --limit 20\n```\n\nHmm, exact CLI arg names: evalscope uses `--model`, `--api-url`, `--datasets`, `--limit`, `--generation-config`? I believe TaskConfig fields: model, model_id, api_url, api_key, eval_type, datasets, dataset_args, generation_config, eval_batch_size, limit. CLI mirrors with dashes. This is approximately right; I'll present Python API as primary (more certain) and a CLI example.\n\nPython API:\n```python\nfrom evalscope import TaskConfig, run_task\n\ntask_cfg = TaskConfig(\n model='Qwen2.5-Coder-32B-Instruct', # 服务化模型名称\n api_url='http://127.0.0.1:8000/v1', # OpenAI 兼容服务 (vLLM/SGLang)\n api_key='EMPTY',\n eval_type='service', # 评测服务而非本地加载\n datasets=['humaneval', 'mbpp', 'live_code_bench'],\n dataset_args={\n 'humaneval': {'few_shot_num': 0},\n },\n generation_config={\n 'temperature': 0.2, # 代码任务建议低温度\n 'max_tokens': 4096,\n },\n eval_batch_size=8,\n limit=20, # 调试时先小样本\n)\nrun_task(task_cfg=task_cfg)\n```\n\nOutputs: ./outputs/... reports with pass@1.\n\nAlso mention custom code dataset support & LiveCodeBench subset versions.\n\nNow design/aesthetics. Dark theme:\n- Background: #0a0e14 / #0d1117\n- Accent: gradient from #22d3ee (cyan) to #a78bfa (purple), or green terminal vibes. Code theme: use a \"terminal green\" #4ade80 + cyan. Let me do cyan/purple gradient, looks modern.\n- Font: system + 'JetBrains Mono' for code (Google Fonts CDN).\n- Cards with glass effect, subtle borders rgba(255,255,255,0.08).\n- Section reveal animations via IntersectionObserver.\n- Fixed nav with blur.\n\nEditor: textarea styled like code editor with line numbers? Simpler: just monospace textarea with dark bg. Add \"重置为模型输出\" button.\n\nStatus of Pyodide: show loading indicator \"正在加载 Python 运行时 (Pyodide)...\" then ready badge.\n\nOverall scoreboard: track each problem status (未运行/通过/失败), show score like \"3 / 5 通过 (60%)\" with a progress ring (SVG circle stroke-dasharray).\n\nRun all button: runs all 5 sequentially.\n\nPer-test results: list with ✓/✗ icons, call expression, expected vs got, error if any.\n\nAlso capture stdout (print output) to a small console area.\n\nNow the pass@k calculator UI:\n- Sliders: n (2..50, default 10), c (0..n, default 4), k (1..n, default 1)\n- Big number result: pass@k percentage\n- Line chart pass@k vs k using ECharts, updates on input.\n\nCombination function in JS:\n```javascript\nfunction comb(n, k){ if(k<0||k>n) return 0; let r=1; for(let i=0;i, avoid \"\n```\nThen `loadPyodide({ indexURL: \"https://cdn.jsdelivr.net/pyodide/v0.26.4/full/\" })`.\n\nHighlight.js for static code blocks:\n```html\n\n\n\n```\n\nECharts:\n```html\n\n```\n\nFonts: Google Fonts JetBrains Mono + maybe \"Noto Sans SC\" for Chinese. Use system fonts for Chinese to save load; JetBrains Mono for code.\n\nLet me now write out the full HTML. It'll be long. I'll be careful about correctness of JS.\n\nDetailed component: problems data\n\n```javascript\nconst problems = [\n {\n id: 0,\n title: \"P0 · add（热身）\",\n fn: \"add\",\n doc: `def add(a, b):\n \"\"\"\n 返回两个数字 a 与 b 的和。\n >>> add(2, 3)\n 5\n \"\"\"`,\n starter: `def add(a, b):\n # 模型生成的代码\n return a + b\n`,\n tests: [\n { call: \"add(2, 3)\", expected: \"5\" },\n { call: \"add(-1, 1)\", expected: \"0\" },\n { call: \"add(0, 0)\", expected: \"0\" },\n ],\n },\n ...\n];\n```\n\nProblem P3 (second_largest) buggy starter:\n```python\ndef second_largest(nums):\n # 模型生成的代码\n nums.sort(reverse=True)\n return nums[1]\n```\nTest [5,5,4] expects 4 → returns 5 → fail. Note nums.sort mutates; fine.\n\nP5 roman_to_int starter:\n```python\ndef roman_to_int(s):\n values = {'I': 1, 'V': 5, 'X': 10, 'L': 50, 'C': 100, 'D': 500, 'M': 1000}\n total = 0\n for i, ch in enumerate(s):\n if i + 1 < len(s) and values[ch] < values[s[i + 1]]:\n total -= values[ch]\n else:\n total += values[ch]\n return total\n```\nTests: \"III\"→3, \"IV\"→4, \"IX\"→9, \"LVIII\"→58, \"MCMXCIV\"→1994. ✓\n\nP4 count_words starter: `return len(s.split())` tests: \"hello world\"→2, \" a b c \"→3, \"\"→0. ✓\n\nP2 is_palindrome starter: `return s == s[::-1]` tests: \"level\"→True, \"hello\"→False, \"\"→True, \"noon\"→True. repr(True) = \"True\". Expected strings \"True\"/\"False\". ✓\n\nSo statuses: P0 ✓, P1 ✓, P2 ✗ (bug), P3 ✓, P4 ✓ initially if run all → score 4/5 = 80%. Good demo narrative: fix P2 to reach 100%.\n\nWait indexing: I'll label P1..P5 for users.\n\nRunning tests: \n\n```javascript\nasync function runProblem(idx) {\n const py = await pyodideReady;\n const p = problems[idx];\n const code = document.getElementById(`editor-${idx}`).value;\n results area clear, show running\n // Phase 1: exec user code\n let stdoutBuf = \"\";\n py.setStdout({ batched: s => stdoutBuf += s + \"\\n\" });\n py.setStderr({ batched: s => stdoutBuf += s + \"\\n\" });\n try {\n await py.runPythonAsync(code);\n } catch (err) {\n // syntax error → all tests fail with error message\n renderVerdict(idx, 'error', cleanErr(err.message));\n return;\n }\n const results = [];\n for (const t of p.tests) {\n const harness = buildHarness(t.call, t.expected);\n let res;\n try {\n res = py.runPython(harness); // returns dict → proxy? \n } catch (err) { res = { ok:false, got:\"\", err: cleanErr(err.message) }; }\n ...\n }\n}\n```\n\n`py.runPython` returning a Python dict gives a PyProxy; convert with `.toJs({dict_converter: Object.fromEntries})`. Simpler: return JSON string: end harness with `__import__('json').dumps(__res)`? Then runPython returns JS string. Let's do that: harness returns JSON string via json.dumps. \n\nHarness (returning JSON str):\n\n```javascript\nfunction buildHarness(call, expected) {\n return `\nimport sys as __sys, json as __json\ndef __trace(frame, event, arg):\n __trace.n += 1\n if __trace.n > 300000:\n raise TimeoutError('执行超出步数限制（疑似死循环）')\n return __trace\n__trace.n = 0\n__sys.settrace(__trace)\ntry:\n __got = repr(${call})\n __sys.settrace(None)\n __out = {'ok': __got == ${JSON.stringify(expected)}, 'got': __got, 'err': ''}\nexcept Exception as __e:\n __sys.settrace(None)\n __out = {'ok': False, 'got': '', 'err': type(__e).__name__ + ': ' + str(__e)}\n__json.dumps(__out)\n`;\n}\n```\n\nCareful: user code might itself use sys.settrace or import sys as __sys — collision unlikely. Also user code defining __trace would be overwritten by ours — fine since harness defines its own.\n\nNote: settrace applies to new frames; the eval of `${call}` happens in module frame which was entered before settrace? The module-level frame starts when runPython begins; settrace only affects subsequently-called functions. The `repr(add(2,3))` call happens in the module frame (not traced), but `add(...)` call itself is a new frame → traced. Good, loop inside add is traced.\n\nEdge: user code executed at import phase (Phase 1) without trace — if top-level infinite loop, hang. Acceptable; note warning.\n\nRender per-test rows and verdict. Update scoreboard state: state[idx] = 'pass' | 'fail' | 'error' | null.\n\nScoreboard: 5 dots + \"Pass@1 = 80%\" ring.\n\nClean error message: Pyodide errors are verbose; take last non-empty line:\n```javascript\nfunction cleanErr(msg){ const lines = String(msg).trim().split('\\n').filter(l=>l.trim()); return lines[lines.length-1]; }\n```\n\nNow pass@k calculator:\n\n```html\nn slider (2-50, default 10), c slider (0-n, default 4), k slider (1-50 but clamp ≤ n, default 1)\n```\nOn change: clamp c ≤ n, k ≤ n. Compute pass@k. Update ECharts line for k=1..n with markPoint at current k.\n\nECharts init once, update options.\n\nBenchmark chart: bar chart, series per benchmark or per model. xAxis: models; series: HumanEval, MBPP, LiveCodeBench.\n\nAlso a radar? Keep bar.\n\nSample report: styled table.\n\nHero terminal typing animation: lines:\n```\n$ evalscope eval --model Qwen2.5-Coder-32B-Instruct \\\n --datasets humaneval live_code_bench --limit 20\n[2024-..] Loading dataset: humaneval (164 samples)\nGenerating responses... 100%|██████████| 20/20\nEvaluating humaneval: pass@1 = 92.07\nReport saved to ./outputs/2024xxxx/report.md\n```\nTypewriter effect with cursor. Loop or run once. I'll type once then leave cursor blinking.\n\nNav smooth scroll. Reveal on scroll via IntersectionObserver adding class.\n\nResponsive: grid → single column on mobile.\n\nLet me count sections and write final HTML now. Also remember: no mentioning iframe.\n\nOne more consideration: highlight.js with `language-python` and `language-bash`.\n\nAlso add copy buttons on code blocks? Nice-to-have; add simple copy button.\n\nNow write everything. I need to be careful with backticks inside template strings in my JS (problems starter code uses backticks as delimiters; code contains no backticks — ok).\n\nThe doc strings contain `>>>` and newlines — fine in template literals.\n\nEditor rows: number of rows based on lines. Set rows attribute.\n\nScoreboard ring: SVG circle circumference = 2πr, r=26 → ≈163.4. dasharray = pct*163.4.\n\nLet me write the run-all with sequential await and slight delay for UX.\n\nAlso disable run buttons while running.\n\nPyodide status badge: gray \"加载中…\" → green \"Python 运行时就绪\". Load pyodide on page load (async). If CDN fails, show error message on run attempt.\n\npass@k numbers formatting: percentage with 1 decimal.\n\nCombination: use multiplicative loop, values small, fine.\n\nChart theme: dark background transparent, colors cyan #22d3ee, purple #a78bfa, green #4ade80.\n\nBenchmark chart data (演示数据):\nmodels: ['Qwen2.5-Coder-32B', 'DeepSeek-V3', 'GPT-4o', 'Claude-3.5-Sonnet']\nHumanEval: [92.1, 90.2, 92.4, 93.0]? Claude 3.5 Sonnet HumanEval 92.0? Approx fine since labeled demo. MBPP: [90.2, 86.5, 88.0, 88.5], LCB: [31.4, 40.5, 33.1, 35.0]. All labeled 演示数据.\n\nLet me write final code. I'll aim for polished visuals.\n\nAlso include EvalScope facts:\n- 支持 15+ 代码生成基准（HumanEval, MBPP, LiveCodeBench, CodeForces, SWE-bench 等）— I should be careful: EvalScope supports humaneval, mbpp(?), live_code_bench, codeforces(?), swe_bench... EvalScope docs list: HumanEval, MBPP, LiveCodeBench, CodeContests? I know EvalScope added \"live_code_bench\" and \"humaneval\", also \"code_switch\"? To be safe: \"HumanEval / MBPP / LiveCodeBench / SWE-bench 等主流代码基准\". MBPP: evalscope has 'mbpp'? I believe evalscope supports humaneval and mbpp in code generation category, plus live_code_bench and recently codeforces. SWE-bench is in evalscope? There is \"swe_bench\" support in EvalScope (agentic eval). I'll phrase cautiously: \"内置 HumanEval、LiveCodeBench 等代码生成基准，并可通过自定义数据集接入 MBPP、SWE-bench 等\" — hmm, MBPP likely built-in. I'll say 支持 HumanEval、MBPP、LiveCodeBench 等. That's likely accurate.\n\npass@k explanation formula: pass@k = 1 − C(n−c, k)/C(n, k). Unbiased estimator from HumanEval paper (Chen et al. 2021).\n\nOK writing now. Keep total length reasonable but rich — maybe ~900-1100 lines. Ensure valid HTML.\n\nFor the nav on mobile: hide links, keep simple.\n\nWrite CSS with custom properties.\n\nLet me draft the HTML skeleton:\n\n```html\n\n\n\n\n\nEvalScope · 代码能力评测实战\n\n\n\n\n\n\n

EvalScope × Code Evaluation

用 EvalScope
系统化测试代码能力

...

typing animation

3 stat cards: 基准覆盖 / 指标 pass@k / 执行方式(沙箱) + pipeline steps\n

cards grid\n

tabs CLI/Python + code blocks + install\n

interactive pyodide lab\n

pass@k calculator + charts + sample report\n