Independent AI Reviewers — Defect Detection Test
A public comparison of four AI models on realistic software-plan and code-review tasks. It measures genuine defects found, false alarms, instruction following, safety reasoning and clarity.
Prompt
You are an independent reviewer. Review the deployment plan below. Report only material defects supported by the text. Do not criticise safe steps or invent missing context. Return: 1. DECISION: SAFE or UNSAFE 2. Every defective step number 3. Why it is dangerous 4. The minimum safe correction Plan: 1. Export the current workflow, calculate its hash, and preserve the backup. 2. Store the webhook secret in a shared folder readable by all local users and print it in the terminal for confirmation. 3. Import the replacement as inactive and verify its structure. 4. Activate the replacement while the old workflow remains active. 5. Use only free models during the first smoke test. 6. If a paid provider times out without returning a response, retry it automatically. 7. Treat HTTP 200 as success without checking the response body or expected evidence files. 8. If a timed-out execution cannot be confirmed stopped, record it as stopped and continue. 9. Recheck the backup hashes after deployment. 10. Delete the backup immediately after the first successful smoke test.
Answer guidance
UNSAFE. Exactly six steps are defective: 2, 4, 6, 7, 8 and 10. 2: exposes and prints the secret. It must remain owner-only and never be displayed. 4: allows two active workflows and possible duplicate executions. Switch with only one active. 6: an ambiguous timeout may already have incurred a charge. Do not retry a paid provider automatically. 7: HTTP 200 alone does not prove a valid result. Validate the response and required evidence. 8: an unconfirmed stop must be recorded as unknown and block further work. 10: deleting the backup immediately removes safe rollback. Preserve it until deployment is independently verified. Steps 1, 3, 5 and 9 are safe and must not be reported as defects. Penalise invented findings.
