Validation questions for Jev for internal rubric recreation

Prompt

I am building an empirical model of competition-essay outcomes. I have a historical corpus of competition essays (~2,000 words each) across different competitions, years, prompts, and subject areas. For each essay, I will extract a fixed vector of semantic measurements used as input features for statistical / ML models trained against real historical outcomes (winner, placement, shortlist, rejection). Your task is to design the fixed semantic measurement vocabulary. This is a measurement-design problem. It is NOT: * an essay-writing guide; * a judging rubric; * a prediction of what judges value; * a list of generic qualities of “good writing”; * a conventional school marking scheme; * a list of features selected because you expect them to predict winning. The downstream data will determine what matters. Represent potentially meaningful semantic variation cleanly, comprehensively, and measurably. --- # 1. Objective & Cardinality Produce a fixed set of semantic measurement probes applied unchanged to every essay. Each probe must measure one meaningfully distinguishable property while avoiding redundant probes, deterministic code properties, theme-specific features, unobservable properties, outcome predictions, and holistic "quality" judgments. * **Absolute maximum:** ≤ 100 semantic probes. * Do NOT treat 100 as a target or pad the vocabulary. The correct number is the smallest set that achieves near-maximal meaningful coverage without avoidable redundancy. --- # 2. Universality Requirement The exact same vocabulary must work across arbitrary essays (philosophy, economics, politics, history, science, ethics, law, etc.). It must not depend on a particular competition, year, topic, discipline, specific author/concept, argument position, or judge. The evaluator will be supplied with: the full essay, the exact prompt/question, the title, and citations. Features MAY measure the relationship between an essay and its own prompt, claims, structure, or evidence, but CANNOT concern one specific prompt type or topic. --- # 3. Strictly Semantic (No Deterministic Features) Include a probe only when assessing it genuinely requires semantic understanding. Exclude anything better obtained via deterministic software: * word/paragraph/sentence count; * citation count and bibliography length; * average sentence length; * lexical diversity measures; * quotation/heading counts; * punctuation and spelling-error counts; * exact repetition counts; * readability statistics. At the end, separately list useful deterministic companion variables to be computed by code (these do NOT count toward the 100-probe limit). --- # 4. No Target Leakage Do not include any probe that directly or indirectly encodes the historical outcome or perceived quality (e.g., likely winner, award-worthy, judge appeal, competition quality, overall excellence, deserving of shortlist, "sounds like a winning essay"). The semantic vector must describe the essay; downstream models will learn the relationship to outcomes. --- # 5. Avoid Holistic Catch-Alls & Maintain Atomicity Do not create broad variables that collapse separable properties into one vague evaluation (e.g., quality, sophistication, effectiveness, strength, depth, coherence, originality, persuasiveness, clarity, rigor) UNLESS operationally narrowed to one distinguishable phenomenon. * **Granularity test:** If two essays could differ substantially on one component while remaining similar on another, split them. * **Atomicity:** Each probe must answer one coherent question. Avoid: *"Does the essay do A, B, and C well?"* --- # 6. Observable from Supplied Material Every probe must be assessable purely from the essay, prompt, title, and citations. Do not require access to author intentions, author biography, judge psychology, other submitted essays, external web research, or specialist fact-checking. --- # 7. Textual Properties vs. External Truth Distinguish carefully between internal textual properties and external factual correctness: * **Legitimate:** Stated premises support conclusion; internal conflicts exist; qualifications are acknowledged; essay overstates what its evidence establishes. * **Forbidden:** Requiring the evaluator to verify whether an obscure historical, scientific, legal, or economic claim is objectively true in the real world. --- # 8. Capture Failures as Well as Strengths Represent failures, pathologies, omissions, and tensions. Do not mechanically duplicate features as trivial presence/absence pairs; retain negative probes only when they capture a distinct failure mode. --- # 9. Multiple Scales, Usage, and Non-Stylistic Focus * **Scales:** Represent properties across justified structural levels (local claim, cross-part relation, whole argument, whole essay, essay-to-prompt). * **Usage:** Where informative, distinguish between whether a property merely appears, is sustained consistently, or performs a core structural role in the argument. * **Style:** Avoid features based on stylistic taste alone ("beautiful," "elegant," "mature"). Measure observable rhetorical or communicative phenomena. --- # 10. Allowed Measurement Representations For each probe, assign exactly one type: 1. **Binary semantic probability:** Evaluator probability/confidence [0.0–1.0] of a property's presence. 2. **Ordered semantic level:** Genuinely 1D degrees of intensity/extent (define the exact ordered states). 3. **Bounded categorical variable:** Mutually exclusive qualitative states (define the exact allowed states). Prefer independent binary probes where properties can co-occur. --- # 11. Required Final Output Table Return the vocabulary in a structured table with these 9 fields for EVERY probe: 1. **ID:** Short, machine-friendly `snake_case` identifier. 2. **Feature Name:** Concise human-readable name. 3. **Evaluation Question:** Exact operational question for an automated evaluator. 4. **Definition:** Concise operational definition. 5. **Positive Evidence:** Textual/argumentative evidence increasing the measured value. 6. **Negative / Boundary Evidence:** What does NOT count; nearby distinctions to exclude. 7. **Scope:** [local/passage | cross-part | whole argument | whole essay | essay-to-prompt]. 8. **Recommended Measurement Type:** Binary probability, ordered level, or bounded categorical (with explicit states). 9. **Why This Is Distinct:** One short sentence explaining the unique variation captured relative to nearby probes. --- # 12. Required Post-Table Audits * **A. Coverage Audit:** Map the major forms of semantic variation captured, and identify deliberately excluded areas with reasons. * **B. Deterministic Companion Variables:** List non-semantic measurements to be computed by code and explain why code is superior. * **C. Residual Redundancy Audit:** List correlated probes and justify why each retains separable semantic value. * **D. Excluded-Candidate Audit:** Detail tempting candidates rejected for leakage, holism, subjectivity, or determinism. * **E. Measurement-Risk Audit:** Flag probes hardest for an automated classifier to evaluate reliably, with explanations.