You are a senior AI systems architect, reliability engineer,...

Prompt

You are a senior AI systems architect, reliability engineer, adversarial tester, and prompt-engineering specialist. I am building an automated AI pipeline that discovers external material, extracts useful signals from it, transforms those signals into original structured outputs, evaluates them through several quality gates, compares them against historical outputs, selects a small final portfolio, and persists enough history to reduce repetition across future runs. The exact product domain, creative concept, source list, repository, providers, prompts, and business purpose are intentionally withheld. I do NOT want you to redesign the product idea. I want you to audit the SYSTEM LOGIC. Assume that a first implementation already exists and appears to work in normal tests, but I am concerned that it may still contain hidden assumptions, incomplete guarantees, silent failure modes, or edge cases that only appear after many production runs. Your task is to think like an adversarial production engineer before deployment. ## PRIMARY OBJECTIVE Find every important assumption that could make the system behave incorrectly while still appearing successful. Do not limit yourself to obvious software bugs. Look for: * assumptions about identity * assumptions about uniqueness * assumptions about persistence * assumptions about history * assumptions about external data * assumptions about AI model behavior * assumptions about retries * assumptions about concurrency * assumptions about partial failure * assumptions about ordering * assumptions about provenance * assumptions about semantic similarity * assumptions about configuration * assumptions about pricing or resource eligibility * assumptions about APIs changing * assumptions about incomplete or malformed responses * assumptions about what can and cannot actually be guaranteed ## SYSTEM SHAPE At a high level, the pipeline behaves approximately like this: Input → history-aware planning → external discovery → signal extraction → factual or rule validation → original transformation → quality validation → structural fingerprinting → historical comparison → portfolio-level diversity selection → final outputs → persistent history update The implementation may also run repeatedly in an ephemeral CI environment and may write state back to persistent storage. External AI services and external information sources may be involved. ## AUDIT REQUIREMENTS Do not assume that because something has a check, the check is sufficient. For every stage ask: 1. What does the implementation probably assume? 2. Under what condition is that assumption false? 3. What incorrect behavior could result? 4. Would the failure be visible, or could the run still report success? 5. How should the invariant actually be defined? 6. Can it be enforced deterministically? 7. If not, what is the strongest honest guarantee we can make? 8. What test would prove that the protection really works? Pay special attention to identity and deduplication. For example, think about whether two different URLs can represent the same material, whether the same material can appear with tracking parameters or mirrors, whether slightly modified content bypasses deduplication, whether a source can accidentally generate multiple descendants, whether rejected candidates can reappear during the same run, and whether historical comparison accidentally examines only a limited recent window. Also examine lineage. The system should be able to reason about relationships such as: external item → extracted signal → transformed candidate → accepted output Ask whether one ancestor can unintentionally produce multiple final descendants, whether lineage survives retries, and whether provenance can be lost when data is transformed. ## CONCURRENCY AND TRANSACTIONAL SAFETY Assume two executions may start close together. Audit race conditions such as: * both processes reading the same old history * both reserving the same external item * both generating equivalent outputs * one run overwriting another run's history * partial commits * process termination between generation and history persistence * retry after an uncertain commit * CI jobs running concurrently * duplicate commits or merge conflicts Determine which operations require: * locks * leases * reservations * idempotency keys * atomic writes * transactional semantics * compare-and-swap behavior * recovery markers Do not simply say "use a lock." Explain exactly what invariant the lock protects. ## RETRY AUDIT Retries are particularly dangerous. For every retryable stage ask whether retrying can: * consume the same source again * create another descendant from the same source * duplicate final outputs * bypass diversity checks * lose provenance * create inconsistent state * turn an uncertain failure into a double write Classify operations as: SAFE TO RETRY SAFE ONLY WITH IDEMPOTENCY NOT SAFE TO RETRY AUTOMATICALLY ## HISTORY AUDIT Treat long-term history as a critical subsystem. Check whether: * comparisons use the full required history rather than an arbitrary recent subset * history can grow indefinitely without making the system unusable * old schema versions remain readable * corrupted records are detected * incomplete runs contaminate permanent history * rejected candidates and accepted outputs are stored differently * temporary run memory survives the situations where it needs to * permanent uniqueness rules remain valid across future versions Distinguish clearly between: run-local memory and cross-run persistent memory. ## DUPLICATION AUDIT Do not define duplication only as equal text. Consider several levels: * exact identifier duplicate * normalized URL duplicate * content hash duplicate * near-copy duplicate * same underlying external event from different publishers * same extracted signal * same transformation mechanism * same causal/event structure * same decisive reasoning pattern * cosmetically different output with structurally identical logic Explain which levels can be checked deterministically and which require semantic judgment. Do not pretend semantic duplication can be perfectly solved. ## EXTERNAL FACT / RULE VALIDATION If an external fact, property, rule, or constraint affects the generated output, distinguish: "The external statement is true" from "The generated output uses that statement correctly." Audit whether the pipeline separately validates both. Consider: * outdated information * ambiguous wording * missing conditions * exceptions * incorrect units * timing assumptions * extrapolating beyond what the source supports * a model turning a probabilistic statement into an absolute statement ## EXTERNAL SERVICE SAFETY Assume an external AI routing/service layer may change behavior without notice. Audit: * requested model vs actual model used * unavailable pricing information * changed model identifiers * silent fallback * routing to an unintended resource * malformed usage metadata * timeout after remote execution succeeded * partial responses * provider-side retries * API schema changes If the system has a "zero-cost only" or similar hard resource constraint, treat it as a security-style invariant. The correct philosophy should be FAIL CLOSED: uncertainty must stop the run rather than silently violate the constraint. But examine whether that guarantee is actually enforceable with the metadata exposed by the service. If it cannot be absolutely guaranteed, say exactly where the capability boundary lies. ## AI OUTPUT VALIDATION Never assume that structured AI output is valid simply because it parses. Consider outputs that are: * syntactically valid but logically contradictory * missing required evidence * duplicated under different wording * internally inconsistent * based on invented facts * impossible to validate * maliciously shaped by source material * semantically empty while satisfying the schema * technically compliant but unusable downstream Recommend deterministic validators wherever possible. For semantic judgments, recommend independent checks, disagreement handling, or rejection policies rather than fake certainty. ## SOURCE-INPUT SECURITY Treat all discovered external content as untrusted data. Audit risks including: * prompt injection inside retrieved content * instructions embedded in web pages or feeds * malicious JSON/XML fields * huge documents * malformed encodings * redirect loops * unexpected MIME types * HTML pretending to be another format * poisoned source metadata External content must never become system instructions simply because an AI model reads it. ## PORTFOLIO-LEVEL AUDIT The system selects multiple final outputs per run. Therefore evaluate not only whether each candidate is good individually, but whether the selected SET is genuinely diverse. Look for cases where outputs differ in names, settings, surface topic, or vocabulary but share the same deeper mechanism or structure. Also consider the opposite failure: over-aggressive diversity rules that force the system to select weaker outputs merely to fill a quota. The system should prefer producing fewer valid outputs or retrying discovery rather than silently lowering quality. ## CRASH RECOVERY For every important state transition, consider a crash: BEFORE the operation DURING the operation AFTER the external side effect but BEFORE local acknowledgement AFTER local persistence but BEFORE the next stage Determine whether recovery produces: * duplication * lost work * corrupted history * incorrect reservations * permanently blocked items * inconsistent lineage ## CONFIGURATION AUDIT Assume configuration itself can be wrong. Check: * missing fields * unknown fields * contradictory options * unsafe defaults * version mismatches * empty allowlists * invalid thresholds * accidental fallback behavior * production accidentally using development configuration * secrets leaking into logs or generated artifacts The system should reject invalid configuration before beginning expensive work. ## OBSERVABILITY A production system must explain why it did what it did. Recommend enough structured logging to reconstruct: * what was discovered * what was reserved * what was rejected * why it was rejected * what produced each candidate * what validation passed or failed * what historical comparison occurred * why the final portfolio was selected * what was committed to persistent history But do not recommend storing unnecessary full source content if identifiers, hashes, provenance, and minimal evidence are sufficient. ## TESTING STANDARD For every important guarantee you identify, propose at least one adversarial test. Prefer failure-injection tests over happy-path tests. Examples of the style of test I want: * same material through two different URLs * same content with a tracking query parameter * same content appearing in two discovery batches * process killed after reservation * process killed immediately before commit * two concurrent runs attempting to consume the same item * history containing more records than an old implementation's comparison limit * AI response that is valid JSON but semantically contradictory * external service returning missing or contradictory metadata * network timeout after a remote request may already have succeeded * output with different names but the same abstract event chain * source content containing prompt-injection instructions Do not stop after giving these examples. Find additional failure cases yourself. ## REQUIRED CLASSIFICATION For every major invariant classify it as one of: A — DETERMINISTICALLY ENFORCEABLE B — ENFORCEABLE WITH EXTERNAL ASSUMPTIONS C — SEMANTIC / PROBABILISTIC D — CANNOT BE GUARANTEED WITH THE CURRENT ARCHITECTURE I care greatly about this classification. Do not call something "guaranteed" if it actually depends on an AI judgment or third-party behavior. ## RESPONSE FORMAT Return the audit in this order: ### 1. Executive Verdict PASS PASS WITH REQUIRED HARDENING or NOT PRODUCTION READY Explain why in a few paragraphs. ### 2. Hidden Assumptions Register Create a table: ID Subsystem Hidden assumption Failure scenario Severity Detectability Required protection Guarantee class ### 3. Critical Invariants Write the exact invariants the implementation should enforce. Use precise language such as: "An external item that has entered state X may never produce more than one accepted descendant across retries and concurrent executions." Avoid vague advice. ### 4. Failure-State Analysis Walk through major stages and identify crash, retry, concurrency, and partial-write behavior. ### 5. Deduplication & Lineage Audit Go deeply into identity, canonicalization, source lineage, candidate lineage, historical comparison, and structural duplication. ### 6. External-Service Boundary Audit Identify what can truly be verified and what remains dependent on third-party behavior. ### 7. Security / Untrusted-Input Audit Include prompt injection and malformed external data. ### 8. Adversarial Test Matrix For each invariant provide: Test Injected failure Expected system behavior What would constitute FAIL ### 9. Claims We Must NOT Make List guarantees that would be dishonest or technically impossible. ### 10. Production Gate Give a concrete checklist of conditions that must all be satisfied before deployment. ## IMPORTANT BEHAVIORAL INSTRUCTION Be skeptical. Do not reward the architecture for being sophisticated. Try to break it. Whenever you see a protection, ask how that protection itself can fail. Whenever you see persistent state, ask what happens under concurrency. Whenever you see a retry, ask whether the operation is idempotent. Whenever you see an AI judgment, distinguish it from deterministic proof. Whenever you see external metadata, ask whether it is authoritative enough to support the claimed guarantee. Whenever you see "duplicate detection," ask: duplicate according to which identity layer? Whenever you see "history," ask: which history, how much of it, and what happens after years of operation? Whenever the system claims an invariant, demand a test that would fail if the invariant were broken. Do not redesign the hidden product. Audit the reliability architecture surrounding it. The final objective is not merely for the system to work during a demonstration. The objective is for it to fail safely, detect uncertainty, preserve provenance, avoid silent duplication, survive retries and concurrency, and make only guarantees that the implementation can genuinely enforce.