Plan COmpare

Prompt

# Frozen History (Review Part 4 + Point-in-Time Audit) Implementation Plan > **Agentic workers:** use superpowers:subagent-driven-development (or executing-plans), task by task. **Goal:** Every as-of screen, report and export offers three views of a past date: **Restated** (default; all entries by effective date, as known today), **As visible on** (entries made by end of the as-on date or a chosen later date), **Closed period** (entries made by the moment the fund closed the period covering it). Posting always uses everything entered so far. **Architecture:** Bitemporal. Each fact has a business-effective date (receipt `value_date`, trade `settlement_date`, amendment `effective_date`, rate `effective_from`, snapshot `valuation_date`) and a recorded timestamp (`posted_on`, `confirmed_on`, `applied_on`, `approved_on`, `run_on`, `updated_on`; reversals/supersessions carry their own). Pure rules in `point_in_time.py` decide visibility; every reader takes one `cutoff` (datetime string, or `None` = everything). One resolver, no per-report filters. User endpoints turn `view` + `visible_on` into the cutoff; engine reads and writes keep `None`. **Specs:** `NCD_Review_UI_Finance_History_2026-09.md` Part 4 (G1–G10, R1–R6); `C:\Users\9025\Downloads\UTI_NCD_Point_in_Time_Audit.html` (F-01–F-09, T01–T15, migration §10, acceptance §11). ## Decisions 1. Restated is default (user's choice, 29 Sep 2026), so screens behave as today. Example: banked 30 Mar, entered 3 Apr, viewed for 31 Mar → Restated: received; As visible on: unpaid; Closed period: as entered when March closed. 2. `resolve_cutoff`: Restated → `None`. As visible on → end of `as_of`, or end of optional `visible_on` (earlier than `as_of` is refused). Closed period → moment the closed date last moved from before `as_of` to on/after it, if still closed (Task 9 log; latest close counts). Not closed → Restated, banner `"{fund}: {as_of} is not in a closed period, so current entries are shown."` Cutoff ≥ now → `None`. Series-only views use the earliest holder-fund close; multi-fund views resolve per fund. 3. User-facing reads (dashboards, drill-downs, schedules, reports, exports) get the toggle. Engine reads (`run_valuation`, previews, posting, close prep, checks) keep `None`; a write given a cutoff throws. 4. Recorded times are exact timestamps; a UI date becomes end of day. 5. Reversals: cutoff before reversal recorded → original counts; after, or Restated → both legs drop. A reversed amendment governs while it stood; Restated uses the prior version. 6. Snapshots get a Live/Superseded event log; the `superseded` flag (kept as cache) can't say when, and reinstatement has no new `run_on`. 7. Legacy marked, not guessed: patch sets `uti_ncd_history_complete_from`; earlier cutoffs get `legacy_reconstruction: true` + banner. Backfill: mirror timestamps → Version log → `modified`. 8. Missing historical terms are disclosed (`terms_unverified` in payload and banner; F-08/T14). 9. Payloads carry a `manifest` (schedule version, receipts, trades, amendments, expectations, rate approval, snapshot, fingerprint): "Audit inputs" drawer, stored on packs, used for T13. 10. Revaluation after back-dated entries is offered, not automatic; a manager supersedes Confirmed packs with a reason. 11. Master-data labels are not frozen (user's choice); T15 out of scope. 12. Expectations take effect the day set. Rate approvals have `effective_from`: back-datable, not into a closed period or future. 13. Past deal status from `Version` log; fund-position label from acquisition/exit dates; foreclosure-quote label current-only (P3). 14. Stale mark: input effective ≤ valuation date recorded after `run_on`. Badge in Restated only. 15. Each close/reopen logs old/new date, when, who, why. ## Global Constraints - Frappe 16.25.0, ERPNext 16.26.1, MariaDB 10.6, Redis 7 pinned in `docker/docker-compose.yml`; don't change. Python `unittest`, vanilla JS. - Pure modules (`calculation`, `servicing`, `trades`, `valuation`, `point_in_time`, `checks`, `exceptions`, `scenario`) never import `frappe`. Frappe code uses **tabs**. Strings via `frappe._()`; `frappe.throw` states the rule in a sentence. - Money via `calculation.money()` (Decimal, ROUND_HALF_UP, 2 dp); `DUST = 0.05` = settled. Boundary dates: ISO/`date` via `calculation.as_date()`. Cutoffs: `"YYYY-MM-DD HH:MM:SS"`. - No history comments. No new permission hooks (only child tables + a report). After JSON: `bench --site uti.local migrate`. **Leave work uncommitted.** - Tests: edit existing first; pure while building; only touched bench modules; no `--app`, no `bench build`; UI at http://localhost:8000 (hard refresh). - Pure (from `uti_ncd/`): `python -m unittest -q <module> 2>&1 | grep -E "^(FAIL|ERROR):|^Ran |^OK|^FAILED"` - Bench (repo root): `docker compose -f docker/docker-compose.yml exec bench bench --site uti.local run-tests --module <module>` - Known failures: close-pack suites fail when a Confirmed pack exists in the dev DB; `test_nav_tieout` never passes. ## Files - Pure: `servicing.py`, `point_in_time.py`, `valuation.py`, `checks.py`. - Adapters: `*_ops.py` (point_in_time, servicing, valuation, trades, cio, close, checks, overdue), `operations.py`, `exports.py`. - New: child doctypes `ncd_valuation_rate_approval`, `ncd_valuation_snapshot_event`, `ncd_closed_period_change`; report `ncd_restatement`; patch `patches/v1_0/backfill_history_timestamps.py` (in `[post_model_sync]` of `patches.txt`). - Changed: series, receipt, trade, amendment, snapshot, close pack, expectation, closed period, holder doctypes; 13 reports; `ncd_common.js`, `ncd_series_operations.js`, dashboard JS. **Build order:** 0→17. Tasks 1, 4, 5, 6 are independent. 2 and 3 precede 7, 8, 11+. 11 precedes 12–16. Each task: failing test → implement → PASS. ## Tasks ### 0 Baseline - [ ] Run pure suite (`python -m unittest discover -q -s uti_ncd/tests/engine -t .`), note failures; `git status --short`, don't stash. ### 1 R3 penal interest bound to the as-on date (G3, G10) - [ ] `servicing.penal_charged(receipts, period_number, as_of) -> float`: sum `default_interest_amount` where `period_anchor == period` and `value_date <= as_of`, money-rounded. - [ ] `_penal_charged`, `expectation_amounts`, `apply_saved_expectations` gain `as_of`/`receipts=None` (raw SQL removed). Suggestion uses `value_date`; `overdue_ops` passes today. - [ ] G10: `inputs_json.receipts` counts receipts with `value_date <= valuation_date`. - [ ] Helpers `insert_posted_receipt`/`delete_receipt` in `seed_demo_deal.py` (pattern: `test_as_at_reproducibility.py:67-81`). - [ ] Tests: pure (400 at 31 Mar, 1300 at 20 Apr, 0 before); bench: later penal doesn't reduce an earlier date. ### 2 Reversal timestamps, pack cutoff, legacy boundary (G8; F-02/03/04; §10) - [ ] `reversed_on` on receipt, trade, amendment (stamped in `mark_reversed`, `reverse_amendment`); `knowledge_cutoff`, `input_manifest` on close pack. - [ ] Recorded-time sources: receipt `posted_on`→`creation`; trade `confirmed_on`→`creation`; amendment `applied_on`; reversals `reversed_on`. - [ ] Idempotent patch (nulls only): receipt/trade from mirror timestamp; amendment from Version log else `modified`; pack cutoff = `prepared_on`; set `uti_ncd_history_complete_from`. Tasks 3–5, 9 add blocks. - [ ] Tests: `reversed_on` matches mirror; no nulls remain. ### 3 Snapshot events (F-05, T08) - [ ] Child `NCD Valuation Snapshot Event` (`event`, `event_on`, `caused_by`, `legacy`) as table `live_events`. - [ ] `run_valuation` logs every flag change: new/reinstated → Live; displaced → Superseded. - [ ] Patch: Live at `run_on`; Superseded at next same-key `run_on`, `legacy=1`. - [ ] Test: A, B, A again → A: Live, Superseded, Live; B: Live, Superseded. ### 4 R2 dated rate approvals (G1, F-06/T09; G7) - [ ] Child `NCD Valuation Rate Approval` (`action` Approve/Clear, `rate_pct`, `effective_from`, `approved_on`, `approved_by`, `reason`); series table `valuation_rate_approvals`, append-only. - [ ] `approved_rate_on(approvals, date, cutoff=None)`: latest `(effective_from, approved_on)` effective ≤ date (approved ≤ cutoff); None for Clear or rate ≤ 0. - [ ] `_automatic_rate(..., cutoff=None)`. Approve/clear take `effective_from` (default today; ≤ today, ≥ issue, not closed) and `reason`, append a row, refresh the four legacy fields. Approved Rate with none in force throws. - [ ] Patch seeds a row from legacy fields; JS dialog adds Effective From, Reason. - [ ] Tests: pure (in force; back-dated hidden under earlier cutoff; same date later wins); bench: no repricing of earlier marks, future refused; run `test_close_blockers`. ### 5 R2 expectation history (G2, F-07/T10; G7) - [ ] `NCD Overdue Expectation` append-only: add `cleared`; drop one-per-period check; edits/deletes throw. Saves and clears insert records. - [ ] `point_in_time.latest_by_period(records, on, cutoff=None)`: per period latest `updated_on` ≤ `on` (and ≤ cutoff); drop if cleared. `saved_expectations(series, on=None, cutoff=None)`. Patch: `updated_on = modified` where null. - [ ] Tests: later change doesn't move earlier view; edit refused. Browser: save/change/clear → three records. ### 6 R2 terms and tranches in force (G5, G6) - [ ] Pure `invested_by(terms, on)` (tranches drawn by `on`, else principal once issued); `tranche_flows(terms)`; `_holder_cost_yields` uses in-force terms. Callers: `run_valuation`, `_fund_slice`, `exports.py`. - [ ] Bench: `test_point_in_time_ops`, `test_run_valuation_all`, `integration.test_exports`. ### 7 R4 marks restated since struck (G4) - [ ] `restating_entries(valuation_date, run_on, entries)`: effective ≤ date, entered > `run_on`. `input_entries(series)` builds `{kind, name, effective_date, entered_on}` for receipts, trades, amendments (and reversals), rates, expectations. - [ ] Check `mark_restated` (after `valuation_age`) lists the entries; dashboards show the badge. - [ ] Tests: receipt posted after mark's `run_on` → warn naming it. ### 8 R4 alert and one-click revaluation - [ ] `affected_marks(series, date)` → live snapshots and Confirmed packs dated ≥ it; returned as `restates` by every post, reverse, confirm, apply and rate action. - [ ] Whitelisted `revalue_marks(series, dates)` (NCD User/Manager): one valuation per date in a savepoint → `{revalued, skipped}`. - [ ] JS `restatement_prompt`: Revalue / Later. ### 9 R5 closed-period log; holder lock (G7) - [ ] Child `NCD Closed Period Change` (previous/new `closed_until`, `changed_on`, `changed_by`, `reason`, `legacy`) as `changes`. - [ ] A move needs a reason; reopening needs a manager; each move appends an immutable row. - [ ] Pure `closed_cutoff(changes, as_of)`: `changed_on` of the last move from before `as_of` to on/after it; None if now before or never covered. Patch rebuilds rows from Version log (`legacy=1`). - [ ] Holders with positions can't change type/fund. - [ ] Tests: reason, manager, log; close moment, reclose, reopen → None. ### 10 R6 status on a date (G9) - [ ] Pure `status_on(changes, as_of, current)`; `status_history`/`status_as_of` from Version; used in living schedule and funds list. ### 11 R1 pure bitemporal rules - [ ] `cutoff_at`; `resolve_cutoff` → `{view, cutoff, source}` (`current`/`end of day`/`chosen date`/`period closed`/`period not closed`); bad input raises `PointInTimeError`. - [ ] `known_at(entered_on, reversed_on, cutoff)`; `amendments_as_known`; `version_known_on` (replay ≤ cutoff; missing `applied_on` raises); `snapshot_live_at`; `pick_snapshot`. - [ ] Pure tests cover T01–T03, T06–T08. ### 12 R1 thread `cutoff` through readers + manifest - [ ] `cutoff=None` keeps today's path. With a cutoff: `resolve_schedule` uses `version_known_on`/`amendments_as_known`; `_sanctioned_rows` throws with `for_write` ("Money can only be posted against what is known now."); `_posted_receipts`, `_series_context`, `cio_ops.fund_shares` (F-03) filter via `known_at`; fund membership keeps positions acquired by `as_of` with units > 0 from known trades (T04/T05); `_latest_snapshots` uses `pick_snapshot`; views use `approved_rate_on(..., cutoff)`. `run_valuation` never takes a cutoff. Living-schedule payload adds `cutoff`, `terms_unverified`, `manifest`. - [ ] `view_cutoff(as_of, view, visible_on, fund_company, ncd_series)` wraps `resolve_cutoff` (Closed period via `closed_cutoff`), adds `fund` and `legacy_reconstruction`. `history_manifest(series, as_of, cutoff)` returns the Decision 9 fields plus a fingerprint. - [ ] Tests: T01–T03, T13 (`test_as_at_reproducibility`), T06, T04/T05 (`test_trade_lifecycle`). Thread layer by layer; `test_book_parity` proves `None` unchanged. ### 13 R1 view toggle, banner, audit drawer - [ ] `get_phase7_dashboard`, `get_fund_series_detail`, `get_book_dashboard` (per fund), new `get_schedule_view` take `view`/`visible_on`; payload adds `history`, `manifest`. Cards stay Restated; posting dialogs untouched. - [ ] JS: view/visible_on filters, `history_banner`, `audit_inputs_drawer`. - [ ] Test the user's example via `_closed_at` (close after entry = Restated; before = As visible on). ### 14 Close packs record cutoff and inputs (F-09, T11) - [ ] `prepare_close_pack` fixes cutoff = now, stores it and `input_manifest`, passes it to `_sections`. - [ ] Test: pack-date view reproduces pack figures and fingerprints after a back-dated receipt. ### 15 Reports and exports share the cutoff (T12) - [ ] 11 as-of reports (arrears, concentration, watchlist, fund cashflow, income, maturity ladder, NAV bridge, roll-forward, schedule cashflow, yield, movements) add view filters, resolve `view_cutoff` per fund, pass it to every reader, show the banner. Registers list entries recorded by the cutoff; schedule-changes report unchanged. Exports take view/visible_on; Checks sheet states them. - [ ] Test: dashboard, ladder, roll-forward agree per view. ### 16 Restatement report - [ ] Pure `restatement_lines(then, now, causes)`: outstanding/overdue/received/fair_value changes > DUST, with causes. - [ ] Report `ncd_restatement`: chosen view vs Restated, causes = entries after the cutoff. ### 17 Final verification - [ ] Pure suite; touched bench modules singly; restore workspace JSON if deleted. - [ ] Check §11 on screen: frozen views stable; later reversals don't erase them; Restated labelled; provenance shown; closed month reconciles to pack. Record checks; leave uncommitted. ## Coverage **Audit:** F-01/T01–02, F-02/T03, F-03/T04–05, F-04/T06–07 → 2, 11, 12 · F-05/T08 → 3, 11, 12 · F-06/T09 → 4 · F-07/T10 → 5 · F-08/T14 → 12, 13 · F-09/T11 → 14 · T12 → 12, 13, 15 · T13 → 12 · T15 out of scope · §10 → 2, 3 · §11 → 13, 14. **Review Part 4:** G1 → 4 · G2 → 5 · G3 → 1 · G4 → 7, 8 · G5/G6 → 6 · G7 → 4, 5, 9 · G8 → 2, 11, 12 · G9 → 10 · G10 → 1 · R1 → 2, 3, 11–16 · Review tests 1–5 → 12, 4, 5, 1, 7.