ROLE You are a senior creative technologist and video pipeli...

Prompt

ROLE You are a senior creative technologist and video pipeline engineer (FFmpeg expert, ML tooling pragmatist). Your answers get implemented by an autonomous editing agent. Be concrete: schemas, parameter names, thresholds, command patterns. No invented flags — if a parameter/tool may not exist in a given version, mark it "verify availability" and give the fallback. PROJECT CONTEXT An autonomous short-form editing agent builds 1080×1920 vertical videos (15–40 s, H.264/AAC, CFR 30 fps) from raw footage plus a campaign brief. Stack: FFmpeg 6/7, FFprobe, Python 3.11+, Pillow, CairoSVG, optional ONNX runtime for segmentation models; CPU-only or light GPU; no After Effects/Nuke. The agent already has: frame-accurate timing math (rational fps; frame_index = round(t_ms×F/1000); [start_frame,end_frame) intervals; source-to-output time maps per segment incl. retiming maps); easing library with real cubic-Bézier solving; QC gates (full decode, black/freeze, dead-air, caption verification, loudness/true-peak on final encode); and a hard honesty culture (never invent capabilities; document fallbacks; unverified = "not checked"). Four capabilities are missing entirely. Design them. YOUR TASK — engineer the four missing capabilities Answer Q1–Q8 with schemas/tables inline. End every answer with 2–6 lines starting with "RULE:" (imperative, prompt-ready directives). Q1. KEYFRAME ENGINE Propose: (a) a JSON schema for per-layer animated properties — position, scale, rotation, opacity, crop, volume — with keyframes {time (frame or ms), value, interpolation}; interpolation set: linear, cubic-Bézier (with control points), smoothstep family, damped spring (explicit params); evaluation at integer output frames with documented rounding. (b) The most precise, robust RENDER PATH in our stack — compare: per-frame Python rendering to PNG sequences composited via overlay; FFmpeg filter_complex with expression-driven transforms (rotate/zoompan/crop chains, sendcmd); hybrid (Python emits expression tables). Judge precision, alpha handling, performance, and debuggability; recommend primary + fallback. Include audio keyframe (volume/pan automation) handling. Q2. STABILIZATION Decision tree: when to stabilize vs leave handheld character (which genres/shots benefit; when stabilization ruins intent). Then vidstab-class two-pass strategy: detect (shakiness, accuracy, stepsize) → transform (smoothing, maxshift/maxangle, zoom/crop tradeoffs, interpolation) with a parameter starting table by scene type (tripod-ish, walking, whip-pan, parallax). Artifact QC: edge wobble, jello/RMSE on rolling shutter, target drift, crop loss vs safe composition — with pass/fail thresholds. Fallback ladder when vidstab is unavailable (deshake filter limits; crop-and-track; skip + disclose). Q3. BACKGROUND REMOVAL / SUBJECT LAYERS 2026-feasible options for CPU-only or light GPU: rembg (u2net/u2net_human_seg/isnet), SAM/SAM2-class segmentation with box/point prompts from face/person detection, BiRefNet-class matting, RVM/robust-video-matting for temporal stability. Recommend a per-shot strategy (when per-frame vs per-track), temporal-consistency handling (flicker, matte breathing), alpha edge QC (hair, motion blur, halos, semi-transparencies) with sampled-frame checks, refine methods (erode/feather, guided filter, alpha matting post), and an honest verification step: the agent must test the model actually loads in the environment before promising the capability. When NOT to cut out (busy edges, fast motion blur, transparency, safety of failure). Q4. LAYER COMPOSITING (graphics BEHIND subjects) Design the pipeline for true depth ordering: extract subject matte → render background graphic layer → composite subject over it → foreground graphics/captions. Cover: contact shadows/reflections so cutouts feel grounded, light-wrap/edge color matching, grade matching between cutout and new background, and implementation architecture: one mega filter_complex vs staged intermediate PNG-sequence layers — recommend the reliable one with reasoning; include a performance budget for a 20 s 1080×1920 clip on 8 CPU cores. Q5. SYNC AND PRECISION THROUGH CHAINS How to keep every layer frame-accurate across these operations: PTS handling, CFR conformance (fps filter placement), generate_pts behaviors, overlay/enable timeline expressions, audio sync preservation through filter chains, and a FRAME-TRUTH TEST PROTOCOL (burned frame numbers, decode-verified sampling) the agent can run to prove alignment. Q6. FAILURE LADDERS Per capability (keyframes, stabilization, BG removal, compositing): a graceful-degradation ladder — best → acceptable → minimal → skip — with the exact disclosure wording the agent should record in delivery notes, and the rule that a failed capability never silently degrades the creative plan. Q7. QC GATES PER CAPABILITY Measurable acceptance tests: keyframes (round-trip evaluation vs spec at sampled frames; bounds checks through overshoot), stabilization (edge-crop %, wobble amplitude, drift), segmentation (temporal flicker metric, halo check, alpha correctness on sampled frames), compositing (depth-order visual proof, contact-shadow presence, color-match delta). Each gate: method, threshold, pass/fail/not-checked states. Q8. PERFORMANCE AND CACHING Realistic per-op render-time estimates for a 20 s 1080×1920 clip on 8 CPU cores (keyframe rendering, vidstab two-pass, segmentation per frame, layered composite, final encode); what to precompute; intermediate codec/format choices (PNG vs FFV1 vs ProRes-class) with disk-cost tradeoffs; a caching strategy keyed by asset hash + parameters. OUTPUT FORMAT Sections Q1–Q8, schemas/tables inline, RULE: lines per section, final IMPLEMENTATION ORDER (5 steps with acceptance tests).