**Planning Key Rotation** I'm currently strategizing the provider key rotation across three hosts, carefully mapping out the dependencies involved to ensure a smooth transition with minimal disruption. The focus is on a phased approach. **Analyzing Rotation Constraints** I'm currently dissecting the constraints for Host/Service A, B, and C. Host C's lack of a rollback path is a significant concern, requiring meticulous planning around its irreversible state change. The dependency of Service B on Service A's health post-cutover is also a key focus. **Investigating Key Overlap** I'm currently investigating the critical ambiguity regarding whether the upstream provider supports dual active keys concurrently. This will drastically alter the approach, potentially eliminating downtime for Service A and influencing the order for Service C. I'm also probing the specifics of Host C's irreversibility. **Clarifying Key Dependencies** I'm now probing the exact nature of the provider key usage: is it a single shared key for A, B, and C, or are they independently managed? Furthermore, I'm deeply analyzing the dependency topology between hosts A, B, and C, and refining the definition of 'downtime' for Service A, as these factors are critical for structuring the subsequent phases. **Validating Rotation Phases** I'm now validating the proposed three-phase rotation strategy. Phase one focuses on Host A's update within a strict downtime budget, including crucial health and soak checks. Phase two, dependent on phase one's success, will then address Host B, with clear rollback contingencies in place for both. **Finalizing Rotation Strategy** I'm now solidifying the phased rotation plan, with a particular focus on the irreversible nature of Host C's update and defining clear rollback boundaries. I'm also refining the presentation for clarity, including explicit timers and rollback conditions. --- ## 1. Critical Assumptions & Architectural Model To make this plan executable, we assume: 1. **Key Overlap**: The upstream provider supports two active concurrent keys ($K_{\text{old}}$ and $K_{\text{new}}$) during the migration window. 2. **Key Scope**: $A$, $B$, and $C$ share the same provider key (or synchronized key pair). 3. **Execution Order**: Derived from constraints $\rightarrow$ **Host A $\rightarrow$ Host B $\rightarrow$ Host C**. * Host A must go first to establish the 60-second health baseline for Host B. * Host C is placed last because it has **no rollback path**; we must not enter an irreversible state until all reversible components ($A$ and $B$) are stable on $K_{\text{new}}$. --- ## 2. Key Rotation Execution Plan ``` [Phase 0: Pre-Flight] ──> [Phase 1: Host A Cutover] ──> [Gate 1B: 60s Soak] │ [Phase 3: Host C (NO ROLLBACK)] <── [Gate 2: B Stable] <── [Phase 2: Host B Cutover] │ ▼ [Phase 4: Invalidate Old Key] ``` --- ### Phase 0: Pre-Flight & Provisioning (T - 15m) * **Step 0.1**: Generate $K_{\text{new}}$ in the provider control plane. **Do not disable $K_{\text{old}}$.** * **Step 0.2**: Stage $K_{\text{new}}$ on Hosts $A$, $B$, and $C$ in an inactive configuration file / secret store. * **Step 0.3**: Establish baseline telemetry: latency, error rates, and health checks across $A$, $B$, and $C$. * **GATE 0 (Pre-Flight Validation)**: * Validate $K_{\text{new}}$ works via out-of-band canary/API call. * Confirm rollback tooling and automated scripts for $A$ and $B$ are armed. * **Pass Condition**: $K_{\text{new}}$ valid, telemetry green, zero traffic disruption. --- ### Phase 1: Host A Cutover & Soak Window (Downtime Budget: $\le 300\text{s}$) * **Step 1.1**: Start **Downtime Timer** ($T_{\text{down}}$). * **Step 1.2**: Drain/stop Service A traffic, swap configuration to $K_{\text{new}}$, and restart/reload Service A. * **Step 1.3**: Validate Service A local health check. Stop $T_{\text{down}}$. * **GATE 1A (Downtime & Basic Health)**: * **Timer Check**: $T_{\text{down}} \le 180\text{s}$ (Internal target to protect the hard 300s SLA). * **Health Check**: Synthetic traffic against Service A on $K_{\text{new}}$ returns HTTP 200/OK. * **FAIL Trigger**: If $T_{\text{down}} > 180\text{s}$ or health check fails $\rightarrow$ **Trigger Rollback Boundary 1**. * **Step 1.4**: Start **Host B Prerequisite Soak Timer** ($T_{\text{soak}}$). * **GATE 1B (Service A Health Soak Gate)**: * Service A must maintain 100% healthy telemetry for **$T_{\text{soak}} \ge 60\text{s}$ continuous seconds**. * **FAIL Trigger**: If error rate spikes or health check drops during these 60s, reset $T_{\text{soak}}$ to 0. If cumulative time exceeds 240s without achieving a 60s continuous healthy window $\rightarrow$ **Trigger Rollback Boundary 1**. --- ### Phase 2: Host B Cutover * **Step 2.1**: Update Host B configuration to $K_{\text{new}}$ and reload/restart Service B. * **Step 2.2**: Validate Service B local health and upstream integration with Service A. * **GATE 2 (Host B Verification)**: * Service B health checks pass. * Dependency chain ($B \rightarrow A$) functions normally using $K_{\text{new}}$. * Service B error budget/metrics remain within baseline for 120s. * **FAIL Trigger**: If Service B fails health checks or connection to A fails $\rightarrow$ **Trigger Rollback Boundary 2**. --- ### Phase 3: Host C Cutover (POINT OF NO RETURN) * **CRITICAL GATE (Pre-C Approval)**: * Explicit manual or policy-enforced gate: 1. Gate 1A, Gate 1B, and Gate 2 are marked **PASSED**. 2. Host A and Host B are actively processing production traffic on $K_{\text{new}}$. * **Step 3.1**: Execute non-reversible cutover on Host C to $K_{\text{new}}$. * **GATE 3 (Host C Verification)**: * Verify Host C has successfully bound to $K_{\text{new}}$ and all data/state validation checks pass. * **FAIL Trigger**: If Host C fails $\rightarrow$ **Trigger Forward-Fix Strategy** (Rollback is impossible). --- ### Phase 4: Deprecation & Decommissioning (T + 24h) * **Step 4.1**: Verify no traffic is using $K_{\text{old}}$ via provider access logs. * **Step 4.2**: Revoke/delete $K_{\text{old}}$ at the provider control plane. --- ## 3. Rollback Boundaries & Recovery Matrix | Boundary | Scope | Trigger Condition | Execution Actions | | :--- | :--- | :--- | :--- | | **Rollback Boundary 1** *(Reversible)* | **Host A Only** | - $T_{\text{down}} > 180\text{s}$
- Gate 1A fails
- Gate 1B cannot maintain 60s stability | 1. Swap Host A config back to $K_{\text{old}}$.
2. Restart Service A.
3. Verify Service A healthy on $K_{\text{old}}$ before $T_{\text{down}} = 300\text{s}$.
4. Abort rotation. | | **Rollback Boundary 2** *(Reversible)* | **Host B & Host A** | - Gate 2 fails
- Integration between B and A fails | 1. Revert Host B to $K_{\text{old}}$ and restart.
2. Evaluate Host A: Revert Host A to $K_{\text{old}}$ if failure is shared, or keep A on $K_{\text{new}}$ if isolated.
3. Abort rotation. | | **POINT OF NO RETURN** | **Threshold** | **Commencing Phase 3 (Host C cutover)** | **All rollbacks to $K_{\text{old}}$ are officially revoked.** | | **Failure at Phase 3/Host C** *(Non-Reversible)* | **Host C Only** | - Gate 3 fails | **DO NOT ROLL BACK A OR B.**
1. Engage emergency forward-fix playbook for Host C.
2. Re-issue $K_{\text{new2}}$ if key corrupted, or repair local state on Host C.
3. If Host C is hard-down, failover Host C traffic to a standby/disaster-recovery node pre-seeded with $K_{\text{new}}$. | --- ## 4. Underdetermined Dependencies & Clarifying Questions The following areas are underdetermined and present operational risks: ### 1. Provider Key Concurrency (Dual-Key vs. Single-Key) * **Ambiguity**: Does the provider support multiple concurrently active keys, or does generating/activating $K_{\text{new}}$ instantly invalidate $K_{\text{old}}$? * **Question**: *Can $K_{\text{old}}$ and $K_{\text{new}}$ coexist and authenticate requests simultaneously for a defined transition period (e.g., 24–48 hours)?* ### 2. Nature of Host C's Irreversibility * **Ambiguity**: Why is C non-rollbackable? Does Host C perform a destructive state/schema mutation upon key application, or does Host C's rotation script explicitly revoke $K_{\text{old}}$ globally at the provider level? * **Question**: *What specifically causes Host C to have no rollback path? If Host C fails midway through cutover, does it leave the entire system in a broken state, or is only Host C degraded?* ### 3. Service A Downtime Definition * **Ambiguity**: Does "down max 300s" refer strictly to total process stop/restart time, or does it include the time to drain connections, reload, and re-warm caches? * **Question**: *What exact health check endpoint and metric defines Service A as "down" vs. "healthy" (e.g., HTTP 5xx rate, TCP handshake, synthetic ping)?* ### 4. Dependency Graph Between C and (A, B) * **Ambiguity**: Does Service A or Service B depend on Service C to function? If Host C fails permanently post-cutover, will it cascade and break A and B? * **Question**: *Are there upstream/downstream traffic dependencies between Host C and Hosts A/B, or is Host C a standalone consumer of the key?*

SCB SCB-planning_multi-T1 KIMI