The user wants a migration plan for a container ct-214 between hosts, with rootfs on ZFS, no shared storage, downtime window of 90 seconds. A snapshot has already been replicated; the final delta is 1.8 GiB at 60 MiB/s transfer rate. They want a plan with gates and rollback. Let me think through the technical details: **Key constraints:** - Container (sounds like LXC, given "ct-214" naming — Proxmox-style container naming, or just generic container ID) - Rootfs on ZFS (both hosts presumably, since no shared storage) - No shared storage → must use ZFS send/receive for data transfer - Downtime window: 90 seconds - Snapshot already replicated (initial full send done) - Final delta: 1.8 GiB at 60 MiB/s **Critical math:** - 1.8 GiB = 1.8 × 1024 = 1843.2 MiB - At 60 MiB/s: 1843.2 / 60 = 30.72 seconds ≈ 31 seconds - So the final delta transfer takes ~31 seconds of the 90-second window - That leaves ~59 seconds for: stop container, final sync, import/rename on target, start container, verify Wait — important consideration: while the container is running, the delta keeps growing. The classic approach: 1. Pre-sync while running: send incremental snapshots while container is up (possibly multiple rounds to converge) 2. Stop container (downtime starts) 3. Final incremental send (small, since we pre-synced) 4. Start on target (downtime ends) But the user says "final delta 1.8 GiB at 60 MiB/s" — this might mean the delta between the replicated snapshot and current state is 1.8 GiB, and they want to know if it fits. 1.8 GiB at 60 MiB/s = ~31 s. That fits within 90 s if we stop the container first then send. But better practice: do a pre-sync while running to shrink the final delta, making the stop-start window mostly container stop/start time. Actually, let me consider both approaches: **Option A: Stop-then-send (simple)** - Downtime = stop time + 31 s transfer + zfs receive/import + start time + verification - Stop: ~5-15 s for graceful container stop - Transfer: ~31 s - Receive is concurrent with send (zfs send | ssh zfs receive), so no extra time - Start: ~5-10 s - Total: ~45-60 s — fits in 90 s with margin **Option B: Pre-sync while running (better)** - Take snapshot B while running, send incremental (the 1.8 GiB) while container still up — no downtime cost - During that ~31 s, new writes accumulate (say write rate w MiB/s → delta2 = w × 31 s + fs flush) - Stop container, send final tiny delta (seconds), start on target - Downtime = stop + tiny send + start ≈ 15-30 s Option B is clearly better and is the standard "poor man's live migration" approach. With the given numbers, Option B gives huge margin. Even if the write rate is high, one or two pre-sync rounds converge. But there's a subtlety: the user says "final delta 1.8 GiB at 60 MiB/s" — they may have already computed that sending the delta takes ~31 s and want to know it fits. I should present the plan that uses pre-sync to shrink the in-window transfer, but also show that even a naive stop-then-send fits (with less margin). The gate structure: proceed only if estimated in-window time < threshold (e.g., 60 s, leaving 30 s buffer). **ZFS specifics:** - Initial replication: `zfs send tank/lxc/ct-214@snap0 | ssh target zfs receive tank/lxc/ct-214` — already done per the prompt - Incremental: `zfs send -i @snap0 tank/lxc/ct-214@snap1 | ssh target zfs receive ...` - Use `-w` (raw) if encrypted; `-c` for compressed; `mbuffer` or `pv` for rate monitoring - Resume tokens: `zfs send -t` if interrupted - On target: filesystem is received read-only? No — received filesystems are not automatically read-only unless you set `readonly=on` (common for replication targets). Important: if the target dataset has `readonly=on`, you must `zfs set readonly=off` before starting the container. Actually for replication targets people often set readonly=on to prevent divergent writes. If someone wrote to the target replica, the incremental receive will fail ("destination has been modified") — that's a gate check: `zfs get -H -o value written` on target or check `zfs receive` dry-run... Actually you can check with `zfs get written` or try receive and it fails. Better gate: verify target dataset's `written` property is 0 (or check `origin`/`clones`) before cutover. - Container config: for LXC, config lives outside rootfs (e.g., /etc/pve/lxc/214.conf on Proxmox, or /var/lib/lxc/ct-214/config). Must copy config separately — it's tiny, copy early. Also apparmor profiles, network config (MAC address, IP — if same L2, no change; if different subnet, update config and possibly DNS). Proxmox: `pct migrate` does this automatically, but the user seems to be doing it manually (or wants the plan). I'll keep it generic-LXC with Proxmox notes where relevant since "ct-214" strongly suggests Proxmox VE (VMIDs are numeric like 214, and "ct" is the Proxmox term). I'll mention `pct` commands as an option but give the manual ZFS path since they say "snapshot already replicated" implying manual control. Actually, Proxmox `pct migrate` without shared storage does exactly this: zfs send/recv incrementals. But the user is asking for a plan with gates and rollback — a runbook. I'll write it tool-agnostic with concrete commands. **Network considerations:** - Same L2 segment: keep IP/MAC, just update ARP (send gratuitous ARP after start). Move = start on target, stop on source. Possible brief duplicate-IP risk if both run — never run both simultaneously. - Different subnet: need IP change, DNS update, or tunnel — increases downtime; flag as gate. - MAC address: keep same MAC so DHCP/ARP caches stay valid; send gratuitous ARP on start (e.g., `arping -U` or `arp_send` in container config). **Consistency:** - Container stop flushes filesystem; app-level consistency: databases inside container get clean shutdown if stop is graceful. If the app has its own replication or needs quiescing, gate on it. - Filesystem-level: ZFS snapshots are crash-consistent; after graceful stop, clean. **Rollback plan:** - Don't destroy anything on source until target verified and a soak period passes. - Keep source dataset + snapshots; container config backed up; container set to not autostart. - Rollback = start container back on source (its data is untouched as of stop time). Any writes made on target after cutover would be lost on rollback — unless we reverse-replicate the delta back. Mention this explicitly: rollback is clean only if no new data written, or you send the reverse delta back first. - Rollback decision gates: if target fails to start or health checks fail within X minutes, roll back. **Timeline plan with gates:** Phase 0 — Preflight (T-1h or earlier): - Verify target: ZFS pool free space ≥ rootfs size + headroom; `zpool status` healthy; same/compatible ZFS feature flags (`zpool get feature@...` / send|recv compatibility), ZFS versions. - Verify snapshot exists on both sides: `zfs list -t snapshot` — common snapshot (the replicated one) present on both, and it's the latest common one. - Verify target dataset hasn't diverged: `zfs get -H -o value written tank/...` on target = 0 since receive (or no snapshots/clones created). - Copy container config to target (but container in "disabled/migrated" state so it can't autostart). On Proxmox, config move happens via cluster; standalone: copy /etc/pve/lxc/214.conf — actually on standalone Proxmox that's a pmxcfs path; just copy file. For vanilla LXC: /var/lib/lxc/ct-214/config + fstab + apparmor. - Network: confirm bridge names/VLANs on target match config (bridge name mismatch is a classic failure — e.g., vmbr0 exists on both). - Measure actual throughput: the 60 MiB/s — confirm with current link (`iperf` or observe the pre-sync rate). Gate: measured rate ≥ 50 MiB/s (else reschedule or shrink delta). - Baseline write rate of container to estimate delta growth: `zfs get written` twice over interval, or `zpool iostat`. This determines whether pre-sync rounds converge. - Backups: verify recent backup exists (vzdump or zfs snapshot offsite) — rollback of last resort. - Monitoring/alerting silence windows, notify stakeholders. Gate 0: all preflight checks pass → schedule. Phase 1 — Pre-sync (T-15 min, container running): - Create snapshot @mig1 on source. - `zfs send -i @snap0 @mig1 | mbuffer | ssh target zfs receive -s` (use -s for resumable; or without -s but with resume token handling). - Time it: expect ~31 s for 1.8 GiB at 60 MiB/s. While this runs, new writes accumulate: delta2 ≈ write_rate × 31 s. - Optionally create @mig2 and send again if write rate is high; converge until estimated final delta send < 10 s. Gate 1: incremental completes successfully; estimate of final delta (current `written` since @mig1) × (1/60 MiB/s) + stop time + start time < 60 s (leaving 30 s buffer of the 90). Also confirm no receive errors, target still not diverged. Phase 2 — Cutover (downtime window, T-0): - t=0: Announce/enter window. Stop container gracefully: `pct shutdown 214` (or lxc-stop). Timeout e.g. 30 s then force-stop (lxc-stop -k). Time: ~5-15 s. - Create final snapshot @mig-final. Send incremental from @mig1 → @mig-final. Should be tiny (just writes during stop + since last presync). Expect seconds. - On target: `zfs set readonly=off` (if set), ensure mountpoint, `zfs mount` if needed. - Start container on target: `pct start 214` / lxc-start. - Gratuitous ARP if same L2. - t≈+30-45 s: container up. Gate 2 (in-window go/no-go checkpoint): If final send hasn't completed by t=+45 s, abort → rollback: start container on source (data unchanged there — the source dataset is intact; we only took snapshots). Actually careful: if we stopped the source container and the send is slow, rollback = just start source again. Clean, no data loss, downtime extends but service restored. Phase 3 — Verify (T+0 to T+15 min): - Container running, processes up, app health checks (HTTP 200, DB accepts connections, etc.). - Network reachability: ping gateway, external VIP, DNS resolves. - Data sanity: app-level checks, recent records present. - Performance smoke test. Gate 3: health checks pass within 5 min → declare success. Else rollback. Phase 4 — Commit / cleanup (after soak, e.g., 24h): - Set source container to not autostart; rename/keep dataset as cold copy for N days. - Update inventory/CMDB, monitoring targets, backup jobs to point at new host. - Eventually destroy source dataset + snapshots after retention. Rollback specifics: - R1 (before/during final send, nothing started on target): start on source. Data loss: none. Time to restore: ~10-20 s. - R2 (target started but unhealthy, no significant new writes): stop target, start source. Data loss: any writes since cutover (small window). - R3 (target ran for a while and took writes, then must roll back): reverse incremental send from target back to source, then start source. Adds minutes; hence the soak-period decision point should be early. - Also: never destroy the source snapshots until commit; keep @mig-final as the common snapshot enabling reverse send. **Numbers recap:** - Delta: 1.8 GiB = 1843 MiB; at 60 MiB/s → 30.7 s transfer (plus ssh/mbuffer overhead, call it 35-40 s worst case). - If done as pre-sync while running: in-window transfer is only the residual, likely < 100-300 MiB → 2-6 s. - Downtime budget: stop 10-15 s + final send 5-10 s + target prep 2-5 s + start 5-15 s + ARP/verify 5 s ≈ 30-50 s. Fits 90 s with ~40 s buffer. - Worst case without pre-sync: 15 + 35 + 15 + 5 = 70 s — still fits but thin. Recommend pre-sync. Also mention throughput caveat: 60 MiB/s ≈ 480 Mbps — suggests either a 1 Gbps link with overhead or throttled. If the link is 1 GbE, fine. Compression (`zfs send -c` if datasets have compressible data with compression=on, send sends compressed blocks) can reduce bytes on wire. If the dataset uses encryption, use raw send `-w` which also preserves compression. mbuffer on both ends smooths throughput and gives rate display. Resume tokens: use `zfs receive -s` so an interrupted receive can resume with `zfs send -t ` — good for the big pre-sync; for the tiny final send it doesn't matter much. Feature flags: ensure target pool supports the source's send features (e.g., large_blocks, compressed recordsize, encryption). Check `zpool get all | grep feature` or just that both run same ZFS version. A failed receive in-window would be a rollback trigger; better to validate during pre-sync (which itself proves compatibility — the fact that the initial snapshot already replicated proves baseline compatibility; incremental uses same features). One more subtlety: holds. Put a ZFS hold on the migration snapshots so cleanup scripts don't prune them mid-migration: `zfs hold migration tank/...@mig1`. Also: `written` property to estimate delta: `zfs get -H -o value written pool/ds@mig1` gives bytes written since that snapshot — that's the final delta size estimate. Locks/fencing: ensure the container cannot autostart on source (disable onboot, stop HA if Proxmox HA managed — take it out of HA or set to disabled state, else HA might restart it on source during migration → split brain / duplicate IP). Important gate. Also app-level quiesce: if ct-214 runs a DB, consider app-level flush before stop; the graceful stop should handle it. DNS/TTL: if IP changes, lower TTL beforehand; but with same L2 not needed. Format: This is a technical runbook request from someone clearly fluent (terse prompt, jargon). They want a plan with gates + rollback. Format should be structured: brief feasibility math up front, then phased plan with commands, gates between phases, rollback matrix, timeline table maybe. Markdown with headers, code blocks for commands, a compact table for the downtime budget and/or rollback scenarios. Should be reasonably detailed but not bloated. I'll aim for a thorough but tight runbook. Use headers, numbered steps, code blocks, tables where they help (downtime budget, rollback scenarios). Keep the math explicit. Let me draft the timeline with the numbers: Feasibility math: - 1.8 GiB ÷ 60 MiB/s = 30.7 s - Plan moves that transfer outside the downtime window via pre-sync, leaving only residual delta in-window. Downtime budget table (in-window): | Step | Est. | |---|---| | Graceful stop (30 s timeout, expect 5-15 s) | 10 s | | Final snapshot + incremental send (residual) | 5-10 s | | Target: readonly=off, mount check | 2 s | | Start container | 5-15 s | | Gratuitous ARP + first health check | 5 s | | **Total** | **~30-40 s** | | Buffer in 90 s window | ~50 s | Gates: - G0 preflight - G1 pre-sync converged & in-window estimate < 45 s - G2 in-window abort checkpoint at t+45 s - G3 post-start health verification - G4 soak/commit Rollback matrix: | Trigger | Point | Action | Data impact | RTO | |---|---|---|---|---| | Pre-sync fails/too slow | before window | reschedule | none | n/a | | Final send stalls > 45 s | in window | start on source | none | ~20 s | | Target won't start / health fail | <5 min after start | stop target, start source | none (no new writes committed... well, any writes on target discarded) | ~1 min | | Late failure after writes | within soak | reverse send delta, start source | none if reverse send completes | ~10-30 min | Assumptions to state: same L2 (IP unchanged), config copied, target dataset not diverged, both pools same ZFS version family. Note that initial replication already proves send/recv compatibility. Commands: Source pre-sync: ``` snap() { zfs snapshot tank/ct214@mig$1; } zfs hold mig tank/ct214@mig1 zfs send -c -i @snap0 tank/ct214@mig1 | mbuffer -q -s 128k -m 256M | ssh tgt 'mbuffer -q -m 256M | zfs receive -s -F tank/ct214' ``` Hmm, `-F` on receive forces rollback of target to most recent snapshot — needed if target dataset got mounted/modified... but we want to detect divergence, not clobber it. Use `-F` carefully. For the pre-sync, don't use -F; if receive fails due to modification, that's a gate failure to investigate. Actually `-F` is needed if the target received fs was mounted (atime updates count as modification? mount changes can dirty it). Common practice in replication scripts is to use -F. But for safety in migration, detecting divergence matters. I'll note it: if receive fails with "destination has been modified," do NOT blindly -F — investigate; -F discards target-side changes which is fine only if you're sure nothing wrote there. Delta estimate: ``` zfs get -H -o value written tank/ct214@mig1 # bytes since mig1 = final send size ``` Final: ``` pct shutdown 214 --timeout 30 # or lxc-stop -n ct-214 -t 30 zfs snapshot tank/ct214@migfinal zfs send -c -i @mig1 tank/ct214@migfinal | ssh tgt zfs receive tank/ct214 ssh tgt 'zfs set readonly=off tank/ct214' # if it was set # config already in place; then on target: pct start 214 # or lxc-start -n ct-214 ``` Wait — dataset naming: Proxmox stores CT rootfs as `pool/subvol-214-disk-0` typically (or `pool/vm-214-disk-0`... for containers it's `subvol-214-disk-0`). I'll use a generic `rpool/data/subvol-214-disk-0` or just say `tank/ct-214`. Generic is fine; maybe note Proxmox path. I'll use `tank/ct-214` for readability and note to substitute the real dataset (e.g., `rpool/data/subvol-214-disk-0` on PVE). Also mention: set `onboot=0` / disable HA on source before window; on Proxmox if HA-managed, `ha-manager set ct:214 --state disabled` first, migrate, then re-enable on target. Also locks: `pct migrate` would handle locks; manual: create lock file / set migration lock to prevent accidental start on both ends. At minimum: config on source renamed or `onboot` off; never both running. Health checks: specific to app — I'll use placeholders (systemctl status inside, curl endpoint, pg_isready, etc.). Also consider: app config/bind mounts, `mp0` entries — those are separate datasets that must also be replicated. Gate: enumerate all mount points/volumes of the container, replicate each. Easy to forget. I'll include in preflight. Timezone/time sync, no. Keep focused. Length: This deserves a fairly complete runbook but I should keep it scannable. Headers + tables + code blocks. Aim for maybe 600-900 words equivalent with code. The user is expert; skip explaining what zfs send is. Use terse runbook style. Let me also double check the arithmetic: 1.8 GiB = 1.8 × 1024 MiB = 1843.2 MiB. 1843.2 / 60 = 30.72 s. Yes, ~31 s. If they meant 1.8 GB decimal = 1800 MB... at 60 MiB/s using MiB: 1800e6 / (60×1048576) = 28.6 s. Either way ~30 s. Fine. Residual delta after pre-sync: depends on write rate. If container writes 5 MiB/s during the 31 s pre-sync, residual ≈ 155 MiB → ~2.6 s send. Converges fast. One pre-sync round suffices unless write rate approaches link speed (then it never converges — gate on it: if write rate > ~50% of link, do two rounds or schedule quieter time). In-window abort checkpoint: decide a hard time. Stop at t=0, final send done by t≈25 s worst case. Set abort at t=+40 s: if send not done, start source. Actually if send is in progress and nearly done, aborting wastes it; but the point of the gate is guaranteeing window. Since source data is untouched, restarting source is always safe. Set checkpoint: if at t+40 s the send isn't complete, kill it, start source, reschedule. Post-cutover rollback with reverse send: need common snapshot on both — @migfinal exists on both, so reverse is `zfs send -i @migfinal targetds@rollbackN` from target to source. Fine. One more: on target, before start, confirm the received fs is mounted at the right mountpoint and config's rootfs path matches. For LXC the rootfs path is in config. ZFS mountpoint property transfers with receive (unless -u or -x). Use `zfs receive -u` (don't mount) during replication, then mount at cutover — avoids the target mounting it early and dirtying it. Good practice: receive with -u, and set canmount appropriately. Hmm, for Proxmox subvols, mountpoint is managed. Keep it simple: mention receive -u to avoid target-side mount/atime divergence, and readonly=on as an alternative guard. I'll recommend: after initial replication, set `readonly=on` on target replica (prevents divergence), receive incrementals (works with readonly=on? — yes, zfs receive can update a readonly filesystem; readonly prevents writes via mount, not receive). Actually let me verify: zfs receive into a dataset with readonly=on — receive is allowed since readonly only blocks the POSIX write path, not receive. Yes, that's standard replication practice (e.g., sanoid/syncoid set readonly on target). So: target has readonly=on → divergence impossible → incremental receives fine → at cutover `zfs set readonly=off`. Clean story. I'll present that. Also `canmount=off`? Not necessary to dive deep. Keep readonly recommendation. Structure: 1. Feasibility (math, 2-3 lines) 2. Assumptions (bullet list, short) 3. Phase 0 Preflight + Gate 0 4. Phase 1 Pre-sync + Gate 1 5. Phase 2 Cutover runbook (timed table) + Gate 2 abort checkpoint 6. Phase 3 Verify + Gate 3 7. Phase 4 Commit 8. Rollback matrix table 9. Gotchas (brief) Keep commands generic-LXC with pct alternatives noted once. Let me write it. Variable convention: SRC host, DST host, dataset `tank/ct-214`, common snapshot `@rep0` (already replicated). Pre-sync snaps @mig1, final @migfinal. I'll write the answer now. Use headers, keep tight. Include the math line prominently. Tables for budget and rollback. Code blocks for commands. Gates as bold checkpointswith pass criteria. # Migration plan: ct-214 (ZFS → ZFS, no shared storage) ## Feasibility math - Final delta: 1.8 GiB = 1843 MiB ÷ 60 MiB/s = **~31 s transfer** - Naive stop-then-send: stop (~10 s) + 31 s + start (~10 s) ≈ 55 s — fits, but thin. - **Better: pre-sync while running**, so only the residual delta (writes during pre-sync + shutdown flush) is sent inside the window. Residual typically < 200 MiB → 2–5 s. Budget drops to ~30–40 s with ~50 s of margin. ## Assumptions (confirm in preflight) - Same L2 segment → IP/MAC unchanged, gratuitous ARP on start. If subnet changes, this plan needs a DNS/IP step and the window is at risk. - Target pool has the replicated snapshot `@rep0` as latest common snapshot; target dataset has not diverged. - Throughput 60 MiB/s is measured, not theoretical. - If Proxmox: dataset is typically `rpool/data/subvol-214-disk-0`; substitute below. Commands shown generic + `pct` equivalents. --- ## Phase 0 — Preflight (T-1h) ```bash # Both hosts: common snapshot present zfs list -t snapshot tank/ct-214 # Target: no divergence (must be 0 if readonly was set post-receive) zfs get -H -o value written tank/ct-214 # Target: capacity + health zpool list tank && zpool status tank # Source: measure write rate (delta growth predictor) zfs get -H -o value written tank/ct-214; sleep 60; zfs get -H -o value written tank/ct-214 # Link check: iperf or observe pre-sync rate; require ≥ 50 MiB/s ``` - [ ] Copy container config to target (LXC: `/var/lib/lxc/ct-214/`; PVE: `/etc/pve/lxc/214.conf`) — but ensure it **cannot autostart** on target yet. - [ ] Enumerate **all** volumes/bind mounts (`mp0`, `mp1`…) — each needs its own replication, not just rootfs. - [ ] Bridge names/VLANs on target match config (`vmbr0` mismatch is the classic failure). - [ ] Disable autostart/HA on source: `pct set 214 --onboot 0`; if HA-managed: `ha-manager set ct:214 --state disabled` (prevents split-brain restart). - [ ] Verified backup exists (last resort rollback). - [ ] Set target replica guard: `zfs set readonly=on tank/ct-214` (receive still works; prevents divergence). **GATE 0 — GO only if:** target not diverged, free space ≥ rootfs + 20%, measured link ≥ 50 MiB/s, write rate < ~25 MiB/s (else pre-sync won't converge in one round), config staged, HA/autostart off. ## Phase 1 — Pre-sync, container running (T-15 min) ```bash zfs snapshot tank/ct-214@mig1 zfs hold mig tank/ct-214@mig1 # protect from snapshot pruning zfs send -c -i @rep0 tank/ct-214@mig1 | mbuffer -q -m 256M | \ ssh dst 'mbuffer -q -m 256M | zfs receive -s tank/ct-214' ``` (`-s` = resumable; if interrupted: `zfs get receive_resume_token` on target, resume with `zfs send -t `.) Expect ~31 s. Then estimate the in-window residual: ```bash zfs get -H -o value written tank/ct-214@mig1 # bytes = final send size ``` If residual > ~600 MiB (write-heavy workload), take `@mig2` and send again — it converges geometrically. **GATE 1 — GO only if:** `(residual_bytes / 60 MiB/s) + stop_time + start_time < 45 s`. If not, do another pre-sync round or reschedule. ## Phase 2 — Cutover (downtime window) | t (s) | Action | Command | |---|---|---| | 0 | Graceful stop | `pct shutdown 214 --timeout 30` / `lxc-stop -n ct-214 -t 30` | | ~10 | Final snapshot | `zfs snapshot tank/ct-214@migfinal` | | ~12 | Final incremental (residual) | `zfs send -c -i @mig1 tank/ct-214@migfinal \| ssh dst zfs receive tank/ct-214` | | ~20 | Target prep | `zfs set readonly=off tank/ct-214`; verify mountpoint | | ~25 | Start on target | `pct start 214` / `lxc-start -n ct-214` | | ~35 | ARP + first check | `arping -c 3 -U ` from target; ping gateway | **GATE 2 — hard abort at t+40 s:** if the final send hasn't completed, kill it and **start the container back on source**. Source data is untouched (snapshots only), zero data loss, service restored in ~15 s. Reschedule. ## Phase 3 — Verify (T+0 → T+15 min) - [ ] Container process/app health: service checks, HTTP 200, DB accepts connections, recent records present. - [ ] Network: gateway, external clients, DNS. - [ ] Confirm nothing running on source: `pct status 214` → stopped. **GATE 3 — COMMIT if healthy at T+5 min, else ROLLBACK (see matrix).** ## Phase 4 — Post-migration (after 24 h soak) - Source container stays stopped, `onboot=0`, dataset retained as cold copy. - Update monitoring, backup jobs, CMDB to target host. - After retention: destroy source dataset + snapshots, release holds (`zfs release`). --- ## Rollback matrix | Trigger | State | Action | Data impact | Time to restore | |---|---|---|---|---| | Pre-sync fails / GATE 1 fail | Source running, untouched | Fix + reschedule | None | n/a | | Final send stalls past t+40 s | Source stopped, intact | Start on source | None | ~15 s | | Target starts but health checks fail (≤5 min) | Minimal writes on target | Stop target, start source | Discard post-cutover writes (tiny window) | ~1 min | | Late failure after real writes (within soak) | Target has new data | Reverse-replicate: snapshot on target, `zfs send -i @migfinal …` back to source, start source | None if reverse send completes | ~10–30 min | Never destroy source snapshots before commit — `@migfinal` on both sides is what makes the reverse path cheap. ## Gotchas - **Do not** use `zfs receive -F` blindly during pre-sync — a "destination has been modified" error means investigate divergence, not clobber it. - Keep the same MAC address so ARP caches/DHCP stay valid. - If the dataset is encrypted, use `zfs send -w` (raw) instead of `-c`. - Graceful stop timeout: if the container hangs on shutdown, `lxc-stop -k` after 30 s — ZFS snapshots are crash-consistent, but prefer clean app shutdown; if ct-214 runs a DB, confirm its recovery time fits your verify phase.

SCB SCB-migration-T1 MK3