Let me think through this carefully. The user is asking about migrating a container (ct-214) between hosts where: - rootfs is on ZFS - no shared storage (so we need replication-based migration, not live storage migration) - downtime window is 90 seconds - Snapshot already replicated (so the bulk sync is done) - Final delta is 1.8 GiB at 60 MiB/s First, let me do the math: 1.8 GiB at 60 MiB/s. 1.8 GiB = 1.8 × 1024 = 1843.2 MiB 1843.2 / 60 = 30.72 seconds So the raw delta transfer takes ~31 seconds. But the downtime window is 90s, and the cutover process involves more than just the transfer: - Container shutdown (clean shutdown time varies; could be 5-30s) - ZFS snapshot creation (near-instant, maybe 1-2s) - Final delta replication (30.7s) - Promotion/import on target - Container start on target (service startup) - Possibly IP/DNS updates So we have 90 - 31 = ~59 seconds of budget for everything else. That's feasible but tight if shutdown is slow. This looks like a Proxmox VE environment (ct-214 naming is typical Proxmox; "pct" commands, ZFS local storage, no shared storage). Proxmox has "Storage Replication" (pvesr) which does periodic ZFS snapshot replication to a target node for VMs and containers. The typical migration with replication configured: pct migrate does a final replication pass then moves the config. But the user says "Snapshot already replicated; final delta 1.8 GiB at 60 MiB/s" — this suggests they've already done manual zfs send/recv or pvesr, and now need the final incremental delta during downtime. Let me plan this properly with gates and rollback: **Pre-flight (before window):** 1. Verify state on target: zfs list, snapshot exists on target, last common snapshot GUID matches 2. Verify container config backed up: /etc/pve/lxc/214.conf copied to target (or pct migration will handle) 3. Network: target host has same bridge/vlan, IP conflict avoided 4. Test target host can start container (dry run: import not needed since replication done; but check host's ZFS pool free space for delta + headroom) 5. Confirm delta size estimate: measure write rate to the dataset, confirm 1.8 GiB still accurate; measure actual throughput with a test send (e.g., replicate an earlier inter-snapshot delta and time it) 6. Check in-flight writes: if the app writes continuously at high rate, the final delta may be larger than 1.8 GiB if we take longer. 60 MiB/s transfer while app writes at, say, 10 MiB/s... the snapshot is a point-in-time, so delta is bounded by writes during the window between snapshot creation and shutdown. Actually key: take the final snapshot AFTER shutdown so delta = writes since last replicated snapshot up to shutdown. So the delta is fixed once the container is stopped. Good — that means the 1.8 GiB estimate should be validated against writes since the last replication + expected writes until shutdown. Actually, important detail: the delta is determined by writes between the last replicated snapshot and the final snapshot taken at cutover. So if the last replication was recent and the app writes slowly, the delta will be small. The 1.8 GiB figure presumably came from estimating the write rate. The risk: if we delay, delta grows. Gate: verify current delta estimate before stopping. **Timeline budget (90s):** - T0-T+5: shutdown container (must be quick — use pre-scripted shutdown, e.g., pct shutdown 214 --timeout 20, fallback to pct stop) - T+5: create final snapshot: zfs snapshot rpool/data/subvol-214-disk-0@final — near instant - T+5-T+36: incremental send/recv: zfs send -I @last @final | zfs recv ... (~31s) - T+36: promote/fix up on target (if recv'd as child dataset, zfs promote; typically pvesr handles) - T+36-T+45: move config, set container to target, start: pct start - T+45-T+90: validation gates (service health checks, log tail, smoke test) - Buffer: ~45s Actually wait — the clean shutdown might take longer. Also, does Proxmox's pct migrate handle this? If replication (pvesr) is configured, pct migrate 214 target-node --online does: final replication, config move, start. With replication, offline migration = brief downtime. The window math: 31s transfer + shutdown + start. Feasible. But maybe the user is doing this manually with zfs send/recv (the phrasing "Snapshot already replicated" suggests manual ZFS replication, possibly not pvesr). I should cover both: if pvesr is set up, use pct migrate; if manual, here's the manual sequence. Let me structure the answer: **1. Pre-flight checks (T-24h or T-1h) — gates G0-G3:** - G0: Snapshot lineage intact — last replicated snapshot exists on both, no divergence; target dataset is not modified (no local writes on target — this is critical: receiving side must not have writes or incremental recv will fail or you need rollback) - G0b: Target host capacity: free space ≥ delta × 2 (for promote/space) — actually ZFS incremental recv needs space for new data + possibly snapshots - G1: Measured throughput ≥ 60 MiB/s sustained (test with a probe send or recent replication logs); gate: measured_rate ≥ 1.8 GiB / 50s ≈ 38 MiB/s minimum to fit with margin. Give the math. - G2: Config parity: CPU flags? For LXC, less of an issue than VMs. Network on target: bridge name, VLAN, firewall policy, MTU match. /etc/pve/lxc/214.conf and any mount points (mp0..) — beware bind mounts referenced in config that exist only on source host! Gate: all mp entries exist on target or are replicated. - G3: Rollback rehearsal: documented command sequence; confirm source dataset NOT deleted until final sign-off; backups exist (or at least last snapshot). - Also: DNS/monitoring maintenance mode, alert silencing, downstream deps. - Backup container config: cp /etc/pve/lxc/214.conf somewhere (pve config is replicated across cluster, but keep a copy with .bak timestamps). - Check container has no local-device passthrough (e.g., /dev entries), no hostabsolute bind mounts. **2. The math (go/no-go sizing):** - Delta 1.8 GiB = 1843 MiB ÷ 60 MiB/s ≈ 30.7s transfer - Budget: shutdown ≤ 15s, snapshot ≤ 2s, transfer 31s, recv/promote ≤ 5s, start ≤ 10s, health ≤ 20s → ~83s total. Tight. Options to widen: pre-shutdown quiesce (application checkpoint) to reduce writes so we could take a pre-shutdown snapshot to trim the final delta; or measure shutdown time in advance (do a rehearsal shutdown/start on the source during an earlier maintenance — that also tells you boot time). Ways to compress: - Option A (two-stage delta): take snapshot S2 ~5-10 min before window while running, replicate it during uptime (send -I from S1 to S2 — that's larger but done before downtime). Then final delta S2→S3 at shutdown is only ~30-60 MiB → a few seconds. This shrinks downtime to ~30-40s. This is exactly how pvesr works (periodic replication). Strongly recommend this: the 1.8 GiB "final delta" is only acceptable if measured shutdown is fast; better to do a last pre-window replication pass. Wait, the user says "final delta 1.8 GiB at 60 MiB/s" — they may have computed the downtime as ~31s for the delta and want a plan fitting 90s. My plan should use their numbers but note the two-stage option to shrink risk. - Also: mbuffer + compression to speed transfer. zfs send | ssh with compression (lz4 for stream, or zfs send -c for embedded per-block compression). If the data is compressible, -c can meaningfully increase effective throughput. But careful: -c requires target pool support (modern ZFS fine). mbuffer -s 128k avoids slow-start/ssh cipher stalls. If 60 MiB/s is network-bound (e.g., 1 GbE ≈ 112 MiB/s theoretical), 60 MiB/s suggests ssh single-stream overhead; mbuffer + maybe parallel ssh ciphers (aes hardware) could push higher. But don't over-promise; keep the plan with measured 60. **3. Execution runbook with gates:** Phase 0 — Pre-window (before T0): - Put app into maintenance mode / stop batch jobs, flush caches (reduce dirty data and shutdown time) - Optional: take and replicate pre-final snapshot (stage delta) during uptime - Silence monitoring alerts on ct-214 - Record current snapshot list: zfs list -t snapshot -r rpool/data/subvol-214-disk-0 - Confirm no one else writes to target dataset Phase 1 — T0: Quiesce and stop (gate: hard stop by T+20) - pct shutdown 214 --timeout 15 (or via console; if timeout exceeded → pct stop 214 — app-level data integrity is presumably handled by app) - Gate G4: container is stopped on source; all processes gone; mountpoint freed Phase 2 — Final snapshot (T+20): - zfs snapshot rpool/data/subvol-214-disk-0@migrate-final - Record size estimate: zfs send -nvI @last-replicated @migrate-final → prints estimated size. GATE G5: if estimate > budget (i.e., > ~3 GiB given remaining time), ABORT & ROLLBACK (start on source). Because transfer time = size/rate; abort criterion precomputed: max_delta = rate × (window_deadline - elapsed - start_allowance - safety). Let me compute abort threshold: need transfer done by ~T+55 to leave 35s for start+validate. At 60 MiB/s from T+22, available 33s → 1.98 GiB max. So gate: estimated ≤ 1.9 GiB else abort. Since user says 1.8 GiB, it passes with slim margin. Emphasize: verify with zfs send -nv (dry-run) BEFORE committing — this is the key gate because if the app wrote more than expected (e.g., logs, DB), the estimate updates in seconds. Hmm — but note: zfs send -nv estimate is quick (metadata walk), taking maybe seconds for a dataset with many files. Fine within window. Actually for a 1.8 GiB delta on a container rootfs, the dry-run estimate is fast (a few seconds at most). Alternative: do the estimate BEFORE stopping? Can't — snapshot must be taken after stop to capture all writes. But you can take a throwaway snapshot right before stopping to estimate. Or better: the two-stage approach — replicate as much as possible pre-window. Phase 3 — Transfer (T+22 → T+53): - Command: zfs send -I @last-sync @migrate-final | mbuffer -s 128k | ssh target "zfs recv -F -u rpool/data/subvol-214-disk-0" - -I includes intermediary snapshots (if any), -u don't mount on recv, -F force rollback on target (dangerous? -F will rollback target dataset and discard divergent snapshots — since target is a pure replica, fine, but document that -F requires target be untouched) - Actually if there are no intermediate snapshots, use -i (incremental) instead of -I. pvesr may have created multiple snapshots; -I @first...@last replays all. - Progress monitoring: mbuffer prints progress; or pv -bartes if no mbuffer. Watch elapsed vs. plan: GATE G6: at T+45, if transfer < 75% complete → prepare abort path (but note: aborting mid-recv leaves partial snapshot? zfs recv of an interrupted stream discards incomplete data automatically on stream termination — yes, if the stream is interrupted, recv fails and cleans up the partial receive; with -s resumable you can resume. Consider -s for resumable send and r flag on recv: zfs send -s, zfs recv -s. But resume doesn't help within a 90s window; still, it makes post-abort retry cheaper — partial state can be resumed! Actually that's a great fallback: if we abort and must roll back to source, the resumable token lets the next attempt skip already-transferred bytes. Worth mentioning as option.) Hmm, one caveat: resumable send (-s) writes resumable state to the dataset property; fine. - Verify transfer integrity: zfs list on target, compare: zfs send -nv (size) vs received used; or compare snapshot GUID: zfs get guid rpool/...@migrate-final on both must match. Cheap and strong. GATE G7: GUID match + recv exit 0. Phase 4 — Cutover config on target (T+53 → T+60): - If Proxmox with pvesr: pct migrate would have done this. Manual: copy config, adjust. Steps: - On source: pct shutdown already; move config: mv /etc/pve/lxc/214.conf handled by cluster (pmxcfs replicates), but manual: edit or use qm/pct commands. For manual ZFS, likely they're not using pvesr. So: - Copy /etc/pve/lxc/214.conf to target (if not clustered, scp) - Fix storage reference in config (rootfs: sourcepool:subvol-214-disk-0 → target pool name), mp entries - Ensure ownership: 100000:100000 mapping if unprivileged (subvol created by recv preserves; fine) - If dataset was received with a different mountpoint property, set mountpoint=none or correct path; Proxmox manages mounts via pct. - Optionally zfs promote if target dataset is a clone chain? With pvesr the received dataset is a child of a placeholder that gets promoted on migration. With manual recv into the same dataset name, no promote needed. Phase 5 — Start & validate (T+60 → T+85): - pct start 214 (or lxc-start -n 214) - GATE G8 (health): process up, port answering (curl smoke test), logs clean for N seconds, app-level check (e.g., DB read/write, login page). Predefine objective checks. - T+85: GO/NO-GO declaration. Phase 6 — Post-migration (after window): - Keep source dataset (with @migrate-final) as instant rollback for 24-48h: rollback = zfs rollback not needed—container just doesn't run there; to roll back: stop on target, start on source (source still has everything). Document the revert path: pct stop on target; move config back; pct start on source. Time to revert ≈ stop+start ≈ 30-60s. - Cleanup: after soak period, delete source dataset + snapshots, remove replication job, re-enable monitoring, DNS TTL already low or revert TTL. **Rollback paths at each gate:** - G4 (won't stop in 20s): force stop; if still failing, abort → start again? If container won't stop, that's a different incident; escalation. - G5 (delta too big): delete @migrate-final snapshot on source (optional keep), pct start 214 on source, end window, investigate write rate, reschedule with pre-window replication pass. Source untouched → zero-risk rollback. - G6 (transfer too slow mid-flight): kill send/recv (partial recv auto-discarded), start on source. With resumable recv (-s), partial progress persists for next attempt. - G7 (integrity fail): same — start on source, retry or reschedule. - G8 (app unhealthy on target): stop container on target, start on source (config back), rollback. Rootfs on source is complete as of shutdown (snapshot @migrate-final is on source too). Rollback time ~1 min — note this may exceed remaining window; so G8 decision must be made by T+~85 with decision owner preassigned. Key insight for rollback: since we never destroy or modify the source dataset during the window, the rollback is always "start on source." The only mutations: adding @migrate-final snapshot on source (harmless) and possibly a config move. So config handling: don't remove the config from source cluster-wide until validation; in Proxmox, pct migrate moves config atomically. Manual approach: keep the original config, create the target config as a copy; on rollback restore. Also important: the 90s window is from user-visible downtime start (shutdown) to service healthy. If G8 fails at T+85 and we roll back, we're at ~2.5 min downtime — must flag that risk and get the business sign-off that "worst case 3 min" is acceptable, or define G8 earlier with auto-fallback. To reduce the worst case: shorten start-validation to essential checks within window; deeper validation post-window. **Risk list:** - Shutdown slower than rehearsed (mitigate: rehearse beforehand; measure; use pct stop fallback) - Delta larger than 1.8 GiB due to write burst (mitigate: app maintenance mode; pre-window extra replication pass; dry-run size gate G5) - Throughput dips (mitigate: mbuffer, dedicated link/VLAN, no concurrent scrub/replication IO, disable other zfs send jobs; check for ZFS scrubs scheduled — pause scrubs on both pools!) - Target pool space (delta + snapshot retention) - Config drift: bind mounts, device nodes, /dev/dri passthrough, nesting, keyctl, hooks scripts (fw not relevant to lxc much), unpriv uid map consistent - IP/MAC conflict during cutover: container starts on target with same IP — ensure source container is stopped before start (it is), no stale ARP on gateway — may need arping gratuitous or just rely on retries; bridges differ (vmbr0 both sides) — check - MTU mismatch on storage/network - SSH known_hosts / bandwidth limits for root ssh between nodes (pbs? no—manual) - Firewall between hosts allows ZFS send port? We're using ssh, fine. - If pvesr was used for the initial replication: replication job direction; after migration, flip the job direction for future DR. Good point: post-migration, reconfigure pvesr job (target→source) so replication continues in the new topology. **Timing table:** | T | Action | Gate | - T-10 min: app maintenance mode; monitoring silenced - T-5: verify last snapshot common on both hosts; verify no active recv on target; free space check; abort thresholds written on runbook printout - T0: pct shutdown (timeout 15) → stop - T+~10: zfs snapshot @final - T+12: zfs send -nv estimate → G5 compare vs threshold (abort if > 1.9 GiB; threshold formula) - T+15: incremental send | mbuffer | ssh recv (31s) - T+46: verify recv (exit code, GUID match) - T+48: config move + fixups - T+50: pct start - T+60-85: smoke tests → G8 - T+85: declare done (or G8 fail → rollback) - Post: leave source intact 48h; flip replication; cleanup The numbers: 31s transfer at 60 MiB/s for 1843 MiB = 30.72s. If we start transfer at T+15 and it takes 31s → done T+46. Recv finalize ~2-3s. Config ~5s. Start ~10s (LXC start is fast, unlike VM boot; container "start" is near-instant but services inside need time — the app's readiness probe matters; e.g., DB container might take 20-30s to accept connections). Validation 20s. Total ≈ 85-90s. Tight but OK, with the caveat that if app readiness is slow, validation spills past 90s. To be safe, strongly recommend the pre-window delta pass: do one more replication 1-2 min before shutdown so the true final delta is ~100-300 MiB → transfer 2-5s. Then the whole cutover is ~45-60s with huge margins. This is the single best improvement. The user's stated numbers still fit, but margin improves massively. I'll present the plan with their numbers as the baseline and this optimization as strongly recommended. Also mention zfs send flags: -L (large blocks, if recordsize large and needed — for consistency with how earlier snapshots were sent; if initial replication used -L, incremental must too? Actually mismatch on large blocks can cause recv errors; be consistent with initial send flags), -c (compressed), -w (raw, for encrypted datasets). Consistency note: incremental recv requires the same property handling; simplest to reuse the exact send flags from the initial replication. If pool has encryption, need -w raw or ensure keys loaded on target. Also: recordsize/block settings — not needed. Proxmox-specific commands: - pct shutdown 214 --timeout 15 - pct stop 214 (hard) - pct migrate 214 --online? For LXC with replication, online migration of containers with replicated ZFS works (shared-nothing live migration is supported for LXC with --online even without shared storage! Proxmox supports LXC live migration via CRIU? No — Proxmox supports online container migration using... Actually pct migrate --online for containers performs checkpoint/restore with CRIU. Yes! Proxmox supports CRIU-based live migration of containers. But CRIU has many limitations (no open file descriptors issues, etc.). Given a 90s window is allowed, offline migration is simpler and safer. I'll mention --online exists (CRIU) but recommend the deterministic offline path given they've already pre-replicated. If pvesr replication is configured (the "snapshot already replicated" strongly suggests pvesr), then `pct migrate 214 targetnode --with-local-disks`? Hmm. For replicated containers, plain pct migrate does: final replication sync, stop, config move, start. The command for offline: pct migrate 214 targetnode. It requires replication job set up with the target. The downtime = final delta transfer + start. Since 31s transfer fits, this one command is the whole cutover. But the user may have done manual replication instead. I'll cover: "If this is Proxmox with pvesr: the entire window is essentially `pct migrate`" and give manual sequence otherwise. Also give the manual ZFS commands since they might be on plain LXC/zfs hosts, not Proxmox. The "ct-214" naming is Proxmox-ish but could be any platform. I'll write it Proxmox-flavored with raw zfs commands, and note adaptations. Actually, let me not assume too hard. I'll structure: runbook with gates, using Proxmox tooling where it exists (pct), raw ZFS for the delta, and note "adjust if not Proxmox." Keep it practical. Abort thresholds — precompute and write on the runbook: - Deadline math: Window 90s. Reserve: start 10s, validate 15s, slack 10s → transfer must complete by T+55. Transfer start realistically T+18 (after stop+snapshot+dry-run). Available transfer time = 37s → max delta at 60 MiB/s = 2220 MiB ≈ 2.1 GiB. Set abort at estimate > 2.0 GiB (or whatever recompute). Their 1.8 GiB leaves only ~0.2 GiB headroom (~3.4s). Thin. Again pushing the pre-window extra pass. Alternatively, improve effective rate: mbuffer + lz4 (ssh -C is too slow CPU-wise; better zfs send -c if blocks compressible). If 60 MiB/s is the ceiling, fine. Health check specifics: define before window: HTTP endpoint curl -f, DB SELECT 1, service count via pct exec, log grep for ERROR since start. Assign a decision owner with a watch/clock. Rollback plan details: - Condition triggers per gate (already above) - Rollback procedure (target→source): 1. pct stop 214 on target (if started) 2. Restore config on source (or pct move back) 3. pct start 214 on source 4. Confirm health - Note: any writes made on target between start and rollback are lost (delta on target only). If app wrote during validation, that data does not exist on source. Mitigation: rollback decision fast; or if data loss matters, zfs send the post-start writes back? Messy — simply state: window between target start and rollback has data on target only; acceptable because validation writes are typically throwaway/test. Flag it. - Keep source intact: do not delete; snapshot @migrate-final on source is the last-known-good running state? Actually source dataset contains all data up to shutdown — identical to what target received. So source is a full hot spare. Keep 24-48h. Edge cases and gotchas: - zfs recv -F on target: if target has snapshots not in source lineage (e.g., someone snapshotted the replica), -F destroys them. Document that target must be replica-only (no manual snapshots). Gate pre-check: compare snapshot lists. - If pvesr is in play: snapshots named __replicate_... — pvesr handles. Manual zfs recv onto a pvesr-managed dataset can conflict; pick one mechanism. If they used pvesr for the initial sync, let pvesr/pct migrate do the final delta too (pvesr trigger manually: pvesr --run-job --force? Actually `pvesr run `; there's `pvesr run --job `. The command is `pvesr run `. I think it's `pvesr run --job `? Let me recall: pvesr has subcommands: create, set, remove, enable, disable, run, status, read, push... `pvesr run ` runs a job. Yes, `pvesr run `. I'll mention it lightly.) - Unprivileged container UID mapping: dataset received keeps file ownership; since same pool type and unpriv settings in config unchanged, fine. If switching privileged↔unprivileged — don't during migration. - Checkpoint: after recv, target dataset mountpoint property may be inherited weirdly; Proxmox pct handles mounting. On plain LXC, set mountpoint correctly in config. - DNS/host keys: container internal unchanged. Host firewall rules for the container's IP on target host must exist. - MAC address: keep same (config carries it). - ARP staleness on gateway after IP moves subnets? Same L2 presumably; flush ARP by arping -U from container after start (post-start hook or manual). - Time/NTP inside container fine. Scrub/IO contention: ensure no scrub running on either pool (zpool status), pause other replication jobs, backups. Also free-space gate: target needs delta + snapshot overhead; ensure ≥ 2× delta free, plus the space freed/deduped—simple: ≥ 10 GiB free or ≥ 4× delta. Throughput gate pre-check: run `zfs send -nv` + measure a small probe? Better: look at the previous replication's actual rate (pvesr logs or shell history). They stated 60 MiB/s — presumably measured. Gate: measured sustained ≥ 45 MiB/s (with margin factor 1.5× on the required 41 MiB/s). Required rate: 1843 MiB / 45s available ≈ 41 MiB/s minimum. Their 60 gives 1.47× margin. Marginal. Recommend the extra pre-window pass to make it robust. Let me also compute exact numbers: 1.8 GiB = 1.8*1024 MiB = 1843.2 MiB; /60 = 30.72s. Round: ~31s. Total budget breakdown (their scenario, no extra pass): - stop: 10-15s (rehearsed) - snapshot + dry-run: 3-5s - transfer: 31s - recv finalize + GUID check: 3s - config: 5s - start: 5-8s (LXC start itself ~2s; wait for network ~5s) - smoke: 15-20s - slack: ~10s Sum: ~83-97s. Right at the edge. Verdict: feasible only with (a) rehearsed fast shutdown, (b) extra pre-window replication pass strongly recommended, (c) strict gates. With extra pass: final delta maybe 100-300 MiB → transfer 2-5s → total ~50-60s, comfortable. Structure of the answer: 1. Feasibility math up front (the budget table) 2. Strong recommendation: extra pre-window delta pass (turns 31s into ~5s) 3. Pre-flight gates (G1..G3) — before window even starts 4. The 90s runbook with timeline, exact commands, in-window gates (G4..G8) with abort criteria and thresholds 5. Rollback at each gate + post-cutover rollback 6. Post-migration cleanup (keep source 48h, flip replication job, unsilence, ARP flush) 7. Gotchas checklist Formatting: This is a technical runbook — headers, a timeline table, code blocks for commands, gates clearly marked with pass criteria. Should be fairly detailed but not bloated. The user is clearly expert-level (terse prompt, knows ZFS, knows their numbers). So: dense, precise, minimal hand-holding. Use their numbers. Markdown appropriate. Reasonable length — this deserves thoroughness since it's an operational plan where missing details cause outages. But avoid fluff. Let me draft the commands precisely: Pre-flight: ``` # source zfs list -t snapshot -o name,guid,creation -r rpool/data/subvol-214-disk-0 zfs get -H guid rpool/data/subvol-214-disk-0@ # target zfs list -t snapshot -o name,guid rpool/data/subvol-214-disk-0 zpool status -v (no scrub, no errors) zfs list -o available rpool/data (>= 4x delta) # throughput probe: replicate one prior inter-snapshot delta and time it time zfs send -i @sA @sB | mbuffer ... | ssh ... ``` Config pre-check: ``` grep -E '^(rootfs|mp[0-9]+)' /etc/pve/lxc/214.conf # ensure referenced storages exist on target: pvesm status on both # unprivileged flag, lxc.idmap consistent ``` Window: ``` T0: pct shutdown 214 --timeout 15 || pct stop 214 # gate: stopped by T+20 else abort? If can't stop, escalate T+10: zfs snapshot rpool/data/subvol-214-disk-0@mig-final T+11: zfs send -nv -I @last-sync @mig-final # size gate ABORT if size > 2.0 GiB (recompute: rate*(T55-now)/1024) T+14: zfs send -I @last-sync @mig-final | mbuffer -s 128k -m 512M | ssh tgt 'zfs recv -Fvu rpool/data/subvol-214-disk-0' # or pv -bter instead of mbuffer T+46: verify: ssh tgt "zfs get -H guid rpool/data/subvol-214-disk-0@mig-final" == source guid T+50: config move: # Proxmox non-pvesr manual: scp /etc/pve/lxc/214.conf tgt:/etc/pve/lxc/214.conf # edit rootfs: line to target pool name (e.g., rootfs: tgtdata:subvol-214-disk-0,size=...) # Actually careful with double config in cluster pmxcfs: if both hosts in same cluster, /etc/pve is shared — config exists once; just edit in place. pct migrate handles. Manual on clustered Proxmox is awkward — better to just use pct migrate if clustered. ``` Hmm, this is where it forks significantly: - Clustered Proxmox + pvesr → `pct migrate 214 ` does everything (final replication sync, config move, start). The runbook gates still apply for abort. - Non-clustered or manual zfs → config copy manually. Given "ct-214" and ZFS and no shared storage, most likely clustered Proxmox with pvesr replication ("snapshot already replicated" = pvesr job ran). Then the honest answer: the window is basically `pct migrate 214 node-b` which triggers one final replication pass (~31s) + config move + start; gates = pre-flight validation and abort/rollback path. I'll present the pct migrate path as primary (Proxmox), and the raw zfs variant for manual setups. That dual path keeps it useful regardless. pvesr specifics: replication job: pvesr list. Manual run: `pvesr run `. Before the window, run it once more (that's the "extra pass" recommendation). pct migrate with replication does the final sync automatically? Yes — for replicated guests, migration does a final replication before switching. Downtime = stop + final delta + start. One more subtlety: pct migrate order: it does final replication first? For offline migrate with pvesr: stop guest on source → final replicate → move config → start on target. Correct. Rollback for pct migrate: since config moved to target, rollback = stop on target, `pct migrate 214 ` back? That would try to replicate again (reverse direction) — could be fine or messy. Simpler: manually move config back: mv config, then pct start on source. In a cluster, the config file is in pmxcfs shared — there's no "copy"; the guest is assigned to a node by which node has the file? No — in clustered pmxcfs, all configs exist in one shared FS; the "node" is implied by the path? /etc/pve/nodes//lxc/214.conf — yes! The config lives at /etc/pve/nodes//lxc/214.conf. Moving between nodes = mv between node dirs. pct migrate does that. Manual rollback: mv /etc/pve/nodes/NEW/lxc/214.conf /etc/pve/nodes/OLD/lxc/214.conf then start on old. Also pvesr job direction would need flipping back or the replication job targets node B... For rollback within the soak period, just disable the pvesr job, keep source dataset (it's still there — pct migrate does NOT delete the source dataset? Does pct migrate delete source? With replication migration, the source dataset remains with its snapshots on the source; migration via replication just makes target authoritative and config points there. Actually I believe pct migrate after successful migration removes the source dataset? Hmm. Let me think. With pvesr replicated volumes, when you migrate, the source copy... I recall that after migrating a replicated guest, the old volume on source stays (it becomes the replication *target* when you flip the job direction). pvesr requires the job to be re-created/edited post-migration (direction flip is manual: edit job to new source→target... actually pvesr docs say after migration you must re-create the replication job with the new direction; the old dataset on source remains until you flip). So the source dataset survives — good for rollback. I'm fairly confident source volume is retained (pvesr migration doesn't destroy it; you flip the job afterward). Yes — Proxmox docs: "After a successful migration, remove/flip the replication job." So rollback is trivial. I should be careful not to assert too many Proxmox internals with certainty. Keep the statements safe: "source dataset is retained after replication-based migration; flip the pvesr job direction post-migration; verify before deleting anything." Alright, also the abort threshold formula to print in the runbook: max_delta_MiB = rate × (T_deadline_transfer − now) / 1024... let me just give numbers: transfer must finish by T+55; if send starts at T+15 → 40s × 60 = 2400 MiB ≈ 2.3 GiB hard cap; with safety 2.0 GiB abort line. Their 1.8 GiB passes with 3.4s of margin. Hence the pre-window pass recommendation. Let me also account: zfs send dry-run of estimate takes ~1-3s for this size. Fine. Snapshot delta growth risk between estimate and send: none — snapshot is frozen. Good: the estimate is exact-ish (send -nv estimates bytes; actual stream may differ a bit with -c etc.). Note: with embedded compression -c, the estimate reflects uncompressed size? zfs send -nv estimates the stream size; with -c it estimates compressed? I believe the estimate accounts for -c? Not sure — safer to estimate without -c and treat estimate as upper bound if using -c. Don't get too deep. mbuffer sizing: -m 512M maybe too big; default 2M. Use -m 64M fine. Or pv. Keep simple: `| pv -bter` or mbuffer -s 128k -m 64M. Also ssh cipher: chacha20 vs aes-ni; if CPU-bound, use -c aes128-gcm@openssh.com. Minor tuning note. One more in-window trick if delta must shrink further: `zfs send -c` compressed blocks (if dataset has compression=on and earlier sends were also -c... incremental recv of -c streams onto datasets received without -c is fine actually; flags must be consistent across the chain? Mixing can cause issues with -L large blocks mainly. Keep the "use same flags as initial replication" rule.) Now the timeline table with their numbers, no extra pass: | Time | Action | Exit gate | | T-60m | pre-flight checklist G1-G3 | | T-5m | app maintenance mode, silence alerts, confirm no scrub/other jobs, verify last common snapshot | | T-2m | (recommended) run extra replication pass → delta shrinks to tens/hundreds MiB | | T0 | pct shutdown --timeout 15 (fallback pct stop) | G4: stopped ≤ T+20 | | T+12 | zfs snapshot @mig-final | | | T+13 | zfs send -nv size check | G5: ≤ 2.0 GiB else abort→rollback | | T+15 | send|recv with pv/mbuffer | G6: ≥75% by T+45 else abort | | T+47 | GUID/exit-code verify | G7 | | T+50 | config cutover | | | T+52 | pct start | | | T+60→T+85 | smoke checks | G8 by T+85 | | T+85 | declare success / rollback decision | | post | keep source, flip replication, cleanup at +48h | Also define the "abort = start on source" universal rollback and its cost (~30-45s), and note G8-rollback lands ~T+120 worst case → needs pre-agreed extended outage tolerance. Also note data written during validation on target won't be on source after rollback — decision guidance: keep validation read-only or accept loss. Include a compact "gotchas" list: bind mounts/mp, device passthrough, unpriv idmap, firewall/bridge/VLAN on target, MAC/IP, gateway ARP flush (post-start arping -U), pvesr job direction, no local snapshots on replica (recv -F), scrubs paused, space gate, consistent send flags, encrypted pool keys (send -w), no writes on target replica ever. I think length: this needs to be a proper runbook. Aim for a thorough but tight answer — maybe 700-1000 words equivalent with tables and code blocks. The user asked for "Plan with gates + rollback" — deliver exactly that. Use headers, tables, code. No fluff intro. Let me finalize precise numbers: - 1.8 GiB / 60 MiB/s = 30.7s ≈ 31s. - Minimum viable rate for 40s budget: 1843/40 ≈ 46 MiB/s. - Margin at 60: 31s vs 40s budget → 9s slack in transfer alone. Total slack across window maybe ~7-10s. Verdict: go, but thin; the pre-pass makes it comfortable. Abort threshold at G5 given transfer start T+15 and finish deadline T+55: 40s → 2400 MiB = 2.34 GiB; apply 20% safety → abort > 2.0 GiB. Write the answer now. Structure: 1. Budget math (small table) 2. Pre-window pass recommendation (the one change that buys real margin) 3. G1–G3 pre-flight gates (day before / T-1h) 4. Runbook T0→T+90 with gates G4–G8, commands 5. Rollback matrix (per gate + post-cutover) 6. Post-window cleanup 7. Gotcha checklist Keep commands Proxmox+ZFS with a note for pure-ZFS/manual. Good. # ct-214 migration runbook — ZFS delta cutover, 90 s window ## Budget check (go/no-go math) | Item | Cost | |---|---| | Clean stop (`pct shutdown --timeout 15`) | ≤ 15 s | | Final snapshot + `send -nv` size probe | ~4 s | | **Delta transfer: 1.8 GiB ÷ 60 MiB/s** | **~31 s** | | recv finalize + GUID verify | ~3 s | | Config cutover + `pct start` | ~8 s | | Smoke validation | ~20 s | | **Total** | **~81 s → fits, with ~9 s slack** | Thin. One change buys real margin: **run one extra replication pass 2–5 min before the window.** Snapshot the running container, replicate while it's still up, and the true final delta drops to tens–hundreds of MiB (2–5 s transfer). Total cutover then lands ~45–60 s. Strongly recommended; plan below assumes it, with the 1.8 GiB figure as worst case. --- ## Pre-flight gates (T−1 h) **G1 — Lineage & target hygiene** - Last common snapshot exists on both sides, same GUID: `zfs get -H guid pool/data/subvol-214-disk-0@` - Target dataset is a *pure replica*: no local writes, no manual snapshots (incremental `recv -F` will destroy divergent snapshots — verify none exist). - No scrub in progress on either pool (`zpool status`); no concurrent replication/backup jobs. **G2 — Capacity & throughput** - Target free space ≥ 4× expected delta. - Measured sustained rate ≥ 46 MiB/s (that's the bare minimum for a 40 s transfer budget); your 60 MiB/s is 1.3× margin. Confirm from previous replication logs, not assumptions. **G3 — Config parity** - Every `mp*:` mount point in `/etc/pve/lxc/214.conf` exists on the target (bind mounts are the classic silent killer). - Unprivileged/`idmap` settings identical; no host-device passthrough that target lacks. - Target host has same bridge/VLAN/MTU/firewall policy for ct-214's IP/MAC (MAC travels with config — keep it). - Backup the config: `cp /etc/pve/lxc/214.conf /root/214.conf.pre-mig` - Silenced monitoring, app in maintenance mode by T−5 min, decision owner named for in-window gates. --- ## Window execution | T | Action | Gate | |---|---|---| | T−2 m | Extra replication pass (running state) — shrink final delta | — | | T+0 | `pct shutdown 214 --timeout 15` → fallback `pct stop 214` | **G4:** stopped by T+20, else escalate | | T+12 | `zfs snapshot rpool/data/subvol-214-disk-0@mig-final` | — | | T+13 | `zfs send -nv -I @last-sync @mig-final` (dry-run size) | **G5:** abort if > **2.0 GiB** (hard cap 2.3 GiB = 40 s × 60 MiB/s; safety margin applied) | | T+15 | `zfs send -I @last-sync @mig-final \| mbuffer -s 128k -m 64M \| ssh tgt 'zfs recv -Fvu rpool/data/subvol-214-disk-0'` (same send flags as the initial replication — don't mix `-L`/`-c`/`-w`) | **G6:** ≥75% complete by T+45, else abort | | T+47 | Verify: recv exit 0; snapshot GUID matches on both hosts | **G7** | | T+50 | Config cutover (see below) | — | | T+52 | `pct start 214` | — | | T+60–85 | Smoke: process up, port answers (`curl -f`), `SELECT 1` / app probe, log tail clean | **G8: declare by T+85** | **Config cutover — two cases:** - **Proxmox cluster with pvesr:** if replication is pvesr-managed, skip the manual send entirely — `pct migrate 214 ` does final replicate → stop → config move → start. Use the raw ZFS path above only if you replicated manually (don't mix mechanisms on one dataset). - **Manual:** `scp /etc/pve/lxc/214.conf` (or `mv /etc/pve/nodes/OLD/lxc/214.conf /etc/pve/nodes/NEW/lxc/214.conf` on a cluster), edit `rootfs:` line to target pool name. --- ## Rollback **Universal rule: the source dataset is never modified during the window** (only the added `@mig-final` snapshot, which is harmless). Rollback is always: *start on source*. | Gate failed | Action | Cost | |---|---|---| | G4 (won't stop) | Hard stop; if still stuck, it's an incident, not a migration | — | | G5 (delta too big) | Discard `@mig-final`, `pct start 214` on source, reschedule after fixing write rate / adding pre-passes | ~30 s downtime total | | G6/G7 (transfer slow/corrupt) | Kill send/recv (interrupted recv auto-discards partial state), start on source. Use `send -s` / `recv -s` resumable tokens so a retry skips transferred bytes | ~45 s | | G8 (unhealthy on target) | `pct stop` on target, move config back, `pct start` on source | Lands ~T+120 — **get pre-agreement that worst case is ~3 min**, and make G8 decision by T+85 sharp | ⚠️ G8 caveat: writes made on target during validation exist only on the target. Keep smoke tests read-only, or accept that loss on rollback. --- ## Post-window - **Keep the source dataset untouched for 24–48 h** — it's your hot spare (identical to shutdown state). Revert path = stop on target, start on source, ~1 min. - Flush stale ARP if needed: `arping -U -I eth0 ` from target host after start. - Flip the pvesr job direction (new source → old host for DR) and re-enable it. - Unsilence monitoring, take app out of maintenance, then delete source dataset + old replication snapshots after soak. --- ## Gotcha checklist - No scrub, no parallel `zfs send`, no backup jobs on either pool during window. - `recv -F` is destructive to divergent target snapshots — G1 covers this, re-check at T−5. - Same send flags as the original replication chain (mixing `-L`/large-block handling breaks incremental recv). - Encrypted dataset → keys loaded on target, `send -w`. - SSH cipher: use `aes128-gcm@openssh.com` if CPU-bound; the mbuffer smoothing matters more. - Container-internal hostnames/keys are unaffected; host-side firewall must allow ct-214's IP on target before T0.

Migrate container ct-214 between hosts, rootfs on ZFS, no sh...