The user is asking for a migration plan for a Proxmox VE container (ct-214) with rootfs on ZFS, between hosts, no shared storage, with a 90-second downtime window. Snapshot is already replicated; final delta is 1.8 GiB at 60 MiB/s. Let me think through the math first: - Final delta: 1.8 GiB at 60 MiB/s - 1.8 GiB = 1843.2 MiB - 1843.2 MiB / 60 MiB/s = ~30.7 seconds So the final delta sync takes roughly 31 seconds. That leaves ~59 seconds of headroom in the 90-second window. That's workable but tight — need to account for: - Container stop time - Final snapshot/delta creation - Delta transfer (~31s) - ZFS receive on target - Container start on target (metadata load, fstab, network bringup) - Validation gate Proxmox context: This is clearly Proxmox VE terminology (ct-214 = VMID 214, rootfs, "migrate container"). For containers on ZFS without shared storage, Proxmox supports: 1. Online migration with storage replication (pvesr) — requires ZFS replication configured, and final delta is synced at migration time. Actually, Proxmox 6+ supports online migration for containers with replicated volumes: `pct migrate` with `--online` uses replication state. The final replication delta runs after the CT is frozen/stopped? Actually for containers: with replication + online migration, the container is frozen (not stopped) for the final sync, then moved. For offline migration, container is stopped, then final sync. 2. Manual approach: zfs send/receive incremental between hosts. Given they said "Snapshot already replicated; final delta 1.8 GiB," this is the standard Proxmox replication flow: pvesr has done incremental replication, and migration will do a final incremental sync. Key considerations: - Downtime window 90s: The critical path is freeze/stop → final delta send/recv → start on target. - With online (restartless-ish) migration: Proxmox freezes the CT (FIFREEZE via cgroup freezer? Actually for containers it uses the freeze functionality — `pct` online migration for containers with replication actually stops the container briefly? Let me recall). Actually, let me be precise about Proxmox behavior: - Container live migration (online) with ZFS + replication: Proxmox docs say "If you have a container with replication configured, you can migrate it online. The migration will freeze the container during the final replication step, so downtime is minimal." This is `--online` flag with `pct migrate` (or the GUI checkbox). The CT is frozen (filesystem frozen via... actually it uses `vmstat`? No). For containers, online migration requires replication because otherwise the disk state changes. The freeze is done via the cgroup freezer or FIFREEZE ioctl on the rootfs so the ZFS dataset is quiesced for a consistent snapshot. - Offline migration: CT stops, then final sync, then start on target. Downtime = stop + sync + start. Given 90s window and ~31s transfer, either works, but online migration with freeze is best: downtime ≈ freeze + delta transfer (~31s) + unfreeze/start + network reattach. Should fit well within 90s. But wait — online migration freeze duration: during freeze, the final snapshot is taken, delta computed, sent. With 1.8 GiB at 60 MiB/s ≈ 31s. Plus ZFS dataset mount, network config on target, start. Realistic total downtime: 35–50s. Fits. If they must do offline migration: stop time depends on in-flight processes (need clean shutdown; can set --force-stop or timeout), transfer 31s, start ~5-10s. Also fits if stop is quick. Rollback plan: If the migration fails mid-way (e.g., final send fails, target receive fails), the CT remains on source (for offline, it stays stopped on source — restart it; need to roll back any replication state). If migration completes but app fails on target, roll back: migrate back (needs reverse replication to have caught up — but if CT started on target and wrote data, rolling back requires re-replicating new delta back to source). Key gates before committing: Gates: 1. Pre-flight gate: - Target host healthy: pvecm status (quorum), pvesm status, zpool health on target (zpool status -x), free space on target zpool ≥ snapshot size + delta + headroom ( replication typically needs ~110%? For full: volsize + slack; here snapshot exists on target already, so need delta + margin). - Version match: pveversion on both nodes, feature compatibility (online migration requires both nodes support it). - Replication state current: `pvesr list` / check job last-sync, verify last snapshot on target matches expected. Confirm incremental delta estimation ~1.8 GiB (zfs send -nvP). - Network: dedicated replication network throughput; verify 60 MiB/s measured (or do a dry-run iperf3 between nodes). Latency low. - App-level: DB quiesce plan (optional), open file handles, big writers paused. - Confirm no snapshot holds (zfs holds), no pending `zfs receive` sessions. - CT 214 config: check `pct config 214`, unprivileged, features (nesting, keyctl), mountpoints other than rootfs? If extra mount points exist, each must be replicated/migrated too. Note that. - Backup recent (vzdump) for worst case. - Dry run: `pct migrate 214 target --online` isn't dry-runnable; but replication is continuous anyway, so final delta only. 2. Go/No-go gate (T-0): - Re-verify quorum and replication freshness (< e.g., 5 min old). - Announce downtime start, stop app writers if needed. 3. Migration execution: - `pct migrate 214 --online` (with replication configured) — final replication + move. - Monitor: `pvesr` logs, `zfs list -t snap`, task log in /var/log/pve/tasks. - Timing checkpoints: at T+45s (transfer should be ~done); if >70s and transfer <80% complete, consider abort criteria. 4. Post-migration validation gate (before declaring success / releasing window): - CT running on target: pct status 214, pct config 214 - Network reachable: ping, service port checks, app health endpoint. - Logs clean on target. - Replication direction now target→source? Actually after migration, replication jobs need updating: job's target must be changed or old job removed; Proxmox warns to reconfigure. Note: after migrating, existing replication jobs referencing old setup will error; update via GUI/API. - Old snapshots on source: cleanup later. Rollback scenarios: A. Failure before freeze/final sync starts (preflight fail): no action, CT stays on source, running. Just abort. B. Failure during final sync (online migration): Proxmox should keep CT on source; frozen then thawed automatically on failure. Verify running, re-run replication to converge, retry later. If CT left in frozen state (edge), `pct` thaw: actually there's `pct` freeze/thaw? There is `lxc-freeze`/cgroup; can do `pct` ... hmm, manually: echo thaw to cgroup freezer: /sys/fs/cgroup/lxc/214/... or `lxc-checkpoint`? Simpler: pct stop/start or use `pct` ... For rollback docs: ensure process not left frozen: check /proc//status or `pct status`. Restart if needed. C. Offline migration failure after stop, before start: CT stopped on source. Roll back: start on source (pct start 214), abandon migration. Data intact on source; target has stale replica. Converge replication later. D. Migration succeeded (CT on target) but app broken / exceeds window: Decision point. Rolling back means migrating back — needs reverse replication of new writes. If downtime window blown and app must be up fast: better to stay on target and fix forward (network/DNS), since rollback costs another ~31s+ sync. If app-level corruption suspected and data on target writes are disposable for the minutes since cutover, can do: stop CT on target, reverse-replicate final snapshot back (zfs send -I target@snap source dataset), move back. Realistically: fix forward within window; rollback only for unrecoverable target-side failure. E. Catastrophic: both sides diverge/broken — restore from vzdump backup taken preflight. Manual zfs alternative (if not using pvesr/pct online): - On source: zfs snapshot tank/subvol-214-disk-0@final (while CT stopped or frozen) - zfs send -I @last @final | ssh target zfs receive -s -F tank/subvol-214-disk-0 - Move config /etc/pve/nodes//lxc/214.conf to target node dir (only valid with CT stopped; pmxcfs handles via pct/qm or mv within /etc/pve — actually you can mv the config file between node dirs when CT is stopped). - Import/adjust: if hostname/machine-specific changes needed. - Start on target. This is the manual fallback; but recommend native pct migrate with --online since replication exists. Timing budget for 90s window: - T-5m to T-0: preflight gates. - T-0: begin (online: freeze). Freeze ~1-2s. - Snapshot + estimate: zfs snapshot is fast; delta send begins ~T+5s. - Transfer: 31s → done ~T+36s. - Receive commit, config move, start: ~T+40–50s. - Validation (ping, ports, app check): T+50–70s. - Declare success by T+75s; abort threshold at T+60s if transfer <90%: abort and thaw/restart on source (unfreeze ~2s, or start ~10s), service restored by T+75s. Wait — abort decision logic: if at T+45s transfer isn't ~complete, something's off with throughput (60 MiB/s assumed). At T+60, if not started on target, consider abort. Abort cost: unfreeze on source (online path) = seconds; CT resumes on source. That's the beauty of online migration: abort is cheap. Also note: 60 MiB/s ≈ 500 Mbps — that's slow for a LAN; likely limited WAN link or SSH encryption overhead. Could use mbuffer or faster cipher (aes128-gcm@openssh.com), or dedicated replication network. But if pvesr measured that, fine. Mention compression: zfs send -c (compressed send) if pool has compression — big win if not already. Actually if replication already running at these speeds, the final delta at 1.8 GiB is post-snap; using raw/compressed sends could cut time. But keep it simple: mention as optimization. Also: the 1.8 GiB delta is the estimate now; during freeze the actual delta may be larger if writes continue until freeze. Online migration freeze minimizes that. If offline migration, writes stop at shutdown so delta fixed after stop. Another subtlety: For online CT migration in Proxmox, requirement: all volumes replicated to target, replication snapshots present, `--online` given. It does freeze via... it actually stops the CT? Let me recall precisely. From Proxmox docs (pct migrate / storage replication chapter): "If you migrate a running container with --online and all volumes are already replicated, the container is frozen during the final replication, so downtime is minimal." Yes — freeze not stop. On failure, thaw. Also note CTs can't do live migration without replication for ZFS volumes (unlike qm with --with-local-disks for offline... containers use replication only). So the plan aligns. DNS/ARP: after cutover, ARP caches may hold old MAC — for CTs on same bridge/network, the VM/CT keeps same MAC if bridge config identical; ARP flush may be needed; gratuitous ARP usually sent by VM/CT networking on start. Minor: mention verifying IP reachability, firewall rules on target node, bridge/VLAN tag match in net0 config. Also check: target node has same bridge name (vmbr0), VLAN aware settings; CT net0 hwaddr preserved. Storage pool mapping: target storage name for rootfs must exist (e.g., local-zfs). `pvesm status` on target. Replication job cleanup post-migration: old job (214-0 to old target) — after migration the job direction must be updated: in Proxmox, after moving, the replication job should be recreated with new source/target (or GUI warns "migration with local replicated volume..."). Post-task, update jobs so future HA/replication works from new host. Also HA: if CT is HA-managed, use ha-manager migrate instead? If `ha-manager status` shows 214, do `ha-manager migrate ct:214 `. Worth a gate check. Let me now structure the answer: Format: This is an ops runbook request. The user is terse and expert. They want: plan with gates + rollback. So deliver a runbook: timing budget (show the math), pre-flight gate checklist with commands, execution sequence with monitoring points and abort criteria, validation gate, rollback matrix by failure phase, and notes/optimizations. Use headers, tight bullets, code blocks for commands. Keep it scannable. Moderate length — comprehensive but no fluff. Tables could help for timing budget and rollback matrix. Markdown is appropriate (ops context, likely rendered). Length: aim for a solid runbook but not bloated. Maybe 400-700 words equivalent with code blocks. The user is expert; keep commands precise. Let me draft: ## Budget (why 90s works) - Final delta 1.8 GiB @ 60 MiB/s ≈ 31s transfer - Freeze/ thaw: ~2s - Start on target + net bring-up: ~5–10s - Validation: ~10–15s - Planned downtime: ~45–55s; ~35–45s headroom = abort/rollback margin. - Abort threshold: T+60s. Caveat: delta grows with writes until freeze; freeze closes it. If offline migration, stop stops writes. ## Gate 0 — T-15m preflight (all must pass) - Cluster: `pvecm status` quorum on both; no node flapping. - Target pool: `zpool status -x` = healthy; `zfs list -o space ` free ≥ 4–5 GiB (delta + slack) — actually target already has snapshot; needs delta + temp. Say ≥2× delta. - Feature parity: `pveversion -v` major/minor equal; check CT features (nesting/keyctl) supported. - Storage name for rootfs exists on target (`pvesm status`). - Replication state: `pvesr list`; last sync fresh (<5 min); target has snapshot @ current. Verify with `zfs list -t snap -r ` on both. - Dry-run delta estimate: `zfs send -nvP -I @lastsnap @latest | tail -1` to confirm ~1.8 GiB. - Throughput: recent replication rate ~60 MiB/s observed; optionally iperf3. - CT config: `pct config 214` — note extra mount points (each must be replicated), net0 bridge/VLAN exists on target, firewall. - Not HA-managed, or handle via ha-manager: `ha-manager status | grep 214`. - Fresh backup: `vzdump 214 --mode snapshot --storage ` — worst-case restore path. - No zfs holds: `zfs holds -r`. - Decide: online (`pct migrate 214 --online`) — freeze-based, recommended. ## Gate 1 — T-5m go/no-go - Re-check quorum, replication freshness, delta estimate. - Quiesce heavy writers / checkpoint DBs (optional but shrinks delta). - Freeze change freeze on CT config (no config edits during window). - Announce start. ## Execution — T-0 - `pct migrate 214 --online` (or GUI with online checkbox). Run in one SSH session to source; monitor task: `tail -f /var/log/pve/tasks/...`. - Checkpoints: - T+5s: task shows final replication started (frozen state). - T+20s: transfer ≥50% per task log. - T+40s: transfer complete, config moved, start issued. - T+60s GO/ABORT: if CT not starting/started on target, abort (see rollback B/C). ## Gate 2 — validation (T+50–75s) - `pct status 214` on target = running. - IP reachable (ping), service port responds (e.g., `ss -tlnp` inside, curl health endpoint from outside). - Logs: `pct exec 214 -- journalctl -b -p err -n 20` clean-ish. - Verify disk is target-local: `pct config 214` rootfs → target node storage; `zfs list` shows dataset with new snaps. - If pass by T+75s: done. Update replication jobs (direction/source), DNS if any node-scoped records, monitoring. ## Rollback matrix - A: Preflight fail → no-op; CT untouched on source. - B: Fail before/during final sync (online) → task auto-thaws; verify `pct status 214` running on source; if left frozen, `pct stop/start 214` or thaw cgroup (`echo 0 > /sys/fs/cgroup/lxc//cgroup.freeze` cgroup v2 or lxc-freeze -u). Converge replication (`pvesr run `), reschedule. - C: Fail after sync, before/at start on target → CT stopped: `pct start 214` on source (data intact), abort migration; total downtime = stop duration so far. - D: On target but broken after cutover → prefer fix-forward (net/bridge/firewall on target) — rollback costs another full reverse sync (~31s+). If unrecoverable by T+85s and writes on target are expendable: stop on target, `zfs send` latest snap back (or `pct migrate 214 --online` if replication reconfigured), start on source. Time cost: ~45–60s more. - E: Data corruption/unsalvageable → restore from preflight vzdump on either node. ## Post - Recreate/redirect replication job (now target→source if you want fallback), remove stale snapshots on old node later, verify monitoring/dashboards, update docs/CMDB. Notes/optimizations: - 60 MiB/s smells like SSH cipher overhead or WAN; if transfer is the risk, use compressed sends (`zfs send -c`) if pool compressed, mbuffer, or dedicated replication net — but don't change mid-window. - During freeze, writes pause; delta measured at freeze is final. If a writer can push >60 MiB/s sustained, delta grows pre-freeze; quiesce big writers at T-5m. One nuance: with `--online`, Proxmox CT migration actually requires that it's not "hotplug"... no. Requirements: online migration of containers is only possible with replicated volumes (ZFS) — yes. Also the command: `pct migrate [--online]`. For HA: `ha-manager migrate ct:214 target` — actually the ct:214 syntax: `ha-manager migrate ct:214 `. Fine. Thaw command: modern cgroup v2: /sys/fs/cgroup/lxc.payload.214/... hmm, on Proxmox, cgroup paths: /sys/fs/cgroup/lxc/214/ (v1) or lxc.payload.214 (v2 hybrid). Simpler to say: `lxc-unfreeze -n 214`? lxc-tools may not be installed; pct doesn't expose thaw directly. Safe guidance: check `pct status` shows running; if processes frozen, `pct restart 214` won't work while frozen? `pct stop` sends signals which may be frozen... In practice, failure during final replication thaws automatically (task abort path does thaw). I'll note the automatic thaw and manual fallback: kill task → thaw, else `pct stop --force-start`... Keep it brief: "task abort path thaws automatically; verify a process is actually scheduled (`pct exec 214 -- true`); if stuck frozen, `pct stop 214 && pct start 214`". Also mention checking that no other replication sync in-flight at T-0 (pvesr job lock) — the migration final sync needs the replication lock; a running job could delay. Gate: check `pvesr list` / no active sync (`ls /run/pve/locks/` or task list showing pvesr running). Extra mount points note: mp0 etc. must also be replicated to target or migration fails/warns. Include in Gate 0. Also bridge check: `cat /etc/network/interfaces` on target or `pvesh get /nodes//network`. Timing math recheck: 1.8 GiB = 1.8 * 1024 = 1843.2 MiB; /60 = 30.72 s. Yes ~31s. Abort threshold: The critical unknown is throughput. If actual drops to 30 MiB/s, transfer = 61s; total ~75s; still inside but validation squeezed. Rule: at T+35s expect ≥50% (proportional). Simpler: hard abort at T+60s if not yet started on target. I'll include a simple checkpoint table. Structure the response with headers, minimal prose, commands in code blocks. A small table for the timeline/checkpoints and one for rollback phases. Keep tight. I should also give the offline alternative one line, since maybe they intend offline stop-migrate-start. But primary plan: online freeze-based. Mention that if they planned offline, downtime = stop time + 31s + start; ensure clean stop within e.g. 20s (`--force-stop 1`? pct migrate has `--force-stop`? There is `--force-stop` for stop mode... `pct migrate` options: --online, --force-stop? Actually `pct shutdown` has timeouts; migrate offline just stops? qm migrate has --force-stop; pct migrate I believe also accepts --force-stop? There's `--force-stop` in qm; for pct, migrate options include --online and --force-stop? Checking memory of API: /nodes/{node}/lxc/{vmid}/migrate with params: target, online, force (bool) — "force" for pct? I recall `--force-stop` exists on qm migrate; for pct it's `--force`? To be safe, I'll not assert the flag name; I can say "online (--online) recommended; if you must go offline, expect stop-time + 31s + start-time." Avoid stating uncertain flags. Actually pct CLI: `pct migrate [OPTIONS]` with --online bool. I'm fairly confident --online exists. --force-stop: qm has it; pct... In PVE::API2::LXC::Status? The migrate endpoint for lxc has 'force-stop' parameter? Hmm. I'll just mention --online confidently and leave offline unflagged. HA check command: `ha-manager status`. If managed: `ha-manager migrate ct:214 ` handles it. Now write it. Keep it disciplined: runbook style, expert tone, no hand-holding explanations of basics. Include the arithmetic up front since they gave numbers — shows the budget works and where the margins are. Length target: this deserves a fairly complete runbook; maybe ~600-800 words with code blocks. Use tables sparingly (timeline + rollback). Headers per gate. Avoid over-nesting bullets. # Migration Plan: ct-214 (ZFS rootfs, no shared storage) ## Budget check (why 90s holds) | Phase | Time | |---|---| | Freeze CT + final snapshot | ~2–5s | | Final delta send/recv (1.8 GiB @ 60 MiB/s) | ~31s | | Config move + start on target | ~5–10s | | Validation | ~10–15s | | **Total planned downtime** | **~50–60s, ~30s margin** | Delta is fixed at freeze (online) or shutdown (offline) — writers before that point add to it. Hard abort line: **T+60s**. **Use `--online` migration** (freeze-based). Abort cost is seconds; offline abort costs a full restart. --- ## Gate 0 — T-15m: Pre-flight (all must pass) ```bash # Cluster + target health pvecm status # quorum on both nodes ssh zpool status -x # pool healthy ssh pvesm status # rootfs storage name exists ssh zfs list -o space # free ≥ 2x delta # Version/feature parity pveversion; ssh pveversion # same major.minor # Replication state fresh — no sync in-flight, last sync < 5 min pvesr list zfs list -t snap -r # on BOTH nodes — matching @last snapshot # Confirm delta estimate matches 1.8 GiB zfs send -nvP -I @

@ | tail -1 ``` Also verify: - `pct config 214`: any extra mount points (`mp0`+) must be replicated too, or migration fails. Net0 bridge/VLAN exists on target with matching config. Same MAC is preserved. - Not HA-managed (`ha-manager status`); if it is, use `ha-manager migrate ct:214 ` instead of `pct`. - No zfs holds on the dataset (`zfs holds -r`). - Fresh backup exists: `vzdump 214 --mode snapshot --storage ` — this is the E-path recovery. - Throughput: 60 MiB/s ≈ what recent replication runs showed. If it's SSH-cipher-bound, note it but **do not change transfer methods mid-window**. ## Gate 1 — T-5m: Go/No-Go - Re-check quorum + replication freshness (a stale replica inflates the final delta). - Quiesce heavy writers / checkpoint DBs — anything writing >60 MiB/s sustained outruns the sync. - Config change freeze on CT 214. Announce window start. ## Execution — T-0 ```bash pct migrate 214 --online # monitor: tail -f /var/log/pve/tasks/index.html # or the specific task log ``` | Checkpoint | Expect | If not | |---|---|---| | T+5s | Task shows final replication, CT frozen | Abort now (cheap) | | T+35s | Transfer ≥ ~50% | Throughput degraded — prep abort | | T+45s | Transfer done, start issued | — | | **T+60s** | **CT starting/started on target** | **ABORT** | ## Gate 2 — Validation (T+50–75s) ```bash pct status 214 # running, on target ping && curl -m 5 pct exec 214 -- journalctl -b -p err -n 20 pct config 214 # rootfs now on target-local storage ``` Pass by T+75s → done. Post-cutover: re-point replication job (now target→source if you want a fallback path), remove stale snaps on old node, update monitoring/CMDB. --- ## Rollback matrix | Phase | Failure | Action | Cost | |---|---|---|---| | A | Any Gate 0/1 fail | No-op — CT untouched on source | 0 | | B | Before/during final sync | Task abort path auto-thaws. Verify: `pct status 214`, `pct exec 214 -- true`. If stuck frozen: `pct stop 214 && pct start 214`. Re-run `pvesr` job to converge, reschedule. | ~5–15s | | C | After sync, before/at start on target | CT is stopped on source with **intact data**. `pct start 214` on source, abandon migration. | seconds | | D | On target, app broken post-cutover | **Fix forward** (bridge/firewall/DNS on target) — rollback costs another ~31s+ reverse sync you don't have. Only if unrecoverable by T+85s *and* post-cutover writes are expendable: stop on target, send latest snap back, start on source. | 45–60s more | | E | Data corruption either side | Restore vzdump from Gate 0. | minutes — this is why the backup gate is mandatory | --- ## Notes - Online migration freeze means the 1.8 GiB estimate only holds if no writer pushes faster than 60 MiB/s between Gate 1 and freeze — that's the main schedule risk. - Biggest failure mode for D is network: MAC is preserved but verify the target bridge is VLAN-tagged identically before you cut. Check that in Gate 0, not at T+50. - 60 MiB/s is slow for a LAN — if it's SSH overhead, `mbuffer` or a faster cipher cuts the 31s meaningfully, but only change it *between* windows, with a test replication run after.

SCB SCB-migration-T1 MUSE