Let me parse this conversation. The user says "Same, plus: one log is unlinked-but-open (63 GiB) and backup was taken without consistency guarantees. Does this change the verdict? Which recovery options are now dead?" This references a prior conversation that I don't have visibility into — the conversation starts with "Same, plus..." implying there was a previous exchange about a data recovery scenario, likely PostgreSQL (given terminology like "log", "unlinked-but-open", recovery options). Actually, the phrasing "one log is unlinked-but-open (63 GiB)" strongly suggests PostgreSQL WAL segments — an unlinked-but-open WAL file is a classic scenario where the file has been deleted from the filesystem but a process still holds an open file descriptor. In PostgreSQL, this happens when pg_wal (pg_xlog) files are deleted while the server is running, e.g., someone runs `rm` on pg_wal files to "free space." The file descriptor remains open in the postmaster's children; the file is still reachable via /proc//fd/ until the process exits. "Backup was taken without consistency guarantees" — a backup (e.g., filesystem-level copy, or maybe pg_basebackup without -X stream/checksums, or a snapshot taken without proper coordination) that isn't guaranteed consistent. Actually "without consistency guarantees" could mean a tar copy of the data directory taken while the server was running, or rsync without --checksum, or a backup without the WAL needed for consistency. Since I don't have the previous conversation, I need to reconstruct the likely scenario from "Same, plus". The word "Same" indicates this is a follow-up. Common scenario: PostgreSQL data recovery question where someone deleted WAL files or lost data files, and the verdict was something like "you can recover via X, Y, Z" and now with these two new facts, the question is whether the verdict changes and which options are dead. Let me think about what a typical prior exchange would be. A plausible scenario: "WAL files deleted from pg_wal while server running; can I recover?" The original verdict might have listed options: (1) take a filesystem-level copy immediately, (2) use pg_basebackup, (3) restore from backup, (4) dump data with pg_dump, (5) crash recovery on clean shutdown. Now the new facts: 1. One log (WAL segment) is unlinked-but-open, 63 GiB — wait, 63 GiB is huge for a WAL segment. Standard WAL segments are 16 MB by default. 63 GiB is not a WAL segment size. Unless it's a custom wal_segment_size — max is 1 GB with pg_wal file... Actually wal_segment_size can be set between 1MB and 1GB. 63 GiB doesn't fit. So maybe "log" refers to the PostgreSQL server log (log file) — but a 63 GiB server log being unlinked-but-open doesn't matter for recovery much; actually it could matter if... hmm. Or maybe the scenario is about MySQL — InnoDB redo logs? MySQL ib_logfile0/ib_logfile1 are typically 48MB or so, but can be huge. 63 GiB... could be. Or maybe it's a generic "database log" in a crash-recovery theory question. Alternatively "log" could be a redo log / transaction log of some database, and the size 63 GiB suggests it's the whole active log, not one segment. Hmm. Alternatively, this could be about a filesystem-level scenario: someone deleted files including the database log, one of which is still held open by a process. The classic advice: before doing anything, copy the unlinked-but-open file from /proc//fd/ using cat or cp to a safe location, because once the process exits, the data is gone. "Backup was taken without consistency guarantees" — e.g., a backup taken with cp/rsync while the DB was running, or a snapshot at the storage layer without quiescing I/O (though crash-consistent snapshots are usually OK for databases if the storage guarantees atomicity at some point in time... "without consistency guarantees" might mean even crash-consistency is not guaranteed, e.g., a backup made over NFS or without fsync ordering, or rsync running while files changed so it's a torn mix of points in time). So the question: does the verdict change? Which recovery options are now dead? Given I lack the prior context, I should handle this gracefully. The best approach: acknowledge I'm continuing the scenario, state my assumptions about what "same" refers to (the prior scenario), and answer conditionally but concretely. But wait — as the assistant, in this simulated conversation, the previous turn would have been my answer. Since I can't see it, I need to reconstruct a plausible "same" and answer. The instruction says the conversation starts here; the human turn references prior content that isn't shown. I should probably reconstruct the likely scenario and answer, perhaps with a brief note of the assumed baseline. Actually, in these "conversation continues" setups, the expected behavior is often to just answer as if I know the prior context, inferring from the message. Let me infer the most likely prior scenario: Prior scenario (reconstructed): A PostgreSQL server where WAL files were deleted (or pg_wal got wiped) while the server was running. Possibly also the disk filled, or someone rm -rf'd part of the data directory. The prior verdict might have been: "Recovery is possible via these routes: (a) copy open FDs from /proc before restarting, (b) pg_dump while still running, (c) restore from backup + WAL, (d) let crash recovery handle it if the deleted files are recreated on restart..." Something like that. New facts and their impact: 1. Unlinked-but-open 63 GiB log: If that's the active WAL/redo log, this is actually GOOD news partially — the data is still reachable via /proc//fd until the process closes it. Immediate action: copy it out NOW (cp /proc//fd/N /safe/place) before any restart or kill. 63 GiB copy takes time — need to make sure there's space. Also note: reading from /proc fd gives the whole file; for a deleted file, reading works fine. But if the log is a log file (like postgresql.log or slow log), it matters less for crash recovery but might contain needed forensic info. Hmm, but "log" in DB recovery discussions typically = transaction log/redo log. 63 GiB though... For MySQL 8, redo log files default innodb_redo_log capacity... In MySQL 8.0.30+, redo is a set of 32 files in #innodb_redo; total capacity can be large. 63 GiB redo is unusual but possible on heavy-write systems. For PostgreSQL, WAL segments are small but pg_wal could hold gigabytes; a single file can't be 63 GiB. So maybe the scenario is MySQL: someone deleted ibdata/undo/redo while mysqld running. The classic MySQL recovery: /proc//fd trick to recover deleted InnoDB files. Alternatively, the scenario might be MongoDB (WiredTiger journal files) — journal files are ~100MB each. 63 GiB doesn't fit either. Or maybe it's about Kafka logs (segments can be 1GB each, not 63). Or Elasticsearch translog (up to 512MB default per... no). Hmm, 63 GiB single log file — could be Oracle redo log? Oracle redo logs can be large, even tens of GB in extreme setups. Or could be a SQL Server transaction log (.ldf) — SQL Server transaction logs absolutely can be 63 GiB! A single .ldf of 63 GiB is plausible on a busy SQL Server. "Unlinked-but-open" — SQL Server on Linux keeps the file open; on Windows you generally can't delete an open file unless opened with FILE_SHARE_DELETE, though SQL Server does allow delete? Actually on Windows, you typically can't delete open files. On Linux, SQL Server files can be deleted while open. Alternatively, the prior conversation might be about MySQL: someone deleted the MySQL data directory files (ibdata1, redo logs, .ibd files) while mysqld was still running. Classic recovery: DO NOT STOP MySQL. Immediately copy everything from /proc//fd. Options: mysqldump while running, or copy files, or take an LVM snapshot. The "verdict" might have been "you're fine, several options: dump, snapshot, copy /proc FDs, stop gracefully." Now new facts: one log unlinked-but-open 63 GiB (so it's huge — copying takes time and space), and backup was taken without consistency guarantees (so the backup is not a reliable restore point — it's a torn/possibly inconsistent copy). Does this change the verdict? Yes, partially: - The backup route is now suspect/dead-ish: restoring an inconsistent backup may not be usable; crash recovery on restore requires a consistent point-in-time image (crash-consistent at minimum). If the backup is a torn copy (files copied at different times, no snapshot), recovery from it is not guaranteed and may fail or corrupt. So "restore from backup" is degraded from "safe option" to "last resort." - The 63 GiB unlinked log: still alive but on a timer — it lives only as long as the process holds the FD. It's fragile: any restart, crash, OOM kill, power loss kills it. Also copying 63 GiB needs 63 GiB free space and time (minutes to hours). But its existence means the redo chain might still be complete if you copy it. Which options are now dead? - If the process is still running and the DB still answers queries: pg_dump/mysqldump still works (if we're in the MySQL scenario, mysqldump while files are deleted works because the server has them open). Not dead. - Graceful shutdown + restart: DEAD. Once the process closes the FD, the 63 GiB log evaporates. A clean shutdown may checkpoint and... in PostgreSQL, shutdown checkpoint writes to WAL then the FD closes at exit; without the file on disk, restart would fail (or PG recreates empty WAL and you get a "clean shutdown" but the data written since deletion? Actually if WAL file is deleted, on shutdown checkpoint PG tries to write to the unlinked FD — fine — then on restart, pg_wal is empty, PG creates a new segment and starts at the checkpoint LSN recorded in pg_control... but pg_control itself was deleted? If only some files deleted...). This is getting too scenario-specific. Let me step back. The safest approach: since I genuinely don't have the prior turn, I should briefly anchor assumptions ("Assuming the baseline we discussed: ...") and answer. The user expects a continuation. The typical persona for such questions is an expert DBA/SRE discussion. The phrase "Does this change the verdict?" and "Which recovery options are now dead?" suggests my previous answer enumerated recovery options and gave a verdict (e.g., "recoverable, low risk"). Now they add two adverse facts and ask if the verdict changes. Given ambiguity about which DB, PostgreSQL is the most common in such discussions, and "log" + "unlinked-but-open" + backup consistency is a classic PG recovery question. But 63 GiB single log file doesn't fit PG WAL (max segment 1GB with --wal-segsize... actually since PG 11, wal segment size can be up to 1GB). Unless it's the log file... "one log is unlinked-but-open (63 GiB)" — maybe it's not a WAL segment but the message log? Hmm, in recovery context, if pg_wal files were deleted, sizes are multiples of segment size. 63 GiB = 64512 MiB; if segments are 1GB (max), that's 63 segments. Not one file. MySQL InnoDB: in 8.0.30+, redo log capacity up to... #innodb_redo files, each 1/32 of capacity, max capacity 128GB. So a single redo file could be 4GB max. Not 63 GiB. SQL Server .ldf: yes, can be 63 GiB. SQL Server on Linux, deleted while open — plausible. Oracle: redo log group members can be huge; 63 GiB possible in extreme cases but rare. Kafka: segment files, max typically 1GB, can be configured to 63GiB in theory — unlikely. Zookeeper transaction log? No. WAL of SQLite? No. Actually maybe the scenario is not database-specific — maybe it's a general filesystem/data-recovery question: "logs deleted while service running; backup without consistency." The term "verdict" suggests a recovery triage. Could be about a generic "log-structured storage" or maybe about Elasticsearch? Elasticsearch translog files can total large but per-file... Hmm, 63 GiB... could this be about PostgreSQL's pg_wal total? "one log is unlinked-but-open (63 GiB)" — one file though. Wait — maybe it's MySQL's general query log or binlog! MySQL binlog files can be large — max_binlog_size default 1GB, but the currently-open binlog is unlimited until rotation. A busy primary could have a 63 GiB binlog if max_binlog_size... no, rotation happens at max_binlog_size (default 1GB, max 1GB). Actually max_binlog_size max is 1GB. Hmm, but the active binlog can exceed? No, it rotates. SQL Server transaction log seems most plausible for 63 GiB single file. But then "backup without consistency guarantees" — SQL Server backups (native) are always consistent unless taken via file copy. A file-copy backup of a running SQL Server = inconsistent. So scenario: someone copied the MDF/LDF files while SQL Server was running (or after crash?), and deleted... this is getting too speculative. Alternative: the prior conversation was generic data-recovery triage, maybe about a VM/host where logs (journald?) ... "recovery options" though implies DB. Let me reconsider: perhaps the prior scenario was about PostgreSQL with a deleted pg_wal, and the "63 GiB" log is... the unlinked-but-open file might be the last WAL segment? No, size wrong. Actually, maybe "log" = the database's transaction log as a whole (e.g., "the log directory"), and one 63 GiB log file is something like an Oracle archive log or an undo tablespace? Or — InfluxDB WAL? InfluxDB WAL files can be large. Or TimescaleDB? Same as PG. Or maybe it's about Kafka again: Kafka log segment... no. Or ClickHouse? Parts/wal... Alternatively — maybe the prior conversation was about MySQL 8 with innodb_redo_log_capacity and someone deleted redo; "one log" = one #innodb_redo file? Sizes wrong. Honestly, the strongest signal for the intended scenario: "unlinked-but-open" + "backup without consistency" + "recovery options" + "verdict" — this is classic PostgreSQL disaster recovery: someone deleted files from pg_wal (or the whole data dir) while PG running. Prior answer likely: options were (1) pg_dump now, (2) pg_basebackup, (3) copy /proc FDs, (4) LVM snapshot, (5) clean shutdown might actually work because PG writes new WAL... and verdict: "recoverable with care; do X first." But 63 GiB doesn't fit PG WAL segment. Unless the PG instance uses huge segment size... max 1GB. So no. Hmm — what if "log" refers to the postmaster's log file (postgresql log), and the deleted-but-open 63 GiB is the server log? Then its relevance: the log might contain evidence, but not recovery-critical. That wouldn't change the verdict much... unless the log is needed to determine the consistency point of the inconsistent backup! Interesting: if the backup was taken without consistency guarantees (e.g., rsync of PGDATA while running), you need WAL covering the backup period to make it consistent on restore. If WAL was also deleted... The 63 GiB open log might be the only surviving WAL-ish artifact. Wait — maybe the DB is Oracle and "log" = redo log; or maybe it's Informix? Or DB2? Alternatively, the scenario might be about a log-structured KV store like RocksDB? RocksDB WAL files can be large. Or LevelDB. Or maybe the prior scenario: "database server, disk full, someone truncated/deleted the transaction log to free space; server still running." Common in SQL Server world: shrink/truncate log. The unlinked-but-open 63 GiB file = the deleted transaction log held by SQL Server on Linux. Backup without consistency = someone copied files while running (not via BACKUP DATABASE). Verdict before: "you can recover: take a proper backup now (BACKUP DATABASE ... WITH COPY_ONLY? or native backup works even after log deletion?), or dump, or..." Now: with the log deleted, SQL Server can still take a native backup (it reads open FDs). But a native backup is consistent. Hmm, "backup was taken without consistency guarantees" — so the only existing backup is bad. I think I'm overanalyzing the specific engine. The user's question is answerable at a general level with concrete mechanics, because the two added facts have generic implications: Fact A: One log file (63 GiB) is deleted on disk but still held open by the running process. Implications: - The data still exists, but only in the page cache + disk blocks referenced by the open inode. It disappears when the last FD closes (process exit, crash, reboot, OOM kill, systemd restart). No fsck/photorec after that (well, extundelete/testdisk might find blocks, but unreliable on a busy fs). - It is recoverable right now via /proc//fd/ (Linux). Copy with cp/cat (or dd). Need 63 GiB free space elsewhere. Time: minutes–hours depending on storage; do it before anything else if that log matters. - Reading from /proc/.../fd/N of a deleted file: you can read the whole file (lseek to 0). Size shown by stat via fd link: the inode size. Works. - Danger: if the process rotates/truncates that log (e.g., log rotation sends SIGHUP, or the DB itself recycles it, e.g., WAL recycling renames... wait rename doesn't unlink; but if the DB's internal rotation unlinks and creates a new file, the FD... the old FD stays but data may be truncated if reopened with truncate). Any of those kills it. - Also: the 63 GiB of disk space is NOT freed; if the original incident was disk-full, df still shows usage. That's a diagnostic clue. Fact B: The backup was taken without consistency guarantees. Implications: - A file-level backup of a running DB without a snapshot/stop/checkpoint coordination is a torn copy: different files captured at different times. For most engines, restoring it requires the transaction log covering the entire capture window, plus the log must be applied from a state where... actually for a torn backup, you need recovery to a single point in time, which requires that the log covers all in-flight changes and that log ordering aligns. In practice: torn file copies often fail to recover or, worse, recover "successfully" with latent corruption. Even crash-consistent (atomic snapshot) backups are fine (DB does crash recovery), but "no consistency guarantees" is worse than crash-consistent — it's a smeared capture. - So restore-from-backup is no longer a trusted primary path. It becomes: last resort, or usable only if you can pair it with logs that span the capture window and the engine tolerates it (usually it doesn't guarantee anything). - If the backup is a tar/rsync taken while running, it may be missing or have partial final files, and the "backup label"/checkpoint LSN pairing is absent. Now: does this change the verdict? The likely prior verdict: "recoverable — several independent options; low panic." New verdict: "still recoverable, but the safety net is gone; you're now in a race; the running process's open FDs are the source of truth; priority order changes; stop making it worse." Which options are now dead (or effectively dead)? 1. Restart / graceful shutdown and let the DB's own crash recovery fix it: DEAD. Shutdown closes the FD; the 63 GiB log vanishes; also crash recovery needs on-disk logs. Restart is now the single worst action. 2. Reboot the host: DEAD (kills FD). 3. Rely on the existing backup as a clean restore point: DEAD or demoted to "untrusted last resort." Restore-from-backup no longer provides a guaranteed consistent baseline. (Unless the engine's own consistent-backup mechanism was used — but "without consistency guarantees" rules that out.) 4. Any action that triggers log rotation/overwrite: dead (e.g., letting log rotation run, or tools that HUP the process; in PG, letting a checkpoint recycle WAL... though WAL recycling renames rather than deletes; but if the file is already unlinked, recycling semantics differ). 5. If "consistency-guaranteed backup" was previously listed as one of the independent recovery routes — that route is dead. Remaining live routes: live dump (pg_dump/mysqldump/export), consistent backup taken NOW via proper mechanism (native backup tooling, pg_basebackup, snapshot with checkpoint coordination), and /proc FD copy of the unlinked log + other open files. 6. Forensic recovery after the fact (extundelete/photorec/scalpel on the freed blocks) — was maybe listed as a fallback before; now it becomes both less necessary (data still held open) and still unreliable; but if the FD closes, that fallback is weak. Also new complications: - Copying 63 GiB takes time; during that window, risk of process death = risk of total loss. Mitigation: first take an atomic-quality snapshot if possible (LVM snapshot, storage snapshot, cp --reflink on btrfs/XFS with reflink? Actually reflink copy of a deleted file via /proc: cp --reflink=auto /proc/PID/fd/N dest — reflink requires same filesystem; copying to another FS won't reflink. On the same FS it can be near-instant and cheap (CoW). But careful: if disk was full, reflink needs little space — good. If btrfs, cp --reflink works across... same fs only. XFS reflink same fs. If destination is another volume, no.) - Check free space before copying: need ≥63 GiB free on destination. - Alternative: use LVM/storage snapshot of the whole volume: instant, captures page cache? No! Snapshot captures disk state, not page cache. Deleted-but-open file's blocks are still allocated on disk (ext4: inode with i_nlink=0 but open FD keeps blocks). An LVM snapshot copies the block device, and the unlinked file's data blocks are still allocated in the filesystem metadata? Hmm: on ext4, when you unlink a file that's still open, the inode is marked deleted (i_nlink=0), blocks are NOT freed until last close. But the directory entry is gone. So in the snapshot, the file is present as an orphan inode (nlink=0). You can't see it via the filesystem namespace, but the blocks are there. Recovering it from the snapshot would need ext4 forensic tools (debugfs can find orphaned inodes? ext4 keeps orphan list in the superblock! Orphan inodes are listed; debugfs can list them). Actually yes: ext4 maintains an orphan inode list precisely so that crash recovery can delete them; debugfs can read it. On XFS, unlinked-but-open inodes are tracked in the AGI unlinked hash — xfs_db can find them. So an LVM snapshot preserves the data even though namespace is gone — and crucially the snapshot is atomic. But then you'd restore the snapshot and use fs tools to extract the file. That's an advanced but valid route. Simpler: /proc copy. - But caution: page-cache dirty pages! The process may have written recently to that log file with buffered I/O; the unlinked inode's data on disk may be stale until flush. Copying from /proc//fd reads through the page cache, so you get current data including dirty pages. An LVM snapshot only has what's been flushed! If the log was written with buffered stdio (like a query log), recent tail may be unflushed. DB redo logs are usually O_DIRECT or O_SYNC...ish (e.g., MySQL redo: O_DIRECT? innodb_flush_method=O_DIRECT applies to data files; redo uses... ; PG WAL uses O_DIRECT|O_DSYNC depending on wal_sync_method, often with pg_wal writes synced). The /proc copy reads page cache → always current. Good point to mention: prefer /proc copy over snapshot for the open file content; snapshot for the rest. So the refined runbook: Phase 0 — Freeze the blast radius: - Do NOT stop/restart the DB, do not reboot, do not kill. Prevent systemd/keepalived/DBA scripts from restarting: e.g., `systemctl mask` the unit, disable cron jobs that touch logs, stop logrotate (or it may HUP/truncate), freeze colleagues. - If the incident was disk-full, note the unlinked file's space is still held; freeing requires... it won't free until close. Don't try to "free space" by other destructive means. - Don't let autovacuum/checkpoint... those are fine. Don't run VACUUM FULL/CLUSTER/pg_repack (extra WAL churn)? Eh. Phase 1 — Preserve evidence (time-critical): - Identify the FDs: `ls -l /proc//fd/ | grep '(deleted)'`; for each deleted file, note offset: `cat /proc//fdinfo/` gives pos (the process's current offset — for an append-only log, pos ≈ high-water mark). - Copy the 63 GiB log: `dd if=/proc//fd/ of=/mnt/safe/log.recovered bs=64M` or `cp --sparse=never`. Verify with size and checksums; compare to fdinfo pos later. Watch out: if the process keeps appending while you copy, the copy may be a moving target — size can grow; that's usually fine (capture at least up to current pos; for a log needed for recovery you want up to a consistent end/LSN; a slightly-longer tail is fine, but a torn last record may need truncation at recovery time — engines typically handle a torn final record by ignoring it after the last checkpoint... depends). - Also copy any other deleted files similarly (data files if any were deleted). - Better: if same-fs reflink possible (`cp --reflink=always /proc//fd/N dest` on btrfs/XFS+reflink), do that first as a cheap instant CoW copy, then do a full copy at leisure. Also consider `tar` no— just cp. Phase 2 — Get a consistent backup while it's alive: - Now that you have the log preserved, take a proper consistent backup via the engine's tooling (pg_basebackup, BACKUP DATABASE, mysqldump, xtrabackup --backup, mongodump, etc.). While the process runs and its data files are open, the DB can produce a consistent backup itself even though underlying files were deleted. - Or dump logical data (slower but robust). Phase 3 — Only after all that: stop/restore/rebuild on fresh storage, using preserved log if needed for point-in-time. Now, which options are dead — enumerate against a presumed prior list. Since I don't know the exact prior list, I'll frame it as: "Compared to the baseline: dead vs alive" with the most plausible prior options: Dead / demoted: 1. Restore from the existing backup as-is — dead as a *trusted* path. A backup with no consistency guarantee (torn file copy of a running DB) either fails recovery or silently yields inconsistent data. You can keep it cold as a forensic last resort. If the engine is one where crash recovery from a *smeared* image is sometimes possible... still not guaranteed. Verdict: not a recovery option you can plan on; treat as evidence, not restore point. 2. Clean shutdown → restart → let crash recovery sort it out — dead. Closing the FD destroys the 63 GiB log. Also if the log is needed for recovery, on-disk it's gone. (Nuance: for some engines, a *graceful* shutdown flushes everything to data files so the log may not be needed on restart — e.g., MySQL: shutdown=1 does slow shutdown full purge + merge; PG: shutdown checkpoint. If the engine can do a full clean shutdown (checkpoint everything, quiesce), then the log's content becomes unnecessary, and restart could work — BUT restart also requires the data files to exist; if only the log was deleted and data files are intact, a fully-graceful shutdown then restart may be fine! Hmm, this depends on what was deleted in the baseline. If baseline = "logs deleted but data files intact, server still up," then a graceful shutdown writes a checkpoint to data files and the log is no longer needed; restart succeeds. The danger is if shutdown is not fully graceful (crash) — then you need the log, which is gone. So option "graceful restart" is not strictly dead, but it's a gamble: it converts a recoverable state into all-or-nothing. With the inconsistent backup, you lose the safety net, so gambling is bad. Verdict: alive but reckless; only after you've extracted the log.) Wait, but the user asks "which recovery options are now dead?" — implying the added facts kill some previously viable options. Let me think about what a prior option list plausibly was, in a PG-flavored scenario: Plausible prior scenario (reconstructed): "PG instance; someone deleted WAL files in pg_wal while the server was running (e.g., to 'free space'); disk now missing WAL; server still up. What now?" Prior answer options: a) While it's running: pg_dump / pg_dumpall → logical export. Works regardless of WAL state (reads buffers/pages). VIABLE. b) pg_basebackup now → consistent physical backup. VIABLE (it streams its own WAL with -X stream). Though it needs... fine. c) Take a filesystem snapshot (LVM/storage) while running → crash-consistent image, then restore like a crash + replay WAL from... but WAL files are deleted! Snapshot of a running PG is crash-consistent only if you have WAL from the snapshot moment onward; pg_wal is deleted → snapshot alone insufficient unless you also grab WAL from /proc. Partially viable with caveats. d) Copy deleted-but-open WAL segments from /proc//fd into place (or a copy), keeping directory layout; then can stop safely. VIABLE — the classic trick. In PG, postmaster children keep WAL segments open... actually which process holds WAL FDs? walwriter holds the current segment open; also checkpointer, etc. When a segment is recycled, it's renamed (pg_wal: recycle = rename to next name), not unlinked, unless not needed. If someone rm'd them, open FDs remain with walwriter etc. e) Restart and hope PG recreates WAL — DEAD generally: PG won't start with missing WAL between checkpoint and shutdown... If server was cleanly shut down at a checkpoint boundary and WAL deleted, PG can start fresh (that's the "pg_resetwal" world). If it crashes, missing WAL = no recovery. f) pg_resetwal as a last-ditch to force start with data loss/corruption — viable last resort. g) Restore from existing backup + archive WAL — depends on baseline. Then "plus: one log is unlinked-but-open (63 GiB)" — in PG-land this would be weird due to segment size, but if we hand-wave "the log" as the DB's transaction log generally... If I answer engine-agnostically but with concrete mechanics (procfs FD copy, fdinfo pos, copy before anything else, no restart, snapshot caveats re page cache, torn-backup implications), the answer is useful regardless of engine. I might note the assumption briefly. Actually, let me reconsider the possibility that the prior conversation was about MySQL/MariaDB with InnoDB and the deleted file is `ib_logfile0` (pre-8.0.30, innodb_log_file_size could be set large — 63 GiB is implausible) or `ibdata1` (can be huge, but that's not "a log") or undo files... Or the scenario: MySQL 8.0.30+ `#innodb_redo` directory: `#ib_redo` files each 3MB... capacity-based. No. SQL Server: `database_log.ldf` 63 GiB — very plausible! And "backup without consistency guarantees" = someone did a file copy of mdf/ldf while SQL Server was running (a classic mistake). And the unlinked-but-open 63 GiB log = someone deleted the .ldf while SQL Server (on Linux) was running, again to "free space." SQL Server holds data and log files open. Recovery: you can still BACKUP DATABASE (native, consistent) while running! That's the key move: take a full backup NOW; SQL Server reads via open FDs. Then restore elsewhere. If the server crashes before that, the .ldf is gone → only the inconsistent file-copy backup remains → you'd have to attempt attach of a torn copy → likely fail or suspect mode. Under SQL Server framing: - Alive: native BACKUP DATABASE ... TO DISK now (consistent, regardless of deleted log, since engine reads via FD); with COPY_ONLY if you care about log chain (but log chain is broken anyway — actually native full backup is fine). - Dead: stopping SQL Server then restarting/attaching — once stopped, the .ldf vanishes; attach of mdf without ldf requires EMERGENCY mode rebuild (ALTER DATABASE ... EMERGENCY; REBUILD LOG) — data loss for uncommitted... well rebuild log loses nothing committed in data files IF data files were checkpointed... In SQL Server, if you lose the log but data files are intact, you can attach with EMERGENCY and rebuild log; committed transactions are in data pages already if checkpointed; un-checkpointed committed changes in log = lost. So not fully dead but lossy. - Dead: the file-copy backup as a restore point: torn copy of running DB = likely unusable or suspect. - The 63 GiB open log: copy from /proc if you want point-in-time/in-flight safety; but if you take a native full backup now, you don't strictly need the log afterward. I can't determine the engine with certainty. The best answer structure: 1) Brief framing: what the two new facts change conceptually: (a) you're now on a timer — the log exists only as long as the process lives; (b) your restore-from-backup safety net is gone — the backup is not a point you can trust. 2) Verdict: changes from "recoverable with several fallbacks" to "recoverable, but single-shot and time-critical; extract the open FD first; treat backup as evidence, not a restore point." The outcome distribution worsens: previously a process crash was annoying; now a process crash before you copy the FD/log may be catastrophic. 3) Which options are dead (list, with reasons): - Restore from the existing backup as a planned recovery path — dead (no consistency guarantee = torn image; engine recovery may refuse or produce suspect DB; even if it comes up, no integrity assurance). Keep it as last-resort evidence. - "Stop the service, restart, let crash recovery replay the log" — dead: closing the FD unlinks the 63 GiB log for good; on-disk it no longer exists. - Reboot/host-level anything (kernel panic, OOM killer, systemd auto-restart) — treat as dead/must-prevent. Also disable auto-restart supervisors (systemctl mask, disable OOM reaper? can't fully; set oom_score_adj for the process; stop logrotate). - "Take a filesystem snapshot and rely on it alone" — partially dead: snapshot captures on-disk state only; a deleted-open file appears as an orphan inode (recoverable only with fs forensics, and unflushed buffered writes are missing). Snapshot ≠ copy of the open file. It's still useful for the rest of the files, but the log must come via /proc. - Any path that involves rotating/truncating that log (logrotate HUP, engine-driven rotation, backup tools that stop/start) — dead. - Point-in-time recovery to arbitrary times via backup+log: dead if the log doesn't span... well the backup isn't consistent so PITR from it is undefined. PITR as a *planned* option: dead. 4) What's still alive (ordered): 0. Stabilize: mask the unit, block logrotate, oom_score_adj -1000, page someone, stop writes if you can afford it (or not — reducing churn helps), ensure 63+ GiB free on another volume. 1. Copy the unlinked log from /proc//fd/N immediately (cat/dd/cp; verify size vs lstat; compare with fdinfo pos; checksum). If same-FS reflink available, take an instant reflink copy first as insurance while the full copy runs. Note: reading via /proc gives page-cache-current content — this is the only method that includes unflushed buffered writes. 2. Copy every other deleted-but-open file the same way (ls -l /proc//fd | grep deleted). 3. Then take a *consistent* backup using the engine's own tooling while the process is still alive (native backup/pg_basebackup/xtrabackup/mongodump...). This is now the primary restore point. 4. And/or a logical dump as a second independent artifact. 5. Only now consider stopping/rebuilding on fresh storage; with preserved log + data files, crash recovery becomes possible again. 6. The old inconsistent backup: keep cold; possibly usable for forensic diff or partial table salvage if everything else fails — but never as the trusted baseline. 5) Nuances/wrinkles: - 63 GiB copy time: at e.g. 200 MB/s ≈ 9 min; on slow network storage hours — calculate and decide destination accordingly; do the reflink trick if possible. - If the log keeps growing during copy (append-only), copy may chase a moving target: read size first (lstat via /proc fd symlink? stat -L /proc/PID/fd/N), copy to that length+slack, then verify final size ≥ pos at start... Actually simplest: copy, then compare tail/checksum. For redo logs engines tolerate a torn tail. - Free space: the deleted log still consumes 63 GiB on the original fs (df shows it) — that's expected; don't "fix" it. - Watch for the process rotating the log itself: many engines close/reopen logs on SIGHUP or size thresholds — after which the old FD closes and data is gone. So speed matters, and suppress signals. - If the process is multi-process (e.g., postmaster model), the FD may be held by several PIDs; copying via any one of them works; but the "owner" dying may close only that FD — fine as long as one remains... careful: in PG, if walwriter dies the postmaster restarts the whole tree (crash loop) — then all FDs close. Prevent child crashes: consider reducing load to lower crash probability. - The inconsistent backup's failure modes: mixture of files from different points in time; recovery (if it runs at all) replays what log it has, but "smeared" data files mean some pages are from *after* the log point → divergence; typical symptoms: recovery fails checksum/LSN checks, or DB opens but with corruption that surfaces later. Hence: never trust it; at most salvage tables row-by-row. - Do NOT fsck the original fs (an fsck on a fs with open deleted files... fsck requires unmounted anyway). - If you're on ext4/XFS and the FD closes before you copy: orphan-inode recovery from a *pre-incident block snapshot* only; after close, blocks are freed and will be overwritten — photorec-style carving is the残り; low odds on busy fs. (i.e., the "recover deleted files afterwards" fallback is now near-dead, which is why the timer matters.) 6) Answer to "does this change the verdict?": Yes — from "several independent, restartable options; a crash is survivable" to "single-threaded, time-boxed recovery where the running process is the only source of truth; a crash before FD extraction is likely catastrophic." Severity goes up one notch; the runbook order changes (evidence preservation jumps to step 0/1; backup-based restore drops off the list). Now, about the engine ambiguity: I should write in a way that works for the common case but ask/flag assumptions. Given "Same, plus" — the user assumes continuity. I'll note briefly: "Assuming the same setup as before (running DB whose transaction log was deleted underneath it)..." and answer. I can keep engine references generic ("redo/WAL/transaction log", "the engine's backup tool"). Maybe include both the generic mechanics and one-liners mapping to PG/MySQL/SQL Server where relevant. Keep it tight. Also handle the subtlety: "backup was taken without consistency guarantees" — I should classify: crash-consistent (atomic snapshot) vs torn (cp/rsync of live files). The phrase "without consistency guarantees" strongly implies torn. If it were merely crash-consistent, that's usually fine for DBs. So I'll state: if it was an atomic snapshot, it's still usable via crash recovery; if it was a file copy with no coordination, treat as unusable. But the user said "without consistency guarantees," so default: torn → dead as a trusted restore point. I could add: even a torn copy may be *partially* salvageable (individual files/tables), so preserve it, but plan as if it doesn't exist. One more consideration: does the unlinked-but-open log change anything *positively*? Yes: it means the log content is still 100% intact and capturable right now — better than "deleted and gone." So one previously-dead option may be resurrected: crash recovery / point-in-time if you copy it out in time. In the baseline (log deleted, gone), crash recovery after stop was dead; now, IF you copy the FD first, crash recovery is possible again. So the verdict: the added facts cut both ways: they kill the backup safety net and put you on a timer, but the open FD offers a window to restore the log to disk. Net: recoverable, but with a hard time dependency and no net. Let me also think: "one log is unlinked-but-open (63 GiB)" — could there be a trap here: 63 GiB log file — maybe it's NOT the transaction log but an application log, and the recovery-relevant logs are already gone? Then the answer: the open 63 GiB file is a red herring for recovery (only useful forensically), and combined with the inconsistent backup, your remaining options are only live-dump. Hmm. But the user asks "does this change the verdict?" — they'd expect analysis of both facts. I'll cover: if that log is the transaction/redo log → as analyzed; if it's just an app/server log → it doesn't restore any recovery option, it only matters forensically; and the inconsistent backup still kills the restore path. I can fold this in briefly. Also worth mentioning: verify whether the backup at least captured a *contiguous* window vs smeared; if the backup tool was e.g. rsync with a second pass (--checksum) it might be closer to a point-in-time after the second pass converges — but no guarantee. Also: an option that may be newly alive: using the preserved log + the data files (still open/intact on disk?) to do a proper crash-consistent capture now. Actually all data files might be intact (only the log was deleted). Then: copy log back to its expected filename/location? You can't "restore" the unlinked file to the original name, but you can copy it to the correct path/name (e.g., recreate path) — but the running process won't switch to it (it keeps using its FD). It matters only for post-stop recovery: after the process dies, place the copied log at the right path so the engine's recovery can use it. Yes — that's the key: pre-stage the copied log into the directory under the expected name so that when you finally stop the DB and it needs recovery, the log is there. (Careful about engines that zero/init logs on start.) Which recovery options are now dead — let me also consider "physical file copy of the live data directory" (cp -a PGDATA while running): in the baseline this might have been listed as a possible-but-inconsistent option; now, given the backup is already inconsistent and the log is unlinked, a naive cp of the directory would MISS the deleted log entirely (it's not in the namespace!) — so "plain file copy of the data dir" is now definitively dead (it silently produces another torn backup missing the log). That's an important call-out: the unlinked file is invisible to normal copies — you must go through /proc. That's a strong point: any filesystem-level backup (cp, rsync, tar, even storage snapshot to a lesser extent) taken now will silently omit the 63 GiB log because it has no name. So those options aren't just degraded — they produce artifacts that look complete but aren't. → "dead in their naive form." So the "dead list": 1. Stop & restart / crash recovery without pre-capturing the log — dead (FD close destroys it; recovery then impossible... well if data files intact and a clean shutdown is achievable it could work, but it's a gamble with no safety net; classify as "gamble, not a plan"). Hmm — I should be careful and precise: it's not that restart always fails; it's that you irrevocably destroy the only intact log before knowing whether you need it. I'll phrase as: "as a *plan*, dead; as a gamble, insane." 2. Restore from the existing backup — dead as trusted restore point (torn); salvage-only. 3. PITR from that backup — dead (needs a consistent base). 4. Naive file-level copy/tar/rsync of the live directory — dead (silently omits the unlinked log; also torn). 5. Storage/LVM snapshot as the sole capture — mostly dead for the log (orphan inode + missing unflushed buffered writes); usable only with fs-level forensics; and note unflushed pages issue. 6. Anything requiring the log to be readable by *name* on disk right now (e.g., pointing the engine's backup tool at the log file path, shipping the log by path) — dead; must go via /proc. 7. Post-mortem undelete after the process exits — near-dead (blocks freed; carving odds low on active fs). So the "if all else fails we can undelete" fallback from the baseline: dead once FD closes. Alive list: A. Immediate /proc FD copy of the 63 GiB log (plus all other deleted FDs) — the critical path. B. Engine-native consistent backup while alive (native BACKUP, pg_basebackup -X stream, xtrabackup, etc.) — the new primary restore point. C. Logical dump/export while alive — independent artifact. D. After A+B/C: controlled shutdown, rebuild, and (if needed) crash recovery using the restored log placed at its proper path. E. Cold salvage from the torn backup — last resort only. Also protective measures: mask the service unit (prevent systemd restart), stop logrotate & cron, oom_score_adj, avoid config reloads/SIGHUP (they may rotate logs), reduce write load if feasible (less log growth → copy converges; lower crash probability), monitor the FD still exists (loop: test -e /proc/PID/fd/N... actually use ls), assign two people: one babysits the FD, one does copies. One more nuance: reading /proc/PID/fd/N requires same-user or root. Note it. Also, copying a 63 GiB file from page cache: the file may not be fully in page cache; reads will hit disk; disk blocks still allocated (unlinked but open keeps blocks). Fine. Another nuance: If the DB is PostgreSQL and the "log" is actually the postgres server log — irrelevant for recovery. But I'll mention the check: confirm which file it is (fdinfo, /proc/PID/cmdline, engine's config) before spending an hour copying an irrelevant query log. Quick check: `stat -L /proc/PID/fd/N`, look at inode/size; map engine's known redo path patterns. Actually since it's deleted, the name is only in "(deleted)" suffix which still shows original name: ls -l shows "/var/lib/mysql/ib_logfile0 (deleted)" — the symlink target includes the original path even after unlink! Yes: /proc/PID/fd/N symlinks show "path (deleted)". So you can identify it instantly. Now the "verdict" language: prior verdict presumably something like "recoverable; worst case restore from backup." New verdict: "recoverable but no longer multi-path; the only intact transaction log in the universe is that open FD; the backup is downgraded to evidence; recovery is now a race with a single point of failure (the running process). Act in this order: preserve FD → consistent backup → dump → then stop." Also: "if the process dies before you capture the FD, you fall back to a torn backup and salvage — i.e., likely data loss." So yes, verdict changes materially: from "guaranteed recoverable with fallbacks" to "probably recoverable if you act now; catastrophic tail risk." Probability framing: risk of total loss now = P(process exits before FD copy) + P(torn backup unsalvageable). Reduce the first to ~0 by copying immediately. Format: This deserves a structured but tight answer. Headers or bold segments: "Verdict: yes, it changes" / "What died" / "What's still alive" / "Do this now, in order" / "Nuances". Length: medium-long; the user is clearly technical and wants specifics. Use bullets, short. Avoid over-hedging. Since I lack the prior turn's exact engine, I'll add one line anchoring assumptions and keep engine terms generic with a couple of parenthetical examples. Keep total maybe 400-600 words? The question has two parts (verdict change? which options dead?). A thorough but crisp answer ~500-700 words with clear lists. Use code formatting for commands. Markdown appropriate here (chat interface, technical). Let me draft. --- Draft: **Short answer: yes — the verdict moves from "recoverable with several independent paths" to "recoverable, but on a timer, with the safety net removed."** Two independent facts, two different effects: **Fact 1 — the 63 GiB unlinked-but-open log.** That file no longer exists as far as the filesystem is concerned; it lives only as long as the process holds the FD. Consequences: - Good news buried in it: it's the only fully intact copy of that log anywhere (the disk blocks are still allocated; the content is current through the page cache). If the log matters for recovery, it is recoverable *right now* and only right now. - The moment the process exits — crash, OOM kill, restart, reboot, log rotation — the inode's blocks are freed and begin being overwritten. After that, carving odds on a busy filesystem are poor. - It is invisible to every path-based tool: `cp`, `rsync`, `tar`, and even a fresh file-level backup will silently omit it and produce something that *looks* complete but isn't. **Fact 2 — the inconsistent backup.** A backup with no consistency guarantee (torn copy of a live tree, files captured at different instants) is not a restore point. Crash recovery can fix "stopped at a single instant," it cannot fix "smeared across 40 minutes." Typical outcomes: recovery refuses (LSN/checksum mismatch) or worse, opens and quietly serves corrupt pages. Downgraded from "fallback" to "cold evidence, salvage only." **Options that are now dead:** 1. **Restore from backup as the plan** (and PITR from it — no consistent base, so PITR is undefined). 2. **Stop/restart and let crash recovery sort it out.** Restart closes the FD; the 63 GiB log evaporates. It's not merely risky — it destroys the only intact log before you know whether you needed it. 3. **Naive file-level copy of the live directory** (cp/rsync/tar) — silently omits the unlinked log *and* is torn anyway. 4. **A storage/LVM snapshot as the sole capture** — the deleted file appears in the snapshot only as an orphan inode (no name; needs debugfs/xfs_db forensics), and anything still in dirty page cache is missing from it. Snapshot ≠ what the process sees. 5. **Anything that rotates/HUPs/truncates that log** — logrotate, config reload, size-based rotation in the engine itself. Also auto-restart supervisors: `systemctl mask` the unit now. 6. **Post-mortem undelete** as the fallback if things go wrong — once the FD closes, the blocks are freed; on an active filesystem, assume gone. **What's still alive, in this order:** 0. Stabilize: mask the unit, stop logrotate/cron, `oom_score_adj -1000` [hmm — oom_score_adj -1000 requires privileges; phrase as "protect from OOM"], no reloads, don't reboot, don't even think about freeing that 63 GiB (df still shows it — that's expected, it's not leaked). 1. Identify the FD: `ls -l /proc//fd | grep deleted` (the symlink shows the original path, so you can confirm it's actually the redo/transaction log and not a 63 GiB query log), and `cat /proc//fdinfo/` for the current write offset. 2. Copy it out, now, to another volume: `dd if=/proc//fd/ of=/safe/log.bin bs=64M` (or cp). 63 GiB at 200 MB/s ≈ 6 minutes; at 20 MB/s ≈ an hour — pick the destination accordingly. If you can reflink on the same filesystem, take the instant CoW copy first as insurance. Reading via /proc is the *only* method that includes unflushed buffered writes. 3. Same for every other deleted-but-open FD in that listing. 4. Now take a *consistent* backup with the engine's own tooling while it's alive (native backup / pg_basebackup / xtrabackup / whatever applies) — this becomes your first trusted restore point of the incident. 5. And/or a logical dump as an independent artifact. 6. Only then: controlled stop, rebuild on fresh storage. If you need crash recovery, place the copied log at the path/name the engine expects before starting — the running process will never switch to it, but post-mortem recovery will. **Verdict delta:** the old answer was "several restartable paths; a crash is survivable." Now: single-threaded and time-boxed. Total loss probability = P(process dies before step 2 finishes) × P(torn backup is unsalvageable). Step 2 is minutes of work that collapses the first term to ~zero — that's why it jumps to the front of the queue. Also: keep the torn backup cold anyway. If everything else fails, individual files/tables can sometimes be salvaged from it; it just can't be the plan. One caveat if "the log" turns out to be an application/server log rather than the transaction log: then it changes nothing about recovery (only forensics), and the effective situation is "no intact transaction log + no consistent backup" — at which point the live dump/backup path is all you have, and it's even more urgent. --- Hmm, this is decent. Let me reconsider length and structure. The user asked two crisp questions; the answer should lead with the verdict change, then the dead list, then briefly what to do / what remains. I think including the runbook is valuable because "act now" is the actionable core. But I shouldn't bloat. Let me trim some. Also reconsider: "Same, plus" — I should open by acknowledging the continuation without inventing too much detail. Something like "Yes — both facts bite, in different ways." I won't restate the baseline scenario extensively; I'll refer to it generically ("the running-instance-with-deleted-logs situation we walked through" — but careful: I shouldn't fabricate memory of specifics I don't have. A light touch: "Assuming the same picture as before — engine still running, logs deleted underneath it —"). Actually, to be safe and honest, I can write the answer so it stands alone: the two new facts are analyzable independently, and the "dead options" list I give can be framed as "relative to the option list from before" using generic option names that would plausibly match (restart/crash-recovery, restore-from-backup, file copy, snapshot, undelete). That works regardless of the exact prior list. I'll include one line: "if your baseline list differed, the classification rule is at the bottom" — nah, too meta. Keep it natural. One more technical check on claims: - /proc/PID/fd/N symlink shows "(deleted)" for unlinked files: correct. And the target string retains the original pathname. Correct. - Reading /proc/PID/fd/N: opening it gives you the file (like reopening the inode) — you can read from offset 0 even if the process's FD is positioned at the end. Correct. Requires permission (same uid or CAP_DAC_OVERRIDE/root). - dd from it reads whole file from start. Correct. - fdinfo shows pos: correct. - Unlinked-but-open: blocks not freed until last close: correct on Linux (ext4/XFS/etc.). df counts it as used. Correct. - Orphan inode in snapshot: ext4 superblock has orphan list; on a *snapshot* (consistent block image), the fs journal/orphan list would reference the unlinked inode... Actually careful: when a file is unlinked-but-open, ext4 adds the inode to the orphan list so that on crash recovery it gets truncated/deleted. If you take an LVM snapshot and mount it, the kernel's ext4 recovery will process the orphan list and DELETE those files! So mounting the snapshot would actively destroy the data. To recover, you'd need debugfs on the unmounted snapshot image to extract the orphan inode's blocks before mounting. That's an important nuance: "don't mount the snapshot; use debugfs to extract." I'll mention it briefly — it strengthens the "snapshot as sole capture is a trap" point. On XFS, unlinked inodes are in the AGI unlinked-bucket; mounting triggers their removal similarly; xfs_db can walk them. Good. - Dirty page cache point: LVM snapshot captures disk only; buffered writes not yet flushed are absent. The /proc read goes through page cache, so you get current data. Correct and important for logs written with buffered I/O; many DB logs are O_DIRECT/O_SYNC (PG WAL with wal_sync_method=open_sync etc.; MySQL redo with innodb_flush_log_at_trx_commit=1 uses sync). So the snapshot might be fine for a synced log but you can't verify easily — the /proc copy is strictly better. Keep one sentence. - Re: "reflink copy via /proc fd": cp --reflink=always /proc/PID/fd/N dest on btrfs (same fs) or XFS with reflink=1 (same fs): the source inode is the unlinked file — reflink (FICLONE ioctl) works on it? FICLONE on an unlinked-but-open file should work since it operates on the inode via the fd. On the same filesystem only. If the fs supports it, it's instant CoW. Worth mentioning as "if same-fs reflink is available." But note: if the incident was disk-full (common reason for deleting logs), same-fs may have no space; reflink needs some metadata space but tiny. Fine — one clause. - Growth during copy: if the process keeps appending, the copy may be a moving target. For a redo log, capturing a bit extra is fine; engines ignore a torn tail record (PG stops at last valid record; MySQL stops at... it errors or truncates? MySQL 8 stops at the "flushed" point...). Keep one sentence: "if it's still being appended to, copy to at least the fdinfo pos you saw; a torn last record is normally tolerated by redo replay." - systemctl mask: prevents restart even on dependency triggers. Good advice. Also `systemctl stop` would kill it — do NOT stop. Mask only. - oom_score_adj -1000 makes it unkillable by OOM killer (root-only). Reasonable advice; caveat: could worsen system-wide OOM behavior, but in an incident, fine. SQL Server specifics: native BACKUP DATABASE while files deleted (open) works — reads via FDs. pg_basebackup works similarly. xtrabackup copies files by *path* though! XtraBackup opens data files by path — if data files are deleted, xtrabackup can't open them by path... wait, in our scenario only the log is deleted, data files presumably intact on disk, so xtrabackup fine. But note: xtrabackup needs the redo log by path too (it tails ib_logfile/#innodb_redo). If the redo log is the deleted file, xtrabackup can't tail it → xtrabackup route is dead in MySQL-land; use mysqldump or native... MySQL has no native hot physical backup other than clone plugin (8.0.17+) — CLONE INSTANCE... clone plugin copies locally? CLONE LOCAL DATA DIRECTORY — does it read files by path or FD? The clone plugin reads via... internal file access, probably by path → also affected? Not sure. So for MySQL specifically, physical hot backup may be dead and mysqldump (logical) is the route. This level of engine detail is too speculative; keep generic: "use the engine's own consistent backup tool — verify it doesn't need the log by path." Hmm, actually most physical backup tools need to read the log. pg_basebackup with -X stream reads WAL via the walsender (process FDs) not by path — works. SQL Server native backup reads via engine — works. I'll keep the generic advice: "engine-native backup (logical dump if physical tooling needs the log by path)." Also the live-logical-dump option: works while the process is up regardless. OK also, what about "does this change the verdict" — maybe the prior verdict was "you're fine, just restart"? Unclear. I'll answer: "Yes — it changes along two axes: urgency (timer) and certainty (no fallback)." That's robust to whatever the prior verdict was, as long as it was more optimistic. If the prior verdict was already "grim," the answer would be "actually, slightly better than pure-deleted because of the open FD, but the backup is now worthless." Hmm. Given "Same, plus" reads like adding complications to test whether the verdict flips, the prior was probably "recoverable." My answer: "still recoverable, but now conditionally and on a clock; several previously listed options die; one previously-dead option (crash recovery using the intact log) comes back to life *if* you copy the FD out in time." That last point is nice: the unlinked-but-open log can *resurrect* an option (recovery using the log) that was dead if the log were fully deleted. Good to state explicitly: net effect on the option set: -N (backup-related) +1 (log-dependent recovery, conditional on capture). Let me now also double check "63 GiB" handling: mention free-space requirement (need ≥64 GiB on destination), time estimate. Also that you should checksum after (sha256 of 63 GiB ~ a few minutes). Also two-destination copies if paranoid. Structure plan: 1. One-line verdict: Yes — two different shifts: timer + lost safety net. Net option set: some die, one revives conditionally. 2. "What each fact does" — brief, maybe fold into verdict or dead list. 3. Dead options (numbered, with one-line reasons). ~6 items. 4. What survives / new order of operations (compact numbered runbook). 5. Traps & nuances (bulleted, short): snapshot-orphan-mount trap, page-cache vs snapshot, path-based tools omit the file, log still growing, mask systemd, df still shows 63 GiB used, keep torn backup as evidence. 6. Closing: the risk equation, and the caveat re: which log it is. Trim to keep it readable. Use bold headers instead of ## maybe; or small headers. I'll use bold lead-ins and lists. Code spans for commands. Aim ~600-750 words. It's a technical emergency-adjacent question; thoroughness is warranted but no fluff. Let me write the final. Careful wording on "restart" death: Nuance — if all *data* files are intact on disk and only the log was deleted, a fully graceful shutdown (which checkpoints everything) followed by restart *can* succeed without the log, because the log's contents become redundant after a final checkpoint. But: (a) you can't guarantee the shutdown will be graceful; (b) some engines still need the log file to exist to start (PG will create new WAL if cleanly shut down; MySQL recreates redo logs if innodb_force_recovery... by default InnoDB will complain about log file size mismatch/creation—actually MySQL recreates redo on startup if missing? Historically, deleting ib_logfile* while stopped and restarting recreates them — yes, MySQL recreates redo logs if absent, but then any unflushed changes are lost; if shutdown was clean, nothing lost. SQL Server: deleting the ldf and restarting → database won't recover; EMERGENCY rebuild). So "graceful restart" is engine-dependent: sometimes a gamble that works. With no safety net (torn backup), the gamble is unacceptable as a plan. I'll phrase it as "not dead in every engine, but demoted from plan to coin-flip — and it burns the log permanently before you learn the outcome." Also: which options are dead if the 63 GiB file is a *server/query* log, not the transaction log? Then: recovery-relevant, nothing revives; dead list still holds for backup; the FD copy matters only forensically. One sentence at the end. Also should I ask a clarifying question instead? The user asked a direct question expecting a direct answer; asking would be unhelpful. I'll answer with clearly stated assumptions embedded minimally ("if that log is the redo/transaction log" etc.). Given the flow, the assistant previously presumably knew the engine. I'll write engine-agnostically with parentheticals. Fine. Final check on commands: - `ls -l /proc/$PID/fd/ | grep deleted` — yes; better: `ls -l /proc/$PID/fd/ | grep '(deleted)'`. - `cat /proc/$PID/fdinfo/$FD` — shows pos and flags. - `dd if=/proc/$PID/fd/$FD of=/mnt/safe/log.recovered bs=64M status=progress` — good. Or `cp /proc/$PID/fd/$FD /mnt/safe/log.recovered` (cp follows the symlink; works). - `sha256sum` after. - `systemctl mask `; `systemctl edit`/`systemctl unmask` later. - Protect from OOM: `echo -1000 > /proc/$PID/oom_score_adj` (root). - Stop logrotate: `systemctl stop logrotate.timer` or move config; also cron. Snapshot trap detail: "if you do take a block snapshot, do NOT mount it and let the fs journal replay — ext4/XFS will see the open-but-unlinked inode as an orphan and delete it; extract with debugfs/xfs_db on the unmounted image instead." Correct enough. ext4: orphan inode cleanup on mount — yes, ext4_orphan_cleanup deletes them. XFS: on mount, unlinked inodes in AGI buckets get truncated/removed — yes (xfs_mountfs → xfs_iunlink... in older kernels; there's also an inodegc). Good. Also mention: `stat -L /proc/$PID/fd/$FD` gives size (st_size) — useful to know it's really 63 GiB and to verify copy length. And note the FD offset from fdinfo (pos) tells you the high-water mark if append-only. One more: multiple processes may hold the same log FD (forked children). Copy via any holder; but the relevant writer might be one process; if THAT one dies and others keep FD open, data survives. In crash-restart designs (postmaster), one child crash triggers full restart → all close. So: reduce crash odds (lower load), and mask supervisors. Already covered. I think the answer is ready. Length target ~650-800 words. Format with a few bold headers, numbered dead list, numbered runbook, short nuance bullets. No giant intro. Start with the direct verdict sentence. **Yes, the verdict changes — in two independent ways.** The unlinked-but-open log puts you on a timer; the inconsistent backup removes your safety net. And they interact badly: previously a crash mid-recovery was annoying, now it's likely catastrophic. **What the open 63 GiB file means** That log no longer exists in the filesystem — it lives only as long as some process holds the FD. Consequences: - The blocks are still allocated and the content is current (reads via `/proc` go through the page cache). If that's the transaction/redo log, it's the only fully intact copy in existence, and it's capturable *right now and only now*. The moment the process exits — crash, OOM kill, restart, reboot, rotation — the blocks are freed and get overwritten on a busy filesystem. This actually *revives* one option that was dead if the log were fully deleted (log-dependent crash recovery), but only conditionally on capturing it first. - It's invisible to every path-based tool. `cp`, `rsync`, `tar`, and any new file-level backup will silently omit it and produce something that looks complete but isn't. **What the inconsistent backup means** A backup with no consistency guarantee (torn copy of a live tree, files captured at different instants) is not a restore point. Crash recovery can fix "stopped at one instant"; it cannot fix "smeared across 40 minutes." Outcomes are either refusal (LSN/checksum mismatch) or — worse — it opens and quietly serves corrupt pages. **Options that are now dead** 1. **Restore from the existing backup as the plan** — and PITR from it, since there's no consistent base to roll forward from. Keep it cold as salvage material/evidence, nothing more. 2. **Stop/restart and let crash recovery sort it out.** Restart closes the FD and the 63 GiB log evaporates. In some engines a fully graceful shutdown would checkpoint everything and make the log redundant — but you can't guarantee graceful, and you'd be burning the only intact log before learning whether you needed it. That's a coin flip, not a plan, with no net below you now. 3. **Naive file-level copy of the live directory** — omits the unlinked log *and* is torn anyway. Doubly dead. 4. **Block/LVM snapshot as the sole capture** — the deleted file appears only as an orphan inode with no name, and anything still dirty in page cache is missing from it. Trap: if you mount such a snapshot, ext4/XFS journal replay sees the unlinked-open inode as an orphan and *deletes it*. Extraction requires `debugfs`/`xfs_db` on the unmounted image. 5. **Anything that rotates/HUPs/truncates that log** — logrotate, config reload, engine-internal rotation. Also auto-restart supervisors: `systemctl mask` the unit immediately. 6. **Post-mortem undelete as the fallback** — once the FD closes, blocks are freed; assume gone on an active filesystem. **What's still alive, in this order** 0. Stabilize: mask the unit, stop logrotate/cron, `oom_score_adj` the process to −1000, no reloads, no reboot. Don't "reclaim" the 63 GiB — `df` still counting it is expected, not a leak. 1. Identify it: `ls -l /proc//fd | grep deleted` (the symlink shows the original path — confirm it's actually the transaction log, not a 63 GiB query log) and `cat /proc//fdinfo/` for the write offset. 2. Copy it out now, to another volume: `dd if=/proc//fd/ of=/safe/log.bin bs=64M status=progress`. 63 GiB ≈ 6 min at 200 MB/s, an hour at 20 MB/s — choose the destination accordingly. Same-fs reflink (`cp --reflink=always`) gives you an instant CoW insurance copy first if the filesystem supports it. Verify size against `stat -L`, checksum after. If it's still being appended to, capture at least past the offset you saw; a torn final record is normally tolerated by redo replay. 3. Same for every other deleted-but-open FD in that listing. 4. Take a *consistent* backup with the engine's own tooling while it's alive (native backup, pg_basebackup-style, etc. — verify the tool doesn't need the log *by path*; if it does, use a logical dump instead). This is your first trusted restore point of the incident. 5. A logical dump as an independent second artifact. 6. Only then: controlled stop, rebuild. If you need crash recovery, place the copied log at the path/name the engine expects *before* starting — the running process will never switch to it, but post-mortem recovery will. **Net verdict shift:** from "several independent, restartable paths; a crash is survivable" to "single-threaded and time-boxed — the running process is the source of truth." Total loss probability ≈ P(process dies before step 2 completes) × P(torn backup unsalvageable). Step 2 is minutes of work that collapses the first term to ~zero, which is why it jumps the queue ahead of everything else. One caveat: if that 63 GiB file turns out to be an application/server log rather than the transaction log, it changes nothing for recovery — then you're in "no intact redo + no consistent backup" territory, the dead list above still holds, and the live-dump/consistent-backup-while-running path is all you have, with even more urgency.

Same, plus: one log is unlinked-but-open (63 GiB) and backup...