Recycled log files are leaked forever after an unclean shutdown

[Manphil]

Summary

Files held in recycled_files keep their original .raftlog name while the
process is running. If the process dies without running Drop (SIGKILL, OOM
kill, panic=abort), they are re-scanned as regular append files on restart,
detected as a "hole", dropped from the file list — but never deleted from
disk. They are then unreachable forever and the leak accumulates across unclean restarts.

Reproduce

1. Default config (format-version = 2, enable-log-recycle = true,
   prefill-for-recycle = false).
2. Write until purge has run at least once with more files than
   recycle_capacity(), so some files are physically deleted and a gap in
   sequence numbers is created.
3. Keep writing until recycled_files is non-empty. ls now shows two
   non-contiguous ranges, e.g. 1963917-1963953 (recycled) and
   1964131-1964213 (active).
4. kill -9 the process, restart.
5. The first range is still on disk and stays there across every subsequent
   restart. du never goes down.

Root cause

Introduced by #310 @tabokie

purge_to() intentionally does not rename recycled files:

raft-engine/src/file_pipe_log/pipe.rs

Lines 445 to 450 in 8c8624c

	// Recycle purged files whose version meets the requirement.
	if f.format.version.has_log_signing() && recycled_len < remains_capacity {
	new_recycled.push_back(f);
	recycled_len += 1;
	continue;
	}

So the only on-disk marker of "this file belongs to the recycle pool" is the .raftlog.reserved suffix written by Drop for SinglePipe, which never runs under SIGKILL.

On restart those files land in append_file_names, and the hole check discards them without deleting:

raft-engine/src/file_pipe_log/pipe_builder.rs

Lines 173 to 180 in 8c8624c

	for (i, file_pair) in files.windows(2).enumerate() {
	// If there exists a black hole or duplicate scenario on FileSeq, these
	// files should be skipped and cleared.
	if file_pair[1].seq - file_pair[0].seq != 1 {
	invalid_idx = i + 1;
	}
	}
	files.drain(..invalid_idx);

[drifting-shard]

Hi, can I take this? I checked the recovery path and it looks like files before the sequence gap are removed from the list but not from disk. I’ll fix that and add a recovery regression test test.

[waterWang]

Submitted PR #408 to fix this: when the hole check drains the invalid file prefix on restart, the corresponding physical log files are now deleted too (for append/rewrite queues; reserved/recycled files keep the existing no-delete behavior). Includes a regression test that simulates the unclean-shutdown state (renames reserved files back to plain .raftlog) and asserts no leaked files remain on disk.