Reclaim hotblocks disk: range-tombstone cleanup + startup DeleteFilesInRange

[mo4islona]

What & why

Fixes the NET-819 mainnet full-disk incident and NET-798 (blocking disk cleanup on startup).

During the incident the hotblocks volume hit 100% and compaction could not reclaim space: at a full disk compaction deadlocks (it needs scratch space to write merged output before deleting its inputs), and write-based deletion (tombstones) only grows the DB until compaction runs. The one operation that frees space without writing is DeleteFilesInRange, which unlinks whole SST files at the filesystem level.

Approach

Two independent mechanisms.

Phase 1 — routine logical cleanup (logical_cleanup, every 10s). Replaces the old per-key delete loop with a single range tombstone per deleted table. Snapshot-safe (range tombstones respect MVCC — no grace needed). Frees no space directly; it lets compaction drop the data. This is the normal-operation reclaim path.

Compaction tuning (now load-bearing, since routine reclaim relies entirely on it):

• compact-on-deletion collector + 24h periodic compaction → find tombstone-heavy / stale SSTs;
• max_background_jobs=8, max_subcompactions=4, level_compaction_dynamic_level_bytes → the default 2 background jobs could not keep up during the incident.

Physical reclaim — startup only (reclaim_disk_space). Unlinks whole CF_TABLES SST files below the min-live-TableId watermark via DeleteFilesInRange. No writes, no scratch space → frees disk even at 100%, where compaction deadlocks. This is the "reclaim before ingest" step (NET-798). Crash-orphaned DIRTY_TABLES markers are purged at startup so they don't pin the watermark.

Trade-off & deferred work

The file unlink ignores snapshots, so it can break an in-flight query whose snapshot predates a deletion. We therefore run it only at startup, where there are no live readers — it never corrupts the DB, only risks an in-flight read, and only at a moment when there are none.

A runtime disk-pressure trigger (free-space threshold via statvfs, or on a write ENOSPC) — the emergency reclaim that compaction cannot deliver at a full disk — is documented in code as a future improvement, not built here.

Changed from the earlier iteration

This supersedes the grace-period design. Removed: per-table deletion timestamps, the wall-clock grace gate (--reclaim-grace-secs), the runtime orphan reaper (--max-build-secs), and the two-phase bookkeeping. That eliminates a class of wall-clock data-loss bugs (clock jumps, slow clients outliving the grace, malformed-value coercion) in exchange for running the unlink only at startup. reclaim_disk_space and Phase 1 are now fully decoupled (watermark vs. work-queue).

Testing

crates/storage/tests/cleanup_reclaim.rs (8 tests): snapshot-safe logical delete, file unlink below the watermark, live-table watermark pinning, idempotency, startup orphan purge, reclaim/Phase-1 decoupling, and the documented "unlink breaks a live pre-deletion reader" trade-off. Full sqd-storage suite green (cleanup 8, compaction 5, database_ops 10, write_read 15, lib 1).