APFS Zero-Cost Deduplication vs Deleting Duplicates: Why Block-Sharing Wins

The Duplicate Dilemma for Developers and Creators

Every developer and content creator ends up with duplicate files: shared ML model weights in Ollama and LM Studio, duplicated 4K video clips across editing timelines, identical Docker tarballs, or cloned git repositories. In total, duplicates often consume 20 to 100 GB of expensive Apple Silicon SSD storage.

The traditional solution offered by standard Mac cleaner tools is simple: delete the duplicates. But deleting files creates major problems:

• Broken Paths & Scripts: Hardcoded scripts (python train.py --weights ~/models/llama.bin) crash if the file is gone.
• Symlink Fragility: Replacing duplicates with symlinks fails when files are moved, backed up, or read by apps that don't follow symlinks.
• Accidental Deletion of User Work: Automated cleaner rules often delete the wrong copy of a file from an active project directory.

The APFS Secret: Copy-on-Write Clones (clonefile)

Apple File System (APFS) was built from the ground up with Copy-on-Write (CoW) architecture. On APFS, two files at completely different paths can share the exact same physical storage blocks on your SSD.

When you clone a file using Darwin's native clonefile(2) system call:

1. A new directory entry and inode are created instantly.
2. Both files point to the exact same physical SSD blocks.
3. Disk space consumed: 0 bytes.
4. If either file is modified in the future, APFS only writes the modified blocks (Copy-on-Write). Neither file breaks the other.

How Progressive Multi-Tier Hashing Works

Comparing millions of files by computing SHA-256 hashes upfront would burn CPU and take hours. MacPilot uses a Progressive 3-Tier Hashing Engine:

• Tier 1: Exact File Size Filter: Files with different byte counts cannot be identical. Over 98% of non-duplicate files are eliminated in <50ms.
• Tier 2: 4KB Header Hash: Reads only the first 4,096 bytes of files with matching sizes. Eliminates 95% of remaining candidates.
• Tier 3: Full SHA-256 Verification: Computes cryptographic SHA-256 only on files that passed both tiers, guaranteeing 100% byte accuracy.

Smart Auto-Selection Invariants

When duplicates are found, MacPilot provides intelligent auto-selection rules ("Keep Oldest", "Keep Newest", "Keep in Preferred Location") backed by strict safety invariants:

• Invariant 1: Personal Documents Protection: Files located in ~/Documents, ~/Desktop, and ~/Pictures are NEVER automatically selected for permanent deletion.
• Invariant 2: APFS Deduplication over Deletion: MacPilot allows you to choose APFS CoW Deduplication, freeing physical disk space immediately while keeping both files present and functional.
• Invariant 3: Audit Trail: Every deduplication action is logged with paths and bytes saved.