Finding "Duplicates Through Time"

Finding “Duplicates Through Time”: How I Cleaned Up 300GB of Photos Without Losing Quality

Lately, I have been playing around with my personal photo and video library (around 300GB of files, spanning 10+ years of memories). I had a specific type of problem: I needed to save space on my SSD, but I did not want to pay any third-party cloud provider for extra storage.

To make things trickier, I was traveling in a remote location without any electronics stores nearby. Buying an external hard drive was out of the question. I had my laptop, tablet, and smartphone. Together, they had enough cumulative storage, but my laptop drive was suffocating.

What if I just made the library smaller?

The Compression Route (and a Nod to Middle-Out)

There are different ways to make files smaller, and the most obvious route is compression.

Whenever compression comes up in software engineering, I am always surprised by how many developers have not seen HBO’s Silicon Valley. If you have not watched it yet, take a break and go check it out. Great show.

In the real world, though, lossy re-encoding means sacrificing quality. When dealing with a decade of personal memories, heavy compression feels like the wrong compromise. You do not want high-resolution vacation shots turning into pixelated JPEGs just to save a few gigabytes.

Stage 1: Deleting Big Files (The Easy Part)

My next thought was simple: prune the clutter and remove the massive files.

Finding large video files was easy enough. A quick shell command or custom script sorted everything by size, allowing me to offload old movies and raw video dumps.

That gave me some breathing room, but it hit a wall quickly. Depending on how your library is structured, video pruning only works up to a certain point. The core of my storage problem was thousands of individual photographs.

The “Aha!” Moment: Burst Mode & Over-Shooting

A couple of days later, a friend asked me to send her a picture we took when we visited the Empire State Building in New York.

When I opened my library, I realized something: she asked for one picture, but I had taken seven nearly identical shots.

I always do this on holiday so people can choose the best angle or expression. A quick burst of 5 to 10 photos, just to capture one good moment.

That was the lightbulb moment:

How can I programmatically find all the “short burst” photos across 10+ years of media?

Or phrased better: How do I detect duplicates through time?

Enter Siegu

This problem led me to start building siegu, a local tool designed to tackle time-clustered, visually similar photos across massive personal libraries.

Instead of looking for exact byte-for-byte duplicates (which standard deduplication tools do via MD5 or SHA256 hashes), siegu approaches the problem using two dimensions:

1. Temporal Proximity (Time Windows): Grouping photos shot within seconds or milliseconds of each other using EXIF metadata (DateTimeOriginal).
2. Visual Similarity (Perceptual Hashing): Comparing photos within those candidate windows using perceptual hashes (like pHash or dHash) to see if they depict the same scene, regardless of minor lighting changes or slight hand movements.

By combining timestamps with perceptual hashing, you do not waste CPU cycles comparing a picture from 2014 in New York with a picture from 2022 in Tokyo. You only analyze candidate groups that were actually taken in short bursts.

In Part 2, I will dive deep under the hood: breaking down the Rust implementation, showing how siegu extracts EXIF metadata concurrently, and walking through the exact perceptual hashing algorithm used to flag burst duplicates.

What is Siegu?