Solo · 2025
Web Archive Data Fetcher
Recover pages that are no longer online.
A tool for pulling archived snapshots of a site out of the Wayback Machine and extracting structured data from them, built to recover content from a site that had gone offline.
When a site disappears, its content often still exists in the Internet Archive, but as thousands of individual snapshots that are painful to go through by hand.
This walks the archive's index for a domain, fetches the snapshots, and extracts the structured content out of them, turning a scattered archive back into usable data.
Problems worth solving
The archive rate-limits you long before you finish
Fetching thousands of snapshots as fast as possible gets throttled almost immediately, and a naive retry loop makes it worse rather than better.
Requests are paced and retried with backoff, and progress is checkpointed so an interrupted run resumes instead of starting the whole crawl again.
What came of it
- Recovered structured content from an offline site via archived snapshots.