NYC's Digital Archives Push Forward on Duplicate Image Problem — Here's Where Things Stand This Week

New York City's scramble to modernize its digital record-keeping hit a specific snag this week: thousands of duplicate images clogging the archives of at least three municipal agencies, slowing public-access portals and inflating storage costs at a moment when city budgets are stretched thin. The issue, low-profile for years, moved closer to the surface after the Department of City Planning flagged the problem in internal workflow reviews tied to the city's ongoing rezoning documentation push.

The timing matters. With the 2026 FIFA World Cup bringing an estimated 1.5 million additional visitors to the New York metropolitan area between June and July, city agencies and cultural institutions have been racing to make digital assets — maps, venue images, neighborhood guides — publicly accessible and searchable. Duplicate files jam that process, forcing archivists to manually sort through records before anything can be cleanly published or shared with media partners.

Who Is Affected and What They're Doing About It

The New York Public Library, which holds one of the largest publicly searchable photo archives in the country through its Digital Collections portal on Fifth Avenue, confirmed this spring that it had identified a significant volume of near-duplicate scans in collections digitized between 2018 and 2023. The library has been running deduplication software across roughly 900,000 image records in that window, a project that staff expect to continue through the fall. No completion date has been formally announced.

At the city level, the Department of Records and Information Services — DORIS — manages the Municipal Archives on Chambers Street in Lower Manhattan, which houses more than 2.2 million photographs dating back to the 19th century. Staff there have been piloting an AI-assisted image-matching tool since early 2026 to flag redundant scans before they enter the public-facing catalog. The tool compares pixel-level similarity scores and metadata timestamps. Early results, according to agency documentation reviewed by The Daily New York, identified duplicate rates running as high as 12 percent in certain digitized batches from the 1970s and 1980s.

The congestion pricing rollout, which finally began charging drivers entering Manhattan below 60th Street, has added a new layer of digital documentation demands to transportation agencies. The MTA's communications team has been building out a public image library documenting station upgrades and new signage. Sources familiar with the project — who were not authorized to speak publicly — say duplicates have appeared in that archive too, particularly in photos taken during rushed documentation sessions at stations including 72nd Street on the Upper West Side and Atlantic Avenue-Barclays Center in Brooklyn.

The Costs Add Up Fast

Cloud storage is not cheap at scale. Amazon Web Services and Microsoft Azure both charge commercial rates that, for large institutional accounts storing millions of high-resolution image files, can run into tens of thousands of dollars per month. Deduplication is not just a cleanliness exercise — it has a direct line to the budget. For cash-strapped agencies operating under the Adams administration's fiscal constraints, trimming redundant storage has become a genuine line-item conversation.

The Mayor's Office of Technology and Innovation, based at 253 Broadway, has been pushing a city-wide data hygiene initiative since late 2025. That effort covers databases and documents broadly, but image deduplication was added to the scope earlier this year following recommendations from a working group that included representatives from DORIS, the city's 311 service, and the Office of Emergency Management.

For residents and researchers who rely on public archives, the practical advice right now is to check upload dates carefully when downloading from portals like the NYPL Digital Collections or the Municipal Archives catalog. Metadata on older scans may still be incomplete while the deduplication work is underway. Librarians at the Mid-Manhattan branch on 40th Street can help navigate the catalog and flag known problem batches on request. Staff at the Municipal Archives on Chambers Street accept inquiries by appointment and can verify whether a specific image record has passed through the new quality-control process. Both institutions expect their public-facing catalogs to reflect cleaner, deduplicated data by early 2027 at the latest.