New York Is Quietly Purging Duplicate Images From Its Public Records Systems — Here's How It Compares to London and Tokyo

New York City's Department of Records and Information Services has been working since early 2025 to eliminate tens of thousands of duplicate digital images stored across the municipal archive network — a bureaucratic housekeeping effort that has taken on new urgency as the city's data storage costs have climbed sharply. The effort, which touches everything from scanned building permits held at 31 Chambers Street to crime scene photographs logged by the NYPD's Digital Evidence Management System, has moved slower than city planners anticipated when the project launched in January of last year.

The timing matters. New York is running three overlapping digitization drives simultaneously: one tied to the MTA's capital program documentation, one connected to the ongoing congestion pricing implementation records, and a broader push by the Adams administration to modernize city data infrastructure by the end of fiscal year 2027. Duplicate images — identical or near-identical files stored in multiple locations across the city's servers — inflate storage costs, slow retrieval times, and create legal liability when record requests surface conflicting versions of the same document. With the FIFA World Cup bringing an estimated 1.5 million additional visitors to the metro area this summer, the pressure to have clean, accessible public-facing databases has intensified.

How New York Stacks Up Against London and Tokyo

London's Government Digital Service completed a comparable deduplication exercise across borough-level planning records in 2023, finishing an 18-month project that eliminated roughly 2.3 million redundant files from the Greater London Authority's archive. The GLA publicly credited a centralized hashing algorithm — a method that assigns each image a unique digital fingerprint so copies can be flagged automatically — for speeding the process. Tokyo's Bureau of General Affairs wrapped a similar initiative in March 2025, targeting the city's ward-level civil registration photograph archives, and reported cutting storage overhead by approximately 34 percent across 23 special wards.

New York has no comparable citywide figure to point to yet. The Department of Records has not published a midpoint progress report, and a FOIL request filed by The Daily New York in April for project metrics has not been fulfilled within the statutory 20-business-day window. What is publicly known: the city's Digital Backbone Initiative, a program announced by the Office of Technology and Innovation in late 2024, lists deduplication as a Tier 2 priority item, meaning it receives funding but not the emergency-level staffing of Tier 1 projects. The program is being managed in part through a contract with a vendor whose work is coordinated out of the city's technology hub at 2 Metrotech Center in Downtown Brooklyn.

The practical gap between New York and its peers is partly structural. London and Tokyo operate more centralized government IT architectures. New York's 50-plus independent agencies each historically maintained their own records systems, many of which were never designed to communicate with one another. The Brooklyn Public Library's digital collections team, which partnered with the Department of Records on a 2022 photograph scanning grant from the Institute of Museum and Library Services, has described the inter-agency coordination challenge as significant in public budget testimony — though the library has its own deduplication process for its Digitization Lab on Grand Army Plaza that it says is largely complete.

What Comes Next for New York Residents and Record-Seekers

For anyone who regularly files public records requests — housing attorneys working the courts in the Bronx, journalists pulling building inspection histories in Bushwick — the near-term reality is that duplicate records remain a problem that can produce confusing results. Two requests for the same file can still return different image versions if the originating agency hasn't been folded into the centralized deduplication sweep yet.

The Office of Technology and Innovation has indicated that all 15 highest-volume city agencies are expected to complete deduplication integration by the fourth quarter of 2026. That would bring New York roughly three years behind London's 2023 finish line, though officials have pointed out that New York's archive is substantially larger. According to the city's own budget documents for fiscal year 2026, the Digital Backbone Initiative received $47 million in capital funding — a figure that includes but is not limited to the deduplication component. Whether that funding level survives the next budget cycle, given competing demands from the MTA and housing programs, is a question the City Council is expected to revisit in September.