How New York's Public Records Got Buried Under Thousands of Duplicate Images — and What It's Costing the City

New York City's digital archives contain millions of duplicate images. Not a few hundred. Not a rounding error. An internal review circulated among Department of Records and Information Services staff earlier this year identified redundant image files clogging storage infrastructure across at least a dozen agencies, a problem that traces its roots to the fragmented way the city rushed to digitize paper records starting around 2009 under the Bloomberg administration's NYC Digital initiative.

The timing matters. With the city hosting FIFA World Cup matches at MetLife Stadium this month, and tens of thousands of visitors filing permit requests, public-records inquiries, and vendor credentialing paperwork, the strain on city data systems is unusually visible. Backend inefficiencies that once lived quietly inside server rooms on Maiden Lane in Lower Manhattan are suddenly running into real-world deadlines.

A Patchwork Digitization History

The root problem is structural. Between 2009 and 2019, individual city agencies digitized their own paper holdings using different vendors, different file-naming conventions, and different compression standards. The Department of Buildings scanned decades of permit records using one contractor. The Department of City Planning digitized zoning maps through a separate procurement. When the city later tried to unify those holdings under the NYCityMap portal and the Municipal Archives reading room on Chambers Street, deduplication was treated as a secondary concern — something to handle later.

Later never fully arrived. Server costs were low enough through the mid-2010s that storing duplicate TIFFs and JPEGs was cheaper than auditing them. The city's Office of Technology and Innovation, then called DoITT, flagged the redundancy issue in a 2018 infrastructure report but no dedicated funding was allocated to clean it up. By 2023, the city's total unstructured data storage footprint had grown to a scale that was making routine retrieval slower and cloud-migration planning more expensive, according to publicly available budget testimony submitted to the City Council's technology committee.

The Eric Adams administration inherited the backlog. The mayor's 2024 preliminary budget included a line item of roughly $4.2 million for what the Office of Technology and Innovation described as a citywide data-quality program, but that envelope covered far more than duplicate-image removal — it included cybersecurity audits and broadband equity work in neighborhoods like East New York and the South Bronx, leaving relatively little for the deduplication work specifically.

What the Cleanup Actually Involves

Removing duplicate images from a government archive is not simply a matter of deleting files. Each image may be cross-referenced by multiple databases. A scanned deed held by the Department of Finance, for instance, might be indexed separately by block-and-lot number, by grantor name, and by recording date — meaning three database entries point to what are effectively the same image stored three times. Deleting one copy without reconciling the index pointers breaks retrieval for anyone searching by one of those other fields.

The Municipal Archives, which operates its research center at 31 Chambers Street in Civic Center, has been working with a vendor since late 2024 to build deduplication scripts that check hash values before touching any index record. The work is deliberate and slow by design. Archivists have described the process in public presentations at the New York City Bar Association as akin to untangling a knot — you have to understand the shape of the whole thing before pulling any single thread.

The practical stakes are not abstract. Real-estate attorneys filing closing documents in Brooklyn rely on clean retrieval from city land records. Journalists and advocates requesting FOIL responses from agencies like the NYPD or the Department of Social Services depend on staff being able to surface the right version of a document without wading through near-identical duplicates flagged as separate records.

City Council members on the technology and oversight committees have been asked to consider a dedicated appropriation in the fiscal year 2027 budget cycle, which opens formal hearings in the fall. Advocacy groups including the Reinvent Albany government-reform organization have argued publicly that data-quality infrastructure deserves the same budget discipline as physical infrastructure. For now, agencies are managing the overlap manually where they can, and waiting for the longer-term fix to catch up with a backlog that took fifteen years to build.