Under Aretum (Prime), StandardData rebuilt the OCR and metadata pipeline for the Library of Congress's National Digital Newspaper Program — the engine behind Chronicling America. Roughly 300,000 historic newspaper pages have been reprocessed at full federal preservation standards, at about $50 per ~5,000-issue batch, 100× cheaper than commercial OCR and 99% faster than the legacy on-prem path.
That work is the basis for the Library's public announcement that the pipeline will be open-sourced in 2026 for community use and development. Under Aretum's prime contract, our codebase has been prepared to the LOC Office of the CIO's open-source compliance standards — third-party notices, license headers, dependency audits, CIO-level review — the posture any federal cultural-heritage institution would expect of vendor work touching irreplaceable collections.
Same page, reprocessed: clean sentences, recovered named entities (Shipka Pass, Suleiman, Constantinople, Gabrova) — fully searchable for researchers.
Source: Team Aretum, NDNP Open OCR — British Library presentation (10 Sep 2025), slides 13–14. loc.gov ↗
What we built
Distributed serverless cloud compute. The pipeline runs on AWS Batch and processes newspaper pages in parallel — workloads that used to take weeks on a single on-prem server complete in hours in the cloud. The same code path also runs locally on a workstation, on an isolated network, or on any organization's own distributed compute, with no rewrite required. Sovereignty is a deployment choice, not a code rewrite.
Underneath: Tesseract built from source on AMD64, the AmericanStories segmentation model in ONNX for column and line detection, and standards-aligned outputs — ALTO XML word and line bounds, METS structural metadata, Dublin Core descriptive metadata, NISO image headers. Every page emits a searchable PDF plus the metadata payload Chronicling America's full-text search expects.
- Operator-friendly delivery: Library staff submit jobs from a simple Python CLI shipped to PyPI; the pipeline scales up on demand and back to zero when idle.
- Auditable production: every change traceable; production container carries zero critical CVEs; Terraform-managed infrastructure with multi-stage GitLab CI/CD.
- fsspec-generalized I/O: the same pipeline reads from and writes to S3 or local disk with no code change — the foundation for an air-gap deployment when needed.
Why it matters — four reasons this approach is different
Open source & federal-compliant
Delivered to LOC OCIO open-source compliance standards — third-party notices, license headers, dependency audits, CIO-level review. The Library has announced public release in 2026.
Distributed serverless — fast and affordable
AWS Batch processes pages in parallel. Workloads that take weeks on a single on-prem server complete in hours in the cloud — about 99% faster. Pay only for what you use: ~$50/batch vs ~$5,000 commercial (100× cheaper).
On-prem or cloud
The same pipeline runs locally — on a workstation, on an isolated network, or on your own distributed cloud compute. Choose sovereignty (on-prem) for sensitive or donor-restricted material; choose cloud for speed when scale is needed.
Federal preservation standards
Built to the Library of Congress NDNP technical guidelines — ALTO XML, METS, Dublin Core, NISO image metadata. The same standards trusted across federal cultural-heritage preservation.
Inside the pipeline — from pixels to structured, searchable text
The segmentation step is where most legacy OCR fails. Newspapers are dense, multi-column, and frequently mixed with photographs, ads, and maps. Bad zoning means text from one column bleeds into the next; words appear in the wrong reading order; mixed-content pages are abandoned entirely. We retrained the segmentation model on real NDNP pages and fixed the ALTO ordering algorithm so reading order survives multi-column layouts. The result is dense, accurate full-text coverage that downstream search and discovery can actually use.
Source: Team Aretum, NDNP Open OCR — British Library presentation (10 Sep 2025), Library of Congress. loc.gov ↗
How we know it works
ALTO output carries per-word confidence. On real Chronicling America pages, that means recovered proper nouns, intact sentences, and searchable corpora where the 2005-era OCR delivered garbled tokens. A handful of word-level confidences from a single re-OCR'd page: Monterey 0.96, Highland 0.96, County 0.96, December 0.95, Doctrine 0.95, Forcible 0.96, Increase 0.94, Foreign 0.91. That is the difference between an archive scholars can search and one they cannot.
The defence of the Shipka Pass. The blockade maintained by Suleiman's forces. Both sides waiting for more troops, Constantinople, Thursday, Aug. 30, 1877.Recovered text from a reprocessed NY Tribune (1877) page. Source: Team Aretum, NDNP Open OCR — British Library presentation, Library of Congress (10 Sep 2025).
What the Library of Congress got
- Searchable, standards-compliant outputs — every page emits ALTO XML plus a searchable PDF, conformant with the LOC Serials Division's NDNP technical guidelines for preservation and ingest into Chronicling America.
- Operator-friendly, secure, auditable — Library staff submit jobs from a simple command; the pipeline scales up on demand and back to zero when idle; every change traceable; production container carries zero critical CVEs.
- Cost profile makes full-archive reprocessing viable — the kind of project that used to be a multi-year planning exercise is now economically routine.
Why this matters beyond newspapers
The architecture and the standards posture (ALTO, METS, Dublin Core, NISO) extend to any cultural-heritage corpus that needs structured, searchable preservation. The segmentation and OCR stack tuned for English newspapers carries directly to multi-language and multi-script materials, and to source media beyond microfilm.
- Multi-language & multi-script: the OCR stack extends to German (Fraktur and Antiqua), Yiddish (Hebrew script), Polish, Russian, Hungarian, Czech, and Ladino — the core language span of most American cultural-heritage holdings.
- Source-media adaptability: battle-tested on microfilm-scanned newspapers, architected for adaptation to bound volumes, typescripts, photographs with embedded text, and handwritten survivor testimonies.
- Sovereignty by default: on-prem and air-gap-capable deployment for sensitive, donor-restricted, or pre-clearance material; cloud burst when full-archive scale is needed.
- Federal compliance posture: delivered to LOC OCIO open-source compliance standards — the posture any federal cultural-heritage institution would expect of vendor work touching irreplaceable collections.
- Standards-aligned output: ALTO, METS, Dublin Core, NISO — directly ingestible by modern discovery and preservation systems.
Open-source in 2026
The Library has publicly announced the pipeline will be open-sourced in 2026 for community use and development. The codebase has been prepared to LOC OCIO open-source compliance standards — third-party notices, license headers, dependency audits, CIO-level review — the same discipline any federal cultural-heritage institution would expect of vendor work touching irreplaceable collections.
If you have a federal preservation problem, or a commercial one with similar contours — millions of source documents, strict standards, irreplaceable originals — we are happy to walk through how this work would map to your environment.
Further reading. Team Aretum's NDNP Open OCR presentation at the British Library (10 September 2025) is published by the Library of Congress: loc.gov/ndnp/guidelines/docs/BL-ndnp-ocr-20250910.pdf (Dillon Peterson, StandardData, listed as contributor).