Abstract

The global crawl budgets of search indices and foundation model scrapers systematically overlook regional domain spaces, community blogs, and local literary portals in South Asia. This research brief presents a 12-month empirical audit of the discoverability and long-term decay of Bengali cultural and civic archives on the open web.

Key Observations

1. Link Rot in Civic Repositories: Over 38% of regional government notices and municipal agricultural advisories from 2018-2022 are no longer resolvable.
2. Encoding Inconsistencies: Legacy Bijoy and Avro encoding discrepancies prevent semantic indexing in major search engines.
3. Open Preservation Protocols: We propose a decentralized WARC indexing protocol specifically optimized for low-bandwidth mirror nodes in the Global South.

@article{logicdock2024banglaweb,
  title={The Bangla web is a living archive: Preservation, search, and the politics of indexing},
  author={Logicdock Studio and Vernacular Archive Collective},
  journal={Logicdock Research Briefs},
  volume={2},
  year={2024}
}