Md. Masum Khan
AvailableEngage
All Essays
Data
January 20, 2025·8 min read

What Building a System for 400,000 People Taught Me About Scale

National-scale infrastructure forces you to confront every assumption you made at smaller scale. The lessons from designing Bangladesh's student monitoring system for the World Bank are still the most important ones I've learned.

In 2017, I was brought in as IT Consultant for the World Bank's ROSC-II project in Bangladesh. The mandate: design a national monitoring system for the country's secondary education sector. The scope: 400,000+ student records across hundreds of institutions, consumed by government ministries, regional education offices, and international development organizations simultaneously.

I had built large systems before. I had not built a system at national scale. The difference is not quantitative. It is qualitative.

When Assumptions Break

Every system I had designed before had a recoverable failure mode. If the database schema was wrong, you migrated it. If the API contract changed, you updated the consumers. If the access control model was flawed, you patched it.

At national scale, none of these are acceptable answers.

A schema migration affecting 400,000 records across dozens of regional offices is not an afternoon's work. A change to the API contract means coordinating with government ministries, each with their own procurement cycles, IT departments, and change approval processes. A flaw in the access control model — one that exposed student data to unauthorized parties — is not a patch. It is a national data incident.

This forced a discipline I had not fully developed before: you must be right the first time, or design explicitly for the cost of being wrong.

The second option, I came to believe, is usually more honest.

The Data Governance Problem Nobody Talks About

The technical work on ROSC-II was relatively straightforward. The hard problem was data governance.

Who owned the data? The Ministry of Education thought it did. The regional education boards thought they did. The individual institutions that generated the records had their own claims. The World Bank, as funder, had reporting requirements that created their own implied ownership.

In smaller systems, these questions are either answered by organizational hierarchy or never explicitly answered at all — and the ambiguity doesn't cause immediate problems. At national scale, unresolved data ownership is a structural failure waiting to happen.

We designed what I now think of as layered stewardship: a model where ownership and access rights were explicitly defined at each level of the data hierarchy, with no level able to override the constraints set by levels above it. Institutions could access their own records. Regional offices could access aggregate views of their institutions. The Ministry could access national aggregates. No layer could reach down into another layer's granular records without explicit, auditable authorization.

This was not technically complex. The complexity was in getting five different government entities to agree to it — and documenting that agreement in a way that would survive personnel changes over the years the system would operate.

ISO 27001 as a Design Constraint, Not a Compliance Checkbox

I had worked with ISO 27001 frameworks before the ROSC-II project. In most enterprise contexts, it arrives as a compliance requirement — a set of controls to implement to pass an audit.

On this project, I encountered it differently. When you are building infrastructure that will hold national-level data about hundreds of thousands of individuals, security is not a requirement bolted onto the system. It is a constraint that shapes the architecture.

Every design decision — where data resided, how it moved, who could query it, how queries were logged — was made with the security model in mind from the beginning, not reviewed against it at the end.

The practical effect was significant. Systems built with security as an architectural constraint have different properties than systems audited against security requirements post-hoc. The former catches an entirely different class of vulnerabilities — the ones that emerge from the interaction between components, not the ones that exist within individual components.

What Scale Does to Software Quality

Here is a counterintuitive observation: national-scale systems are often more reliable than enterprise-scale systems, not because they are better engineered, but because the cost of failure is so high that corners simply cannot be cut.

At enterprise scale, there is always pressure to ship. Features over stability. Speed over correctness. The implicit assumption is that you can fix problems after deployment.

At national scale, the political cost of a visible failure is so high that it forces a different discipline. You test more. You document more. You design for failure modes that will probably never occur. You build in the redundancy that seems excessive until the day it is not.

I brought this discipline back into every enterprise project I worked on afterward. The question I now ask at every architecture review: if this were a national system, what would we do differently?

The answer to that question is usually worth doing.

The Lesson That Didn't Fit in a Status Report

The most important thing I learned on ROSC-II doesn't appear in any of the project documentation.

I learned that the humans who will use and maintain a system long after you have left are the most important architectural constraint you have.

The system I designed would be operated by government IT staff with limited resources, maintained through political transitions, extended by developers who had not been part of the original design. Every decision I made needed to be explainable to someone who was not in the room when it was made.

This forced a clarity of design that I have tried to preserve in everything since. A system is not just its architecture. It is its documentation, its operational procedures, its failure modes and how they communicate themselves to operators.

The best system is not the cleverest one. It is the one that is still running, correctly, ten years after the team that built it has moved on.