Availability engineering for a public-interest data platform

Who depends on the data

The platform publishes data that other organisations make decisions with: government departments, statistical agencies and research groups. They do not work from a local copy, and they cannot wait one out. A dataset that is unavailable is not a degraded experience — it is a figure that cannot be checked, a briefing that cannot be written, a model that cannot be run.

That changes the reliability conversation. On a consumer product, availability is a question of how many people notice and how annoyed they are. Here the consumers are organisations with their own deadlines, and an outage propagates: a missed update becomes a delayed publication somewhere else, with somebody else explaining it. The availability target has to be set accordingly, and it has to be something the platform is built against rather than hoped for.

What the target means in practice

The design target for the platform was 99.99999% availability, defined over a year. Stated plainly: about three seconds of unplanned downtime across the year, and no more. Five nines allows roughly five minutes; this is a hundred times tighter, which is enough to move it out of operations ambition and into architecture.

Two things follow from writing it down that way. It is a design target and not a measurement: it is the standard the design is held to, and the work was to make the design meet it rather than to report afterwards that it had. And the window matters as much as the number — the same percentage over a month is a different problem from the same percentage over a year, and the annual definition is the one the design answers to.

The engineering it forces

If three seconds a year is the budget, then any component that can be down for longer is a defect in the design, however unlikely its failure. That removes most of the downtime a team normally accepts:

• No single point of failure, including the boring ones — the load balancer that was fine because it had never been restarted, the single writer to a shared database, the DNS record nobody counted as infrastructure, the certificate renewal that needs a person present.
• Deploys that do not take the service down. A release that drains a node before replacing it, a schema change that is compatible one version either side, a feature that can be switched off without a deploy: all of it exists so that shipping stays outside the downtime budget. A maintenance window is downtime, scheduled or not.
• Monitoring and alerting as part of the system. Detection sits inside the budget rather than outside it: if nobody knows the platform is down, the clock keeps running. That means a health signal per dependency, alerts that name a symptom rather than a metric, and somebody whose job it is at three in the morning.
• Failure paths that are exercised. A standby that has never taken over is a hope. Failover, restore and rollback get run deliberately, on the real system, before they are needed, so that the first execution of a procedure is not also the first outage.

How you find out it is working

Availability at this level is a property you maintain, not a state you reach. Treating each failure path as a feature with an owner is what keeps the arithmetic honest: a runbook, a rehearsal, and a record of when it was last exercised and what happened.

That record matters more than it sounds. A design target keeps its value only while the people running the platform can say, without guessing, what would happen if a specific part of it died — and how long ago they last confirmed it. Once that answer becomes vague, the target has quietly turned back into an aspiration, and the arithmetic stops describing the system.

What the work demonstrates

Availability engineering at this level is mostly the discipline of refusing exceptions. Every component that stays in the design without a failover story, every procedure that lives in somebody’s head, every alert nobody owns is a claim against a budget of three seconds, and the claim is usually wrong.

It is the same call as any expensive architecture decision: cheaper to get right before it is frozen than to discover afterwards, and worth a second opinion while the options are still open. That is the shape of architecture and technical advisory work, of which this engagement was the deep end.

If you are being held to an availability target that the current design cannot meet, get in touch.