Aleksey Dorogov freelance software engineer
← All writing Reliability

What I check first when production is down

Jan 2026 1 min read Aleksey Dorogov

Under pressure, people debug the interesting hypothesis first. Almost always it is something boring, so I work the boring list in order, out loud, in whatever channel the client is watching.

What changed. A deploy, a config edit, a DNS change, a dependency that auto-updated, a certificate that expired on a schedule set two years ago. If something changed within the window, that is the suspect until proven otherwise.

What is full. Disk, connection pool, memory, inode count, a queue nobody watches, an API quota that resets monthly. A startling number of outages are a resource at one hundred percent with no alert on it.

What is lying. A health check that returns two hundred while the service is useless, a cache serving a stale success, a load balancer routing to an instance that came back wrong. Trust logs and timestamps over dashboards.

Then, and only then, the code. And once it is up: write down what the alert should have been, and add it that week. The value of an outage is entirely in the alert you build afterwards.

Sitting on a version of this problem right now? I'd rather look at it than guess.

Email me
Site down? Message me now Live chat · usually reply in minutes