Every graph was green.

Latency nominal. Error rate flat. Pods healthy, spread across three availability zones. No alerts had fired in hours.

And the service was down — and had been for most of the morning.

This is not a hypothetical. On October 20, Google Cloud’s London region went down for roughly six hours and ten minutes. Internal dashboards at companies running there looked fine for a while. The workloads were distributed. The redundancy was configured. It didn’t matter.

That gap — between the controls you have and what those controls actually guarantee — is the thing I keep finding in this month’s incident reports. Three of them, in the same week, from three unrelated parts of the stack. All three are worth reading as one story.

A signature proves what was built. Not where it came from.

On October 21, the AI gateway project LiteLLM published version 1.107.0 to PyPI and its container registries. It was signed. It came through the official pipeline. And it was malicious.

The root-cause analysis, published October 23, walks the chain. A master key had leaked into a public repository back in May. The commit was removed — the key never was. In October an attacker used it to obtain an organization owner’s GitHub token, and from there maintainer credentials through GitHub Actions.

Then the critical step: the attacker changed the release pipeline’s source repository from BerriAI/litellm to a lookalike, Berri-AI/litellm. The legitimate pipeline did what it was built to do. It built, signed, and published.

The public tag looked clean. The signature was valid. Neither of those facts proved the intended source had been used.

We have spent years getting good at artifact signing, and it is genuinely valuable. What most teams have not solved is source authorization — proving that the build consumed the repository you meant, not merely that the output was signed by your key. A release signature answers “was this built by us?” It does not answer “from what?”

Your secrets don’t live in production anymore. They live in your platform.

On October 20, Railway — the hosting platform — was attacked. Its preliminary report on October 22 said attackers reached internal systems and obtained some customer environment variables. The company estimated at most 1.85% of workspaces — around 100,000 — had potentially exposed variables, and noted the attacker seemed especially interested in AI provider API keys.

That count is potential exposure, not confirmed compromise. Take the number with the caveat. The operational problem is the point.

Railway temporarily disabled builds and deployments. Applications stayed up. So for a period, customers were told: your credentials may be exposed, you should rotate them — while the platform you’d normally deploy key changes through was frozen.

For a small startup with sixty environment variables shared across services, that’s not a rotation. That’s an outage-shaped problem with a security deadline attached.

The deeper issue is where the secrets were to begin with. Modern teams secure their production application reasonably well, then keep powerful, long-lived credentials in a platform’s environment variables, a CI system, and a handful of SaaS integrations. Protect the app, and you’ve covered maybe a third of your exposure.

Redundancy is not independence.

Back to Google London. The preliminary report attributes the initial failure to a power disruption in europe-west2-b. Backup-generator automation failed to transfer the facility load after UPS batteries were depleted. One zone went dark.

The outage didn’t stay in that zone. Two mechanisms carried it out:

1. Regional control-plane components were making synchronous calls to metadata and catalog systems in the failed zone. Those calls timed out, and the timeouts propagated.
2. Failover traffic and database activity saturated the surviving zones. The capacity to absorb a zone’s worth of load flatly wasn’t there.

Both of these are ordinary architectural failures dressed up as bad luck. Anyone running “three AZs” as their resilience story has, in the first case, an unnoticed synchronous dependency on the failed zone; in the second, an untested assumption about failover capacity.

Three availability zones tell you that one physical zone can vanish and you’ll still have running infrastructure. They do not tell you that your traffic, your control plane, and your dependent services are independent of that zone. Those are different guarantees, and only the first one comes free with a multi-AZ setup.

The pattern

Read these three together and something sharper than “infrastructure is hard” falls out.

Each control is real. Each one is worth having. Each one answers a narrower question than we assume:

• Signing answers “did our key sign this?” — not “did we intend to build it?”
• Multi-AZ answers “can we lose a zone?” — not “are we independent of one?”
• A green dashboard answers “are our systems responding?” — not “are they telling the truth?”

None of this is a reason to throw the controls away. It is a reason to stop treating them as proof of something they were never designed to prove.

The same week produced quieter versions of this in AI systems. SGLang issues showed an inference server answering requests correctly at the API level while returning wrong output underneath — a retracted request resuming with another request’s cache state, KV-cache scales failing to transfer. A healthy endpoint and a correct one are not the same thing. Bayer’s production writeup of its agent platform reported a similar shape: HTTP-successful responses that represented failed agent executions.

Different layers. Same gap.

What to actually do about it

If the thesis is right — if our controls are answering narrower questions than we assume — then the work isn’t buying more controls. It’s writing down what each one actually guarantees, and finding where the guarantee runs out.

In practice, that means a handful of uncomfortable questions:

• For every green signal you trust: what specific claim does it support, and what does it not cover? Where’s the nearest thing it can’t see?
• For your release pipeline: can you prove the intended repository was the one built, or only that the output was signed?
• For your secrets: if your platform told you to rotate everything tomorrow, how long would it take — and could you deploy the changes while it was down?
• For your resilience story: do you know what your system synchronously depends on in the zone you claim to survive, and have you measured failover capacity under real load?
• For your AI endpoints: are you testing correctness, or just availability and tokens per second?

These are not exotic questions. They’re the questions our controls have been quietly letting us skip.

The gap between a control and its guarantee is where the next round of breaches and outages is going to live. Green dashboards are not going to tell you when you’re standing in it.