UptimeProject
Back to blog

The endpoint you monitor decides the outage you can detect

8 September 2026 UptimeProject
methodology monitoring reliability

Once this leaderboard had 90 days of measurements behind it, we audited them for the boring failure modes: gaps in coverage, probes that disagreed too often, aggregates that did not reconcile with the raw rows underneath them.

We found something worse than a gap. Seventeen of the fifty services we were monitoring at the time were structurally incapable of recording an outage. Not unlikely to. Not slow to. Incapable. Every one of them would have reported 100.00% availability through a total, hours-long failure of the API being measured, and the number would not have been a lie by any rule we had written down. It would have been the correct output of a check that was asking the wrong question.

Nothing was broken. No probe had crashed, no data was missing, no code was misbehaving. The 17 services had been measured continuously and correctly for three months, and the resulting availability figures were worthless. Everything worked and the answer was meaningless. That combination is the part worth writing about, because it does not announce itself. There is no alert for it, no dashboard that turns red, and no amount of uptime on your monitoring system will surface it.

This is what happened, why the rule that caused it was defensible, and the general form of the mistake, which we suspect is sitting in a lot of production monitoring right now.

The rule

An HTTP check has to decide what counts as success. Ours, until this week, used the split that most monitoring tools use and that most people would write down if you asked them at a whiteboard:

Response Verdict
1xx, 2xx, 3xx, 4xx up
5xx down
timeout, TLS error, DNS failure, connection refused down

The interesting line is 4xx. Treating it as success looks wrong at first glance and is usually right. A 4xx is the server telling you, in detail, that it understood your request and is declining it. 401 Unauthorized means the request reached the application, the routing worked, the TLS terminated correctly, and an auth layer made a decision about it. A service that returns a crisp 401 in 40 milliseconds is not having an outage. It is working perfectly and you are not logged in.

Get this wrong in the other direction and you generate constant false alarms. Point a check at any authenticated API without credentials, call 4xx a failure, and you will page someone every sixty seconds forever. So the rule is sound. We chose it deliberately in April, wrote down the reasoning, and it was the right call for the question we were asking at the time.

The URL

Here is the other half. This was the check for OpenAI:

checks:
  - id: openai-api
    type: http
    target: https://api.openai.com/
    interval: 60s

https://api.openai.com/ is the bare origin. It looks like the obvious thing to monitor. It is the hostname the API lives on, it resolves, it has a valid certificate, and if you curl it you get a response rather than an error. It seems like a reasonable proxy for “is the OpenAI API reachable”.

It is not, because there is no route at /. The API serves /v1/chat/completions, /v1/models, and so on. A request to the bare origin never reaches anything that knows what the OpenAI API is. It hits the edge, finds no matching route, and gets rejected.

Now put the two halves together. The check asks a URL that can only ever return 4xx. The rule says 4xx is up. The check therefore returns “up” on every single execution, permanently, regardless of the state of the service. Its output is a constant. Across the 17 affected services, roughly 655,000 checks had run over three months and every one of them had returned 4xx and scored as success.

To be precise about what this does and does not mean: those services were not immune to all failure. The check would still have caught a DNS failure, an expired certificate, a connection timeout, or the edge going away entirely, because those are transport-level failures that never produce an HTTP status at all. What it could not catch was any failure behind the edge. A completely broken API sitting behind a healthy load balancer scores 100%. And “the application is broken but the front door is fine” describes a large share of real outages, arguably the majority of the ones that matter, since the transport-level ones tend to be brief and loud while application failures are long and quiet.

So the 17 services were being measured for the failure modes least likely to happen, and were blind to the ones most likely to.

The one that was worse

Two services, OpenAI and xAI’s Grok, were not returning an ordinary 4xx. They were returning 421 Misdirected Request.

421 is unusual and worth knowing about. It means the server received a request on a connection it does not consider valid for that hostname, commonly a TLS or HTTP/2 connection coalescing mismatch, where the client reused a connection for a host the server will not serve on it. It is a connection-level rejection. Nothing in the application saw the request. It is about as far from evidence of health as an HTTP status can get while still being an HTTP status.

And 421 is in the 4xx range, so we were scoring it as up.

This is the failure mode at its purest: a status code whose entire meaning is I did not process this request being recorded as proof that the request was processed successfully.

What we changed

There were two possible fixes and they solve different halves of the problem.

Point the check at a real route. This is the more important one, and it is the one that would have prevented the whole thing. https://api.openai.com/v1/models is an actual endpoint. An unauthenticated request to it reaches the application, passes through routing, and gets a considered 401 from the auth layer. That 401 is meaningful in a way the previous response never was: it means a specific chain of infrastructure worked.

Let a check assert the status it expects. Retargeting alone gets you a meaningful response, but the rule still says any 4xx passes, so the check would still accept 403, 404 or 429 interchangeably. So we added an optional field:

checks:
  - id: openai-api
    type: http
    target: https://api.openai.com/v1/models
    interval: 60s
    expected:
      status_codes: [401]

When status_codes is present it replaces the 1xx–4xx rule for that check. Anything outside the set fails, with a new error code, http_unexpected_status. When it is absent (the case for 33 of the 52 services now on the board), the original semantics apply unchanged. The 4xx rule was never wrong in general, and we did not want to break the services where it was doing the right thing.

The assertion is what converts a steady 401 from a curiosity into evidence. A stable response only tells you something if a change away from it would be treated as a failure. Without the assertion, “this endpoint returns 401 every minute for 90 days” is a fact you are not acting on. With it, that stability becomes the baseline, and any departure from it (the route disappearing, the auth layer failing open, the edge starting to serve an error page) registers as an outage.

We verified every expected value across all five of our probes before writing it down, so that no assertion encodes a response that only one network path happens to see. The seventeen landed as:

A sample of what the retargeting looked like:

The 421 responses are gone. OpenAI and Grok now answer 401 from all five vantage points.

Reading the results honestly

Two things follow from this that we would rather state ourselves than have someone else point out.

The first is that availability figures for those 17 services before September 2026 should be treated as unverified rather than as measurements. They are not fabricated: the checks ran, the responses were real, the timestamps are accurate. But the number could only ever have gone down for reasons unrelated to the health of the API, so a 100.00% in that window carries almost no information. We have not deleted the data, because deleting inconvenient history is its own kind of dishonesty, but we are not going to defend it either.

The second is that the services carrying an assertion (19 of 52 today, these seventeen plus two added since) are measuring something slightly different from the other 33, and are not perfectly comparable on error-code breakdown, because only they can emit http_unexpected_status. That is a real if minor wrinkle in the dataset and it is documented rather than smoothed over.

The general form

Strip out the specifics and the shape of this is portable, which is why it is worth your fifteen minutes rather than just ours.

A check that has never failed is not necessarily a healthy service. It might be a check that cannot fail. These look identical on every dashboard ever built, a flat green line, and the only way to tell them apart is to ask what response the check is actually getting and whether any plausible outage would change it. If you have a monitor that has been green for a year, that is the first one to audit, not the last.

The bare origin is almost never the right target for an API. It is the most natural URL to type and it usually exercises none of the application. If your target has no path component, you are probably measuring your provider’s load balancer.

Prefer a status assertion over a status class. “Not a 5xx” is a weak claim. “Exactly a 401” is a strong one. The stricter assertion costs one line of config and converts a passive observation into an active test. The tradeoff is real: if a vendor legitimately changes its unauthenticated response from 401 to 403 during an auth rewrite, your check fires and you edit one line. That is the assertion doing its job, and a far better failure mode than silence.

Watch for statuses that mean “I did not process this”. 421 is the vivid example, but 404 from a CDN, 502 from a proxy that never reached your origin, and 403 from a WAF all share the property of being generated by infrastructure in front of the thing you meant to measure. A status code tells you which component answered, and that is not always the component you were asking about.

Verify from more than one place before you encode an expectation. We check every assertion against all five probes precisely because a response that differs by network path is a response you should not be hard-coding.

The uncomfortable part is that no monitoring system can find this for you. Every layer of ours was working: the probes ran on schedule, the data landed, the aggregates computed, the API served. A check that returns a constant is indistinguishable from a check on something that never breaks, and no amount of observability tooling closes that gap. The only thing that finds it is someone sitting down with the actual response bodies and asking whether the check would notice if the service died.

What we still cannot see

In the same spirit, the current setup has limits we have not solved.

Our checks are unauthenticated. We can confirm that an API routes a request and that its auth layer responds correctly, and we cannot confirm that an authenticated call would return the right data. An API that has lost its database but still rejects anonymous requests correctly will read as up here. Closing that gap means holding credentials for fifty vendors, which brings problems of its own (cost, key rotation, the risk of our measurement traffic being treated differently from real traffic), and we have chosen not to for now.

We also measure reachability and correctness of response, not latency degradation or partial failure. A service serving correct responses at ten times its normal latency is up by our definition. There is a reasonable argument that it should not be, and it is on the list.

Everything above is in our methodology, which we keep current, including the parts that are unflattering. We keep a written decision log covering both this change and the earlier rule it overrides, and we do not edit old entries to look smarter in retrospect. The earlier decision was not wrong about the question it asked. It just did not ask what its rule would do to a leaderboard where a third of the entries were pinned to 100% by construction.

If you run monitoring of any kind, go and look at your longest-green check today. There is a reasonable chance it is not measuring what you think it is.