Hunting a connection leak: from an OOM to a one-line fix in a third-party library

A GraphQL gateway sits between our web app and several backend services. The browser opens one sync subscription over Server-Sent Events; the gateway fans that out to a per-service SSE subscription each, and merges them back into a single stream. A fairly standard proxied-subscription topology:

browser ──SSE──▶ gateway ──SSE──▶ service A
                        ├──SSE──▶ service B
                        └──SSE──▶ service C

The bug: one of the backend services kept running out of memory in production and dying. This is the story of chasing that from “the service OOMs” down to a single wrong method call in a dependency, and the several confident-but-wrong conclusions I made along the way.

1. Symptoms and observations

The first signal was crude: a backend service OOM’d in production, a few days after we’d shipped an unrelated reconnect fix. Heap limit reached, process killed, restarted automatically.

The load balancer metrics told a cleaner story once I looked at the right one:

• The active-connection count climbed monotonically — a slow ramp of a few hundred connections an hour, all day, sawtoothing back to near-zero only when the process restarted.
• Request volume was almost flat and almost zero. The service was doing barely any work.
• At the peak before a crash: ~10,000 concurrent connections against fewer than 200 requests an hour.

Before the fix. Active connection count (right) ramps monotonically and only sawtooths back to zero when the process restarts; request volume (left) stays low. Connections climbing while the service does almost no work is the signature of a leak, not load.

Two early mistakes worth naming, because they nearly sent me the wrong way:

• I read the connection metric with the wrong aggregation and got 0. That metric is only meaningful summed, not maxed — and reading it wrong briefly hid a ~10,000-connection leak. (If your monitoring shows zero for something you can plainly see happening, suspect the query before the system.)
• I then divided an hourly sum by 60 to get a “concurrent” figure and stated it as fact. It’s murkier than that — the metric is sampled several times a minute. The absolute number was never the point; the ratio and the monotonic climb were.

The observation that actually cracked it came later: a second service — near-idle, no large payloads, no heavy workload — showed the identical sawtooth. That ruled out every theory that depended on payload size or how much work a service was doing. Whatever this was, it lived in the code path all the services shared: the gateway’s sync subscription.

Lesson already forming: a control case you didn’t design is worth more than any amount of reasoning. That near-idle service was the control, and it was the single most useful data point in the whole investigation.

2. First assumptions and guesses

The initial mental model — reasonable, and partly right — was:

A heavy request exhausts CPU → sync falls behind → the gateway hammers the service trying to re-establish connections → it falls over.

So the first guesses were all about load and payload:

• A pathological request with a huge amount of work behind it.
• The gateway broadcasting a large object to many subscribers (“fan-out amplification”).
• The browser reconnecting aggressively and creating a connection storm.

The reconnect angle had real history behind it — we’d genuinely had reconnect storms before — so it was a credible anchor. That credibility is exactly what made it hard to let go of.

3. First wrong theories (a partial list)

I want to be honest about the volume of dead ends, because the false-confidence pattern is the interesting part:

• “Send a smaller payload on the once-a-day broadcast path.” There’s a code path that periodically broadcasts a full object. I was sure this was the amplifier. It turned out that payload is load-bearing — the client relies on it to reconcile its local state — so “just send less” would have been a silent data-corruption bug. Killed by reading the consumer, not the producer.
• Blaming observability overhead. At one low point I attributed a memory gap to tracing/profiling/log volume. It was hand-waving to cover “I don’t know yet,” and I was rightly called on it. If your explanation is “the monitoring did it,” you almost certainly haven’t found the cause.
• “The stream-merging helper doesn’t clean up its children.” The gateway merges the per-service streams with a small combinator. I suspected it leaked children on teardown. I read the installed source: it calls return() on every child in a finally. It’s correct. Not it.
• “The gateway’s subscribe path has an open-window race.” My first actual code change. There’s a window where the client can disconnect while the resolver is still opening the upstreams. I fixed that window, shipped it, and the production curve didn’t move. It was a real bug — but a tiny fraction of the leak. I’d found a bug, convinced myself it was the bug, and the metrics said otherwise.

Lesson: “I found a bug and it’s plausible” is not “I found the bug.” The only arbiter is a measurement that changes.

4. Buying stability, then a deliberate stopgap

Two things about the sequencing here, because they’re the part I’d actually recommend to someone else.

First, we took the pressure off. Rather than debug a burning production, we put in a simple automatic guard — restart the affected component when connections crossed a safe threshold, well below the danger zone. Ugly, but it turned “prod is crashing” into “prod is stable and self-healing,” which bought the thing that matters most in an investigation like this: the calm to be slow and correct instead of fast and wrong.

Second — knowing the true root cause would take real time to find, and not wanting to rush it — I designed a deliberate workaround as a bridge. By this point I understood the leak as “abandoned upstream subscriptions never get torn down,” and I knew a property of the system I could exploit:

The gateway’s SSE consumer sits parked, await-ing the next event from the upstream. On a quiet stream no events arrive, so it parks indefinitely and its teardown never runs — but any event unparks it and lets teardown complete.

So the workaround was: nudge each upstream to emit a tiny no-op event, which wakes every parked consumer for that scope and lets it clean itself up. A targeted, reversible bridge — explicitly a stopgap, never meant to be the final answer.

I built it, then had it torn apart by several independent review passes before it went anywhere. That review earned its keep — it caught things I’d have shipped:

• Every one of my new tests was structurally incapable of failing — I’d asserted against a field that didn’t exist in that context, copy-pasted across places.
• The nudge fanned out with the wrong cost characteristics on the reconnect path — the same amplification shape as the incident itself.
• It only cleaned up scopes that saw new activity; genuinely idle ones would still leak.
• A try/catch around a fire-and-forget promise couldn’t actually catch a rejection — on the runtime’s defaults that’s a process crash waiting for the right trigger.

None of that made the workaround wrong — a bridge is allowed to be imperfect. But because prod was already stable, there was no reason to ship an imperfect bridge in a hurry. The right move was to spend the calm we’d bought on the actual root cause. So that’s what I did.

Lesson: stabilise first so you can investigate without a clock running. A crude auto-recovery that buys you calm is worth more than a clever fix shipped under pressure.

5. Finding the root cause

Two tools made the difference, and I should have reached for both far earlier.

A reproduction harness

I stopped guessing against production and built a local one:

• A tiny script that opens N real SSE subscriptions through the local gateway and holds them.
• A counter that reads the kernel’s own TCP table (/proc/net/tcp) inside the gateway container and reports established connections per upstream, keyed by the exact local ports my script opened — so unrelated background traffic couldn’t contaminate the count.

Now the experiment was: open N, kill -9 the client, watch those specific sockets. Sixty seconds, deterministic, repeatable. Every candidate fix became a one-minute test instead of a multi-day production wait.

Instrumentation you can trust

When you patch a dependency in place to test a theory, it’s surprisingly easy to edit a build the runtime never loads. A package can ship both an ESM and a CommonJS build, and more than one version can be resolved in the tree — so the file you opened and the file actually executing aren’t necessarily the same one. A negative result from the wrong file looks exactly like a negative result from a bad fix.

So when patching a dependency in place, I treat one thing as non-negotiable: stamp the code you think you’re running with a version string and a fresh random nonce per test, and confirm the stamp appears in the logs before trusting any result. With that in place:

PATCHMARK v3.3.0 nonce=9C8F -> reader.cancel()   × 12
my sockets: 12  →  0 of 12 ESTABLISHED at t+5s, t+20s, t+45s

Twelve sockets, twelve stamps, matching nonce. No ambiguity.

The actual bug

With correct instrumentation, the trace was unambiguous. In the version we ran, the SSE consumer was a bare async generator with no cleanup at all — parked on await, and because a suspended generator can’t process a queued return(), teardown was literally unreachable while it waited for a chunk that (on a quiet stream) never came. A newer version had fixed that — rewritten it with a proper teardown path.

But the newer version still leaked, and here’s the crux. Its teardown does this:

stop.then(() => {
  subscriptionCtrl?.abort();
  if (body.locked) {
    reader.releaseLock();   // ← detaches the reader; the body is never cancelled
  }
});

ReadableStreamDefaultReader.releaseLock() releases the reader’s lock on the stream. It does not cancel the stream. The response body stays alive, so the underlying HTTP connection is never released, so the socket stays ESTABLISHED. The one call that actually tears the connection down — reader.cancel() — was only on the error path, never the normal one.

Two separate bugs, stacked: the old version made teardown unreachable; the new version made it reachable but ineffective. Conflating them was what kept “just upgrade the dependency” looking like it might work when it couldn’t.

Proof, same harness, same file, back to back:

Build	Sockets still open after client death
version we ran	0 of 11 closed
latest, stock	0 of 12 closed
latest + releaseLock() → cancel()	12 of 12 closed within 5s

6. Applying the fix

The fix is one line:

-      reader.releaseLock();
+      reader.cancel().catch(() => {});

(.catch(() => {}) because cancelling an already-errored stream can reject, and teardown shouldn’t surface that.)

After the one-line fix — the same metrics, over a comparable window. Connection count (right) now sits in the tens and tracks actual request volume instead of climbing without bound. No more sawtooth, no more restarts.

Rollout came in two parts:

1. Immediately, in our own repo: a pinned dependency version plus a package-manager patch applying the one-line change, so we replaced the crude restart guard with a real fix without waiting on anyone. Verified through the real install path — not a hand-edited node_modules — again with the version+nonce stamp.
2. Upstream: the honest fix belongs in the library. A wrinkle: the package had moved repositories between the version we ran and the current one, so the obvious repo was the wrong one — worth checking the package metadata rather than assuming. I fixed it at the source (not the build output), added the project’s required changeset, and ran their full CI matrix locally — format, lint, types, peer-deps, build, bundle, unit tests — all green (after verifying that two already-failing test files fail identically on a clean checkout, so I wouldn’t be blamed for them). Then a fork, a pull request, and an issue documenting the leak with the evidence table.

One more sting in the tail: while cleaning up, a service failed to start — a generated artifact was stale, so its schema and its code disagreed. My typecheck, lint, and unit tests had all passed. The process just didn’t boot. Passing checks are not running software. I’d have caught it by starting the thing once.

7. Lessons learned

• Stabilise first, then investigate without a clock running. A crude auto-recovery that keeps prod alive is worth more than a clever fix shipped under pressure. The calm it buys is what lets you be correct instead of fast.
• Build the reproduction harness first, not last. The single highest-leverage thing I did was the local “open subscriptions, kill the client, count the sockets” loop. It turned days of production inference into 60-second experiments. I built it embarrassingly late.
• Prove your instrumentation is live before trusting a negative result. When you patch a dependency in place, stamp it with a version and a per-run nonce and confirm the stamp shows up before believing anything — a package can ship multiple builds and resolve multiple versions, so it’s easy to measure a file the runtime never loads. A negative result from unverified instrumentation is worth nothing.
• A control case you didn’t engineer is gold. The near-idle service with the same leak is what collapsed the search space from “workload” to “shared subscription path.”
• A deliberate stopgap is not a failure; a stopgap shipped in a panic is. Knowing the root fix would take time, building an explicit reversible bridge was the right instinct — and having bought stability first meant I could choose the root fix over the bridge instead of being forced onto it.
• Distinguish “what makes it unfixable” from “what makes it leak.” There were two stacked bugs. Treating them as one made an upgrade look like it might be a total fix when it couldn’t be.
• Adversarial review earns its keep. Independent reviewers found tests that couldn’t fail, a latent process-crash, and a bad amplification profile — cheap insurance against your own momentum.
• releaseLock() is not cancel(). Releasing a stream reader’s lock detaches the reader; it does not close the stream or its connection. If you want the resource freed, you must cancel it. A small API distinction with ~10,000 leaked sockets and an OOM downstream of getting it wrong.
• Passing checks ≠ running software. Typecheck, lint, and tests were green on a service that crashed on startup. Boot the thing.
• Own the “I don’t know” honestly. The lowest-quality moment was inventing a plausible cause to avoid admitting I was stuck. The investigation only moved once I stopped doing that and started measuring.

---

The fix upstream: one line, releaseLock() → cancel(), in the SSE teardown of a widely-used GraphQL-over-HTTP executor. Still present in the latest published version at the time of writing — so if you run a gateway that proxies SSE subscriptions, you may be leaking connections right now.