A production bug, explained

The singleton that wasn’t

Why a real-time feed worked on my laptop, passed CI, and delivered nothing in production.

Charlene Athena · github.com/lilyofthevaleys

The symptom

I was helping take a sports-recruiting product live. Athletes had a dashboard with a live activity feed: when a coach left feedback, a counter ticked up and the feed updated without a refresh. It was built the obvious way. A mutation published an event to an in-memory event bus, and a long-lived Server-Sent Events (SSE) connection subscribed to that same bus and pushed events to the browser.

Locally it worked. Tests passed. In production, the SSE connection opened fine and stayed open, but no event ever arrived. No error, no crash: just silence.

// lib/bus.ts — "there is only one of these", or so I thought
export const bus = new EventEmitter();

// a Server Action (the mutation)
bus.emit("activity", { type: "coach_feedback", athleteId });

// a Route Handler (the SSE stream)
bus.on("activity", (e) => stream.write(`data: ${JSON.stringify(e)}\n\n`));

The wrong mental model

The code assumes that import { bus } gives every file the same object. That is true inside one module graph in one process. It quietly stops being true in two situations:

  1. Separate module graphs. A framework like Next.js can compile Server Actions and Route Handlers into different bundles. Each bundle evaluates lib/bus.ts on its own, so there are two buses. The Server Action shouts into one; the SSE stream listens to the other.
  2. Separate processes. On a platform like Cloud Run, a second server instance has its own memory. Even a truly process-wide object is invisible to a user whose SSE connection landed on the other instance.

Try it. Publish an event and watch where it lands.

Press the button to publish an event.

With a module-level bus, the event never leaves the Server Action’s copy. Moving the bus onto globalThis is a common patch: it makes one object per process, which fixes the bundle problem. Then switch to two instances and it fails again, but only for some users, which is worse because it looks random.

The fix

The fix was to stop treating process memory as shared state. The mutation now writes an ActivityEvent row to the database, and the SSE route polls for new rows and streams them. The database is the one thing every bundle and every instance already agrees on.

The trade-off: polling adds up to one poll interval of latency and some database reads. For an activity feed that is fine. If it ever is not, the next step is a real pub/sub layer (for example Redis or Postgres LISTEN/NOTIFY) rather than going back to memory.

What I took from it