User Connectivity: Making the System Scale with Event Hub Partitions, ACA, and KEDA

Part 3 of the User Connectivity Architecture series.

Introduction

The first post in this series described the pattern: a heartbeat on a timer, an Event Hub, a worker writing sessions into Redis, and Redis key expiration driving facility online/offline status.

One detail matters later. The heartbeat interval is not hard-coded in the client. The API tells the client when to call next, and the default is 30 seconds.

The second post covered two years of running that in production.

This post is about the month it stopped working.

In January 2026 our heartbeat traffic went from boring to terrifying and stayed there for about four weeks. This is the story of what broke, why the original design had a ceiling we never noticed, and the changes that fixed it: more Event Hub partitions, Azure Container Apps, and KEDA.

The Storm

A normal day looked like this:

• 51,000-58,000 heartbeats per hour, hour after hour
• Roughly 15-16 events per second at idle
• Flat, predictable, forgettable

On January 5, around 7:00 AM PST, it stopped being flat.

Time (PST) Heartbeats/hour Baseline ~57,000 12:00 PM 80,005 1:00 PM 216,351 5:00 PM 343,480 9:00 PM 466,760

That is eight times normal event volume in a single hour, and it was still climbing.

Events were only half the story. SignalR connection counts told the other half. At the worst of it we were holding roughly eleven times the connections we normally maintain, and every one of those was a browser session we had to track, keep alive, and report status for.

It did not spike and recover. It stayed elevated for weeks while we hunted for the cause.

When we finally found it, the answer was almost funny: 507 zombie sessions that never ended, running months-old cached client code, and a single user account responsible for 33% of all our token API traffic.

One account. Eight times the load. Four weeks.

What Eight Times Load Actually Did

Here is the part that matters, and it has nothing to do with the number itself.

Our Event Hub had one partition.

In Event Hubs, the partition count sets how much you can read in parallel. Within a consumer group, each partition is owned by one processor at a time.

One partition means one owner. Everything else waits its turn. More instances, more CPU, better code, none of it helps. There is only one lane.

So while events poured in at well over a hundred per second, HeartbeatMonitor processed them single-file. A handful per second. Sometimes it felt like one.

Incoming and outgoing stopped matching, and the backlog only grew. Facility connectivity status, the thing hospitals and EMS actually look at, started lagging reality.

Then the Consumption plan added its own failure mode: SNAT port exhaustion.

Under sustained storm traffic, outbound connections tied up SNAT ports faster than they were released. We ran out. New connections slowed, then failed outright. Failed calls made clients retry. Retries made more load. More load exhausted more ports.

We had built a feedback loop, and it was feeding itself.

Consumption gave us nothing to work with either. No instance visibility, no per-instance control, no way to reason about what any single worker was doing. And since scale-out was capped by the partition count anyway, more instances would not have helped even if we could see them.

The pattern was fine. The plumbing under it had a ceiling of one.

Buying Time

Before rewriting anything, we needed to survive.

That heartbeat interval from the intro is a variable, and the API hands it to every client on every call. It normally sits at 30 seconds. We set it to 90 seconds, which cut incoming events by roughly two thirds. No deploy, no client update.

It worked. It was also a tourniquet. Connectivity status got noticeably coarser, but it kept the lights on while we built the real fix.

Worth stealing regardless of your architecture: let the server tell the client how often to call. That one design decision was the difference between a bad month and an outage.

Fix 1 – Give the Queue More Lanes

On our Event Hub tier we could not change the partition count on the existing hub, so we created a new one: heartbeat-p8, with 8 partitions and a dedicated heartbeat-monitor consumer group.

That is the whole fix, and it is the foundation for everything after it. Eight partitions means eight streams can be processed in parallel instead of forcing all the work through one. A noisy workload has far less ability to stall the entire pipeline.

Eight lanes only help if you have eight drivers, though. That is the next part.

Fix 2 – Why KEDA

KEDA watches how many unprocessed events are sitting in the Event Hub and scales the worker for you. No timers, no guessing, no manual scale-out.

Here is the behavior that sold us. As lag grows, KEDA adds replicas. The Event Hub processor then spreads partition ownership across whatever replicas are running, each one checkpointing independently. Events fan out across partitions, replicas fan out to match.

Set maxReplicas to the partition count. This is the one number worth getting right.

There are only eight partitions to own, so a ninth replica has nothing to take. At best it sits idle. At worst it adds avoidable rebalancing, which means brief churn and duplicate processing from the last checkpoint. Extra replicas do not buy throughput.

All of it lives in Bicep, so every environment gets the same behavior with different numbers:

scale: {
  minReplicas: minReplicas
  maxReplicas: maxReplicas
  rules: [
    {
      name: 'eventhub-keda-rule'
      custom: {
        type: 'azure-eventhub'
        metadata: {
          consumerGroup: toLower(eventHubConsumerGroup)
          unprocessedEventThreshold: kedaUnprocessedEventThreshold
          activationUnprocessedEventThreshold: '0'
          checkpointStrategy: 'blobMetadata'
          blobContainer: 'heartbeat-checkpoints'
        }
        auth: [
          {
            secretRef: 'eventhub-connection'
            triggerParameter: 'connection'
          }
          {
            secretRef: 'storage-connection'
            triggerParameter: 'storageConnection'
          }
        ]
      }
    }
  ]
}

Enter fullscreen mode Exit fullscreen mode

Two things in there cost us real time, so take them for free:

• checkpointStrategy: 'blobMetadata' – get this wrong and KEDA reads your lag as zero and never scales
• If an old Function App is still running against the same hub and consumer group, it keeps competing for partition ownership and the Container App never settles. Stop it first.

Non-production environments run 2 partitions with minReplicas: 0, so they scale to zero and cost nothing when nobody is testing.

Fix 3 – Make the Image Small Enough to Matter

Fast scale-out is a lie if your container takes 45 seconds to start.

So the HeartbeatMonitor image got stripped down:

• No ingress. It is a worker. Nothing calls it. It has no reason to have a networking surface.
• No health check libraries. ACA probes handle liveness.
• No extra packages. If it is not on the hot path between Event Hub and Redis, it is not in the image.

The result is a small image sitting in ACR, ready to go. During a storm, the time between “KEDA notices lag” and “a new instance is processing events” is measured in seconds, not minutes. That difference is the entire point.

There is a quieter benefit too. Moving off Consumption to a container we control means we can actually see and tune each instance: logs, probes, resource limits, startup behavior. After a month of flying blind, that visibility was worth as much as the scaling.

SessionMonitor – A Different Job, Different Rules

Not everything should scale out.

SessionMonitor reconciles Redis session state against SQL and pushes facility status changes. Running two of them means doing the same work twice. So it moved to ACA as well, but fixed at exactly one instance.

What it got instead:

A speed knob in Redis. The poll interval is read from Redis at runtime, not from config. Speed it up, slow it down, or effectively pause it, mid-incident, with no redeploy.

redis-cli SET SessionMonitor:PollIntervalSeconds 15

Enter fullscreen mode Exit fullscreen mode

Catch-up on startup. Every session lives in Redis as two keys: a short-lived one that expires when the user goes quiet, and a long-lived one that sticks around as the record.

The short key expiring is the signal that someone went offline. But Redis expiration events are fire-and-forget. If the container is down at that moment, the signal is gone and that session stays marked online forever. In the old Function App, that is exactly what happened.

So SessionMonitor now starts by comparing the two sets of keys. Anything holding a long-lived key with no short-lived partner expired while we were away. It clears those first, then starts listening for live events.

Simple idea, and it means a restart or a deploy no longer costs us accuracy.

Proving It

Honest answer first: we could never reproduce the real storm in a load test. Not even close. Four weeks of a broken client hammering production is not something you manufacture with JMeter on a Tuesday.

So we stopped trying to recreate the storm and tested the thing that actually mattered, which is the response.

The method was simple:

1. Stop the Container App entirely
2. Run the JMeter load test for several minutes against the heartbeat API
3. Let the queue pile up to tens of thousands of unprocessed events
4. Start the Container App and watch

Production normally keeps one replica running at all times. We stopped it on purpose so the test would show the worst case: a cold start into a full queue.

KEDA drove it up to all eight replicas almost immediately. All eight partitions were consumed in parallel, the backlog drained, and once the queue was clear it scaled back down on its own.

That is the behavior we needed to see. Not “can we survive a storm we already survived,” but “when the backlog appears, does the system react instantly and without anyone being awake.”

Where We Landed

• Production reads and writes to the queue stay in sync, including under heavy load
• The heartbeat interval is back to normal, no tourniquet needed
• No SNAT port exhaustion since the migration
• Processing capacity scales with demand instead of collapsing under it
• Every environment is defined in Bicep, so dev, QA, staging, training, and prod behave identically at different sizes

The architecture from the first post, heartbeat to Event Hub to Redis expiration to SignalR, was never the problem. It held up for two years and it holds up today.

It just needed more than one lane, and something smart enough to fill them.

Previous posts in this series:

1. User Connectivity: A Real-time Web Solution for Online and Offline User Status
2. User Connectivity: Two Years in Production – Lessons Learned and New Patterns