Building Resilient Real-Time Systems: WebSockets, Redis, and High Availability

Originally published on tamiz.pro.

Real-time systems are foundational to modern interactive applications, from chat platforms and collaborative tools to live dashboards and gaming. The core challenge lies not just in establishing a real-time connection but in ensuring that connection, and the data flowing through it, remains resilient, scalable, and highly available even under adverse conditions. This deep-dive explores how combining WebSockets for persistent, bidirectional communication with Redis for robust state management and message brokering can address these challenges.

The Real-Time Challenge: Beyond Basic Connections

Establishing a WebSocket connection is straightforward. Libraries and frameworks across languages offer excellent support. However, a single WebSocket server is a single point of failure and a scalability bottleneck. As user numbers grow, you inevitably need multiple WebSocket server instances, which introduces new complexities:

• State Management: Where do you store user session data? It needs to be accessible by any server instance a user might connect to.
• Message Broadcasting: How do you send a message from one user to another if they are connected to different WebSocket servers? How do you broadcast a message to all connected users across all servers?
• High Availability & Failover: What happens if a WebSocket server crashes? Users need to be able to reconnect seamlessly to another available server without losing critical context.
• Scalability: How do you add more WebSocket servers as demand increases without re-architecting the entire system?

WebSockets: The Persistent Connection

WebSockets provide a full-duplex communication channel over a single, long-lived TCP connection. This eliminates the overhead of HTTP request/response cycles, making them ideal for low-latency, high-frequency data exchange. A typical WebSocket handshake upgrades an HTTP connection:

GET /chat HTTP/1.1
Host: example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13

Enter fullscreen mode Exit fullscreen mode

Upon successful handshake, the server and client can send messages back and forth independently. In a distributed setup, clients will typically connect to a load balancer, which then routes them to one of many WebSocket server instances.

Redis: The Backbone for Resilience and Scale

Redis is an in-memory data store known for its speed and versatility. It offers several features critical for building resilient real-time systems:

• Pub/Sub (Publish/Subscribe): This mechanism is perfect for broadcasting messages across multiple WebSocket server instances. When a message needs to be sent to a user or a group of users, the originating WebSocket server publishes it to a Redis channel. All other WebSocket servers subscribed to that channel receive the message and can then forward it to their connected clients.
• Key-Value Store: Redis can store session data, user preferences, and other ephemeral state information. This allows any WebSocket server to retrieve a user’s context, making sessions stateless from the individual server’s perspective and enabling seamless failover.
• Distributed Locks/Semaphores: For coordinating operations across servers or ensuring atomic updates to shared resources.
• Streams: A more advanced, log-like data structure that can act as a robust message queue, offering consumer groups and persistent message storage, ideal for event sourcing or guaranteed message delivery scenarios.

Architectural Overview

Let’s visualize the architecture:

graph TD
    Client1[Client A] --> LB(Load Balancer)
    Client2[Client B] --> LB
    Client3[Client C] --> LB

    LB --> WS1[WebSocket Server 1]
    LB --> WS2[WebSocket Server 2]
    LB --> WS3[WebSocket Server 3]

    WS1 -- Pub/Sub --> Redis(Redis Cluster)
    WS2 -- Pub/Sub --> Redis
    WS3 -- Pub/Sub --> Redis

    WS1 -- Get/Set State --> Redis
    WS2 -- Get/Set State --> Redis
    WS3 -- Get/Set State --> Redis

    Redis -- Persistence --> DB(Persistent Database)

Enter fullscreen mode Exit fullscreen mode

In this setup:

1. Clients connect to a Load Balancer, which distributes connections among WebSocket Server instances.
2. Each WebSocket Server handles a subset of client connections.
3. Redis Cluster acts as the central hub for:
   • Message brokering (Pub/Sub): When Client A sends a message to Client B (who might be on WS2), WS1 publishes the message to a Redis channel. WS2 (and all other WS servers) are subscribed to this channel, receive the message, and WS2 then forwards it to Client B.
   • Session State: User session data (e.g., connected user ID, current room) is stored in Redis. If WS1 crashes, Client A reconnects to WS2, which can retrieve Client A‘s session context from Redis.

Implementing Redis Pub/Sub for Message Broadcasting

Here’s a simplified example of how WebSocket servers can use Redis Pub/Sub.

WebSocket Server (Node.js with ws and ioredis)

// server.js
const WebSocket = require('ws');
const Redis = require('ioredis');

const wss = new WebSocket.Server({ port: 8080 });
const redisPublisher = new Redis(); // Used to publish messages
const redisSubscriber = new Redis(); // Used to subscribe to channels

// Map to store connected clients by their user ID
const clients = new Map(); // userId -> WebSocket instance

redisSubscriber.subscribe('global_chat_channel', (err, count) => {
    if (err) console.error('Failed to subscribe:', err);
    console.log(`Subscribed to ${count} channels`);
});

redisSubscriber.on('message', (channel, message) => {
    console.log(`Received message from Redis on channel ${channel}: ${message}`);
    // Parse message to determine target client(s)
    const parsedMessage = JSON.parse(message);

    // Example: broadcast to all connected clients on this server
    wss.clients.forEach(client => {
        if (client.readyState === WebSocket.OPEN) {
            client.send(JSON.stringify({ type: 'broadcast', data: parsedMessage }));
        }
    });

    // Example: target specific client (if tracking users on this server)
    // if (parsedMessage.targetUserId && clients.has(parsedMessage.targetUserId)) {
    //     clients.get(parsedMessage.targetUserId).send(JSON.stringify(parsedMessage));
    // }
});

wss.on('connection', ws => {
    console.log('Client connected');

    // Simulate user login and store connection
    const userId = Math.random().toString(36).substring(7); // Assign a dummy user ID
    clients.set(userId, ws);
    ws.userId = userId; // Attach userId to WebSocket instance for easier lookup
    console.log(`Client ${userId} connected.`);

    ws.on('message', message => {
        console.log(`Received message from client ${userId}: ${message}`);
        // When a client sends a message, publish it to Redis
        const msgPayload = {
            senderId: userId,
            text: message.toString(),
            timestamp: new Date().toISOString()
        };
        redisPublisher.publish('global_chat_channel', JSON.stringify(msgPayload));
    });

    ws.on('close', () => {
        console.log(`Client ${ws.userId} disconnected`);
        clients.delete(ws.userId);
    });

    ws.on('error', error => {
        console.error('WebSocket error:', error);
    });
});

console.log('WebSocket server started on port 8080');

Enter fullscreen mode Exit fullscreen mode

Explanation:

1. redisPublisher & redisSubscriber: We use two ioredis instances. This is a common pattern because ioredis (and Redis clients in general) block when in subscriber mode. One instance is dedicated to publishing, and another to subscribing.
2. redisSubscriber.subscribe('global_chat_channel'): Each WebSocket server instance subscribes to a common Redis channel (e.g., global_chat_channel).
3. redisSubscriber.on('message', ...): When a message is published to global_chat_channel by any WebSocket server, all subscribed servers receive it. They then parse the message and can forward it to relevant clients connected to their instance.
4. ws.on('message', ...): When a client sends a message to this WebSocket server, the server takes the message and publishes it to the global_chat_channel via redisPublisher. This ensures all other servers (and thus all other clients) eventually receive it.

Managing Session State for Seamless Failover

To achieve seamless failover, the WebSocket servers must be stateless. All critical session information (e.g., which user is connected, their current room, permissions) should be stored externally, ideally in Redis.

Example: Storing User-to-Server Mapping

// In your WebSocket server (server.js)

// When a client connects:
ws.on('connection', async ws => {
    const userId = 'user_' + Math.random().toString(36).substring(2, 9);
    ws.userId = userId;
    // Store mapping: userId -> current server instance ID
    await redisPublisher.set(`user:${userId}:server_id`, process.env.SERVER_ID || 'server_1');
    // Store other session data, e.g., 'user:userId:room_id'
    await redisPublisher.hset(`user:${userId}:session`, 'room_id', 'general');

    // ... rest of your connection logic ...
});

// When a client disconnects:
ws.on('close', async () => {
    // Clean up mapping (optional, depending on desired behavior)
    await redisPublisher.del(`user:${ws.userId}:server_id`);
    await redisPublisher.del(`user:${ws.userId}:session`);
    // ...
});

// When sending a targeted message from another server:
// Suppose another server wants to send a message to 'targetUserId'
async function sendTargetedMessage(targetUserId, message) {
    const serverId = await redisPublisher.get(`user:${targetUserId}:server_id`);
    if (serverId) {
        // Publish to a specific channel that only the target server listens to
        // Or, if using a global channel, include targetUserId in the payload
        redisPublisher.publish('global_chat_channel', JSON.stringify({ 
            type: 'private_message', 
            targetUserId, 
            message 
        }));
    } else {
        console.log(`User ${targetUserId} not found or not connected.`);
    }
}

Enter fullscreen mode Exit fullscreen mode

Resilience Through Statelessness:

If WS1 crashes, Client A‘s connection is dropped. The client-side code should include a reconnection mechanism (e.g., using a library like reconnecting-websocket). When Client A reconnects, the load balancer routes them to, say, WS2. WS2 can then retrieve Client A‘s userId (perhaps sent during reconnection) and then fetch all necessary session context from Redis. This ensures that even though the underlying server changed, the user’s experience is largely uninterrupted.

Redis High Availability and Persistence

For Redis itself to be resilient, you need to run it in a highly available configuration:

• Redis Sentinel: Provides monitoring, notification, and automatic failover for Redis instances. If a primary Redis instance fails, Sentinel promotes a replica to primary and reconfigures clients.
• Redis Cluster: For horizontal scaling and sharding data across multiple Redis nodes, offering even greater availability and performance for larger datasets. It automatically shards data and handles failover for individual shards.

Furthermore, Redis offers persistence options (RDB snapshots and AOF append-only file) to prevent data loss in case of a complete system restart, ensuring that even ephemeral session data can survive restarts if configured correctly.

Considerations for Production Deployments

• Load Balancing: Use a robust load balancer (e.g., Nginx, HAProxy, AWS ALB) configured for sticky sessions (if session affinity is desired, though a stateless design with Redis reduces this need) and WebSocket proxying.
• Authentication & Authorization: Integrate with your existing auth system. Tokens (JWTs) can be passed during the WebSocket handshake and validated by the server, potentially storing user roles/permissions in Redis.
• Error Handling & Logging: Implement comprehensive error handling for both WebSocket and Redis connections. Centralized logging is crucial for debugging distributed systems.
• Monitoring: Monitor WebSocket connection counts, message rates, Redis latency, memory usage, and CPU load across all components.
• Security: Always use wss:// (WebSockets over TLS/SSL). Sanitize all incoming client messages.
• Backpressure: Implement mechanisms to handle situations where consumers (clients or servers) cannot process messages as fast as they are produced, preventing memory exhaustion.
• Client-side Reconnection Logic: Clients must be designed to gracefully handle disconnections and attempt intelligent reconnections, potentially with exponential backoff.

Conclusion

Building resilient real-time systems requires careful consideration of scalability and fault tolerance beyond a single server. By coupling WebSockets for efficient communication with Redis for distributed state management and message brokering, engineers can construct highly available, scalable, and robust real-time applications. Redis’s Pub/Sub and key-value capabilities make it an indispensable tool for ensuring that messages reach their intended recipients across a cluster of WebSocket servers and that user sessions can seamlessly persist through server failures, providing a foundation for truly resilient interactive experiences.