How We Scaled to 1 Million Connections — and the Silent Kernel Bug That Almost Broke It

No crashes. No errors. Just connections that vanished — until we found where the kernel was hiding them.

The Incident That Started This

At around 800K concurrent connections on our chat cluster, something strange started happening. No crashes. No error spikes in the app logs. Just connections that never landed. Clients retried, some got through, most didn’t. CPU was fine. Memory was fine. The app itself was healthy.

The culprit was nf_conntrack — the kernel’s connection tracking table for iptables — silently full, printing one line into dmesg that nobody was watching:

ip_conntrack: table full, dropping packet

Enter fullscreen mode Exit fullscreen mode

No exception. No 5xx. No alert. Just packets vanishing at the kernel level, invisible to every application metric we had.

The lesson that reframed everything: past a certain connection count, the kernel is part of your application’s failure surface. You have to instrument and tune it like one.

Where connections silently die:

[Client] → [NIC] → [Conntrack Table] → [Socket Buffers] → [Application]
                          |
                          ⚠ Table full → packet silently dropped.
                             No error. No log. App never sees it.

Everything before "Application" is invisible to app-level monitoring.

Enter fullscreen mode Exit fullscreen mode

This post covers the real tuning work — file descriptors, TCP buffers, interface queueing, connection tracking, JVM flags — that got a Java-based server stack (blob, chat, Couchbase, video) to hold 1 million concurrent connections. The actual values, the math behind them, and the parts we never fully closed the loop on.

The Quick Wins

Two changes, five minutes, no math required.

File descriptors. Every socket is a file descriptor, and the Linux default isn’t close to enough.

ulimit -n 1000000
echo 1000000 > /proc/sys/fs/nr_open

Enter fullscreen mode Exit fullscreen mode

Persist it in /etc/security/limits.conf:

root  -  nofile   1000000

Enter fullscreen mode Exit fullscreen mode

Interface queue length. For paths with RTT > 50ms, the default txqueuelen of 1000 is too conservative:

ifconfig int0 txqueuelen 5000

Enter fullscreen mode Exit fullscreen mode

Do this across every interface. Pair it with IRQ balancing — interrupt servicing for different interfaces should land on different CPUs, not stack on core 0 (irqbalance usually handles this; verify with cat /proc/interrupts).

Neither of these requires understanding your traffic pattern. Do them first, always.

The Math You Can’t Skip: TCP Buffers

This is the step people guess at instead of calculating, and it’s the one that actually determines your capacity ceiling.

Bandwidth-delay product:

buffer size = RTT * interface_speed / 8

Enter fullscreen mode Exit fullscreen mode

At 1 Gbps with a 130ms RTT (our Amsterdam ↔ India path):

0.130 * (1024)^3 / 8 ≈ 17.5 MB

Enter fullscreen mode Exit fullscreen mode

That number drives the tuning:

sysctl -w net.ipv4.tcp_rmem="4096 87380 17448960"
sysctl -w net.ipv4.tcp_wmem="4096 65536 17448960"
sysctl -w net.core.rmem_max="17448960"
sysctl -w net.core.wmem_max="17448960"

Enter fullscreen mode Exit fullscreen mode

Rule that matters more than the numbers: send buffer must be ≥ receive window. If it isn’t, the sender stalls the moment unacked data equals the receive window — the rest of the send buffer sits idle, unusable, waiting on an ack stuck behind a too-small window.

For chat specifically, we sized rmem around 9K — enough for two “blurs” (message chunks). Worst case (6 blurs) hits ~30KB, but the common case is a single 4K blur. We sized for the common case with headroom, not the worst case.

The number that has to be in your capacity spreadsheet: 4KB per socket × 2M connections ≈ 46GB, just in TCP buffer memory. Discover this in planning, not when your boxes start swapping.

Choice Benefit Cost Larger tcp_rmem/tcp_wmem Higher throughput, fewer stalls under latency Linear memory cost × connection count Larger txqueuelen Absorbs bursts on high-RTT paths More queueing latency if truly saturated (bufferbloat) Aggressive dirty-page ratios Higher sustained write throughput Larger stalls when sync threshold hits

The One That Bit Us: Connection Tracking

Back to the incident. nf_conntrack has a hard ceiling, and once full, the kernel drops new connections silently — invisible to app-level monitoring. Size it deliberately:

sysctl -w net.netfilter.nf_conntrack_max=1000000
sysctl -w net.netfilter.nf_conntrack_buckets=250000
echo "250000" > /sys/module/nf_conntrack/parameters/hashsize

Enter fullscreen mode Exit fullscreen mode

Bucket count is generally max / 4, keeping hash lookups fast as the table fills.

If you take one thing from this post: alert directly on conntrack table fill percentage. Don’t rely on discovering it through user complaints — by the time complaints arrive, you’ve already been silently dropping connections for a while.

Don’t Stop at the OS: JVM Tuning

Kernel tuning alone doesn’t get a JVM-based server there. Two flags mattered:

-XX:+UseCompressedOops
-XX:ObjectAlignmentInBytes=16

Enter fullscreen mode Exit fullscreen mode

UseCompressedOops compresses 64-bit object pointers to 32 bits — real memory savings, but capped at a 32GB heap. Past that, ObjectAlignmentInBytes=16 extends compressed-pointer support beyond the default ceiling, letting a blob server use its full 32GB without silently falling back to 64-bit pointers everywhere.

What We Deliberately Left Alone

Knowing what’s already correct saves as much time as knowing what to change. Quick list, no elaboration needed:

• tcp_moderate_rcvbuf — auto-tunes receive window. Leave enabled.
• tcp_window_scaling — required for windows beyond 64KB. Don’t touch.
• tcp_sack — selective acks. Keep enabled.
• tcp_timestamps — 10 bytes overhead, but meaningfully better congestion control. Worth it.
• net.ipv4.tcp_mem — auto-calculated at boot from RAM. Leave it alone.
• rmem_default / wmem_default — govern non-TCP sockets. Don’t touch.

What We Never Fully Resolved

The part usually missing from tuning writeups — and probably the most useful part for whoever does this next.

rmem_max/wmem_max doubling behavior. Some documentation suggests the kernel doubles the requested SO_SNDBUF/SO_RCVBUF internally via setsockopt(). We flagged this with a literal ??? in our own runbook and never verified it against our kernel version — meaning our effective buffer sizes may have been 2x what we calculated. Verify this on your kernel before trusting the bandwidth-delay math precisely.

tcp_no_metrics_save. We ran with this enabled (default — ssthresh isn’t cached across connections). There’s a real argument that disabling it helps repeat connections between the same host pairs ramp up faster. Flagged as “worth investigating,” never benchmarked. If your workload has lots of repeat client-server pairs on the same route, test this properly instead of trusting our default.

Couchbase-specific tuning. We had a dedicated script (tuneCb.sh) and never fully documented how it diverged from the generic network tuning above. Don’t assume generic socket tuning is sufficient for Couchbase’s connection and replication patterns — give it its own pass.

Systematic packet-drop diagnosis. We built monitoring to catch drops but never built a repeatable runbook for isolating which layer dropped a packet (NIC ring buffer, backlog queue, socket buffer, conntrack). We diagnosed these case-by-case with netstat -s and dtrace — it worked, but it was tribal knowledge, not process.

How We Verified It

Two tools worth keeping on hand rather than a full monitoring writeup:

sudo modprobe tcp_probe port=443 full=1
sudo chmod 444 /proc/net/tcpprobe
cat /proc/net/tcpprobe > /tmp/output.out &

Enter fullscreen mode Exit fullscreen mode

Live TCP internals — snd_cwnd, ssthresh, rcv_wnd — the fastest way to check your buffer math against real traffic instead of trusting the spreadsheet.

For ongoing monitoring, apply the USE method to the network layer: Utilization (throughput vs. negotiated speed), Saturation (retransmits, overruns — your earliest warning, well before user-facing errors), Errors (checksums, short frames, collisions).

When You Actually Need Any of This

This level of kernel tuning earns its cost when you’re running long-lived, high-fan-out connections — chat, streaming, pub/sub — where the constraint is concurrent connection count, not request throughput. If your workload is short-lived request/response traffic, adding instances behind a load balancer will get you further, faster, than any of the above. Reach for this playbook specifically when connections, not requests, are what’s running out.

Closing Thoughts

Scaling to a million connections isn’t one setting — it’s file descriptor limits, buffer memory sized from real math, interface queueing, connection tracking, JVM heap layout, all moving together. Just as important: knowing which defaults are already right, and being honest about what you never fully verified.

None of the numbers here are copy-paste config. The 17.5MB buffer, the 46GB budget, the 9K chat buffer — all came from our RTT, our bandwidth, our workload. Recompute yours before shipping. And keep a list of what you’re still unsure about — ours turned out to be just as useful to the next engineer as the settings we were confident in.