The failover that fooled me

My reverse proxy had VRRP failover. I killed the process directly to test it, and the traffic black-holed anyway.

The setup that looked complete

I run a Traefik pair with keepalived holding a shared virtual IP between them. The idea is simple and standard: if the active node dies, the IP moves to the standby and traffic keeps flowing. And it does, if the whole node dies. Pull the power on the master and the VIP is on the backup in a couple of seconds.

The gap is what happens when the box lives but the application does not.

Node up, app down, VIP stuck

keepalived on its own watches only the VRRP heartbeat between the two peers. That heartbeat proves the box is alive and on the network. It says nothing about whether the application on that box is actually serving. So when I killed the Traefik process but left the machine running, the heartbeat kept ticking, keepalived saw a healthy master, and the virtual IP sat happily on a node that was no longer answering a single connection. The failover I thought I had covered exactly the failure that is least likely (a whole machine dying) and missed the one that is most likely (a process crashing).

The five lines that fix it

The fix is a health check that keepalived runs on a timer and folds into its own priority:

vrrp_script chk_traefik {
  script "curl -sf http://localhost/ping || exit 1"
  interval 2
  fall 2
  rise 3
}

Enter fullscreen mode Exit fullscreen mode

Reference it from the vrrp_instance track_script block, and now keepalived lowers its own priority the moment the app stops answering, which hands the VIP to the standby. The same pattern works for nginx, haproxy, or any single process sitting behind a virtual IP.

What it comes back to

The heartbeat proves the box is up. The track_script proves the app is up. If your VIP fronts an application, you need both, and the only reliable way to know you have both is to test by killing the process, not the machine. The machine-death test passes on a broken config. The process-death test is the one that tells the truth.