Your Kubernetes Controller Is Silently Dead and Nobody Knows

Kubernetes controllers built on client-go have a silent failure mode that’s easy to miss. When a projected ServiceAccount token becomes permanently invalid, your controller keeps running, passes health checks, shows Ready, and does absolutely nothing.

The Problem

For in-cluster auth, client-go reads the projected ServiceAccount token from file via CachedFileTokenSource and re-reads it on a fixed interval. But it does not check the response status. If the API server returns 401, the same token gets sent again. The error is logged, but the process never exits. The pod stays Running/Ready.

// client-go's default: log and retry. Forever.
func DefaultWatchErrorHandler(ctx context.Context, r *Reflector, err error) {
    // logs the error
    // returns (triggering a retry with backoff)
}

Enter fullscreen mode Exit fullscreen mode

Where I Found It

During a Helm upgrade via ArgoCD on KAI-Scheduler (CNCF Sandbox GPU scheduler), the scheduler’s ServiceAccount got deleted and recreated. Running pods still held tokens bound to the old SA UID. The scheduler kept retrying 401s indefinitely while showing Running/Ready. ArgoCD reported the application as Synced/Healthy the entire time.

Full incident: #1751. The 401 handling gap was tracked separately in #1817.

The Fix

Submitted a PR that adds a transport wrapper on rest.Config that exits on 401:

type unauthorizedRoundTripper struct {
    rt http.RoundTripper
}

func (t *unauthorizedRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
    resp, err := t.rt.RoundTrip(req)
    if err == nil && resp.StatusCode == http.StatusUnauthorized {
        log.InfraLogger.Errorf("API server returned 401 Unauthorized, exiting to trigger pod restart")
        os.Exit(1)
    }
    return resp, err
}

Enter fullscreen mode Exit fullscreen mode

One call in the startup path:

config := clientconfig.GetConfigOrDie()
config.Wrap(wrapExitOnUnauthorized)

Enter fullscreen mode Exit fullscreen mode

Every client created from this config goes through the wrapper. On 401, the process exits, kubelet restarts it, fresh token is mounted.

Why os.Exit and not graceful shutdown? If your token is invalid, you can’t release your leader election lease either (that’s an API call too). The lease expires on its own in 15 seconds. os.Exit matches what kube-scheduler upstream does for unrecoverable errors.

If you build controllers on client-go, worth checking whether yours handles 401. Without an exit or a failed health probe, the pod stays Running and nothing self-heals.