Debugging a Python "Memory Leak" That Was Actually a Measurement Bug (ru_maxrss vs VmRSS)

I was running a pipeline for building a RAG knowledge base: crawl web articles, split them into chunks, create embeddings, and push them into Qdrant.

Then my memory usage logs started looking like this:

  50/1287 pushed... (RSS 1594MB)
  ⚠️ RSS 1594MB exceeded the 1024MB limit — aborting

Enter fullscreen mode Exit fullscreen mode

Every time I resumed the process, the RSS number kept climbing. I was convinced the embedding loop was leaking memory. I sprinkled gc.collect() calls around, added more del statements — nothing helped. The number never went down. Not once.

Spoiler: nothing was leaking except my measurement method.

What was actually happening

Here’s the RSS measurement code I started with:

import resource

def rss_mb() -> int:
    return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024

Enter fullscreen mode Exit fullscreen mode

At a glance, it looks like it returns the current RSS.

It doesn’t. ru_maxrss is the maximum resident set size since the process started. It’s a peak value, a high-water mark.

Once your process touches a memory peak — even for a moment — this number will never go down, no matter how much memory you free afterwards.

Why using ru_maxrss for monitoring goes wrong

Because it’s a peak value, using it to monitor current memory leads you straight into misdiagnosis.

Even when memory is actually being freed, your logs look like this:

RSS 1200MB
RSS 1500MB
RSS 1800MB
RSS 1800MB
RSS 1800MB

Enter fullscreen mode Exit fullscreen mode

Looking at this log, memory appears to grow monotonically. In reality it only says “at some point, this process used 1800MB.”

The consequences:

• gc.collect() appears to do nothing
• del appears to free nothing
• It looks exactly like a memory leak
• If you use it for a limit check, one peak means every subsequent check fails forever

That last one is exactly what hit me.

On Linux, read VmRSS instead

For the current RSS on Linux, read VmRSS from /proc/self/status:

def rss_mb() -> int:
    # VmRSS = current RSS
    # ru_maxrss is a peak value — don't use it for live monitoring
    try:
        with open("/proc/self/status") as f:
            for line in f:
                if line.startswith("VmRSS:"):
                    return int(line.split()[1]) // 1024  # kB -> MB
    except OSError:
        # Fallback for non-Linux / unreadable /proc
        import resource
        return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024

    return 0

Enter fullscreen mode Exit fullscreen mode

After the fix, the logs looked like this:

  850/1105 pushed... (RSS 1974MB)
  900/1105 pushed... (RSS 1809MB)
  950/1105 pushed... (RSS 1811MB)
  1000/1105 pushed... (RSS 1862MB)

Enter fullscreen mode Exit fullscreen mode

The number goes up and down now. It turned out memory was stable at around 1.8GB after model load. No explosion. No leak. It had been fine the whole time.

The 1GB limit was never realistic anyway

There was a second problem hiding underneath.

Right after loading the embedding model, RSS was already around 1.6GB. The model was intfloat/multilingual-e5-base — that’s just what it costs to hold it in memory.

So my 1GB RSS limit was dead on arrival. Two mistakes stacked on top of each other:

1. I assumed ru_maxrss was a current value
2. I set the RSS limit to 1GB without ever checking the post-model-load baseline

Before suspecting a memory leak, check your baseline.

Bonus fix: stop encoding one chunk at a time

While investigating, I found another inefficiency — chunks were being encoded one at a time:

# Before: one encode() call per chunk
for r in batch:
    vector = model.encode(prefix + r["text"], normalize_embeddings=True)

Enter fullscreen mode Exit fullscreen mode

I switched to batched encoding, 50 chunks at a time:

import gc
import torch

PUSH_BATCH = 50
prefix = "passage: "  # required prefix for e5-family models (indexing side)

for batch_start in range(0, len(rows), PUSH_BATCH):
    batch = rows[batch_start : batch_start + PUSH_BATCH]
    texts = [prefix + r["text"] for r in batch]

    with torch.no_grad():
        vectors = model.encode(
            texts,
            normalize_embeddings=True,
            batch_size=PUSH_BATCH,
        )

    for r, vector in zip(batch, vectors):
        upsert(
            collection=collection,
            vector=vector.tolist(),
            payload=payload_of(r),
        )

    del vectors, texts
    gc.collect()

    if rss_mb() > RSS_LIMIT_MB:
        print("RSS limit exceeded — aborting. Remaining chunks resume next run.")
        break

Enter fullscreen mode Exit fullscreen mode

Batching amortizes the tokenize/forward overhead, and throughput improved significantly.

Also note torch.no_grad(): inference doesn’t need gradient buffers, and forgetting this is a way to get a real memory leak.

Don’t forget the e5 prefixes

intfloat/multilingual-e5-base is an e5-family model, and e5 models expect a prefix on every text.

Indexing side uses "passage: ":

prefix = "passage: "
texts = [prefix + r["text"] for r in batch]

Enter fullscreen mode Exit fullscreen mode

Query side uses "query: ":

query_vector = model.encode(
    "query: " + query,
    normalize_embeddings=True,
)

Enter fullscreen mode Exit fullscreen mode

Skip the prefixes and embeddings still get created — your search quality just quietly degrades.

Design for interruption and resume

When the RSS limit is exceeded, the process stops instead of failing outright — and it can resume where it left off.

The trick: record a qdrant_id on every chunk that’s been pushed. On the next run, only process the ones that haven’t been:

SELECT *
FROM chunks
WHERE qdrant_id IS NULL
ORDER BY id;

Enter fullscreen mode Exit fullscreen mode

RAG ingestion jobs tend to grow in size, so building them “interruption-first” from the start pays off quickly.

Summary

Symptom Root cause Fix RSS appears to grow monotonically ru_maxrss is a peak, not a current value Read VmRSS from /proc/self/status gc.collect() doesn’t lower the number You’re looking at a high-water mark Measure current RSS separately RSS is high right after model load That’s the embedding model itself Check the post-load baseline Embedding is slow One encode() call per chunk Batch encode with batch_size Interruption means starting over Push state isn’t persisted Reprocess only qdrant_id IS NULL

The lesson

Before suspecting a memory leak, check whether the value you’re measuring is a current value or a peak value.

My mistake was reading the name ru_maxrss and assuming “current RSS”. It literally stands for maximum resident set size. The max was right there.

Memory wasn’t exploding — I was watching a high-water mark and calling it a leak. Get the measurement wrong and you’ll chase bugs that don’t exist.

Suspect your metrics before you suspect your memory.

That was the real takeaway.

This is an English version of my Japanese article on Zenn.