The background job worker looked perfect when I deployed it. A free model available through MonkeyCode generated the code, my tests passed, and the health endpoint returned 200 every time I poked it. Disclosure: This article was prepared as part of MonkeyCode’s product outreach. Then I submitted a batch of jobs at 9 PM, went to bed, and woke up to an empty queue with no error in sight.
This is the story of how a free model and a free server conspired to lose my data, and which debugging habits finally exposed the culprit. The most dangerous part was that nothing looked broken, which is exactly why the failure took me an entire morning to understand.
The symptom that made no sense
The API accepted every job and returned 202 Accepted, so the client side was happy. The worker log showed each job being claimed and processed, and then the entries simply stopped around 2 AM. My first instinct was to blame the process, so I checked the restart policy, the memory limit, and the exit code, but the platform reported the instance as healthy the whole night.
The most confusing part was the health check, because it kept saying everything was fine. I curled the endpoint from my laptop and got {"status": "ok"}, which made me wonder whether the worker was actually dead or just quiet. That question, it turns out, was the entire problem in miniature.
The wrong hypothesis: a crashed process
I spent an hour assuming the worker had crashed, because that is the default explanation when logs stop. I added more logging, I set up a restart hook, and I even wrote a small watchdog that would relaunch the process if it died, but none of that changed the morning outcome. Every one of those changes was a reasonable thing to do, and every one of them addressed the wrong layer.
# step 1: check the process was really gone
curl -s http://localhost:8080/health
# {"status": "ok"} <- still green
# step 2: check the logs for a crash
journalctl -u worker --since "01:00"
# no output after 02:03
Enter fullscreen mode Exit fullscreen mode
The logs stopped, yet the health check stayed green, which should have told me that the two signals were measuring different things. I was so focused on the process that I forgot to ask a much simpler question: where does this worker keep its state?
The real root cause: state on a disk that doesn’t exist
The generated code used SQLite as the job queue, which is a reasonable choice for a small prototype and a terrible one for this deployment target. The free server option that I was using turned out to run instances on ephemeral storage, so when the platform recycled the instance overnight, the entire jobs.db file vanished along with every pending job.
# generated by a free model — the queue is a local file
import sqlite3
def claim_job():
conn = sqlite3.connect("jobs.db") # this file does not survive a recycle
row = conn.execute(
"SELECT * FROM jobs WHERE status='pending' ORDER BY created_at LIMIT 1"
).fetchone()
if row:
conn.execute("UPDATE jobs SET status='running' WHERE id=?", (row[0],))
conn.commit()
return row
Enter fullscreen mode Exit fullscreen mode
The worker never crashed, because there was nothing to crash into; the process was simply holding a handle to a file that the platform had deleted. The health check returned 200 because it only verified that the Python process was alive, and the process was very much alive, just pointing at a database that no longer existed.
The fix: make state survive the instance
The first change was to move the job queue out of the instance entirely, because any state that lives on the local disk is state that the platform can take away. I switched the generated code to a managed Postgres database, which meant changing one connection string and rewriting the few SQL queries to use the same schema.
# before: local file, dies with the instance
DB_PATH = "jobs.db"
# after: external store, survives instance recycling
import os
DATABASE_URL = os.environ["DATABASE_URL"] # managed Postgres, not local disk
Enter fullscreen mode Exit fullscreen mode
The second change was a startup invariant check, because an empty queue can be legitimate and a missing database never is. The worker now refuses to start if the job table is absent, which turns a silent data loss into a loud, obvious failure.
def verify_job_store():
try:
conn = sqlite3.connect(DATABASE_URL)
conn.execute("SELECT COUNT(*) FROM jobs")
except sqlite3.OperationalError:
raise SystemExit("job store missing — refusing to start")
Enter fullscreen mode Exit fullscreen mode
The third change was the health check, because a liveness probe that cannot tell the difference between alive and useful is worse than no probe at all. I added a canary job that runs every five minutes and writes a heartbeat to the external database, so a green health endpoint now means the whole pipeline is functioning.
The reusable debugging checklist
If you take one thing from this post, let it be the questions I should have asked before trusting the green checkmark.
- Ask what survives a restart: list every file and connection the code touches, then check which ones exist only on the instance.
- Verify that your health check tests the right thing: a process can be alive while being completely useless.
- Test the deployment target, not just the code: the same worker behaved perfectly on my laptop and lost everything on the free server.
- Add a canary that exercises the full path: a heartbeat written to external storage catches failures that a liveness probe never will.
Who should not use this approach
If you are building a throwaway prototype or a demo that can tolerate losing state, ephemeral storage is fine and the free server option is a great way to test ideas without paying. If your jobs must survive an instance recycle, or if losing a queue would cost you real users, then do not rely on local disk, and do not assume a free tier gives you durability guarantees it never promised.
The free model wrote reasonable code, and the free server ran it exactly as advertised; the failure was mine, because I never checked what the platform guarantees about storage. Next time a model hands me a worker, I will ask where the state lives before I ask for any new features. If your health check has ever lied to you, start by asking what it actually verifies.