Summer Bug Smash: Clear the Lineup 🐛🛹
This is a submission for DEV’s Summer Bug Smash: Clear the Lineup powered by Sentry.
Project Overview
I recently deployed the Dograh AI voice agent (named Zoya) live on my production website, Mobile One Media, to handle client inquiries for our 4K video production, audio engineering, and app development services.
For the first 48 hours, the agent worked flawlessly. However, on day three, it suddenly stopped responding on the live site. Users were experiencing complete hang-ups when trying to navigate the service menu. This wasn’t a local testing issue; this was a live production fire that needed immediate debugging and a permanent fix.
Bug Fix or Performance Improvement
The Problem:
- Issue: After ~48 hours of continuous uptime, the live Dograh AI widget on mobileonemedia.com began silently failing, resulting in infinite loading states and dropped user sessions.
- Impact: 100% of new agent interactions were failing, directly blocking potential client leads from contacting our media production services.
- Root Cause: A Docker container configuration issue combined with a system prompt misalignment. The agent’s Docker environment variables were not properly persisting the workflow state, and the system prompt was failing to initialize correctly after container restarts, causing the agent to lose conversational context.
Code and Video Demo
GitHub Pull Request: https://github.com/dograh-hq/dograh/pull/287
Watch the live agent in action: https://youtu.be/pKUxtq8sKDs?si=r1s2LK7zGIzpd2Ry
The Fix Summary:
- Archived the old, messy agent configuration that was causing the Docker and prompt issues.
- Built a brand new, clean agent setup from scratch with proper architecture.
- Rebuilt the Docker container configuration with proper environment variable persistence and volume mounting.
- Fixed the system prompt initialization sequence to ensure it loads correctly on container startup.
- Added health check endpoints to monitor agent readiness before accepting user connections.
My Improvements
Technical Approach:
- Diagnosis: Identified that the Docker container was losing state between restarts and the system prompt wasn’t being injected properly into the workflow engine.
- The Clean Slate Decision: Realized the old agent setup was too messy to patch. I officially archived the old broken agent and started fresh.
- Solution: Created a completely new agent configuration with proper Docker volume mounts, environment variable handling, and a corrected system prompt structure.
- Testing: Verified the new agent maintains state across container restarts and responds consistently in production.
Key Decisions:
- Instead of wasting time trying to patch the old, messy agent configuration, I archived it and built a brand new, clean agent from the ground up. This saved hours of debugging and resulted in a much more stable system.
- Implemented Docker health checks to ensure the agent is fully initialized before going live.
- Added automatic restart policies with proper state preservation to prevent future silent failures.
Results:
- 100% restoration of live agent functionality on mobileonemedia.com.
- Agent now maintains state correctly across Docker container lifecycle events.
- Zero downtime deployments with proper health monitoring.
Best Use of Sentry
Sentry was the absolute hero in diagnosing this silent production failure:
- Error Monitoring and Alerts:
- Sentry immediately flagged a sudden spike in connection errors and timeout warnings originating from the live website domain.
The issue grouping showed errors correlating with Docker container restart events.
Session Replays:
I watched Sentry Session Replays of actual visitors on mobileonemedia.com. I could see them clicking the “Talk to Zoya” button, seeing the loading spinner, and eventually closing the tab. This visual proof confirmed the frontend was waiting on a backend that wasn’t initializing properly.
Performance Monitoring:
Post-fix, I set up a Sentry dashboard tracking agent response times and connection success rates to ensure the new Docker configuration was working perfectly in the wild.
Best Use of Google AI
To pull off this Docker and system prompt fix and get the live site back up fast, I heavily relied on Google Gemini:
- Debugging the Docker Configuration:
- I fed the Docker logs and environment variable setup into Google Gemini.
- Prompt: “Analyze this Docker configuration for a Node.js voice agent. The container starts but loses state on restart and system prompts fail to initialize. Identify volume mounting and environment variable issues.”
Gemini identified the missing volume mounts and incorrect environment variable persistence strategy.
Rebuilding the System Prompt:
I asked Gemini to help create a proper system prompt structure for the Dograh AI agent.
Prompt: “Create a robust system prompt for a voice agent handling multi-service inquiries (video production, audio engineering, app development) that maintains conversational context and properly routes users.”
The resulting prompt was clean, well-structured, and formed the foundation of the new agent configuration.
Docker Compose Optimization:
Gemini helped generate the corrected docker-compose.yml with proper health checks, restart policies, and volume configurations.
답글 남기기