Hi, I’m Gowtham M.
Over the last 10 days, I participated in the 10 Days of Voice Agents — VoiceForBharat Edition and built MoneyBuddy, an AI-powered voice assistant designed to make financial and government scheme-related information more accessible through natural conversations.
What started as a basic voice interaction gradually evolved into a system involving real-time AI, voice processing, memory, tool usage, human escalation, analytics, and multi-agent handoffs.
The goal was simple:
Can users get help with financial and government scheme-related questions through a natural voice conversation instead of navigating multiple websites and forms?
What started as a basic voice interaction gradually evolved into a system involving real-time AI, voice processing, memory, tool usage, human escalation, analytics, and multi-agent handoffs.
The idea behind MoneyBuddy
Users often need answers to questions like:
- Am I eligible for a scheme?
- What information do I need?
- What should I do next?
- Can this be explained in a simpler way?
Instead of searching through multiple sources, MoneyBuddy is designed to let users speak naturally to an AI assistant.
The core flow looks like this:
User Voice
↓
Speech-to-Text
↓
MoneyBuddy AI Agent
↓
Tools / Memory / Agent Routing
↓
Response Generation
↓
Text-to-Speech
↓
User
Enter fullscreen mode Exit fullscreen mode
The 10-day journey
Day 1 — Building the foundation
I started by understanding how a real-time voice agent works.
The basic pipeline was:
Voice Input → AI Processing → Voice Output
Enter fullscreen mode Exit fullscreen mode
This became the foundation for MoneyBuddy.
Day 2 — Designing better conversations
A voice assistant cannot simply read long chatbot-style responses.
I focused on making interactions:
- Short
- Conversational
- Clear
- Natural for voice
This made me think more about user experience, not just AI responses.
Day 3 — Defining the agent’s role
I worked on giving MoneyBuddy clear instructions.
The agent needed to understand:
- What it is responsible for
- How it should respond
- When to ask follow-up questions
- When not to guess
- When another action is required
One important lesson: an AI agent performs better when its responsibilities are clearly defined.
Day 4 — Adding tools
Some information should not be generated or guessed by an LLM.
So I explored structured tool usage.
User Request
↓
Agent Understands Intent
↓
Tool Required?
↓
Call Tool
↓
Get Result
↓
Explain Result to User
Enter fullscreen mode Exit fullscreen mode
This was an important shift from a simple conversational AI system toward an AI agent capable of taking structured actions.
Day 5 — Working with memory
I explored how MoneyBuddy could remember useful conversation context.
The goal was to avoid making returning users repeat the same information.
At the same time, memory requires responsible handling.
The agent should ask for consent before storing relevant information.
This taught me that AI memory is not just a database feature—it is also a user trust feature.
Day 6 — Human escalation
Not every problem should be handled completely by AI.
For situations requiring additional help or manual intervention, I explored human escalation.
A useful AI system should not pretend to know everything.
Sometimes the best response is knowing when to say:
This requires additional assistance.
My key learning:
Good AI systems should understand their limitations.
Day 7 — Thinking beyond a chatbot
At this stage, I started looking at MoneyBuddy as a complete system.
The focus moved toward:
- Conversation flows
- Agent states
- Error handling
- Different user paths
- System reliability
This was when the project started becoming more than a basic voice demo.
Day 8 — Analytics and outcomes
Building an agent is only the first step.
I also explored tracking what happens during interactions.
Examples of useful metrics include:
- Total conversations
- Successful outcomes
- Escalations
- Types of user requests
- Agent handoffs
This helped me understand an important engineering principle:
If you cannot observe a system, it becomes difficult to improve it.
Day 9 — Multi-agent architecture
This was one of the biggest upgrades to MoneyBuddy.
Instead of asking one agent to handle everything, I added a dedicated Government Scheme Specialist.
The architecture became:
User
↓
MoneyBuddy Main Agent
↓
Is specialist help needed?
↙ ↘
No Yes
↓ ↓
Main Agent Scheme Specialist
↘ ↙
Final Response
↓
User
Enter fullscreen mode Exit fullscreen mode
The main agent handles:
- Greetings
- General conversations
- Initial request understanding
- Basic assistance
- Deciding whether a handoff is required
The specialist handles:
- Government scheme-related questions
- More detailed scheme discussions
- Specialized assistance
The important part was context handoff.
The user should not have to explain the entire problem again after being transferred.
This was my biggest takeaway from multi-agent systems:
Specialized agents with clear responsibilities can be easier to manage than one agent trying to do everything.
Day 10 — Bringing everything together
By the end of the challenge, MoneyBuddy had evolved into a system involving:
Voice AI
+
LLM
+
Clear Instructions
+
Structured Tools
+
Memory
+
Consent
+
Human Escalation
+
Analytics
+
Specialist Routing
+
Multi-Agent Handoff
Enter fullscreen mode Exit fullscreen mode
What I learned
Building MoneyBuddy taught me that a voice AI application is much more than:
LLM + Microphone + Speaker
Enter fullscreen mode Exit fullscreen mode
A useful AI agent needs:
- Clear objectives
- Good prompts
- Natural conversation design
- Reliable tools
- Responsible memory
- User consent
- Error handling
- Real-time performance
- Human escalation
- Observability and analytics
- Clear agent responsibilities
- Agent routing
- Continuous testing
Challenges I faced
Some of the biggest challenges were:
Real-time latency
A voice pipeline contains multiple stages:
Speech → STT → LLM → Tools → TTS → Audio
Enter fullscreen mode Exit fullscreen mode
A delay at any stage affects the overall user experience.
Agent handoff
Transferring a conversation between agents requires clear routing logic and context management.
Without proper context transfer, users may need to repeat themselves.
Defining agent responsibilities
As the system grew, it became increasingly important to clearly define what each agent should handle.
What’s next for MoneyBuddy?
I would like to continue improving the project by exploring:
- More specialist agents
- Better tool integration
- More verified data sources
- Improved latency
- Better multilingual support
- More advanced analytics
- Stronger memory management
- Better mobile experience
- Public deployment
Final thoughts
These 10 days changed the way I think about AI applications.
I started with the idea of building a voice assistant.
I ended up learning about real-time systems, AI agents, tools, memory, user consent, escalation, observability, and multi-agent architectures.
My biggest takeaway:
A useful AI product is not built by choosing only a powerful model. It is built by connecting the right components with clear responsibilities, good user experience, reliable systems, and continuous testing.
MoneyBuddy is still a work in progress, but these 10 days gave me a much stronger foundation for building AI agents that can interact with users in more natural and useful ways.