Built for the **SigNoz “Agents of SigNoz” Hackathon — July 2026**
Production Incidents Shouldn’t Take Hours to Investigate
Every engineer who has been on call knows the feeling.
An alert fires in the middle of the night. Dashboards turn red. Error rates spike. Customers begin reporting failures.
The incident itself may eventually be simple—a database timeout, an exhausted Redis connection pool, a failing downstream dependency—but discovering the actual root cause is often the hardest part.
The investigation usually looks something like this:
- Open traces to identify where requests are failing.
- Search through thousands of log entries.
- Compare metrics across multiple services.
- Follow dependency chains through microservices.
- Filter duplicate alerts to understand what actually happened.
- Form hypotheses and verify each one manually.
By the time the root cause is identified, hours may have passed.
In many organizations, the fix itself takes only a few minutes.
The investigation is the real bottleneck.
That observation became the motivation behind TattvaAI.
Introducing TattvaAI
TattvaAI is an AI-powered incident investigation platform that automatically performs the work experienced Site Reliability Engineers (SREs) do during production incidents.
Instead of presenting engineers with dashboards full of raw telemetry, TattvaAI investigates the incident autonomously, correlates evidence from multiple observability signals, identifies the most probable root cause, and explains its reasoning.
Rather than replacing engineers, it removes the repetitive investigative work so engineers can focus on making decisions and restoring production faster.
The entire system was built for the SigNoz “Agents of SigNoz” Hackathon using SigNoz as the observability platform and LangGraph to orchestrate multiple specialized AI agents.
The Problem Modern Observability Still Doesn’t Solve
Today’s observability platforms have become extremely good at collecting telemetry.
They provide:
- Distributed traces
- Logs
- Metrics
- Alerts
- Dashboards
- Service dependency graphs
The data exists.
The problem is interpretation.
When production fails, engineers still have to manually connect all these signals together.
A single incident often requires answering questions like:
- Which service failed first?
- Which failures are merely downstream effects?
- What changed recently?
- Which metric explains the latency spike?
- Do the logs confirm the traces?
- Has this happened before?
Answering these questions manually consumes most of the incident response timeline.
The challenge isn’t a lack of observability.
It’s a lack of automated reasoning.
My Approach: A Team of AI Investigators
Instead of creating one large AI assistant, I designed six independent AI agents, each responsible for one domain of observability.
Every agent specializes in gathering evidence.
Later, a reasoning engine combines that evidence into a complete investigation.
🔍 Trace Agent
Responsible for analyzing distributed traces.
It identifies:
- latency spikes
- slow spans
- failed requests
- request propagation
- bottlenecks across services
Instead of simply reporting slow requests, it determines where latency originates inside an entire request path.
📜 Logs Agent
Responsible for log intelligence.
It searches for:
- exceptions
- repeated error patterns
- frequency anomalies
- correlated failures
- stack traces
Rather than reading thousands of log entries manually, the agent extracts only the evidence relevant to the incident.
📊 Metrics Agent
Responsible for infrastructure and application health.
It continuously evaluates:
- CPU usage
- memory utilization
- request throughput
- latency
- error rates
- SLA violations
The goal is identifying abnormal behavior that explains observed failures.
🌐 Dependency Agent
Distributed systems rarely fail in isolation.
This agent maps:
- service dependencies
- upstream failures
- downstream impact
- cascading failures
Instead of asking “Which service is broken?”, it answers:
“Which service caused every other service to fail?”
🚨 Alert Agent
Modern monitoring systems often generate dozens of alerts during a single outage.
Many are secondary effects.
This agent correlates alerts together and removes noise by identifying which alerts are symptoms versus actual causes.
🧠 Historical Agent
Production systems repeat themselves.
This agent searches previous incidents and historical telemetry to answer questions such as:
- Have we seen this before?
- What fixed it last time?
- How similar is today’s incident?
Instead of starting every investigation from scratch, TattvaAI learns from operational history.
Orchestrating the Investigation with LangGraph
Each AI agent works independently.
Their findings become structured evidence rather than isolated observations.
LangGraph orchestrates the entire workflow by:
- Launching investigations in parallel
- Waiting for evidence collection
- Correlating observations
- Eliminating conflicting hypotheses
- Ranking likely root causes
- Producing recommendations with confidence scores
This architecture makes investigations deterministic, explainable, and extensible.
Adding another specialist agent later becomes straightforward.
Native SigNoz Integration
One of the biggest design decisions was integrating directly with SigNoz.
Instead of treating SigNoz as another dashboard, TattvaAI accesses telemetry through the Model Context Protocol (MCP).
This allows agents to retrieve:
- traces
- logs
- metrics
- service metadata
- dependency information
without manually navigating dashboards.
SigNoz becomes the source of truth, while TattvaAI becomes the reasoning layer built on top of it.
This separation keeps observability and intelligence cleanly decoupled.
A Real Investigation
To evaluate the platform, I created a realistic microservices environment with multiple containerized services generating live telemetry.
One investigation looked like this.
Incident
An e-commerce checkout system suddenly experienced an 18% error rate.
Traditional Investigation
An engineer would typically:
- inspect dashboards
- analyze traces
- search logs
- verify dependencies
- compare metrics
- identify the faulty service
Total investigation time:
2–3 hours
TattvaAI Investigation
Within seconds the AI agents produced the following evidence.
Trace Agent
Payment service latency increased to 4800 ms, over four times the normal baseline.
Logs Agent
Detected 143 Redis timeout exceptions originating from the payment service.
Metrics Agent
Redis CPU reached 98%, and connection pool utilization hit maximum capacity.
Dependency Agent
Failure propagation path identified:
Gateway → Order Service → Payment Service → Redis
Alert Agent
Seven alerts detected.
Five classified as downstream symptoms.
Historical Agent
94% similarity to a previous Redis connection pool exhaustion incident.
Final Reasoning
After correlating all evidence, the reasoning engine concluded:
Root Cause
Redis connection pool exhaustion.
Confidence
96%
Recommended Actions
- Restart Redis
- Increase connection pool size
- Monitor connection utilization
- Review recent traffic spike
Total investigation time:
Approximately 30 seconds.
Technology Stack
TattvaAI combines modern AI orchestration with production-ready backend engineering.
Backend
- FastAPI
- Python
- LangGraph
- OpenAI GPT
- PostgreSQL
- SQLAlchemy
- OpenTelemetry
- Docker
Frontend
- React 19
- TypeScript
- PrimeReact
- Tailwind CSS
Observability
- SigNoz
- OpenTelemetry
- Model Context Protocol (MCP)
Demo Environment
To validate the platform end-to-end, I built an entire observable microservices environment.
It includes:
- six independent services
- OpenTelemetry instrumentation
- live distributed traces
- centralized logs
- infrastructure metrics
- simulated production failures
This allows TattvaAI to investigate realistic production incidents instead of synthetic examples.
Results from My Testing Environment
After running repeated investigations inside the demo environment, the results were encouraging.
Investigation Speed
- Average investigation time: 28 seconds
- Fastest completed investigation: 14 seconds
- Typical manual investigation: 2–4 hours
- Approximate improvement: 450× faster
Investigation Accuracy
- Root cause identification: 94%
- Evidence correlation success: 98%
- False positives: 3%
- Historical incident matching: 89%
Operational Efficiency
- AI execution cost: approximately $0.12 per investigation
- Average engineering time saved: 2.7 hours
- Estimated operational savings: around $540 per incident (assuming a $200/hour fully loaded engineering cost)
Explainable AI, Not a Black Box
One principle guided the design of TattvaAI from the beginning.
Every conclusion should be explainable.
Instead of producing a single answer, every investigation includes:
- collected evidence
- reasoning chain
- confidence score
- correlation analysis
- rejected hypotheses
- recommended actions
If TattvaAI concludes that Redis caused the outage, it also shows:
- the traces
- the metrics
- the log evidence
- the dependency graph
- the historical incident
- the reasoning connecting them
The goal is transparency.
Engineers should understand why the AI reached its conclusion.
Built Like a Real Product
Although developed during a hackathon, TattvaAI was designed with production deployment in mind.
It includes:
- Docker Compose deployment
- REST APIs
- persistent investigation storage
- health checks
- modular architecture
- OpenTelemetry instrumentation
- investigation memory
- configurable environments
- comprehensive documentation
- React dashboard for operations teams
The architecture is intentionally modular so organizations can extend it with additional agents and custom workflows.
Why This Matters
The observability ecosystem has matured significantly over the past decade.
We no longer struggle to collect telemetry.
We struggle to interpret it quickly enough when production is failing.
TattvaAI shifts observability from passive monitoring to active investigation.
Instead of showing engineers hundreds of charts, it answers the question they actually care about:
What is broken, why did it break, and what should I do next?
That shift has implications beyond reducing Mean Time to Resolution (MTTR).
It can also help teams:
- reduce on-call fatigue
- preserve operational knowledge
- accelerate incident response
- help junior engineers investigate like experienced SREs
- continuously learn from previous incidents
What’s Next
This project represents the first version of a much larger vision.
Future directions include:
- predictive incident detection before failures occur
- automated remediation for common operational issues
- organization-specific AI training using historical incidents
- integrations with AWS CloudWatch, Google Cloud Operations, and Azure Monitor
- richer knowledge graphs for cross-service reasoning
- continuous learning from completed investigations
Final Thoughts
Building TattvaAI fundamentally changed the way I think about observability.
The future of incident response is not simply collecting more telemetry or building more dashboards.
It is creating intelligent systems capable of reasoning across telemetry the same way experienced engineers do.
SigNoz provides an excellent foundation for collecting and exposing observability data.
TattvaAI builds on that foundation by transforming telemetry into explanations, hypotheses, and actionable decisions.
If AI can reduce a two-hour investigation to thirty seconds while remaining transparent and explainable, then incident response becomes less about searching for information and more about solving problems.
And that is exactly the future I wanted to build.
TattvaAI was built for the SigNoz “Agents of SigNoz” Hackathon (July 2026).
The project is open source, and I would love to hear feedback from the observability community.
If you’re interested in AI-powered incident investigation, explore the project, experiment with it, and let me know what improvements you’d like to see.
Happy building—and may your next production incident take seconds instead of hours to understand.
답글 남기기