Learn how we built an observable AI-powered restaurant operating system using OpenTelemetry, SigNoz, Prisma, and multi-agent architecture.
Building RestaurantOS AI: Observable Multi-Agent Restaurant Orchestration with OpenTelemetry and SigNoz
Modern restaurants generate enormous amounts of operational data every day. Forecasting demand, tracking inventory, minimizing food waste, and purchasing ingredients are all interconnected decisions that traditionally require significant manual effort.
To automate these processes, we built RestaurantOS AI an AI-powered Restaurant Operating System driven by specialized autonomous agents.
But there was one major challenge.
Large Language Model (LLM) agents don’t behave like traditional software.
They are probabilistic.
They make decisions.
They call multiple services.
They generate intermediate reasoning.
Without observability, debugging becomes nearly impossible.
That is why OpenTelemetry and SigNoz became core components of our architecture rather than optional monitoring tools.
In this article, we’ll walk through how we built an observable multi-agent system that makes every AI decision transparent.
Table of Contents
- Why RestaurantOS AI?
- Multi-Agent Architecture
- Integrating OpenTelemetry
- Custom Agent Tracing
- LLM Observability
- SigNoz MCP Integration
- End-to-End Trace Flow
- Real-world Problems We Solved
- Key Learnings
Why RestaurantOS AI?
Restaurant managers constantly answer questions like:
- How many customers should we expect today?
- Which ingredients will run out?
- Which food items are close to expiry?
- What should we purchase from suppliers?
- How can we reduce food waste?
Instead of solving these manually, RestaurantOS AI coordinates multiple specialized AI agents that collaborate together.
Multi-Agent Architecture
RestaurantOS AI uses five specialized agents orchestrated through a sequential pipeline.
+--------------------------------------+
| User Request |
| e.g. "Tomato supply shortages" |
+------------------+-------------------+
|
v
+--------------------------------------+
| Supervisor Agent |
| Routes, Orchestrates & Synthesizes |
+------------------+-------------------+
|
v
┌────────────────┐ ┌─────────────────┐ ┌────────────────┐ ┌─────────────────┐ ┌────────────────┐
│ Stage 1 │ ---> │ Stage 2 │ ---> │ Stage 3 │ ---> │ Stage 4 │ ---> │ Final Output │
└────────────────┘ └─────────────────┘ └────────────────┘ └─────────────────┘ └────────────────┘
| | | | |
v v v v v
┌────────────────┐ ┌─────────────────┐ ┌────────────────┐ ┌─────────────────┐ ┌────────────────┐
│ Demand Agent │ ---> │ Inventory Agent │ ---> │ Waste Agent │ ---> │ Purchase Agent │ ---> │ Response │
│ │ │ │ │ │ │ │ │ Synthesis │
│ Forecast POS │ │ Detect Stock │ │ Reduce Waste │ │ Generate POs │ │ Final Decision │
│ Sales Demand │ │ Deficits │ │ & Promotions │ │ Supplier Logic │ │ Returned │
└────────────────┘ └─────────────────┘ └────────────────┘ └─────────────────┘ └────────────────┘
Enter fullscreen mode Exit fullscreen mode
Supervisor Agent
Acts as the orchestrator.
Responsibilities:
- Understands user intent
- Selects workflow
- Coordinates agents
- Synthesizes final response
Demand Forecast Agent
Uses historical POS sales to estimate:
- Customer traffic
- Menu demand
- Peak hours
Inventory Intelligence Agent
Compares projected demand against current inventory.
Detects:
- Ingredient shortages
- Overstock
- Critical stock levels
Waste Reduction Agent
Identifies ingredients approaching expiry.
Suggests:
- Chef specials
- Promotional meals
- Smart inventory utilization
Purchase Decision Agent
Creates supplier purchase recommendations by considering:
- Stock deficits
- Vendor pricing
- Lead times
- Bulk purchasing
Integrating OpenTelemetry
To trace every component of our backend, we initialized the OpenTelemetry Node SDK before the Express application starts.
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
const sdk = new NodeSDK({
traceExporter,
instrumentations: [
new HttpInstrumentation(),
new ExpressInstrumentation(),
new PrismaInstrumentation(),
getNodeAutoInstrumentations()
]
});
sdk.start();
Enter fullscreen mode Exit fullscreen mode
This automatically traces:
- HTTP requests
- Express middleware
- Prisma queries
- External network calls
Custom Agent Instrumentation
Auto instrumentation isn’t enough for AI systems.
We created custom spans around every agent execution.
return tracer.startActiveSpan(`Agent ${agent.name}`, async (span) => {
span.setAttribute("agent.name", agent.name);
const result = await agent.execute(input);
span.setStatus({
code: SpanStatusCode.OK
});
span.end();
});
Enter fullscreen mode Exit fullscreen mode
Each span records:
- Agent name
- Execution duration
- Success/failure
- Events
- Exceptions
This lets us visualize every agent independently inside SigNoz.
LLM Observability
LLM requests deserve their own telemetry.
We adopted OpenTelemetry GenAI Semantic Conventions.
span.setAttribute("gen_ai.request.model", model);
span.setAttribute(
"gen_ai.usage.total_tokens",
response.usage.total_tokens
);
span.setAttribute(
"gen_ai.latency_ms",
durationMs
);
Enter fullscreen mode Exit fullscreen mode
Now every LLM request captures:
- Prompt tokens
- Completion tokens
- Total tokens
- Model
- Latency
Exactly what production AI applications need.
Leveraging SigNoz MCP
During development we also used the official SigNoz MCP Server.
It enabled:
- Dashboard generation through natural language
- Trace exploration
- Log investigations
- Alert creation
- Querying ClickHouse metrics directly
Instead of manually configuring dashboards, the AI assistant generated many of them automatically.
End-to-End Trace Flow
When a user submits a request:
POST /ai/query
│
├── Demand Agent
│ └── LLM Request
│
├── Inventory Agent
│ ├── Prisma Query
│ └── LLM Request
│
├── Waste Agent
│ └── LLM Request
│
└── Purchase Agent
├── Prisma Query
└── LLM Request
Enter fullscreen mode Exit fullscreen mode
Inside SigNoz we can immediately identify:
- Slow database queries
- Slow LLM calls
- Token-heavy prompts
- Failed agents
- Error propagation
This transformed debugging from hours into minutes.
Real-world Problems We Solved
1. OpenTelemetry Loaded Before dotenv
Since OpenTelemetry was initialized using Node’s --import, it executed before our application loaded environment variables.
As a result, the OTLP exporter ignored our cloud configuration.
Solution
import dotenv from "dotenv";
dotenv.config();
Enter fullscreen mode Exit fullscreen mode
Load environment variables before initializing OpenTelemetry.
2. PostgreSQL Port Conflict
Native PostgreSQL on Windows occupied port 5432, causing Prisma to connect to the wrong instance.
Solution
Expose Docker PostgreSQL on 5435 instead.
ports:
- "5435:5432"
Enter fullscreen mode Exit fullscreen mode
3. Docker Volume Credential Caching
Changing POSTGRES_PASSWORD had no effect because PostgreSQL initializes credentials only once.
Solution
Delete the existing volume.
docker rm -f restaurantos-postgres
docker volume rm restaurantos_ai_postgres_data
docker compose up -d postgres
Enter fullscreen mode Exit fullscreen mode
Key Learnings
Building RestaurantOS AI taught us several important lessons.
✅ AI systems require domain-specific observability.
✅ OpenTelemetry makes every agent execution traceable.
✅ SigNoz provides a complete end-to-end visualization.
✅ Token usage should be monitored just like CPU or memory.
✅ Database spans reveal bottlenecks that AI latency often hides.
Most importantly:
Observability turns AI from a black box into production software.
Conclusion
RestaurantOS AI combines specialized AI agents with production-grade observability.
By integrating OpenTelemetry and SigNoz, every HTTP request, database query, LLM call, and autonomous decision becomes traceable.
Instead of wondering why an agent behaved a certain way, we can inspect the complete execution path in seconds.
As AI systems continue becoming more autonomous, observability will become just as important as the models themselves.
If you’re building AI applications for production, start instrumenting early you’ll thank yourself later.
Happy Building! 🚀
답글 남기기