I Had 5 Servers Sending Metrics But Couldn’t Tell Which One Was Which
What happened
I had one server. It sent metrics (CPU, errors, etc.) to my monitoring dashboard. Fine.
Then I added more servers same service, running on 4-5 machines now.
All the metrics from all the machines showed up in the same dashboard, but with no way to tell which machine sent which metric. If CPU spiked, I couldn’t tell if it was server 1 or server 4. If errors went up, same problem.
This is useless when something breaks. “Something is wrong” isn’t helpful I need “server 3 is wrong.”
Why it happened
Each server was supposed to tag its own metrics with its own name like a label saying “this metric came from server-3.”
I had set up a config value meant to do exactly that. But when I checked, that name was never actually reaching the code that sends metrics. The setting existed in a file, but it wasn’t making it into the running program the way I assumed it would.
So every server, silently, was sending metrics with no name attached or the same default name. No error. No warning. It just quietly didn’t work.
How I fixed it
Instead of hoping a config file setting would “just work” and reach the right place automatically, I made it explicit:
- When a server starts up, its name gets set directly, as a hard requirement.
- That name gets passed by hand, step by step, into the exact piece of code that sends metrics.
- Nothing is assumed to “just be available” it’s handed over directly, on purpose, every time.
I checked it worked by looking at the actual metrics in the dashboard afterward and confirming each server’s name showed up correctly.
The real lesson
If your app depends on a setting “just working” through some automatic config-loading magic double check it’s actually arriving where you think it is. Don’t assume. Go look.
And before you scale from 1 server to many: make sure you can tell them apart in your monitoring before you actually need to, during a real problem, at 2am, with no way to know which machine to even look at.