PostgreSQL was active.
Nginx was active.
PHP-FPM was active.
The application still returned:
HTTP 503
Enter fullscreen mode Exit fullscreen mode
The actual root cause was a completely full filesystem backing a PostgreSQL tablespace.
I built this incident deliberately in my Ubuntu/KVM support lab to test a common troubleshooting mistake:
Treating process state as proof of service health.
The incident became a useful example of why application health, dependency health, and process health need to be investigated separately.
Lab architecture
I used three Ubuntu VMs connected through a private libvirt network:
Client
|
v
vm-web-01
192.168.100.10
Nginx + PHP-FPM
|
| TCP/5432
v
vm-db-01
192.168.100.20
PostgreSQL 14
|
| node_exporter
v
vm-monitor-01
192.168.100.30
Prometheus + Alertmanager
Enter fullscreen mode Exit fullscreen mode
The database VM also had a dedicated disk mounted at:
/mnt/inc012-db
Enter fullscreen mode Exit fullscreen mode
That filesystem was used for the PostgreSQL tablespace, which meant I could create storage pressure without filling the VM’s root filesystem.
Establishing the baseline
Before introducing the failure, I checked the application, database, filesystem, and monitoring state.
The PostgreSQL tablespace filesystem was:
8.03% used
Enter fullscreen mode Exit fullscreen mode
The application returned:
HTTP 200
Enter fullscreen mode Exit fullscreen mode
Database writes succeeded.
Prometheus showed:
DBDiskNearlyFull state=inactive health=ok
Enter fullscreen mode Exit fullscreen mode
This gave me a known-good state to compare against during the incident.
Monitoring detected the problem first
I then consumed storage on the dedicated database filesystem in a controlled way.
When usage reached:
88.49%
Enter fullscreen mode Exit fullscreen mode
Prometheus changed the storage alert to:
DBDiskNearlyFull firing
Enter fullscreen mode Exit fullscreen mode
But the application was still returning:
HTTP 200
Enter fullscreen mode Exit fullscreen mode
This was one of the most important observations in the test.
The monitoring system had identified a capacity problem before the customer-facing service failed.
That is exactly what useful monitoring should do: create an intervention window before an operational condition becomes an outage.
Then the filesystem reached 100%
I continued the controlled storage consumption until the dedicated filesystem reached:
100%
Enter fullscreen mode Exit fullscreen mode
The database VM’s root filesystem remained healthy.
PostgreSQL also continued reporting:
active
Enter fullscreen mode Exit fullscreen mode
So at a quick glance, the database server could appear healthy.
It wasn’t.
Testing the operation that actually mattered
A connectivity test alone wasn’t enough.
PostgreSQL could still accept connections, but when I performed a write that required additional filesystem blocks, it failed with:
ERROR: could not extend file
No space left on device
HINT: Check free disk space.
Enter fullscreen mode Exit fullscreen mode
The PostgreSQL logs showed the same ENOSPC condition.
That exposed the distinction I was looking for:
PostgreSQL process health
!=
PostgreSQL write health
!=
Application health
Enter fullscreen mode Exit fullscreen mode
A running database process does not necessarily mean the database can complete the operations the application depends on.
The user-facing failure appeared at the web tier
The actual failure originated in database storage.
But the visible symptom appeared somewhere else.
The web endpoint changed from:
HTTP 200
Enter fullscreen mode Exit fullscreen mode
to:
HTTP 503
{"status":"degraded","database":"write_failed"}
Enter fullscreen mode Exit fullscreen mode
At the same time:
Nginx active
PHP-FPM active
PostgreSQL active
Enter fullscreen mode Exit fullscreen mode
All three processes were alive while the application was degraded.
If my investigation had stopped at:
systemctl is-active nginx
systemctl is-active postgresql
Enter fullscreen mode Exit fullscreen mode
I could easily have concluded that the web and database layers were fine.
They weren’t.
My investigation path
I worked down the dependency chain instead:
HTTP 503
↓
Check Nginx and PHP-FPM
↓
Processes active
↓
Check PostgreSQL reachability
↓
Database reachable
↓
Test an actual database write
↓
Write fails with ENOSPC
↓
Inspect PostgreSQL error
↓
Check tablespace filesystem
↓
Filesystem 100% full
↓
Correlate with Prometheus
↓
Storage alert already firing
Enter fullscreen mode Exit fullscreen mode
The key shift was moving from:
Is the process running?
Enter fullscreen mode Exit fullscreen mode
to:
Can this dependency perform the operation the application requires?
Enter fullscreen mode Exit fullscreen mode
That narrowed the failure to storage rather than networking, web-server availability, process crashes, or general VM capacity.
Recovery
Because the disk pressure had been generated intentionally for the lab, I removed the temporary filler data from the isolated database filesystem.
I did not restart PostgreSQL.
After the storage was released, filesystem usage returned to:
8.27%
Enter fullscreen mode Exit fullscreen mode
PostgreSQL writes succeeded again.
The application returned:
HTTP 200
Enter fullscreen mode Exit fullscreen mode
Prometheus returned the storage alert to inactive, and Alertmanager had no remaining active DBDiskNearlyFull alert.
Before, failure and recovery
Check Baseline Failure Recovery DB filesystem 8.03% 100% 8.27% PostgreSQL process Active Active Active PostgreSQL writes Successful ENOSPC Successful Application HTTP 200 HTTP 503 HTTP 200 Prometheus alert Inactive Firing Inactive Alertmanager Clear Active ClearThe most interesting line is still:
PostgreSQL process
Active → Active → Active
Enter fullscreen mode Exit fullscreen mode
The process never stopped.
The service still failed from the application’s perspective.
What I took away from the incident
1. Process status is only one health signal
systemctl is-active tells me whether systemd considers a process active.
It does not prove that the service can perform useful work.
2. Check the dependency the application actually uses
For a PostgreSQL-backed application, TCP connectivity or pg_isready is useful, but it may not be enough.
The application needed database writes.
So write capability was the meaningful test during this incident.
3. Monitoring should create time to respond
The alert fired at 88.49% while the application was still healthy.
That gap between warning and outage is operationally valuable.
Capacity alerts should give engineers enough time to investigate and remediate before the filesystem reaches exhaustion.
4. Validate recovery through the entire stack
Freeing disk space was not enough for me to call the incident resolved.
I checked:
Filesystem capacity
→ PostgreSQL write
→ application HTTP response
→ Prometheus state
→ Alertmanager state
Enter fullscreen mode Exit fullscreen mode
Only after all of those recovered did I consider the incident validated.
5. Restarting isn’t automatically the fix
PostgreSQL did not need a restart.
The process wasn’t broken.
Its storage dependency was.
Restarting it would have treated the symptom instead of the root cause.
Commands I would use during a similar investigation
Check filesystem block capacity:
df -hT /mnt/inc012-db
Enter fullscreen mode Exit fullscreen mode
Check inode capacity:
df -i /mnt/inc012-db
Enter fullscreen mode Exit fullscreen mode
Find large consumers:
sudo du -xhd1 /mnt/inc012-db | sort -h
Enter fullscreen mode Exit fullscreen mode
Check PostgreSQL process state:
systemctl is-active postgresql
Enter fullscreen mode Exit fullscreen mode
Check PostgreSQL connectivity:
pg_isready
Enter fullscreen mode Exit fullscreen mode
But after those checks, I would still test the actual database operation required by the application.
Final takeaway
The entire incident can be summarized as:
Prometheus warned first.
PostgreSQL stayed active.
Database writes failed.
The application returned HTTP 503.
The real root cause was exhausted database storage.
Enter fullscreen mode Exit fullscreen mode
The biggest lesson for me was simple:
“The service is running” and “the service is healthy” are not the same statement.
This experiment was performed entirely in an isolated Ubuntu/KVM lab using controlled fault injection and manual troubleshooting.
Full lab and incident report
The full INC012 incident report and the rest of my 16-incident Ubuntu/KVM support lab are available here: