ChronicleOps: Autonomous Chaos Engineering & Self-Healing Telemetry Engine published

What I Built

ChronicleOps is an autonomous cloud chaos engineering, real-time observability, and self-healing platform built natively on Zerops.

Instead of manually troubleshooting downtime, ChronicleOps lets developers spin up target application stacks, inject controlled fault vectors (such as process kills or container crashes), observe real-time telemetry degradation, and let the system automatically recover services while generating Gemini AI root-cause diagnoses.

The Problem & Solution

• The Problem: Microservice failures in cloud environments are often chaotic and hard to diagnose. Reproducing fault states safely without breaking persistent infrastructure is difficult, and measuring exact Mean Time to Recovery (MTTR) requires heavy monitoring setups.
• The Solution: ChronicleOps automates the entire chaos-to-recovery loop. With a single click, it provisions isolated Zerops environments, kills target processes, measures recovery latency down to the millisecond, and feeds real-time container log streams into vector-matched AI for automated incident reports.

Architecture & Tech Stack

ChronicleOps is structured as a multi-service architecture running directly on Zerops infrastructure:

Component Stack Description Orchestrator FastAPI (Python 3.11), Uvicorn Primary API gateway, environment lifecycle orchestrator, and Gemini AI diagnosis engine. Worker Python 3.11, Zerops CLI (zcli) Background polling daemon that monitors target health, triggers recovery restarts, and runs TTL resource pruning. Ingestion Engine FastAPI, Pub/Sub Log collector and real-time event pipeline for telemetry streams. Database & Cache PostgreSQL, Valkey Vector store for historical incident embeddings and real-time state synchronization. Frontend UI HTML5, Modern CSS3, Canvas, Vanilla JS Zero-dependency Command Center featuring CRT scanline aesthetic, real-time Canvas telemetry charts, and interactive controls.

                     ┌─────────────────────────────────────────┐
                     │          ChronicleOps Dashboard         │
                     │    (Vanilla JS + HTML5 Canvas RTC)      │
                     └────────────────────┬────────────────────┘
                                          │
                                          ▼
                     ┌─────────────────────────────────────────┐
                     │          Orchestrator Service           │
                     │          (FastAPI / Python 3.11)        │
                     └───────┬─────────────────────────┬───────┘
                             │                         │
                             ▼                         ▼
┌─────────────────────────────────────────┐   ┌─────────────────────────────────┐
│              Worker Service             │   │       Ingestion & AI Engine     │
│   (Zerops CLI / Health & TTL Poller)    │   │  (PostgreSQL + Gemini Vector RAG)│
└─────────────────────────────────────────┘   └─────────────────────────────────┘

Enter fullscreen mode Exit fullscreen mode

Key Features

• 1-Click Demo Quickstart: Instantly provisions an isolated application stack on Zerops using a demo repository without local configuration.
  Controlled Chaos Injection: Injects process termination signals (SIGKILL / 502 Bad Gateway) into active containers to test fault tolerance.

• Real-time Canvas Telemetry: Tracks 1.0s RTC uptime signals with visual telemetry drop and step-up curves.
  Autonomous Self-Healing & MTTR: Detects service outages in real time, executes automated restarts via zcli, and calculates exact Mean Time to Recovery metrics.

• Gemini AI Root-Cause Diagnosis: Correlates log tails against historical incident vector embeddings stored in PostgreSQL/Valkey to generate root-cause reports, confidence scores, and recommended fixes.

• TTL Environment Pruning: Automatically cleans up expired demo environments and Zerops project resources to keep platform quotas clean.

How Zerops Powers ChronicleOps
ChronicleOps relies on Zerops for its deployment pipeline and multi-service orchestrations:

1. Isolated Multi-Service Deployments: Deploys orchestrator, worker, and ingestion services independently via zerops.yaml.
2. Dynamic Infrastructure Provisioning: Uses Zerops CLI (zcli) inside the worker service to spin up and tear down temporary user environments on demand.
3. Internal Service Discovery & Public Ingress: Connects microservices using internal Zerops environment variables (VALKEY_HOST, POSTGRES_URL) while exposing public ingress endpoints for target apps.

YAML
zerops:
  - setup: orchestrator
    build:
      base: [email protected]
      buildCommands:
        - pip install -r orchestrator/requirements.txt
      deployFiles:
        - ./orchestrator
    run:
      base: [email protected]
      ports:
        - port: 8080
          httpSupport: true
      start: uvicorn orchestrator.app.main:app --host 0.0.0.0 --port 8080

Enter fullscreen mode Exit fullscreen mode

Links & Live Demo
Live Dashboard: https://orchestrator-2b86-8080.prg1.zerops.app
Main GitHub Repository: https://github.com/siddarth709/ChronicleOps
Demo App Target Repository: https://github.com/siddarth709/ChronicleOps_Demo