Overwatch

AI-powered Docker log monitor with a live web UI. Overwatch tails all running container logs, detects errors automatically, and uses local Ollama models to diagnose problems and propose fixes — no cloud APIs, no cost per query.

Designed to run alongside Dozzle on a home server or VPS. Dozzle gives you raw log access; Overwatch gives you the AI layer on top.

---

Features

• Live log streaming — all Docker containers, color-coded by severity, filterable by container and log level (all / info / warning / error)
• Automatic error detection — regex pre-filter catches errors/exceptions/OOM/timeouts before touching the LLM
• Adaptive anomaly scoring — findings can trigger on spikes/novel signatures/severity even below static line thresholds
• Pre-emptive risk scoring — drift-aware risk score + risk horizon warns before hard failure signatures
• AI analysis — suspicious log windows are sent to a local Ollama model for structured diagnosis (severity, summary, root cause)
• Diagnostic plans — a second model generates step-by-step investigation steps and proposed fix actions
• Context-enriched prompts — analysis and planning include bounded runtime context with redaction
• Local model failover — optional local fallback model routing with AI health visibility
• One-click fixes — restart a container or exec any AI-suggested command directly from the UI, with a confirm dialog and inline output
• Fingerprint clustering — repeated incidents merge into one finding with occurrence count and last-seen updates
• Incident correlation — related findings are grouped into incident clusters with confidence and evidence
• Blast-radius hints — findings show likely impacted peer services to speed triage
• Lifecycle workflow — finding states support open / investigating / mitigated / regressed / resolved / dismissed
• Policy guardrails — high-risk exec actions require explicit approval before execution
• Safe auto-remediation — policy-profile-driven low-risk restart automation with verification and escalation
• Sidebar filtering — group containers by Compose stack, expand/collapse stacks, filter to unhealthy containers only
• Findings filtering — toggle between active-only and all (including dismissed) findings
• Operational status bar — live connection state, AI health, detected container count, and server uptime
• Policy templates — switch recommendation-only / conservative / default / aggressive profile from the UI
• Prioritized work queue — risk + blast radius + age based incident ordering via API
• Shift summary export — one-click markdown handoff summary from audit tab
• Audit log — every finding, plan, and executed action is persisted to SQLite
• Fully local — uses qwen3:8b for analysis and devstral-small-2 for planning by default; both configurable

---

Screenshot

---

Requirements

• Docker + Docker Compose (on the target host)
• Ollama reachable on the LAN (same host or any other machine), with at least one capable model pulled
• LAN access to the VPS/server from your browser

Recommended models (pull these before starting):

ollama pull qwen3:8b          # analysis — fast, strong reasoning
ollama pull devstral-small-2  # planning — code/DevOps-focused

Any instruction-following model works. Edit config/overwatch.yaml to use what you have.

---

Installation

Production (recommended)

git clone https://github.com/your-username/overwatch.git
cd overwatch

# Set up environment
cp .env.example .env
nano .env                      # adjust OLLAMA_HOST if on Linux (see note below)

# Optional: adjust models, thresholds, allowed actions
nano config/overwatch.yaml

docker compose up -d --build

The UI is available at http://<host-ip>:8090.

Linux VPS note: host.docker.internal does not resolve on Linux by default. Set OLLAMA_HOST in .env to your host's LAN IP:

OLLAMA_HOST=http://192.168.1.x:11434

Or add this to the backend service in docker-compose.yml instead:

extra_hosts:
  - "host.docker.internal:host-gateway"

Development

# Terminal 1 — backend with hot reload (port 8000)
docker compose -f docker-compose.dev.yml up backend

# Terminal 2 — frontend dev server with HMR (port 5173)
cd frontend && npm install && npm run dev

The Vite dev server proxies /api and /ws to the backend automatically.

---

Configuration

Environment variables

Copy .env.example to .env — Docker Compose loads it automatically.

Variable	Default	Description
OLLAMA_HOST	http://host.docker.internal:11434	Ollama API URL. Can point to any host on the LAN, not just the VPS itself.
OVERWATCH_PORT	8090	Port the web UI is exposed on.

config/overwatch.yaml

ollama:
  host: http://host.docker.internal:11434  # overridden by OLLAMA_HOST env var
  analysis_model: qwen3:8b        # used for: severity / summary / root cause
  planning_model: devstral-small-2 # used for: diagnostic steps + proposed actions

monitor:
  log_window_seconds: 30          # rolling window size for grouping errors
  min_error_lines_to_trigger: 3   # how many suspicious lines trigger an analysis
  finding_severity_threshold: WARNING  # minimum severity to generate a plan
  cooldown_minutes: 10                 # suppress duplicate findings per container for this long
  anomaly_score_threshold: 2.0         # emit early when anomaly score crosses threshold
  risk_score_threshold: 65.0           # emit pre-emptive warnings above this risk level
  auto_remediation_profile: recommendation_only  # recommendation_only|conservative|default|aggressive
  auto_remediation_window_minutes: 30
  auto_remediation_max_per_window: 2

allowed_actions:
  - type: docker_restart
    description: Restart a container
  - type: docker_exec
    commands:
      - "*"   # allow any AI-suggested exec command
      # Replace "*" with an explicit list to restrict, e.g.:
      # - "nginx -s reload"
      # - "curl -v https://example.com"

Use "*" under docker_exec.commands to allow any AI-suggested exec command (suitable for a trusted LAN). Replace it with an explicit list to restrict what can be run.

---

How it works

Docker socket
    │
    ▼
Log monitor — tails all running containers via Docker SDK (one thread per container)
    │
    ▼
Error detector — cheap regex pre-filter (ERROR, FATAL, Exception, OOM, timeout, ...)
  │
  ├── Fingerprint + anomaly scoring (novel signature / spike / severity)
  │      emits finding metadata: score + trigger reasons
  │
  ├── Drift baseline + risk scoring
  │      emits risk_score + risk_horizon
  │
  │  (threshold OR anomaly trigger OR risk trigger)
    ▼
AI analyzer ──► Ollama qwen3:8b
  │              returns: severity, summary, root cause, confidence
  │              uses context-enriched, size-bounded, redacted prompts
    ▼
SQLite (findings table) + WebSocket broadcast → UI
    │
  ├── Correlation engine
  │      groups findings into incident_group + blast_radius
  │
    └── if severity ≥ threshold:
          AI analyzer ──► Ollama devstral-small-2
                           returns: diagnostic steps + proposed actions
               actions reordered by historical outcome ranking
                        SQLite (plans table) + WebSocket broadcast → UI

User clicks "Execute" in UI → confirm dialog → POST /api/plans/.../execute
    │
    ▼
Action executor (docker restart / exec, via Docker SDK)
    │
  ├── optional low-risk auto-remediation by policy profile
  │      verification + escalation if recovery checks fail
  │
    ▼
SQLite (audit_log table) + WebSocket broadcast → UI

The regex pre-filter plus anomaly gate means Ollama is only invoked when logs look suspicious enough to justify analysis.

---

API highlights

• GET /api/server-status — backend start time + uptime seconds (used by top-bar uptime)
• GET /api/ai-health — local model health/failure snapshot
• GET /api/anomaly — latest anomaly evaluation snapshot
• GET /api/risk — persisted per-container risk snapshots and threshold
• GET /api/work-queue — prioritized actionable incidents (risk/blast-radius/age)
• GET /api/summary/shift — markdown shift handoff summary
• POST /api/policy-template — switch auto-remediation profile at runtime
• POST /api/findings/{id}/status — lifecycle status transition endpoint

---

Upgrading

git pull
docker compose up -d --build

The SQLite database in data/ persists across upgrades automatically. If a schema change is needed in a future version, it will be noted in the release notes.

Restricted-network VPS note

If your VPS cannot reach GitHub or Docker Hub directly, a plain git pull or docker compose up --build may not refresh code/images.

In that environment, use this workflow instead:

1. Build and test locally.
2. Create and transfer a git bundle (git bundle create ... + scp ...).
3. On VPS: git fetch ./bundle main && git reset --hard FETCH_HEAD.
4. Recreate stack with docker compose -p overwatch up -d --force-recreate.

If frontend/backend images are stale and cannot be rebuilt due to blocked registry access, you can temporarily hot-sync artifacts/source into running containers (docker cp ...) and restart services. This workaround is not persistent across future recreate operations.

---

Debugging

UI shows "connecting..." and never connects

Check that the backend started successfully:

docker compose logs backend

On Linux, verify Ollama is reachable from inside the container:

docker exec overwatch-backend-1 curl http://host.docker.internal:11434/api/tags

If it fails, set OLLAMA_HOST to your host's LAN IP or add extra_hosts: ["host.docker.internal:host-gateway"] to the backend service in docker-compose.yml.

No containers appear in the sidebar

The backend needs access to the Docker socket. Verify the volume mount:

docker inspect overwatch-backend-1 | grep -A5 Mounts

The socket /var/run/docker.sock must be present. On some systems you may need to add the container user to the docker group.

No findings are generated despite visible errors

The pre-filter requires at least 3 suspicious lines within the configured window (default: 30 seconds). You can trigger a synthetic finding to test the full pipeline:

docker exec <any-container> sh -c \
  'for i in $(seq 1 5); do echo "ERROR: synthetic test failure $i" >&2; sleep 2; done'

Wait up to 30 seconds for the analysis to appear.

Ollama requests are slow or time out

Switch to a smaller model in config/overwatch.yaml:

ollama:
  analysis_model: qwen3:1.7b
  planning_model: qwen3:8b

qwen3:1.7b (1.4 GB) is very fast and sufficient for log analysis.

An action button shows an error instead of executing

The error message is shown inline on the button. A Not Permitted error means the command is not in the allowed_actions allowlist. The default config uses "*" (allow all), so this only occurs if you have restricted the list. Add the command to config/overwatch.yaml and restart the backend:

docker compose restart backend

Findings keep firing for the same ongoing problem

Findings are clustered by fingerprint. Repeated matching incidents update occurrence count and last_seen_at on the existing finding. Cooldown is still used to dampen truly new non-novel incidents in the same container.

Database is missing or corrupted

The database lives at data/overwatch.db. To reset:

docker compose down
rm data/overwatch.db
docker compose up -d

---

Project structure

overwatch/
├── backend/
│   ├── main.py              # FastAPI app, WebSocket hub, API routes
│   ├── log_monitor.py       # Docker log tailing + window accumulation
│   ├── error_detector.py    # Regex pre-filter
│   ├── ai_analyzer.py       # Ollama HTTP client (analysis + planning)
│   ├── action_ranking.py    # historical outcome ranking + explainability metadata
│   ├── action_executor.py   # docker restart / exec
│   ├── correlation.py       # incident grouping + blast-radius inference
│   ├── database.py          # SQLAlchemy async + SQLite models
│   └── config.py            # YAML config loader
├── frontend/
│   └── src/
│       ├── App.tsx           # Three-panel layout + tab bar
│       ├── store/index.ts    # Zustand global state
│       ├── hooks/useWebSocket.ts
│       └── components/
│           ├── ContainerGrid.tsx   # Sidebar with health dots
│           ├── LogStream.tsx       # Live log view
│           ├── FindingsPanel.tsx   # AI finding cards
│           ├── PlanView.tsx        # Diagnostic plan + action buttons
│           └── AuditLog.tsx        # History table
├── config/
│   └── overwatch.yaml       # Models, thresholds, allowed actions
├── data/                    # SQLite database (gitignored)
├── .env.example             # Environment variable template
├── docker-compose.yml       # Production
└── docker-compose.dev.yml   # Development with hot reload

---

Credits

Built with:

• FastAPI — Python async web framework
• Ollama — local LLM inference runtime
• qwen3:8b by Alibaba — log analysis model
• devstral-small-2 by Mistral AI — diagnostic planning model
• Docker Python SDK — container log streaming and action execution
• SQLAlchemy + aiosqlite — async SQLite persistence
• React + Vite — frontend framework and build tool
• Tailwind CSS — styling
• Zustand — frontend state management
• Dozzle by Amir Raminfar — the log viewer this was designed to complement

Designed and implemented by Claude (Anthropic claude-sonnet-4-6), based on a specification by Niels Emmer.