diff --git a/docs/fixes-todo.md b/docs/fixes-todo.md deleted file mode 100644 index 4e4c4ef..0000000 --- a/docs/fixes-todo.md +++ /dev/null @@ -1,228 +0,0 @@ -# Fixes & Hardening TODO - -Kritisches Review-Backlog vor Produktiveinsatz und OSS-Release. -Sortiert nach Risikoklasse. Datei-/Zeilenreferenzen beziehen sich auf den Stand bei Review-Erstellung. - ---- - -## 🔴 Kritisch — vor Produktion blockierend - -- [x] **1. Default-Admin `admin/admin` entfernen** ✅ erledigt - - `management/src/db.js:83` — wirft jetzt Exception bei leerem `AUTH_PASSWORD`. Kein Silent-Fallback mehr. - - README/`docs/deployment.md` entsprechend anpassen (noch offen). - -- [x] **2. `API_TOKEN` im Node-Agent zur Pflicht machen** ✅ erledigt - - `agent/src/config.js:44` — Start schlägt fehl wenn Token fehlt oder kürzer als 32 Zeichen. - - `agent/src/index.js` — `if (!config.apiToken) next()` entfernt; Token-Prüfung immer aktiv. - -- [x] **3. HTTPS zwischen Management und Agent erzwingen** ✅ erledigt - - `management/src/routes/nodes.js:102-112` — `validateBaseUrl` erzwingt `https://`. Escape-Hatch `ALLOW_INSECURE_AGENT_HTTP=true` nur für Dev. - - `management/src/agentClient.js` — eigene CA über `AGENT_CA_FILE` konfigurierbar. - - Agent-Server unterstützt TLS via `HTTPS_ENABLED`, `TLS_CERT_FILE`, `TLS_KEY_FILE`. - -- [x] **4. CSRF-Risiko durch reflektierendes CORS schließen** ✅ erledigt - - `management/src/index.js:18-27` — Nur Origins aus `CORS_ORIGINS`-Env zugelassen. - - `management/src/auth.js:57` — Cookie auf `SameSite=Strict` gesetzt. - -- [x] **5. Brute-Force-Schutz beim Login** ✅ erledigt - - `management/src/routes/auth.js:7-9` — In-Memory Rate-Limit: 5 Versuche / 15 min / IP+User. - - Audit-Event `login_blocked` und `login_failed` implementiert. - - Hinweis: Kein externer `express-rate-limit` nötig, eigenständige Implementierung ausreichend. - -- [x] **6. Restore atomarisieren — kein direktes `dd` auf Produktiv-Volume** ✅ staged Restore umgesetzt - - Datei: `agent/src/executor.js:92` (`streamResticDumpToDd`), `agent/src/routes/restore.js` - - Umsetzung: Restore schreibt zuerst in ein temporäres ZFS-Volume, prüft die Größe vorab und tauscht danach per `zfs rename` gegen das Produktiv-Volume. - - Das alte Volume bleibt als `*.pre-restore-*` Rollback-Kopie erhalten. - - Bei Fehlern vor dem Swap wird nur das temporäre Volume entfernt; bei Fehlern nach dem Swap versucht der Agent den alten Volume-Namen wiederherzustellen. - -- [ ] **7. Backup-Verifikation einbauen** - - Datei: `agent/src/routes/backup.js` (VM-Pfad ~Z.71, Container-Pfad ~Z.137) - - Aktuell: Bei Pipe-Fehler (`zfs send` !=0, `createReadStream`-Abbruch) committed restic ggf. einen unvollständigen Snapshot mit Status „success". - - Fix: - - Vor Start erwartete Größe ermitteln (`zfs get volsize` / `used`). - - Nach Restic-Commit `restic stats ` vergleichen. - - Bei Mismatch oder Pipe-Fehler: Restic-Snapshot per ID `forget`+`prune`en, Job auf `failed`. - -- [x] **8. Bearer-Token aus Frontend-`localStorage` entfernen** ✅ erledigt - - `frontend/src/api.js` — nur noch Cookie-Auth (`withCredentials: true`). `localStorage`-Token und `VITE_API_TOKEN` entfernt. - -- [x] **45. Secrets in `GET /settings` maskieren (Write-only)** ✅ erledigt - - Datei: `agent/src/config.js` (`readEnvSettings`/`writeEnvSettings`), `frontend/src/components/Settings.jsx` - - Problem: `readEnvSettings` gab den Wert *aller* Felder zurück — inkl. `RESTIC_PASSWORD`, `AWS_SECRET_ACCESS_KEY`, `API_TOKEN`. Über den Management-Proxy konnte damit jeder eingeloggte UI-User das Restic-Verschlüsselungspasswort und die S3-Credentials enthüllen (= alle Backups entschlüsseln und löschen). - - Fix: Secret-Felder liefern beim Lesen keinen Wert mehr, nur `hasValue: true|false`. Beim Schreiben gilt ein leeres Secret-Feld als „unverändert" (bestehender Wert bleibt erhalten). Frontend zeigt Platzhalter „gesetzt – leer lassen zum Beibehalten". - ---- - -## 🟠 Hoch - -- [~] **9. Session-Cookie `Secure`-Flag** ⚠️ teilweise erledigt - - `management/src/auth.js:56` — `Secure`-Flag via `SESSION_COOKIE_SECURE` konfigurierbar, auto-aktiviert bei `NODE_ENV=production`. - - Offen: `management/.env.example` hat `SESSION_COOKIE_SECURE=false` — in Deployment-Doku explizit als "in Produktion auf `true` setzen" dokumentieren. - -- [ ] **10. Race beim Sichtbarmachen des Snapshot-Devices** - - Datei: `agent/src/routes/backup.js:57` - - Aktuell: 2 s `sleep` reicht nicht garantiert. - - Fix: Polling-Schleife auf `fs.access(snapshotDevice)` mit Timeout; zusätzlich `udevadm trigger && udevadm settle`. - -- [x] **11. Persistenter Job- und Lock-Store im Agent** ✅ erledigt - - Datei: `agent/src/jobs.js` - - Umsetzung: Agent speichert Jobs in SQLite (`AGENT_DATABASE_PATH`). - - Beim Start werden `running`/`queued` Jobs als `failed` markiert, damit Management einen finalen Zustand pollen kann und VM-Locks nicht hängen bleiben. - - Offen für später: ressourcenspezifische Cleanup-Recovery für abgebrochene Host-Operationen. - -- [ ] **12. Toten `SESSION_SECRET` aufräumen** - - Datei: `management/src/config.js:8` - - Aktuell: `SESSION_SECRET` wird geladen, aber nirgendwo genutzt — toter Code. - - Fix: Variable und zugehörigen `.env.example`-Eintrag entfernen, oder Session-IDs HMAC-signieren und Variable dann sinnvoll nutzen. - -- [ ] **13. Snapshot-ID-Prefix-Matching eindeutig machen** - - Datei: `agent/src/validators.js:66` - - Aktuell: `startsWith` — bei Prefix-Kollision wird stillschweigend der erste Treffer genommen. - - Fix: Bei >1 Treffer 409 zurückgeben; UI auf 12-Hex-Prefix umstellen. - -- [ ] **14. RBAC einführen** - - Aktuell: ein User, alle Rechte, kein Read-only. - - Fix: Mindestens Rollen `admin` / `operator` / `viewer`. Restore nur für `admin`. - -- [ ] **15. ENV-Escaping in `writeEnvSettings` verbessern** - - Datei: `agent/src/config.js:105` - - Aktuell: Nur `\` und `"` escaped; `$`, Backticks, Newlines nicht. - - Fix: Eigener Serializer mit korrektem Escaping aller Sonderzeichen. - -- [ ] **16. Settings-Endpunkt sperren bis `API_TOKEN` initial gesetzt ist** - - Datei: `agent/src/routes/settings.js` - - Hängt mit Fix #2 zusammen — nach #2 automatisch erfüllt, hier zur Sicherheit dokumentieren/testen. - -- [x] **46. Agent-Token-Vergleich timing-safe machen** ✅ erledigt - - Datei: `agent/src/index.js:28` - - Problem: `header === \`Bearer ${config.apiToken}\`` — normaler String-Vergleich am root-Agent (das wertvollste Ziel), während das Login-Passwort bereits `timingSafeEqual` nutzte. - - Fix: Vergleich über `crypto.timingSafeEqual` mit Längen-Guard. Leerer/fehlender konfigurierter Token verweigert weiterhin (`config.apiToken &&`). - -- [ ] **47. Pre-Restore-Volumes aufräumen (Disk-Space-GC)** - - Datei: `agent/src/routes/restore.js` - - Aktuell: Jeder erfolgreiche Restore behält das alte Volume als `*.pre-restore-*` Rollback-Kopie — es gibt aber keinen Cleanup. Jeder Restore verdoppelt den Plattenbedarf der VM dauerhaft; bei mehreren Restores läuft der ZFS-Pool voll. - - Fix: Aufbewahrungsregel (z.B. „keep last N pre-restore/failed-restore Volumes pro VM") oder expliziter Cleanup-Schritt/UI-Aktion. Mindestens im Health/Operations-View sichtbar machen. - -- [ ] **48. Agent-Tokens im Management-SQLite nicht im Klartext speichern** - - Datei: `management/src/store.js` (`nodes.token`), `management/src/db.js` - - Aktuell: `nodes.token` liegt im Klartext in der Management-DB. DB-Diebstahl = alle Agent-Tokens = root auf allen Hosts. - - Fix: Verschlüsselung at-rest mit einem Management-Key (z.B. aus `SESSION_SECRET`/dediziertem Key abgeleitet), oder zumindest DB-Dateipermissions/Disk-Encryption-Anforderung im Deployment-Doc verbindlich dokumentieren. - ---- - -## 🟡 Mittel - -- [ ] **17. `Math.random()` durch `crypto.randomBytes` ersetzen** - - Datei: `management/src/db.js:93` (`cryptoId`). - - Noch vorhanden: `Math.random().toString(16)` für DB-interne User-IDs — kein akutes Sicherheitsproblem, aber nicht kryptografisch sicher. - -- [ ] **18. Systemd-Hardening für den Agent** - - Datei: `deploy/systemd/incus-backup-agent.service` - - Hinzufügen: `NoNewPrivileges=true`, `ProtectSystem=strict`, `ProtectHome=true`, `PrivateTmp=true`, `ReadWritePaths=/dev/zvol /var/lib/incus /opt/incus-backup-ui/agent`, `CapabilityBoundingSet=...`, eingeschränkte `AmbientCapabilities`. - -- [ ] **19. `npm start` durch direkten `node`-Aufruf ersetzen** - - Datei: beide `deploy/systemd/*.service` - - `ExecStart=/usr/bin/node src/index.js` — kein npm-Wrapper-Prozess, kein PATH-Risiko. - -- [ ] **20. CORS am Agent entfernen** - - Datei: `agent/src/index.js:16` - - Agent wird nie aus dem Browser angesprochen → Angriffsfläche raus. - -- [ ] **21. `schedules.json`-Pfad explizit konfigurierbar** - - Datei: `agent/src/scheduler.js:6` - - Aktuell: `process.cwd()`-abhängig. - - Fix: über Env (`SCHEDULES_PATH`) absolut konfigurieren, Default unter `/var/lib/incus-backup-agent/`. - -- [ ] **22. Audit-Log-Integrität** - - Datei: `management/src/store.js:71` - - Append-only Constraint + Hash-Chain (jeder Event hat `prev_hash`). Optional Off-Site-Versand (Webhook). - -- [ ] **23. Passwort-Hashing auf argon2id umstellen** - - Datei: `management/src/crypto.js` - - Aktuell: `scryptSync` mit Node-Defaults, ohne explizite Parameter. - - Fix: `@node-rs/argon2` oder explizite scrypt-Parameter dokumentieren + Migration-Pfad. - -- [ ] **24. Rate-Limit / Quoten für Backups** - - Pro VM und global (z.B. max N parallele Streams nach S3). Disk-Space-/Quota-Check vor Start. - -- [ ] **25. `incus snapshot delete` Retry verallgemeinern** - - Datei: `agent/src/routes/backup.js:303` - - Aktuell: Substring-Match auf englische Stderr — bricht bei lokalisierten Builds. - - Fix: Generischer Retry (n Versuche, Backoff) bei nicht-0 Exit-Code. - -- [ ] **26. Restic-`ls` streamen statt vollständig in RAM laden** - - Datei: `agent/src/routes/snapshots.js:19` - - Für Container-Backups mit vielen Files relevant. - -- [ ] **27. Container-Restore implementieren oder Container-Backup deaktivieren** - - Datei: `agent/src/routes/restore.js:21` - - Aktuell: 501. Backups laufen, aber nicht wiederherstellbar = Backup-Theater. - - Fix: Sicheren `zfs receive`-Workflow umsetzen oder Container-Backup im UI/API ausschalten bis fertig. - -- [ ] **28. Pre-Backup VM-Zustand prüfen** - - Live-Migration, laufende interne Snapshots, fehlende Berechtigungen → klare Fehler statt halb durchgeführter Pipeline. - -- [ ] **49. Abgelaufene Sessions serverseitig löschen + Rotation** - - Datei: `management/src/store.js:12` (`getUserBySession`), `createSession` - - Aktuell: Abgelaufene Sessions werden beim Lesen nur gefiltert, nie aus der DB entfernt → unbegrenztes Tabellenwachstum. Außerdem keine Session-Rotation nach erfolgreichem Login. - - Fix: Periodischer Cleanup (`DELETE FROM sessions WHERE expires_at <= now`) und neue Session-ID nach Login ausstellen. - ---- - -## 🟢 Niedrig / Aufräumen - -- [ ] **29. `.DS_Store` aus Repo entfernen und in `.gitignore` aufnehmen** -- [ ] **30. `frontend/dist/` ist eingecheckt** — ignorieren und löschen. -- [ ] **31. `incus-backup-ui-plan.md` auf Secrets/Bucket-Namen prüfen** bevor OSS. -- [ ] **32. Container-Limit (Restore fehlt) prominent in README dokumentieren.** -- [ ] **33. Scheduler-Jitter einbauen** (`management/src/scheduler.js`, `agent/src/scheduler.js`) — sonst belasten viele Nodes synchron S3. -- [ ] **34. `formatBytes` deduplizieren** (`agent/src/routes/backup.js`, `restore.js`) → `executor.js` oder `utils.js`. -- [ ] **35. `.env`-Dateipermissions dokumentieren** — `chmod 600` im Deployment-Doc verlangen. -- [ ] **36. Schema-Versionierung statt `addColumnIfMissing`** — z.B. `schema_version`-Tabelle + nummerierte Migrationen. -- [ ] **37. `audit_events.details` Größenlimit** oder JSON-Spalte (SQLite hat JSON1). - ---- - -## 📦 OSS-Release-Voraussetzungen - -- [ ] **38. Tests einführen** — aktuell keinerlei Tests im Repo. Mindestens: - - `validators.js` (vmName-/snapshot-Regex, Pfad-Validation). - - `crypto.js` (hash/verify Roundtrip, Timing-Safe). - - `jobs.js` (Locking, Trimming). - - Pipeline-Smoke-Tests mit gemockten Befehlen. -- [ ] **39. CI-Pipeline** (GitHub Actions): Lint, Tests, `npm audit`, Lockfile-Check. -- [ ] **40. `SECURITY.md`** mit Disclosure-Adresse + GPG-Key. -- [ ] **41. Threat-Model dokumentieren** — explizit machen, was außerhalb des Schutzbereichs liegt (root-Agent, vertrauenswürdiges Netz, etc.). -- [ ] **42. `LICENSE`-Datei ergänzen** (z.B. MIT/Apache-2.0). -- [ ] **43. `CONTRIBUTING.md`** + Code-of-Conduct. -- [ ] **44. Beispiel-`compose.yaml` oder Ansible-Rolle** für reproduzierbares Deployment. - ---- - -## Status-Übersicht kritische Punkte (Stand 2026-06-04) - -| # | Titel | Status | -|---|-------|--------| -| 1 | Default-Admin `admin/admin` entfernen | ✅ erledigt | -| 2 | `API_TOKEN` Pflicht | ✅ erledigt | -| 3 | HTTPS Management↔Agent | ✅ erledigt | -| 4 | CORS-Whitelist + SameSite=Strict | ✅ erledigt | -| 5 | Brute-Force-Schutz Login | ✅ erledigt | -| 6 | Restore atomarisieren | ✅ staged Restore umgesetzt | -| 7 | Backup-Verifikation | ⬜ offen | -| 8 | Bearer-Token aus localStorage | ✅ erledigt | -| 9 | Session-Cookie `Secure`-Flag | ⚠️ teilweise | -| 45 | Secrets in `GET /settings` maskieren | ✅ erledigt | -| 46 | Agent-Token-Vergleich timing-safe | ✅ erledigt | -| 47 | Pre-Restore-Volume-GC | ⬜ offen | -| 48 | Agent-Tokens in DB verschlüsseln | ⬜ offen | -| 49 | Session-Cleanup serverseitig | ⬜ offen | - -## Nächste Prioritäten - -1. **#7** Backup-Verifikation weiter härten — Pipeline-/Stream-Fehler testen und vollständiger absichern. -2. **#47** Pre-Restore-Volume-GC — sonst läuft der ZFS-Pool bei wiederholten Restores voll. -3. **#9** `SESSION_COOKIE_SECURE=true` in Deployment-Doku festschreiben. -4. **#12** Toten `SESSION_SECRET` entfernen. -5. **#15** ENV-Escaping vervollständigen (`$`, Backticks, Newlines). -6. **#11 Folgearbeit** ressourcenspezifische Cleanup-Recovery nach Agent-Crash definieren. diff --git a/docs/issues.md b/docs/issues.md index 285b394..0b73db8 100644 --- a/docs/issues.md +++ b/docs/issues.md @@ -1,103 +1,57 @@ # Issue Backlog -## Status snapshot - Last reviewed against code: 2026-06-04. -Several earlier backlog items have already landed in the codebase: +This is the single source of truth for product work, hardening, release tasks, and production blockers. The previous `docs/fixes-todo.md` list has been merged into this file. + +## Current Status + +Implemented and verified in code: - Management persists accepted agent jobs in `job_history`. - Management polls agent jobs and updates final status, error, finish time, duration, current step, and parsed backup snapshot ID. - Operations page shows job history and audit log. - Node-agent health checks cover required commands, ZFS pool state, `/dev/zvol`, and Restic repository access. -- Backup jobs verify the stored Restic file size and remove failed snapshots on verification errors. +- Backup jobs parse the Restic snapshot ID, verify stored file size, and remove failed snapshots on verification errors. +- VM restore no longer writes directly to the production ZVOL. It writes to a staged ZVOL first, then swaps volumes with `zfs rename`. +- Agent jobs are persisted in SQLite. Active jobs are marked `failed` after an agent restart because subprocesses cannot survive restart. +- Auth hardening landed: no default admin password, required agent token, CORS allowlist, `SameSite=Strict`, login brute-force protection, no frontend localStorage bearer token. +- Settings secrets are write-only/masked in the UI and API. +- Agent bearer-token comparison is timing-safe. +- Agent directory was renamed from `backend/` to `agent/`. -Known caveat: agent-side jobs are persisted, but subprocesses cannot survive an agent restart. Active jobs are marked `failed` on startup; deeper cleanup recovery for partially changed host resources is still future work. +Known caveat: active agent jobs are persisted, but deeper cleanup recovery for partially changed host resources is still future work. -Current pre-production priorities: +## Pre-Production Priorities 1. Harden backup verification and add tests for stream/pipeline failure cases. 2. Validate the staged restore workflow on a disposable Incus VM, including rollback scenarios. -3. Improve agent crash cleanup for partially changed ZFS/Incus resources. -4. Surface detailed node health diagnostics in the UI. -5. Keep the root-running agent tightly network-restricted. +3. Add cleanup and visibility for pre-restore/failed-restore ZVOLs. +4. Improve agent crash cleanup for partially changed ZFS/Incus resources. +5. Surface detailed node health diagnostics in the UI. +6. Keep the root-running agent tightly network-restricted. +7. Add automated tests and CI. -## 1. Persist final agent job status in management +## P0 - Production Blockers -Status: mostly done. - -Management records when a backup or restore was accepted by an agent and now persists the final agent job result when the node-agent remains reachable long enough to be polled. - -### Goal - -Persist reliable end-to-end job status in the management database. - -### Tasks - -- [x] Add polling for accepted agent jobs from management. -- [x] Store final `success` or `failed` status in `job_history`. -- [x] Store duration, finished timestamp, error message, and current step. -- [ ] Store agent job logs summary. -- [x] Store created Restic snapshot ID for successful backup jobs when available. -- [x] Surface final status in the Operations page. -- [x] Persist recent agent-side jobs. -- [x] Mark active agent jobs as `failed` after an agent restart so management can poll a final state. - -### Acceptance Criteria - -- [x] A backup started through management eventually shows `success` or `failed` while the agent remains reachable. -- [x] A restore started through management eventually shows `success` or `failed` while the agent remains reachable. -- [x] Management restart does not lose already persisted history. -- [x] Agent restart does not lose the active job record; active work is marked `failed` because the subprocess cannot survive restart. - -## 2. Expand node-agent health checks - -Status: mostly done. - -The health endpoint now provides deeper operational checks for backup readiness. - -### Goal - -Make `/api/health` useful for diagnosing whether a node can actually run backup and restore operations. - -### Tasks - -- [x] Check that required commands exist: `incus`, `zfs`, `zpool`, `restic`, `udevadm`, `dd`. -- [x] Check that configured ZFS pool exists. -- [x] Check that `/dev/zvol` is accessible. -- [x] Check Restic repository access. -- [x] Check S3/Restic credentials by running a safe Restic command. -- [x] Include ZFS pool capacity and free space. -- [x] Return structured check names and messages. -- [ ] Surface detailed per-node health output in the management UI, not just a compact aggregate. - -### Acceptance Criteria - -- [~] Management UI shows degraded node health with actionable check names. Detailed output exists through node health endpoints; UI can still be improved. -- [x] A missing command, wrong pool, or wrong Restic credentials is visible in health output. - -## 3. Harden backup verification +### 1. Harden Backup Verification Status: partially done. -Backup jobs now verify the backed-up Restic file size after `restic backup` and attempt to remove failed snapshots. This reduces the risk of accepting a truncated backup, but the implementation still needs stronger stream failure handling and broader verification semantics. +Goal: ensure a backup is marked `success` only when the expected source data was fully stored and verified. -### Goal - -Ensure a backup is marked `success` only when the expected source data was fully stored and verified. - -### Tasks +Tasks: - [x] Parse the created Restic snapshot ID after backup. - [x] Verify stored Restic file size against streamed source bytes or expected ZFS size. - [x] Remove failed Restic snapshots with `forget` and `prune` on verification errors. -- [ ] Make stream/pipeline failure handling explicit for all sources. -- [ ] Avoid marking success if source stream closes early but Restic exits successfully. -- [ ] Add a post-backup Restic integrity check strategy, such as targeted `restic check`/`restic stats` usage that is safe for large repositories. +- [ ] Make stream/pipeline failure handling explicit for VM and container sources. +- [ ] Avoid marking success if the source stream closes early but Restic exits successfully. +- [ ] Add a post-backup Restic integrity-check strategy that is safe for large repositories. - [ ] Define an optional restore-probe workflow for critical VMs. -- [ ] Add tests with mocked command failures and short reads. +- [ ] Add tests with mocked command failures, pipe failures, and short reads. -### Acceptance Criteria +Acceptance criteria: - [x] Successful backup jobs include a verifiable Restic snapshot ID in management history. - [x] Size mismatches fail the job. @@ -105,245 +59,436 @@ Ensure a backup is marked `success` only when the expected source data was fully - [ ] A backup can be independently verified without trusting only the successful process exit. - [ ] Verification behavior is covered by automated tests. -## 4. Add per-VM backup policy +### 2. Validate Staged Restore Workflow -Retention and schedule behavior is currently broad. Per-VM policies would make production usage more flexible. +Status: implemented, not proven on real Incus/ZFS hardware in this repo session. + +Goal: prove restore safety before production use. + +Tasks: + +- [x] Require explicit typed confirmation including VM name. +- [x] Validate snapshot ownership before restore. +- [x] Replace direct `dd` to production ZVOL with a staged restore workflow. +- [x] Log restore intent in audit log before dispatch. +- [ ] Add restore preflight endpoint. +- [ ] Validate node health before restore. +- [ ] Show selected snapshot metadata before restore. +- [ ] Show target VM status and disk size before restore. +- [ ] Optionally offer “create backup before restore” when the VM is accessible. +- [ ] Test staged restore on a disposable VM: backup, restore, boot, and confirm data. +- [ ] Test failed staged restore paths: Restic dump failure before swap, rename failure after old volume rename, and VM start failure. + +Acceptance criteria: + +- [x] Restore stream failures happen on the staged volume, not the production disk. +- [ ] UI displays a restore plan before final confirmation. +- [ ] Restore is blocked when required preflight checks fail. +- [ ] Staged restore behavior is validated on a real Incus/ZFS test node. + +### 3. Add Pre-Restore Volume Cleanup + +Status: open. + +Current behavior: every successful VM restore keeps the old volume as `*.pre-restore-*`. This is good for rollback, but without retention or cleanup it can fill the ZFS pool. + +Tasks: + +- [ ] Track pre-restore and failed-restore volumes created by the agent. +- [ ] Add a retention policy, for example keep the last N rollback volumes per VM. +- [ ] Surface retained rollback volumes in Operations or VM detail. +- [ ] Add explicit cleanup action with confirmation. +- [ ] Add health warning when retained restore volumes consume significant pool space. + +Acceptance criteria: + +- [ ] Repeated restores cannot silently consume unbounded pool space. +- [ ] Operators can see and clean retained restore volumes. + +### 4. Harden Root-Running Agent Exposure + +Status: partially done. + +The agent runs with root-level host access because it needs Incus, ZFS, `/dev/zvol`, Restic, and device operations. A compromised agent is a host-level incident. + +Tasks: + +- [x] Require a non-empty `API_TOKEN` of at least 32 characters. +- [x] Support `ALLOWED_MANAGEMENT_IPS`. +- [x] Support HTTPS directly on the agent. +- [x] Support private CA validation from management via `AGENT_CA_FILE`. +- [ ] Require private management-to-agent connectivity in production docs, such as NetBird or a dedicated management network. +- [ ] Prefer HTTPS agents with a private CA; document local HTTP only as development mode. +- [ ] Add installer warning when `HTTPS_ENABLED=false` and no `ALLOWED_MANAGEMENT_IPS` is configured. +- [ ] Add systemd hardening where compatible with Incus/ZFS access. +- [ ] Replace `npm start` in systemd with direct `node src/index.js`. +- [ ] Remove CORS from the agent; browsers should never call it directly. + +Acceptance criteria: + +- [ ] Production install docs do not expose the agent publicly by default. +- [ ] Installer warns on insecure network exposure. +- [ ] systemd unit has a documented minimum hardening baseline. + +### 5. Encrypt or Protect Agent Tokens in Management + +Status: open. + +Current behavior: `nodes.token` is stored in plaintext in the management SQLite database. DB theft means all agent tokens are exposed. + +Tasks: + +- [ ] Decide between at-rest encryption with a dedicated management key or strict documented file/disk protection. +- [ ] If encrypting, add key configuration and migration for existing tokens. +- [ ] Ensure public node API responses never include tokens. +- [ ] Document database file permissions and disk-encryption expectations. + +Acceptance criteria: + +- [ ] Stealing the management database alone is not enough to read every agent token, or the residual risk is explicitly documented and operationally mitigated. + +## P1 - High Priority + +### 6. Add Automated Tests and CI + +Status: open. + +Tasks: + +- [ ] Add a test runner for agent and management code. +- [ ] Test validators for instance names, snapshot IDs, and ambiguous snapshot prefixes. +- [ ] Test job locking, persistence, trimming, and restart behavior. +- [ ] Test backup pipeline failure handling with mocked command/process failures. +- [ ] Test restore staged-volume command sequencing with mocked ZFS/Incus/Restic commands. +- [ ] Test management job polling updates `job_history` for success, failed, timeout, and missing-agent cases. +- [ ] Add CI for install, tests, frontend build, and syntax checks. + +Acceptance criteria: + +- [ ] Pull requests run tests automatically. +- [ ] Simulated backup stream failures fail the job. +- [ ] Simulated restore failures leave the original ZVOL name restored in the command sequence. +- [ ] Agent restart behavior is covered by tests. + +### 7. Improve Agent Crash Cleanup and Resource Recovery + +Status: open. + +Agent jobs are persisted and active jobs are marked `failed` on restart, but a process crash can still leave host resources behind. + +Tasks: + +- [ ] Persist cleanup metadata for backup jobs: temporary Incus snapshot name, ZVOL, snapdev state, and backup type. +- [ ] Persist cleanup metadata for restore jobs: staged ZVOL, backup ZVOL, failed ZVOL, and swap state. +- [ ] On startup, scan failed active jobs and run safe cleanup actions. +- [ ] Log cleanup results into the persisted job logs. +- [ ] Avoid destructive cleanup when state is ambiguous; surface manual action instead. + +Acceptance criteria: + +- [ ] Crashing during backup streaming does not leave `snapdev=visible` or temporary snapshots unnoticed. +- [ ] Crashing before restore swap removes staged restore volumes when safe. +- [ ] Crashing after restore swap does not automatically destroy rollback copies. +- [ ] Management can show cleanup-required states. + +### 8. Improve Health Diagnostics in the UI + +Status: partially done on API, open in UI. + +Tasks: + +- [x] Agent health checks commands, ZFS pool, `/dev/zvol`, and Restic repository. +- [x] Management records compact node health state. +- [ ] Add a detailed health drawer or modal on the Nodes page. +- [ ] Show command, ZFS pool, `/dev/zvol`, Restic repository, and credential check results. +- [ ] Show last health timestamp and the management-side error if the agent is unreachable. +- [ ] Distinguish unreachable, unauthorized, TLS failure, degraded health, and healthy states. +- [ ] Avoid global UI timeout banners when only one enabled node is slow or unreachable. + +Acceptance criteria: + +- [ ] A missing command is visible by name in the UI. +- [ ] Wrong Restic credentials are visible as a Restic health failure. +- [ ] TLS or connectivity failures are distinguishable from degraded agent health. + +### 9. Fix Snapshot Device Visibility Race + +Status: open. + +Current behavior: backup waits a fixed 2 seconds after setting `snapdev=visible`. + +Tasks: + +- [ ] Replace fixed sleep with polling `fs.access(snapshotDevice)` and timeout. +- [ ] Run `udevadm trigger` and `udevadm settle`. +- [ ] Surface timeout as a clear backup failure. + +Acceptance criteria: + +- [ ] Slow device-node creation does not fail randomly. + +### 10. Make Snapshot-ID Prefix Matching Unambiguous + +Status: open. + +Current behavior: prefix matching uses `startsWith`; collisions choose the first match. + +Tasks: + +- [ ] Return conflict when a snapshot prefix matches more than one snapshot. +- [ ] Prefer 12+ hex characters in the UI. +- [ ] Add validator tests. + +Acceptance criteria: + +- [ ] Ambiguous snapshot prefixes cannot restore the wrong snapshot. + +### 11. Settings and Environment Hardening + +Status: partially done. + +Tasks: + +- [x] Mask secret values in `GET /settings`. +- [x] Treat empty secret field as “keep existing value”. +- [ ] Improve `.env` escaping for `$`, backticks, and newlines. +- [ ] Add tests for `readEnvSettings` and `writeEnvSettings`. +- [ ] Document `.env` file permissions and enforce `chmod 600` in installer/deployment. +- [ ] Keep or remove `SESSION_SECRET` deliberately: either use it for signing/encryption or remove dead config. +- [ ] Set and document `SESSION_COOKIE_SECURE=true` for production. + +Acceptance criteria: + +- [ ] Secret settings cannot be exfiltrated through the UI/API. +- [ ] Writing `.env` cannot create shell-expansion surprises. + +### 12. Add RBAC + +Status: open. + +Current behavior: one user model, effectively all permissions. + +Tasks: + +- [ ] Add roles: `admin`, `operator`, `viewer`. +- [ ] Restrict restore and node settings to `admin`. +- [ ] Allow read-only dashboard access for `viewer`. +- [ ] Audit role changes. + +Acceptance criteria: + +- [ ] Restore is admin-only. +- [ ] Viewer cannot trigger backup, restore, node edits, or settings writes. + +### 13. Session and Auth Cleanup + +Status: partially done. + +Tasks: + +- [x] Login brute-force protection. +- [x] `SameSite=Strict`. +- [ ] Server-side cleanup of expired sessions. +- [ ] Session rotation after successful login. +- [ ] Replace `Math.random()` in management user ID creation with `crypto.randomBytes`. +- [ ] Consider argon2id or explicit scrypt parameters and migration path. + +Acceptance criteria: + +- [ ] Expired sessions do not accumulate unbounded in SQLite. +- [ ] Session IDs are rotated after login. + +### 14. Container Restore Decision + +Status: open. + +Current behavior: container backups can be created, but restore returns `501`. + +Tasks: + +- [ ] Implement safe container restore with `zfs receive`, or disable container backups in UI/API until restore exists. +- [ ] Document the limitation prominently. +- [ ] Add tests for unsupported restore behavior. + +Acceptance criteria: + +- [ ] Users cannot mistake an unrestorable container backup for complete protection. + +## P2 - Product and Operations + +### 15. Add Per-VM Backup Policy + +Status: partially done. Current state: schedules can be enabled per VM with interval and time of day. Retention is still global per agent. -### Goal +Tasks: -Allow each VM to define its own backup policy. - -### Tasks - -- Add management database table for VM backup policies. -- Support per-VM retention values: hourly, daily, weekly, monthly. +- [ ] Add management database table for VM backup policies. +- [ ] Support per-VM retention values: hourly, daily, weekly, monthly. - [x] Support per-VM schedule enablement, interval, and time. -- Add optional policy flag: backup only when VM is running. -- Update Scheduler UI to edit policies per node and VM. -- Send policy retention to the agent backup request or apply it in management scheduling. +- [ ] Add optional policy flag: backup only when VM is running. +- [ ] Update Scheduler UI to edit policies per node and VM. +- [ ] Send policy retention to the agent backup request or apply it in management scheduling. -### Acceptance Criteria +Acceptance criteria: -- Two VMs on the same node can have different schedules. -- Two VMs on the same node can have different retention policies. -- Disabled policies do not trigger backups. +- [ ] Two VMs on the same node can have different schedules. +- [ ] Two VMs on the same node can have different retention policies. +- [ ] Disabled policies do not trigger backups. -## 5. Improve restore safety workflow +### 16. Add Failure Notifications -Restore is destructive and should be guarded with a clearer preflight and confirmation flow. +Status: open. -Current state: VM restore requires typed confirmation, validates the snapshot against the VM, checks source/target size, writes the Restic dump to a staged ZFS volume, stops the VM, and swaps the staged volume into place with `zfs rename`. The previous production volume is kept as a rollback copy. +Tasks: -### Goal +- [ ] Add notification settings in management. +- [ ] Support webhook notifications first. +- [ ] Include node, VM, job type, error, and timestamp. +- [ ] Trigger notifications for failed scheduled backups. +- [ ] Trigger notifications for failed manual backups/restores. +- [ ] Add a “send test notification” action. -Reduce the risk of accidental or unsafe restores. +Acceptance criteria: -### Tasks +- [ ] A failed scheduled backup sends one notification. +- [ ] A test notification can be triggered from the UI. +- [ ] Notification failures are visible in Operations or audit logs. -- Add restore preflight endpoint. -- Validate node health before restore. -- Show selected snapshot metadata before restore. -- Show target VM status and disk size before restore. -- [x] Require explicit typed confirmation including VM name. -- Optionally offer “create backup before restore” when the VM is accessible. -- [x] Log restore intent in audit log before dispatch. -- [x] Replace direct `dd` to production ZVOL with a staged restore workflow. -- [ ] Test staged restore on a disposable VM: backup, restore, boot, and confirm data. -- [ ] Test failed staged restore paths: Restic dump failure before swap, rename failure after old volume rename, and VM start failure. -- [ ] Define a safe container restore workflow with `zfs receive` or disable container restore surfaces entirely. +### 17. Implement Real Snapshot File Browsing -### Acceptance Criteria - -- UI displays a restore plan before the final confirmation. -- Restore is blocked when required preflight checks fail. -- Audit log records restore attempts and results. -- [x] Restore stream failures happen on the staged volume, not the production disk. -- [ ] Staged restore behavior is validated on a real Incus/ZFS test node. - -## 6. Add failure notifications - -Operators need to know when scheduled backups or restores fail. - -### Goal - -Send notifications for failed or degraded operations. - -### Tasks - -- Add notification settings in management. -- Support webhook notifications first. -- Include node, VM, job type, error, and timestamp. -- Trigger notifications for failed scheduled backups. -- Trigger notifications for failed manual backups/restores. -- Add a “send test notification” action. - -### Acceptance Criteria - -- A failed scheduled backup sends one notification. -- A test notification can be triggered from the UI. -- Notification failures are visible in Operations or audit logs. - -## 7. Implement real snapshot file browsing +Status: open. Current snapshot browsing shows Restic contents, which for block-level backups is usually only `/vm.raw`. -### Goal +Tasks: -Allow browsing files inside a backed-up VM disk image. +- [ ] Design a safe read-only raw image inspection workflow. +- [ ] Restore or mount raw image read-only in a temporary workspace. +- [ ] Detect partitions and filesystems. +- [ ] Browse directories through management UI. +- [ ] Allow downloading a single file. +- [ ] Ensure cleanup of mounts and temporary files. -### Tasks +Acceptance criteria: -- Design a safe read-only raw image inspection workflow. -- Restore or mount raw image read-only in a temporary workspace. -- Detect partitions and filesystems. -- Browse directories through management UI. -- Allow downloading a single file. -- Ensure cleanup of mounts and temporary files. +- [ ] User can browse a Linux VM filesystem from a snapshot without restoring the VM. +- [ ] Mounted/temporary resources are cleaned up after use. +- [ ] Unsupported or unsafe disk images fail with a clear error. -### Acceptance Criteria +### 18. Add Version Reporting -- User can browse a Linux VM filesystem from a snapshot without restoring the VM. -- Mounted/temporary resources are cleaned up after use. -- The workflow refuses unsupported or unsafe disk images with a clear error. +Status: open. -## 8. Add agent and management version reporting +Tasks: -Multi-node setups need version visibility. +- [ ] Add version field to management API. +- [ ] Add version field to agent health response. +- [ ] Show agent version in Nodes page. +- [ ] Flag unsupported or outdated agents. +- [ ] Document compatibility expectations. -### Goal +Acceptance criteria: -Show software version and compatibility state for management and each agent. +- [ ] Nodes page displays agent version. +- [ ] Management can identify incompatible agents. +- [ ] Health output includes version information. -### Tasks +### 19. Containerized Management and UI Deployment -- Add version field to management API. -- Add version field to agent health response. -- Show agent version in Nodes page. -- Flag unsupported or outdated agents. -- Document compatibility expectations. +Status: open. -### Acceptance Criteria +The node-agent remains a host-level systemd service. Management and frontend are good candidates for containers. -- Nodes page displays agent version. -- Management can identify incompatible agents. -- Health output includes version information. +Tasks: -## 9. Containerized management and UI deployment +- [ ] Add a `management` container image. +- [ ] Add a frontend image that serves the Vite build through a small static server or Nginx. +- [ ] Provide `compose.yaml` with persistent SQLite volume for management. +- [ ] Mount the agent CA certificate into the management container read-only. +- [ ] Document `CORS_ORIGINS`, `SESSION_COOKIE_SECURE`, `DATABASE_PATH`, and reverse-proxy assumptions. +- [ ] Decide whether frontend calls management through same-origin `/api` or separate API origin. +- [ ] Add health checks for both containers. -The node-agent must remain a host-level systemd service because it needs direct Incus, ZFS, `/dev/zvol`, Restic, and device access. Management and the frontend do not need those host privileges and are good candidates for container deployment. +Acceptance criteria: -### Goal +- [ ] Management API and frontend can be started with Compose without installing Node.js on the management host. +- [ ] Management database survives container recreation. +- [ ] Cookie login works behind HTTPS. +- [ ] Management can connect to HTTPS node-agents using the configured CA file. -Provide a production-ready Docker/Compose deployment for the management API and frontend while keeping node-agents installed as systemd services on Incus hosts. +### 20. Backup Scheduling and Quotas -### Tasks +Status: open. -- Add a `management` container image. -- Add a frontend image that serves the Vite build through a small static server or Nginx. -- Provide `compose.yaml` with persistent SQLite volume for management. -- Mount the agent CA certificate into the management container as read-only. -- Document required `CORS_ORIGINS`, `SESSION_COOKIE_SECURE`, `DATABASE_PATH`, and reverse-proxy assumptions. -- Decide whether the frontend calls the management API through the same origin reverse proxy or a separate API origin. -- Add health checks for both containers. +Tasks: -### Acceptance Criteria +- [ ] Add scheduler jitter in management and agent schedulers. +- [ ] Add per-VM and global backup concurrency limits. +- [ ] Add disk-space/quota checks before backup and restore. +- [ ] Add pre-backup VM state checks for live migration, existing snapshots, and permission problems. -- Management API and frontend can be started with Compose without installing Node.js on the management host. -- Management database survives container recreation. -- Cookie login works behind HTTPS. -- Management can connect to HTTPS node-agents using the configured CA file. +Acceptance criteria: -## 10. Add automated tests and CI +- [ ] Many nodes do not all start scheduled backups at exactly the same time. +- [ ] A full pool or too many concurrent streams blocks new jobs with a clear error. -The project currently has no automated tests. This is the biggest engineering gap for a backup system because most dangerous failures happen in error paths, not in the happy path. +## P3 - Cleanup and Refactoring -### Goal +### 21. Systemd and Deployment Cleanup -Catch regressions in validation, locking, backup verification, restore orchestration, authentication, and management polling before deployment. +Tasks: -### Tasks +- [ ] Add systemd hardening compatible with Incus/ZFS access. +- [ ] Use direct `node src/index.js` in service files instead of `npm start`. +- [ ] Make `SCHEDULES_PATH` explicitly configurable. +- [ ] Installer should warn on insecure agent exposure. -- Add a test runner for agent and management code. -- Test validators for instance names, snapshot IDs, and ambiguous snapshot prefixes. -- Test job locking, persistence, trimming, and restart behavior. -- Test backup pipeline failure handling with mocked command/process failures. -- Test restore staged-volume command sequencing with mocked ZFS/Incus/Restic commands. -- Test management job polling updates `job_history` correctly for success, failed, timeout, and missing-agent cases. -- Add CI for install, tests, frontend build, and syntax checks. +### 22. Data Integrity and Schema Cleanup -### Acceptance Criteria +Tasks: -- Pull requests run tests automatically. -- Simulated backup stream failures fail the job. -- Simulated restore failures leave the original ZVOL name restored in the command sequence. -- Agent restart behavior is covered by tests. +- [ ] Add audit-log integrity, such as append-only behavior and optional hash chain. +- [ ] Add `audit_events.details` size limit or JSON column handling. +- [ ] Replace ad-hoc `addColumnIfMissing` with schema versioning and migrations. +- [ ] Stream Restic `ls` instead of loading all output in RAM. -## 11. Harden root-running agent exposure +### 23. Code Cleanup -The agent runs with root-level host access because it needs Incus, ZFS, `/dev/zvol`, Restic, and device operations. A compromised agent is therefore a host-level incident. +Tasks: -### Goal +- [ ] Generalize `incus snapshot delete` retry instead of matching English stderr. +- [ ] Deduplicate `formatBytes`. +- [ ] Review `incus-backup-ui-plan.md` for stale paths, secrets, and obsolete implementation notes. +- [ ] Remove tracked `.DS_Store` files. +- [ ] Ensure `frontend/dist/` remains ignored and untracked. -Reduce the network and systemd blast radius of the root-running node-agent. +## Done -### Tasks +- [x] Default admin `admin/admin` fallback removed. +- [x] Agent `API_TOKEN` required and minimum length enforced. +- [x] HTTPS required for agent URLs in management, with dev-only insecure HTTP escape hatch. +- [x] Management can trust internal agent CA through `AGENT_CA_FILE`. +- [x] Agent supports direct HTTPS. +- [x] CORS allowlist added for management. +- [x] Auth cookie set to `SameSite=Strict`. +- [x] Login brute-force protection added. +- [x] Bearer token removed from frontend localStorage. +- [x] Settings secrets masked/write-only. +- [x] Agent token comparison made timing-safe. +- [x] Agent job persistence added. +- [x] Staged VM restore implemented. +- [x] Node-agent renamed from `backend/` to `agent/`. +- [x] Agent installer script added. +- [x] Backup UI ETA/rate/bytes display added. -- Require private management-to-agent connectivity in production documentation, such as NetBird or a dedicated management network. -- Prefer HTTPS agents with a private CA; document local HTTP only as development mode. -- Enable and document `ALLOWED_MANAGEMENT_IPS` for production. -- Add systemd hardening where compatible with Incus/ZFS access. -- Add explicit installer warnings when `HTTPS_ENABLED=false` and no `ALLOWED_MANAGEMENT_IPS` is configured. -- Consider replacing `npm start` in systemd with direct `node src/index.js`. +## OSS Release Requirements -### Acceptance Criteria - -- Production install docs do not expose the agent publicly by default. -- Installer warns on insecure network exposure. -- systemd unit has a documented minimum hardening baseline. - -## 12. Improve agent crash cleanup and resource recovery - -Agent jobs are persisted and active jobs are marked `failed` on restart, but a process crash can still leave host resources behind, such as temporary Incus snapshots, staged restore volumes, visible snapshot devices, or changed ZFS properties. - -### Goal - -Make agent startup detect and clean up known leftover resources from interrupted jobs where doing so is safe. - -### Tasks - -- Persist cleanup metadata for backup jobs: temporary Incus snapshot name, ZVOL, snapdev state, and backup type. -- Persist cleanup metadata for restore jobs: staged ZVOL, backup ZVOL, failed ZVOL, and swap state. -- On startup, scan failed active jobs and run safe cleanup actions. -- Log cleanup results into the persisted job logs. -- Avoid destructive cleanup when state is ambiguous; surface manual action instead. - -### Acceptance Criteria - -- Crashing during backup streaming does not leave `snapdev=visible` or temporary snapshots unnoticed. -- Crashing before restore swap removes staged restore volumes when safe. -- Crashing after restore swap does not automatically destroy rollback copies. -- Management can show cleanup-required states. - -## 13. Improve health diagnostics in the UI - -The agent health endpoint returns useful structured details, but the management UI currently summarizes this too aggressively. Operators need actionable health reasons without opening logs. - -### Goal - -Show per-node health detail in the UI with concrete failed checks and messages. - -### Tasks - -- Add a detailed health drawer or modal on the Nodes page. -- Show command, ZFS pool, `/dev/zvol`, Restic repository, and credential check results. -- Show last health timestamp and the management-side error if the agent is unreachable. -- Distinguish unreachable, unauthorized, TLS failure, degraded health, and healthy states. -- Avoid global UI timeout banners when only one enabled node is slow or unreachable. - -### Acceptance Criteria - -- A missing command is visible by name in the UI. -- Wrong Restic credentials are visible as a Restic health failure. -- TLS or connectivity failures are distinguishable from degraded agent health. +- [ ] Add tests. +- [ ] Add CI pipeline. +- [ ] Add `SECURITY.md`. +- [ ] Document threat model and trusted network assumptions. +- [ ] Add `LICENSE`. +- [ ] Add `CONTRIBUTING.md`. +- [ ] Add example `compose.yaml` or Ansible role for reproducible deployment.