merged issues and todo.
removed todo
This commit is contained in:
Philipp
@@ -1,228 +0,0 @@
|
||||
# Fixes & Hardening TODO
|
||||
|
||||
Kritisches Review-Backlog vor Produktiveinsatz und OSS-Release.
|
||||
Sortiert nach Risikoklasse. Datei-/Zeilenreferenzen beziehen sich auf den Stand bei Review-Erstellung.
|
||||
|
||||
---
|
||||
|
||||
## 🔴 Kritisch — vor Produktion blockierend
|
||||
|
||||
- [x] **1. Default-Admin `admin/admin` entfernen** ✅ erledigt
|
||||
- `management/src/db.js:83` — wirft jetzt Exception bei leerem `AUTH_PASSWORD`. Kein Silent-Fallback mehr.
|
||||
- README/`docs/deployment.md` entsprechend anpassen (noch offen).
|
||||
|
||||
- [x] **2. `API_TOKEN` im Node-Agent zur Pflicht machen** ✅ erledigt
|
||||
- `agent/src/config.js:44` — Start schlägt fehl wenn Token fehlt oder kürzer als 32 Zeichen.
|
||||
- `agent/src/index.js` — `if (!config.apiToken) next()` entfernt; Token-Prüfung immer aktiv.
|
||||
|
||||
- [x] **3. HTTPS zwischen Management und Agent erzwingen** ✅ erledigt
|
||||
- `management/src/routes/nodes.js:102-112` — `validateBaseUrl` erzwingt `https://`. Escape-Hatch `ALLOW_INSECURE_AGENT_HTTP=true` nur für Dev.
|
||||
- `management/src/agentClient.js` — eigene CA über `AGENT_CA_FILE` konfigurierbar.
|
||||
- Agent-Server unterstützt TLS via `HTTPS_ENABLED`, `TLS_CERT_FILE`, `TLS_KEY_FILE`.
|
||||
|
||||
- [x] **4. CSRF-Risiko durch reflektierendes CORS schließen** ✅ erledigt
|
||||
- `management/src/index.js:18-27` — Nur Origins aus `CORS_ORIGINS`-Env zugelassen.
|
||||
- `management/src/auth.js:57` — Cookie auf `SameSite=Strict` gesetzt.
|
||||
|
||||
- [x] **5. Brute-Force-Schutz beim Login** ✅ erledigt
|
||||
- `management/src/routes/auth.js:7-9` — In-Memory Rate-Limit: 5 Versuche / 15 min / IP+User.
|
||||
- Audit-Event `login_blocked` und `login_failed` implementiert.
|
||||
- Hinweis: Kein externer `express-rate-limit` nötig, eigenständige Implementierung ausreichend.
|
||||
|
||||
- [x] **6. Restore atomarisieren — kein direktes `dd` auf Produktiv-Volume** ✅ staged Restore umgesetzt
|
||||
- Datei: `agent/src/executor.js:92` (`streamResticDumpToDd`), `agent/src/routes/restore.js`
|
||||
- Umsetzung: Restore schreibt zuerst in ein temporäres ZFS-Volume, prüft die Größe vorab und tauscht danach per `zfs rename` gegen das Produktiv-Volume.
|
||||
- Das alte Volume bleibt als `*.pre-restore-*` Rollback-Kopie erhalten.
|
||||
- Bei Fehlern vor dem Swap wird nur das temporäre Volume entfernt; bei Fehlern nach dem Swap versucht der Agent den alten Volume-Namen wiederherzustellen.
|
||||
|
||||
- [ ] **7. Backup-Verifikation einbauen**
|
||||
- Datei: `agent/src/routes/backup.js` (VM-Pfad ~Z.71, Container-Pfad ~Z.137)
|
||||
- Aktuell: Bei Pipe-Fehler (`zfs send` !=0, `createReadStream`-Abbruch) committed restic ggf. einen unvollständigen Snapshot mit Status „success".
|
||||
- Fix:
|
||||
- Vor Start erwartete Größe ermitteln (`zfs get volsize` / `used`).
|
||||
- Nach Restic-Commit `restic stats <snapshotId>` vergleichen.
|
||||
- Bei Mismatch oder Pipe-Fehler: Restic-Snapshot per ID `forget`+`prune`en, Job auf `failed`.
|
||||
|
||||
- [x] **8. Bearer-Token aus Frontend-`localStorage` entfernen** ✅ erledigt
|
||||
- `frontend/src/api.js` — nur noch Cookie-Auth (`withCredentials: true`). `localStorage`-Token und `VITE_API_TOKEN` entfernt.
|
||||
|
||||
- [x] **45. Secrets in `GET /settings` maskieren (Write-only)** ✅ erledigt
|
||||
- Datei: `agent/src/config.js` (`readEnvSettings`/`writeEnvSettings`), `frontend/src/components/Settings.jsx`
|
||||
- Problem: `readEnvSettings` gab den Wert *aller* Felder zurück — inkl. `RESTIC_PASSWORD`, `AWS_SECRET_ACCESS_KEY`, `API_TOKEN`. Über den Management-Proxy konnte damit jeder eingeloggte UI-User das Restic-Verschlüsselungspasswort und die S3-Credentials enthüllen (= alle Backups entschlüsseln und löschen).
|
||||
- Fix: Secret-Felder liefern beim Lesen keinen Wert mehr, nur `hasValue: true|false`. Beim Schreiben gilt ein leeres Secret-Feld als „unverändert" (bestehender Wert bleibt erhalten). Frontend zeigt Platzhalter „gesetzt – leer lassen zum Beibehalten".
|
||||
|
||||
---
|
||||
|
||||
## 🟠 Hoch
|
||||
|
||||
- [~] **9. Session-Cookie `Secure`-Flag** ⚠️ teilweise erledigt
|
||||
- `management/src/auth.js:56` — `Secure`-Flag via `SESSION_COOKIE_SECURE` konfigurierbar, auto-aktiviert bei `NODE_ENV=production`.
|
||||
- Offen: `management/.env.example` hat `SESSION_COOKIE_SECURE=false` — in Deployment-Doku explizit als "in Produktion auf `true` setzen" dokumentieren.
|
||||
|
||||
- [ ] **10. Race beim Sichtbarmachen des Snapshot-Devices**
|
||||
- Datei: `agent/src/routes/backup.js:57`
|
||||
- Aktuell: 2 s `sleep` reicht nicht garantiert.
|
||||
- Fix: Polling-Schleife auf `fs.access(snapshotDevice)` mit Timeout; zusätzlich `udevadm trigger && udevadm settle`.
|
||||
|
||||
- [x] **11. Persistenter Job- und Lock-Store im Agent** ✅ erledigt
|
||||
- Datei: `agent/src/jobs.js`
|
||||
- Umsetzung: Agent speichert Jobs in SQLite (`AGENT_DATABASE_PATH`).
|
||||
- Beim Start werden `running`/`queued` Jobs als `failed` markiert, damit Management einen finalen Zustand pollen kann und VM-Locks nicht hängen bleiben.
|
||||
- Offen für später: ressourcenspezifische Cleanup-Recovery für abgebrochene Host-Operationen.
|
||||
|
||||
- [ ] **12. Toten `SESSION_SECRET` aufräumen**
|
||||
- Datei: `management/src/config.js:8`
|
||||
- Aktuell: `SESSION_SECRET` wird geladen, aber nirgendwo genutzt — toter Code.
|
||||
- Fix: Variable und zugehörigen `.env.example`-Eintrag entfernen, oder Session-IDs HMAC-signieren und Variable dann sinnvoll nutzen.
|
||||
|
||||
- [ ] **13. Snapshot-ID-Prefix-Matching eindeutig machen**
|
||||
- Datei: `agent/src/validators.js:66`
|
||||
- Aktuell: `startsWith` — bei Prefix-Kollision wird stillschweigend der erste Treffer genommen.
|
||||
- Fix: Bei >1 Treffer 409 zurückgeben; UI auf 12-Hex-Prefix umstellen.
|
||||
|
||||
- [ ] **14. RBAC einführen**
|
||||
- Aktuell: ein User, alle Rechte, kein Read-only.
|
||||
- Fix: Mindestens Rollen `admin` / `operator` / `viewer`. Restore nur für `admin`.
|
||||
|
||||
- [ ] **15. ENV-Escaping in `writeEnvSettings` verbessern**
|
||||
- Datei: `agent/src/config.js:105`
|
||||
- Aktuell: Nur `\` und `"` escaped; `$`, Backticks, Newlines nicht.
|
||||
- Fix: Eigener Serializer mit korrektem Escaping aller Sonderzeichen.
|
||||
|
||||
- [ ] **16. Settings-Endpunkt sperren bis `API_TOKEN` initial gesetzt ist**
|
||||
- Datei: `agent/src/routes/settings.js`
|
||||
- Hängt mit Fix #2 zusammen — nach #2 automatisch erfüllt, hier zur Sicherheit dokumentieren/testen.
|
||||
|
||||
- [x] **46. Agent-Token-Vergleich timing-safe machen** ✅ erledigt
|
||||
- Datei: `agent/src/index.js:28`
|
||||
- Problem: `header === \`Bearer ${config.apiToken}\`` — normaler String-Vergleich am root-Agent (das wertvollste Ziel), während das Login-Passwort bereits `timingSafeEqual` nutzte.
|
||||
- Fix: Vergleich über `crypto.timingSafeEqual` mit Längen-Guard. Leerer/fehlender konfigurierter Token verweigert weiterhin (`config.apiToken &&`).
|
||||
|
||||
- [ ] **47. Pre-Restore-Volumes aufräumen (Disk-Space-GC)**
|
||||
- Datei: `agent/src/routes/restore.js`
|
||||
- Aktuell: Jeder erfolgreiche Restore behält das alte Volume als `*.pre-restore-*` Rollback-Kopie — es gibt aber keinen Cleanup. Jeder Restore verdoppelt den Plattenbedarf der VM dauerhaft; bei mehreren Restores läuft der ZFS-Pool voll.
|
||||
- Fix: Aufbewahrungsregel (z.B. „keep last N pre-restore/failed-restore Volumes pro VM") oder expliziter Cleanup-Schritt/UI-Aktion. Mindestens im Health/Operations-View sichtbar machen.
|
||||
|
||||
- [ ] **48. Agent-Tokens im Management-SQLite nicht im Klartext speichern**
|
||||
- Datei: `management/src/store.js` (`nodes.token`), `management/src/db.js`
|
||||
- Aktuell: `nodes.token` liegt im Klartext in der Management-DB. DB-Diebstahl = alle Agent-Tokens = root auf allen Hosts.
|
||||
- Fix: Verschlüsselung at-rest mit einem Management-Key (z.B. aus `SESSION_SECRET`/dediziertem Key abgeleitet), oder zumindest DB-Dateipermissions/Disk-Encryption-Anforderung im Deployment-Doc verbindlich dokumentieren.
|
||||
|
||||
---
|
||||
|
||||
## 🟡 Mittel
|
||||
|
||||
- [ ] **17. `Math.random()` durch `crypto.randomBytes` ersetzen**
|
||||
- Datei: `management/src/db.js:93` (`cryptoId`).
|
||||
- Noch vorhanden: `Math.random().toString(16)` für DB-interne User-IDs — kein akutes Sicherheitsproblem, aber nicht kryptografisch sicher.
|
||||
|
||||
- [ ] **18. Systemd-Hardening für den Agent**
|
||||
- Datei: `deploy/systemd/incus-backup-agent.service`
|
||||
- Hinzufügen: `NoNewPrivileges=true`, `ProtectSystem=strict`, `ProtectHome=true`, `PrivateTmp=true`, `ReadWritePaths=/dev/zvol /var/lib/incus /opt/incus-backup-ui/agent`, `CapabilityBoundingSet=...`, eingeschränkte `AmbientCapabilities`.
|
||||
|
||||
- [ ] **19. `npm start` durch direkten `node`-Aufruf ersetzen**
|
||||
- Datei: beide `deploy/systemd/*.service`
|
||||
- `ExecStart=/usr/bin/node src/index.js` — kein npm-Wrapper-Prozess, kein PATH-Risiko.
|
||||
|
||||
- [ ] **20. CORS am Agent entfernen**
|
||||
- Datei: `agent/src/index.js:16`
|
||||
- Agent wird nie aus dem Browser angesprochen → Angriffsfläche raus.
|
||||
|
||||
- [ ] **21. `schedules.json`-Pfad explizit konfigurierbar**
|
||||
- Datei: `agent/src/scheduler.js:6`
|
||||
- Aktuell: `process.cwd()`-abhängig.
|
||||
- Fix: über Env (`SCHEDULES_PATH`) absolut konfigurieren, Default unter `/var/lib/incus-backup-agent/`.
|
||||
|
||||
- [ ] **22. Audit-Log-Integrität**
|
||||
- Datei: `management/src/store.js:71`
|
||||
- Append-only Constraint + Hash-Chain (jeder Event hat `prev_hash`). Optional Off-Site-Versand (Webhook).
|
||||
|
||||
- [ ] **23. Passwort-Hashing auf argon2id umstellen**
|
||||
- Datei: `management/src/crypto.js`
|
||||
- Aktuell: `scryptSync` mit Node-Defaults, ohne explizite Parameter.
|
||||
- Fix: `@node-rs/argon2` oder explizite scrypt-Parameter dokumentieren + Migration-Pfad.
|
||||
|
||||
- [ ] **24. Rate-Limit / Quoten für Backups**
|
||||
- Pro VM und global (z.B. max N parallele Streams nach S3). Disk-Space-/Quota-Check vor Start.
|
||||
|
||||
- [ ] **25. `incus snapshot delete` Retry verallgemeinern**
|
||||
- Datei: `agent/src/routes/backup.js:303`
|
||||
- Aktuell: Substring-Match auf englische Stderr — bricht bei lokalisierten Builds.
|
||||
- Fix: Generischer Retry (n Versuche, Backoff) bei nicht-0 Exit-Code.
|
||||
|
||||
- [ ] **26. Restic-`ls` streamen statt vollständig in RAM laden**
|
||||
- Datei: `agent/src/routes/snapshots.js:19`
|
||||
- Für Container-Backups mit vielen Files relevant.
|
||||
|
||||
- [ ] **27. Container-Restore implementieren oder Container-Backup deaktivieren**
|
||||
- Datei: `agent/src/routes/restore.js:21`
|
||||
- Aktuell: 501. Backups laufen, aber nicht wiederherstellbar = Backup-Theater.
|
||||
- Fix: Sicheren `zfs receive`-Workflow umsetzen oder Container-Backup im UI/API ausschalten bis fertig.
|
||||
|
||||
- [ ] **28. Pre-Backup VM-Zustand prüfen**
|
||||
- Live-Migration, laufende interne Snapshots, fehlende Berechtigungen → klare Fehler statt halb durchgeführter Pipeline.
|
||||
|
||||
- [ ] **49. Abgelaufene Sessions serverseitig löschen + Rotation**
|
||||
- Datei: `management/src/store.js:12` (`getUserBySession`), `createSession`
|
||||
- Aktuell: Abgelaufene Sessions werden beim Lesen nur gefiltert, nie aus der DB entfernt → unbegrenztes Tabellenwachstum. Außerdem keine Session-Rotation nach erfolgreichem Login.
|
||||
- Fix: Periodischer Cleanup (`DELETE FROM sessions WHERE expires_at <= now`) und neue Session-ID nach Login ausstellen.
|
||||
|
||||
---
|
||||
|
||||
## 🟢 Niedrig / Aufräumen
|
||||
|
||||
- [ ] **29. `.DS_Store` aus Repo entfernen und in `.gitignore` aufnehmen**
|
||||
- [ ] **30. `frontend/dist/` ist eingecheckt** — ignorieren und löschen.
|
||||
- [ ] **31. `incus-backup-ui-plan.md` auf Secrets/Bucket-Namen prüfen** bevor OSS.
|
||||
- [ ] **32. Container-Limit (Restore fehlt) prominent in README dokumentieren.**
|
||||
- [ ] **33. Scheduler-Jitter einbauen** (`management/src/scheduler.js`, `agent/src/scheduler.js`) — sonst belasten viele Nodes synchron S3.
|
||||
- [ ] **34. `formatBytes` deduplizieren** (`agent/src/routes/backup.js`, `restore.js`) → `executor.js` oder `utils.js`.
|
||||
- [ ] **35. `.env`-Dateipermissions dokumentieren** — `chmod 600` im Deployment-Doc verlangen.
|
||||
- [ ] **36. Schema-Versionierung statt `addColumnIfMissing`** — z.B. `schema_version`-Tabelle + nummerierte Migrationen.
|
||||
- [ ] **37. `audit_events.details` Größenlimit** oder JSON-Spalte (SQLite hat JSON1).
|
||||
|
||||
---
|
||||
|
||||
## 📦 OSS-Release-Voraussetzungen
|
||||
|
||||
- [ ] **38. Tests einführen** — aktuell keinerlei Tests im Repo. Mindestens:
|
||||
- `validators.js` (vmName-/snapshot-Regex, Pfad-Validation).
|
||||
- `crypto.js` (hash/verify Roundtrip, Timing-Safe).
|
||||
- `jobs.js` (Locking, Trimming).
|
||||
- Pipeline-Smoke-Tests mit gemockten Befehlen.
|
||||
- [ ] **39. CI-Pipeline** (GitHub Actions): Lint, Tests, `npm audit`, Lockfile-Check.
|
||||
- [ ] **40. `SECURITY.md`** mit Disclosure-Adresse + GPG-Key.
|
||||
- [ ] **41. Threat-Model dokumentieren** — explizit machen, was außerhalb des Schutzbereichs liegt (root-Agent, vertrauenswürdiges Netz, etc.).
|
||||
- [ ] **42. `LICENSE`-Datei ergänzen** (z.B. MIT/Apache-2.0).
|
||||
- [ ] **43. `CONTRIBUTING.md`** + Code-of-Conduct.
|
||||
- [ ] **44. Beispiel-`compose.yaml` oder Ansible-Rolle** für reproduzierbares Deployment.
|
||||
|
||||
---
|
||||
|
||||
## Status-Übersicht kritische Punkte (Stand 2026-06-04)
|
||||
|
||||
| # | Titel | Status |
|
||||
|---|-------|--------|
|
||||
| 1 | Default-Admin `admin/admin` entfernen | ✅ erledigt |
|
||||
| 2 | `API_TOKEN` Pflicht | ✅ erledigt |
|
||||
| 3 | HTTPS Management↔Agent | ✅ erledigt |
|
||||
| 4 | CORS-Whitelist + SameSite=Strict | ✅ erledigt |
|
||||
| 5 | Brute-Force-Schutz Login | ✅ erledigt |
|
||||
| 6 | Restore atomarisieren | ✅ staged Restore umgesetzt |
|
||||
| 7 | Backup-Verifikation | ⬜ offen |
|
||||
| 8 | Bearer-Token aus localStorage | ✅ erledigt |
|
||||
| 9 | Session-Cookie `Secure`-Flag | ⚠️ teilweise |
|
||||
| 45 | Secrets in `GET /settings` maskieren | ✅ erledigt |
|
||||
| 46 | Agent-Token-Vergleich timing-safe | ✅ erledigt |
|
||||
| 47 | Pre-Restore-Volume-GC | ⬜ offen |
|
||||
| 48 | Agent-Tokens in DB verschlüsseln | ⬜ offen |
|
||||
| 49 | Session-Cleanup serverseitig | ⬜ offen |
|
||||
|
||||
## Nächste Prioritäten
|
||||
|
||||
1. **#7** Backup-Verifikation weiter härten — Pipeline-/Stream-Fehler testen und vollständiger absichern.
|
||||
2. **#47** Pre-Restore-Volume-GC — sonst läuft der ZFS-Pool bei wiederholten Restores voll.
|
||||
3. **#9** `SESSION_COOKIE_SECURE=true` in Deployment-Doku festschreiben.
|
||||
4. **#12** Toten `SESSION_SECRET` entfernen.
|
||||
5. **#15** ENV-Escaping vervollständigen (`$`, Backticks, Newlines).
|
||||
6. **#11 Folgearbeit** ressourcenspezifische Cleanup-Recovery nach Agent-Crash definieren.
|
||||
+406
-261
@@ -1,103 +1,57 @@
|
||||
# Issue Backlog
|
||||
|
||||
## Status snapshot
|
||||
|
||||
Last reviewed against code: 2026-06-04.
|
||||
|
||||
Several earlier backlog items have already landed in the codebase:
|
||||
This is the single source of truth for product work, hardening, release tasks, and production blockers. The previous `docs/fixes-todo.md` list has been merged into this file.
|
||||
|
||||
## Current Status
|
||||
|
||||
Implemented and verified in code:
|
||||
|
||||
- Management persists accepted agent jobs in `job_history`.
|
||||
- Management polls agent jobs and updates final status, error, finish time, duration, current step, and parsed backup snapshot ID.
|
||||
- Operations page shows job history and audit log.
|
||||
- Node-agent health checks cover required commands, ZFS pool state, `/dev/zvol`, and Restic repository access.
|
||||
- Backup jobs verify the stored Restic file size and remove failed snapshots on verification errors.
|
||||
- Backup jobs parse the Restic snapshot ID, verify stored file size, and remove failed snapshots on verification errors.
|
||||
- VM restore no longer writes directly to the production ZVOL. It writes to a staged ZVOL first, then swaps volumes with `zfs rename`.
|
||||
- Agent jobs are persisted in SQLite. Active jobs are marked `failed` after an agent restart because subprocesses cannot survive restart.
|
||||
- Auth hardening landed: no default admin password, required agent token, CORS allowlist, `SameSite=Strict`, login brute-force protection, no frontend localStorage bearer token.
|
||||
- Settings secrets are write-only/masked in the UI and API.
|
||||
- Agent bearer-token comparison is timing-safe.
|
||||
- Agent directory was renamed from `backend/` to `agent/`.
|
||||
|
||||
Known caveat: agent-side jobs are persisted, but subprocesses cannot survive an agent restart. Active jobs are marked `failed` on startup; deeper cleanup recovery for partially changed host resources is still future work.
|
||||
Known caveat: active agent jobs are persisted, but deeper cleanup recovery for partially changed host resources is still future work.
|
||||
|
||||
Current pre-production priorities:
|
||||
## Pre-Production Priorities
|
||||
|
||||
1. Harden backup verification and add tests for stream/pipeline failure cases.
|
||||
2. Validate the staged restore workflow on a disposable Incus VM, including rollback scenarios.
|
||||
3. Improve agent crash cleanup for partially changed ZFS/Incus resources.
|
||||
4. Surface detailed node health diagnostics in the UI.
|
||||
5. Keep the root-running agent tightly network-restricted.
|
||||
3. Add cleanup and visibility for pre-restore/failed-restore ZVOLs.
|
||||
4. Improve agent crash cleanup for partially changed ZFS/Incus resources.
|
||||
5. Surface detailed node health diagnostics in the UI.
|
||||
6. Keep the root-running agent tightly network-restricted.
|
||||
7. Add automated tests and CI.
|
||||
|
||||
## 1. Persist final agent job status in management
|
||||
## P0 - Production Blockers
|
||||
|
||||
Status: mostly done.
|
||||
|
||||
Management records when a backup or restore was accepted by an agent and now persists the final agent job result when the node-agent remains reachable long enough to be polled.
|
||||
|
||||
### Goal
|
||||
|
||||
Persist reliable end-to-end job status in the management database.
|
||||
|
||||
### Tasks
|
||||
|
||||
- [x] Add polling for accepted agent jobs from management.
|
||||
- [x] Store final `success` or `failed` status in `job_history`.
|
||||
- [x] Store duration, finished timestamp, error message, and current step.
|
||||
- [ ] Store agent job logs summary.
|
||||
- [x] Store created Restic snapshot ID for successful backup jobs when available.
|
||||
- [x] Surface final status in the Operations page.
|
||||
- [x] Persist recent agent-side jobs.
|
||||
- [x] Mark active agent jobs as `failed` after an agent restart so management can poll a final state.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- [x] A backup started through management eventually shows `success` or `failed` while the agent remains reachable.
|
||||
- [x] A restore started through management eventually shows `success` or `failed` while the agent remains reachable.
|
||||
- [x] Management restart does not lose already persisted history.
|
||||
- [x] Agent restart does not lose the active job record; active work is marked `failed` because the subprocess cannot survive restart.
|
||||
|
||||
## 2. Expand node-agent health checks
|
||||
|
||||
Status: mostly done.
|
||||
|
||||
The health endpoint now provides deeper operational checks for backup readiness.
|
||||
|
||||
### Goal
|
||||
|
||||
Make `/api/health` useful for diagnosing whether a node can actually run backup and restore operations.
|
||||
|
||||
### Tasks
|
||||
|
||||
- [x] Check that required commands exist: `incus`, `zfs`, `zpool`, `restic`, `udevadm`, `dd`.
|
||||
- [x] Check that configured ZFS pool exists.
|
||||
- [x] Check that `/dev/zvol` is accessible.
|
||||
- [x] Check Restic repository access.
|
||||
- [x] Check S3/Restic credentials by running a safe Restic command.
|
||||
- [x] Include ZFS pool capacity and free space.
|
||||
- [x] Return structured check names and messages.
|
||||
- [ ] Surface detailed per-node health output in the management UI, not just a compact aggregate.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- [~] Management UI shows degraded node health with actionable check names. Detailed output exists through node health endpoints; UI can still be improved.
|
||||
- [x] A missing command, wrong pool, or wrong Restic credentials is visible in health output.
|
||||
|
||||
## 3. Harden backup verification
|
||||
### 1. Harden Backup Verification
|
||||
|
||||
Status: partially done.
|
||||
|
||||
Backup jobs now verify the backed-up Restic file size after `restic backup` and attempt to remove failed snapshots. This reduces the risk of accepting a truncated backup, but the implementation still needs stronger stream failure handling and broader verification semantics.
|
||||
Goal: ensure a backup is marked `success` only when the expected source data was fully stored and verified.
|
||||
|
||||
### Goal
|
||||
|
||||
Ensure a backup is marked `success` only when the expected source data was fully stored and verified.
|
||||
|
||||
### Tasks
|
||||
Tasks:
|
||||
|
||||
- [x] Parse the created Restic snapshot ID after backup.
|
||||
- [x] Verify stored Restic file size against streamed source bytes or expected ZFS size.
|
||||
- [x] Remove failed Restic snapshots with `forget` and `prune` on verification errors.
|
||||
- [ ] Make stream/pipeline failure handling explicit for all sources.
|
||||
- [ ] Avoid marking success if source stream closes early but Restic exits successfully.
|
||||
- [ ] Add a post-backup Restic integrity check strategy, such as targeted `restic check`/`restic stats` usage that is safe for large repositories.
|
||||
- [ ] Make stream/pipeline failure handling explicit for VM and container sources.
|
||||
- [ ] Avoid marking success if the source stream closes early but Restic exits successfully.
|
||||
- [ ] Add a post-backup Restic integrity-check strategy that is safe for large repositories.
|
||||
- [ ] Define an optional restore-probe workflow for critical VMs.
|
||||
- [ ] Add tests with mocked command failures and short reads.
|
||||
- [ ] Add tests with mocked command failures, pipe failures, and short reads.
|
||||
|
||||
### Acceptance Criteria
|
||||
Acceptance criteria:
|
||||
|
||||
- [x] Successful backup jobs include a verifiable Restic snapshot ID in management history.
|
||||
- [x] Size mismatches fail the job.
|
||||
@@ -105,245 +59,436 @@ Ensure a backup is marked `success` only when the expected source data was fully
|
||||
- [ ] A backup can be independently verified without trusting only the successful process exit.
|
||||
- [ ] Verification behavior is covered by automated tests.
|
||||
|
||||
## 4. Add per-VM backup policy
|
||||
### 2. Validate Staged Restore Workflow
|
||||
|
||||
Retention and schedule behavior is currently broad. Per-VM policies would make production usage more flexible.
|
||||
Status: implemented, not proven on real Incus/ZFS hardware in this repo session.
|
||||
|
||||
Goal: prove restore safety before production use.
|
||||
|
||||
Tasks:
|
||||
|
||||
- [x] Require explicit typed confirmation including VM name.
|
||||
- [x] Validate snapshot ownership before restore.
|
||||
- [x] Replace direct `dd` to production ZVOL with a staged restore workflow.
|
||||
- [x] Log restore intent in audit log before dispatch.
|
||||
- [ ] Add restore preflight endpoint.
|
||||
- [ ] Validate node health before restore.
|
||||
- [ ] Show selected snapshot metadata before restore.
|
||||
- [ ] Show target VM status and disk size before restore.
|
||||
- [ ] Optionally offer “create backup before restore” when the VM is accessible.
|
||||
- [ ] Test staged restore on a disposable VM: backup, restore, boot, and confirm data.
|
||||
- [ ] Test failed staged restore paths: Restic dump failure before swap, rename failure after old volume rename, and VM start failure.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- [x] Restore stream failures happen on the staged volume, not the production disk.
|
||||
- [ ] UI displays a restore plan before final confirmation.
|
||||
- [ ] Restore is blocked when required preflight checks fail.
|
||||
- [ ] Staged restore behavior is validated on a real Incus/ZFS test node.
|
||||
|
||||
### 3. Add Pre-Restore Volume Cleanup
|
||||
|
||||
Status: open.
|
||||
|
||||
Current behavior: every successful VM restore keeps the old volume as `*.pre-restore-*`. This is good for rollback, but without retention or cleanup it can fill the ZFS pool.
|
||||
|
||||
Tasks:
|
||||
|
||||
- [ ] Track pre-restore and failed-restore volumes created by the agent.
|
||||
- [ ] Add a retention policy, for example keep the last N rollback volumes per VM.
|
||||
- [ ] Surface retained rollback volumes in Operations or VM detail.
|
||||
- [ ] Add explicit cleanup action with confirmation.
|
||||
- [ ] Add health warning when retained restore volumes consume significant pool space.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- [ ] Repeated restores cannot silently consume unbounded pool space.
|
||||
- [ ] Operators can see and clean retained restore volumes.
|
||||
|
||||
### 4. Harden Root-Running Agent Exposure
|
||||
|
||||
Status: partially done.
|
||||
|
||||
The agent runs with root-level host access because it needs Incus, ZFS, `/dev/zvol`, Restic, and device operations. A compromised agent is a host-level incident.
|
||||
|
||||
Tasks:
|
||||
|
||||
- [x] Require a non-empty `API_TOKEN` of at least 32 characters.
|
||||
- [x] Support `ALLOWED_MANAGEMENT_IPS`.
|
||||
- [x] Support HTTPS directly on the agent.
|
||||
- [x] Support private CA validation from management via `AGENT_CA_FILE`.
|
||||
- [ ] Require private management-to-agent connectivity in production docs, such as NetBird or a dedicated management network.
|
||||
- [ ] Prefer HTTPS agents with a private CA; document local HTTP only as development mode.
|
||||
- [ ] Add installer warning when `HTTPS_ENABLED=false` and no `ALLOWED_MANAGEMENT_IPS` is configured.
|
||||
- [ ] Add systemd hardening where compatible with Incus/ZFS access.
|
||||
- [ ] Replace `npm start` in systemd with direct `node src/index.js`.
|
||||
- [ ] Remove CORS from the agent; browsers should never call it directly.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- [ ] Production install docs do not expose the agent publicly by default.
|
||||
- [ ] Installer warns on insecure network exposure.
|
||||
- [ ] systemd unit has a documented minimum hardening baseline.
|
||||
|
||||
### 5. Encrypt or Protect Agent Tokens in Management
|
||||
|
||||
Status: open.
|
||||
|
||||
Current behavior: `nodes.token` is stored in plaintext in the management SQLite database. DB theft means all agent tokens are exposed.
|
||||
|
||||
Tasks:
|
||||
|
||||
- [ ] Decide between at-rest encryption with a dedicated management key or strict documented file/disk protection.
|
||||
- [ ] If encrypting, add key configuration and migration for existing tokens.
|
||||
- [ ] Ensure public node API responses never include tokens.
|
||||
- [ ] Document database file permissions and disk-encryption expectations.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- [ ] Stealing the management database alone is not enough to read every agent token, or the residual risk is explicitly documented and operationally mitigated.
|
||||
|
||||
## P1 - High Priority
|
||||
|
||||
### 6. Add Automated Tests and CI
|
||||
|
||||
Status: open.
|
||||
|
||||
Tasks:
|
||||
|
||||
- [ ] Add a test runner for agent and management code.
|
||||
- [ ] Test validators for instance names, snapshot IDs, and ambiguous snapshot prefixes.
|
||||
- [ ] Test job locking, persistence, trimming, and restart behavior.
|
||||
- [ ] Test backup pipeline failure handling with mocked command/process failures.
|
||||
- [ ] Test restore staged-volume command sequencing with mocked ZFS/Incus/Restic commands.
|
||||
- [ ] Test management job polling updates `job_history` for success, failed, timeout, and missing-agent cases.
|
||||
- [ ] Add CI for install, tests, frontend build, and syntax checks.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- [ ] Pull requests run tests automatically.
|
||||
- [ ] Simulated backup stream failures fail the job.
|
||||
- [ ] Simulated restore failures leave the original ZVOL name restored in the command sequence.
|
||||
- [ ] Agent restart behavior is covered by tests.
|
||||
|
||||
### 7. Improve Agent Crash Cleanup and Resource Recovery
|
||||
|
||||
Status: open.
|
||||
|
||||
Agent jobs are persisted and active jobs are marked `failed` on restart, but a process crash can still leave host resources behind.
|
||||
|
||||
Tasks:
|
||||
|
||||
- [ ] Persist cleanup metadata for backup jobs: temporary Incus snapshot name, ZVOL, snapdev state, and backup type.
|
||||
- [ ] Persist cleanup metadata for restore jobs: staged ZVOL, backup ZVOL, failed ZVOL, and swap state.
|
||||
- [ ] On startup, scan failed active jobs and run safe cleanup actions.
|
||||
- [ ] Log cleanup results into the persisted job logs.
|
||||
- [ ] Avoid destructive cleanup when state is ambiguous; surface manual action instead.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- [ ] Crashing during backup streaming does not leave `snapdev=visible` or temporary snapshots unnoticed.
|
||||
- [ ] Crashing before restore swap removes staged restore volumes when safe.
|
||||
- [ ] Crashing after restore swap does not automatically destroy rollback copies.
|
||||
- [ ] Management can show cleanup-required states.
|
||||
|
||||
### 8. Improve Health Diagnostics in the UI
|
||||
|
||||
Status: partially done on API, open in UI.
|
||||
|
||||
Tasks:
|
||||
|
||||
- [x] Agent health checks commands, ZFS pool, `/dev/zvol`, and Restic repository.
|
||||
- [x] Management records compact node health state.
|
||||
- [ ] Add a detailed health drawer or modal on the Nodes page.
|
||||
- [ ] Show command, ZFS pool, `/dev/zvol`, Restic repository, and credential check results.
|
||||
- [ ] Show last health timestamp and the management-side error if the agent is unreachable.
|
||||
- [ ] Distinguish unreachable, unauthorized, TLS failure, degraded health, and healthy states.
|
||||
- [ ] Avoid global UI timeout banners when only one enabled node is slow or unreachable.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- [ ] A missing command is visible by name in the UI.
|
||||
- [ ] Wrong Restic credentials are visible as a Restic health failure.
|
||||
- [ ] TLS or connectivity failures are distinguishable from degraded agent health.
|
||||
|
||||
### 9. Fix Snapshot Device Visibility Race
|
||||
|
||||
Status: open.
|
||||
|
||||
Current behavior: backup waits a fixed 2 seconds after setting `snapdev=visible`.
|
||||
|
||||
Tasks:
|
||||
|
||||
- [ ] Replace fixed sleep with polling `fs.access(snapshotDevice)` and timeout.
|
||||
- [ ] Run `udevadm trigger` and `udevadm settle`.
|
||||
- [ ] Surface timeout as a clear backup failure.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- [ ] Slow device-node creation does not fail randomly.
|
||||
|
||||
### 10. Make Snapshot-ID Prefix Matching Unambiguous
|
||||
|
||||
Status: open.
|
||||
|
||||
Current behavior: prefix matching uses `startsWith`; collisions choose the first match.
|
||||
|
||||
Tasks:
|
||||
|
||||
- [ ] Return conflict when a snapshot prefix matches more than one snapshot.
|
||||
- [ ] Prefer 12+ hex characters in the UI.
|
||||
- [ ] Add validator tests.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- [ ] Ambiguous snapshot prefixes cannot restore the wrong snapshot.
|
||||
|
||||
### 11. Settings and Environment Hardening
|
||||
|
||||
Status: partially done.
|
||||
|
||||
Tasks:
|
||||
|
||||
- [x] Mask secret values in `GET /settings`.
|
||||
- [x] Treat empty secret field as “keep existing value”.
|
||||
- [ ] Improve `.env` escaping for `$`, backticks, and newlines.
|
||||
- [ ] Add tests for `readEnvSettings` and `writeEnvSettings`.
|
||||
- [ ] Document `.env` file permissions and enforce `chmod 600` in installer/deployment.
|
||||
- [ ] Keep or remove `SESSION_SECRET` deliberately: either use it for signing/encryption or remove dead config.
|
||||
- [ ] Set and document `SESSION_COOKIE_SECURE=true` for production.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- [ ] Secret settings cannot be exfiltrated through the UI/API.
|
||||
- [ ] Writing `.env` cannot create shell-expansion surprises.
|
||||
|
||||
### 12. Add RBAC
|
||||
|
||||
Status: open.
|
||||
|
||||
Current behavior: one user model, effectively all permissions.
|
||||
|
||||
Tasks:
|
||||
|
||||
- [ ] Add roles: `admin`, `operator`, `viewer`.
|
||||
- [ ] Restrict restore and node settings to `admin`.
|
||||
- [ ] Allow read-only dashboard access for `viewer`.
|
||||
- [ ] Audit role changes.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- [ ] Restore is admin-only.
|
||||
- [ ] Viewer cannot trigger backup, restore, node edits, or settings writes.
|
||||
|
||||
### 13. Session and Auth Cleanup
|
||||
|
||||
Status: partially done.
|
||||
|
||||
Tasks:
|
||||
|
||||
- [x] Login brute-force protection.
|
||||
- [x] `SameSite=Strict`.
|
||||
- [ ] Server-side cleanup of expired sessions.
|
||||
- [ ] Session rotation after successful login.
|
||||
- [ ] Replace `Math.random()` in management user ID creation with `crypto.randomBytes`.
|
||||
- [ ] Consider argon2id or explicit scrypt parameters and migration path.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- [ ] Expired sessions do not accumulate unbounded in SQLite.
|
||||
- [ ] Session IDs are rotated after login.
|
||||
|
||||
### 14. Container Restore Decision
|
||||
|
||||
Status: open.
|
||||
|
||||
Current behavior: container backups can be created, but restore returns `501`.
|
||||
|
||||
Tasks:
|
||||
|
||||
- [ ] Implement safe container restore with `zfs receive`, or disable container backups in UI/API until restore exists.
|
||||
- [ ] Document the limitation prominently.
|
||||
- [ ] Add tests for unsupported restore behavior.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- [ ] Users cannot mistake an unrestorable container backup for complete protection.
|
||||
|
||||
## P2 - Product and Operations
|
||||
|
||||
### 15. Add Per-VM Backup Policy
|
||||
|
||||
Status: partially done.
|
||||
|
||||
Current state: schedules can be enabled per VM with interval and time of day. Retention is still global per agent.
|
||||
|
||||
### Goal
|
||||
Tasks:
|
||||
|
||||
Allow each VM to define its own backup policy.
|
||||
|
||||
### Tasks
|
||||
|
||||
- Add management database table for VM backup policies.
|
||||
- Support per-VM retention values: hourly, daily, weekly, monthly.
|
||||
- [ ] Add management database table for VM backup policies.
|
||||
- [ ] Support per-VM retention values: hourly, daily, weekly, monthly.
|
||||
- [x] Support per-VM schedule enablement, interval, and time.
|
||||
- Add optional policy flag: backup only when VM is running.
|
||||
- Update Scheduler UI to edit policies per node and VM.
|
||||
- Send policy retention to the agent backup request or apply it in management scheduling.
|
||||
- [ ] Add optional policy flag: backup only when VM is running.
|
||||
- [ ] Update Scheduler UI to edit policies per node and VM.
|
||||
- [ ] Send policy retention to the agent backup request or apply it in management scheduling.
|
||||
|
||||
### Acceptance Criteria
|
||||
Acceptance criteria:
|
||||
|
||||
- Two VMs on the same node can have different schedules.
|
||||
- Two VMs on the same node can have different retention policies.
|
||||
- Disabled policies do not trigger backups.
|
||||
- [ ] Two VMs on the same node can have different schedules.
|
||||
- [ ] Two VMs on the same node can have different retention policies.
|
||||
- [ ] Disabled policies do not trigger backups.
|
||||
|
||||
## 5. Improve restore safety workflow
|
||||
### 16. Add Failure Notifications
|
||||
|
||||
Restore is destructive and should be guarded with a clearer preflight and confirmation flow.
|
||||
Status: open.
|
||||
|
||||
Current state: VM restore requires typed confirmation, validates the snapshot against the VM, checks source/target size, writes the Restic dump to a staged ZFS volume, stops the VM, and swaps the staged volume into place with `zfs rename`. The previous production volume is kept as a rollback copy.
|
||||
Tasks:
|
||||
|
||||
### Goal
|
||||
- [ ] Add notification settings in management.
|
||||
- [ ] Support webhook notifications first.
|
||||
- [ ] Include node, VM, job type, error, and timestamp.
|
||||
- [ ] Trigger notifications for failed scheduled backups.
|
||||
- [ ] Trigger notifications for failed manual backups/restores.
|
||||
- [ ] Add a “send test notification” action.
|
||||
|
||||
Reduce the risk of accidental or unsafe restores.
|
||||
Acceptance criteria:
|
||||
|
||||
### Tasks
|
||||
- [ ] A failed scheduled backup sends one notification.
|
||||
- [ ] A test notification can be triggered from the UI.
|
||||
- [ ] Notification failures are visible in Operations or audit logs.
|
||||
|
||||
- Add restore preflight endpoint.
|
||||
- Validate node health before restore.
|
||||
- Show selected snapshot metadata before restore.
|
||||
- Show target VM status and disk size before restore.
|
||||
- [x] Require explicit typed confirmation including VM name.
|
||||
- Optionally offer “create backup before restore” when the VM is accessible.
|
||||
- [x] Log restore intent in audit log before dispatch.
|
||||
- [x] Replace direct `dd` to production ZVOL with a staged restore workflow.
|
||||
- [ ] Test staged restore on a disposable VM: backup, restore, boot, and confirm data.
|
||||
- [ ] Test failed staged restore paths: Restic dump failure before swap, rename failure after old volume rename, and VM start failure.
|
||||
- [ ] Define a safe container restore workflow with `zfs receive` or disable container restore surfaces entirely.
|
||||
### 17. Implement Real Snapshot File Browsing
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- UI displays a restore plan before the final confirmation.
|
||||
- Restore is blocked when required preflight checks fail.
|
||||
- Audit log records restore attempts and results.
|
||||
- [x] Restore stream failures happen on the staged volume, not the production disk.
|
||||
- [ ] Staged restore behavior is validated on a real Incus/ZFS test node.
|
||||
|
||||
## 6. Add failure notifications
|
||||
|
||||
Operators need to know when scheduled backups or restores fail.
|
||||
|
||||
### Goal
|
||||
|
||||
Send notifications for failed or degraded operations.
|
||||
|
||||
### Tasks
|
||||
|
||||
- Add notification settings in management.
|
||||
- Support webhook notifications first.
|
||||
- Include node, VM, job type, error, and timestamp.
|
||||
- Trigger notifications for failed scheduled backups.
|
||||
- Trigger notifications for failed manual backups/restores.
|
||||
- Add a “send test notification” action.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- A failed scheduled backup sends one notification.
|
||||
- A test notification can be triggered from the UI.
|
||||
- Notification failures are visible in Operations or audit logs.
|
||||
|
||||
## 7. Implement real snapshot file browsing
|
||||
Status: open.
|
||||
|
||||
Current snapshot browsing shows Restic contents, which for block-level backups is usually only `/vm.raw`.
|
||||
|
||||
### Goal
|
||||
Tasks:
|
||||
|
||||
Allow browsing files inside a backed-up VM disk image.
|
||||
- [ ] Design a safe read-only raw image inspection workflow.
|
||||
- [ ] Restore or mount raw image read-only in a temporary workspace.
|
||||
- [ ] Detect partitions and filesystems.
|
||||
- [ ] Browse directories through management UI.
|
||||
- [ ] Allow downloading a single file.
|
||||
- [ ] Ensure cleanup of mounts and temporary files.
|
||||
|
||||
### Tasks
|
||||
Acceptance criteria:
|
||||
|
||||
- Design a safe read-only raw image inspection workflow.
|
||||
- Restore or mount raw image read-only in a temporary workspace.
|
||||
- Detect partitions and filesystems.
|
||||
- Browse directories through management UI.
|
||||
- Allow downloading a single file.
|
||||
- Ensure cleanup of mounts and temporary files.
|
||||
- [ ] User can browse a Linux VM filesystem from a snapshot without restoring the VM.
|
||||
- [ ] Mounted/temporary resources are cleaned up after use.
|
||||
- [ ] Unsupported or unsafe disk images fail with a clear error.
|
||||
|
||||
### Acceptance Criteria
|
||||
### 18. Add Version Reporting
|
||||
|
||||
- User can browse a Linux VM filesystem from a snapshot without restoring the VM.
|
||||
- Mounted/temporary resources are cleaned up after use.
|
||||
- The workflow refuses unsupported or unsafe disk images with a clear error.
|
||||
Status: open.
|
||||
|
||||
## 8. Add agent and management version reporting
|
||||
Tasks:
|
||||
|
||||
Multi-node setups need version visibility.
|
||||
- [ ] Add version field to management API.
|
||||
- [ ] Add version field to agent health response.
|
||||
- [ ] Show agent version in Nodes page.
|
||||
- [ ] Flag unsupported or outdated agents.
|
||||
- [ ] Document compatibility expectations.
|
||||
|
||||
### Goal
|
||||
Acceptance criteria:
|
||||
|
||||
Show software version and compatibility state for management and each agent.
|
||||
- [ ] Nodes page displays agent version.
|
||||
- [ ] Management can identify incompatible agents.
|
||||
- [ ] Health output includes version information.
|
||||
|
||||
### Tasks
|
||||
### 19. Containerized Management and UI Deployment
|
||||
|
||||
- Add version field to management API.
|
||||
- Add version field to agent health response.
|
||||
- Show agent version in Nodes page.
|
||||
- Flag unsupported or outdated agents.
|
||||
- Document compatibility expectations.
|
||||
Status: open.
|
||||
|
||||
### Acceptance Criteria
|
||||
The node-agent remains a host-level systemd service. Management and frontend are good candidates for containers.
|
||||
|
||||
- Nodes page displays agent version.
|
||||
- Management can identify incompatible agents.
|
||||
- Health output includes version information.
|
||||
Tasks:
|
||||
|
||||
## 9. Containerized management and UI deployment
|
||||
- [ ] Add a `management` container image.
|
||||
- [ ] Add a frontend image that serves the Vite build through a small static server or Nginx.
|
||||
- [ ] Provide `compose.yaml` with persistent SQLite volume for management.
|
||||
- [ ] Mount the agent CA certificate into the management container read-only.
|
||||
- [ ] Document `CORS_ORIGINS`, `SESSION_COOKIE_SECURE`, `DATABASE_PATH`, and reverse-proxy assumptions.
|
||||
- [ ] Decide whether frontend calls management through same-origin `/api` or separate API origin.
|
||||
- [ ] Add health checks for both containers.
|
||||
|
||||
The node-agent must remain a host-level systemd service because it needs direct Incus, ZFS, `/dev/zvol`, Restic, and device access. Management and the frontend do not need those host privileges and are good candidates for container deployment.
|
||||
Acceptance criteria:
|
||||
|
||||
### Goal
|
||||
- [ ] Management API and frontend can be started with Compose without installing Node.js on the management host.
|
||||
- [ ] Management database survives container recreation.
|
||||
- [ ] Cookie login works behind HTTPS.
|
||||
- [ ] Management can connect to HTTPS node-agents using the configured CA file.
|
||||
|
||||
Provide a production-ready Docker/Compose deployment for the management API and frontend while keeping node-agents installed as systemd services on Incus hosts.
|
||||
### 20. Backup Scheduling and Quotas
|
||||
|
||||
### Tasks
|
||||
Status: open.
|
||||
|
||||
- Add a `management` container image.
|
||||
- Add a frontend image that serves the Vite build through a small static server or Nginx.
|
||||
- Provide `compose.yaml` with persistent SQLite volume for management.
|
||||
- Mount the agent CA certificate into the management container as read-only.
|
||||
- Document required `CORS_ORIGINS`, `SESSION_COOKIE_SECURE`, `DATABASE_PATH`, and reverse-proxy assumptions.
|
||||
- Decide whether the frontend calls the management API through the same origin reverse proxy or a separate API origin.
|
||||
- Add health checks for both containers.
|
||||
Tasks:
|
||||
|
||||
### Acceptance Criteria
|
||||
- [ ] Add scheduler jitter in management and agent schedulers.
|
||||
- [ ] Add per-VM and global backup concurrency limits.
|
||||
- [ ] Add disk-space/quota checks before backup and restore.
|
||||
- [ ] Add pre-backup VM state checks for live migration, existing snapshots, and permission problems.
|
||||
|
||||
- Management API and frontend can be started with Compose without installing Node.js on the management host.
|
||||
- Management database survives container recreation.
|
||||
- Cookie login works behind HTTPS.
|
||||
- Management can connect to HTTPS node-agents using the configured CA file.
|
||||
Acceptance criteria:
|
||||
|
||||
## 10. Add automated tests and CI
|
||||
- [ ] Many nodes do not all start scheduled backups at exactly the same time.
|
||||
- [ ] A full pool or too many concurrent streams blocks new jobs with a clear error.
|
||||
|
||||
The project currently has no automated tests. This is the biggest engineering gap for a backup system because most dangerous failures happen in error paths, not in the happy path.
|
||||
## P3 - Cleanup and Refactoring
|
||||
|
||||
### Goal
|
||||
### 21. Systemd and Deployment Cleanup
|
||||
|
||||
Catch regressions in validation, locking, backup verification, restore orchestration, authentication, and management polling before deployment.
|
||||
Tasks:
|
||||
|
||||
### Tasks
|
||||
- [ ] Add systemd hardening compatible with Incus/ZFS access.
|
||||
- [ ] Use direct `node src/index.js` in service files instead of `npm start`.
|
||||
- [ ] Make `SCHEDULES_PATH` explicitly configurable.
|
||||
- [ ] Installer should warn on insecure agent exposure.
|
||||
|
||||
- Add a test runner for agent and management code.
|
||||
- Test validators for instance names, snapshot IDs, and ambiguous snapshot prefixes.
|
||||
- Test job locking, persistence, trimming, and restart behavior.
|
||||
- Test backup pipeline failure handling with mocked command/process failures.
|
||||
- Test restore staged-volume command sequencing with mocked ZFS/Incus/Restic commands.
|
||||
- Test management job polling updates `job_history` correctly for success, failed, timeout, and missing-agent cases.
|
||||
- Add CI for install, tests, frontend build, and syntax checks.
|
||||
### 22. Data Integrity and Schema Cleanup
|
||||
|
||||
### Acceptance Criteria
|
||||
Tasks:
|
||||
|
||||
- Pull requests run tests automatically.
|
||||
- Simulated backup stream failures fail the job.
|
||||
- Simulated restore failures leave the original ZVOL name restored in the command sequence.
|
||||
- Agent restart behavior is covered by tests.
|
||||
- [ ] Add audit-log integrity, such as append-only behavior and optional hash chain.
|
||||
- [ ] Add `audit_events.details` size limit or JSON column handling.
|
||||
- [ ] Replace ad-hoc `addColumnIfMissing` with schema versioning and migrations.
|
||||
- [ ] Stream Restic `ls` instead of loading all output in RAM.
|
||||
|
||||
## 11. Harden root-running agent exposure
|
||||
### 23. Code Cleanup
|
||||
|
||||
The agent runs with root-level host access because it needs Incus, ZFS, `/dev/zvol`, Restic, and device operations. A compromised agent is therefore a host-level incident.
|
||||
Tasks:
|
||||
|
||||
### Goal
|
||||
- [ ] Generalize `incus snapshot delete` retry instead of matching English stderr.
|
||||
- [ ] Deduplicate `formatBytes`.
|
||||
- [ ] Review `incus-backup-ui-plan.md` for stale paths, secrets, and obsolete implementation notes.
|
||||
- [ ] Remove tracked `.DS_Store` files.
|
||||
- [ ] Ensure `frontend/dist/` remains ignored and untracked.
|
||||
|
||||
Reduce the network and systemd blast radius of the root-running node-agent.
|
||||
## Done
|
||||
|
||||
### Tasks
|
||||
- [x] Default admin `admin/admin` fallback removed.
|
||||
- [x] Agent `API_TOKEN` required and minimum length enforced.
|
||||
- [x] HTTPS required for agent URLs in management, with dev-only insecure HTTP escape hatch.
|
||||
- [x] Management can trust internal agent CA through `AGENT_CA_FILE`.
|
||||
- [x] Agent supports direct HTTPS.
|
||||
- [x] CORS allowlist added for management.
|
||||
- [x] Auth cookie set to `SameSite=Strict`.
|
||||
- [x] Login brute-force protection added.
|
||||
- [x] Bearer token removed from frontend localStorage.
|
||||
- [x] Settings secrets masked/write-only.
|
||||
- [x] Agent token comparison made timing-safe.
|
||||
- [x] Agent job persistence added.
|
||||
- [x] Staged VM restore implemented.
|
||||
- [x] Node-agent renamed from `backend/` to `agent/`.
|
||||
- [x] Agent installer script added.
|
||||
- [x] Backup UI ETA/rate/bytes display added.
|
||||
|
||||
- Require private management-to-agent connectivity in production documentation, such as NetBird or a dedicated management network.
|
||||
- Prefer HTTPS agents with a private CA; document local HTTP only as development mode.
|
||||
- Enable and document `ALLOWED_MANAGEMENT_IPS` for production.
|
||||
- Add systemd hardening where compatible with Incus/ZFS access.
|
||||
- Add explicit installer warnings when `HTTPS_ENABLED=false` and no `ALLOWED_MANAGEMENT_IPS` is configured.
|
||||
- Consider replacing `npm start` in systemd with direct `node src/index.js`.
|
||||
## OSS Release Requirements
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Production install docs do not expose the agent publicly by default.
|
||||
- Installer warns on insecure network exposure.
|
||||
- systemd unit has a documented minimum hardening baseline.
|
||||
|
||||
## 12. Improve agent crash cleanup and resource recovery
|
||||
|
||||
Agent jobs are persisted and active jobs are marked `failed` on restart, but a process crash can still leave host resources behind, such as temporary Incus snapshots, staged restore volumes, visible snapshot devices, or changed ZFS properties.
|
||||
|
||||
### Goal
|
||||
|
||||
Make agent startup detect and clean up known leftover resources from interrupted jobs where doing so is safe.
|
||||
|
||||
### Tasks
|
||||
|
||||
- Persist cleanup metadata for backup jobs: temporary Incus snapshot name, ZVOL, snapdev state, and backup type.
|
||||
- Persist cleanup metadata for restore jobs: staged ZVOL, backup ZVOL, failed ZVOL, and swap state.
|
||||
- On startup, scan failed active jobs and run safe cleanup actions.
|
||||
- Log cleanup results into the persisted job logs.
|
||||
- Avoid destructive cleanup when state is ambiguous; surface manual action instead.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Crashing during backup streaming does not leave `snapdev=visible` or temporary snapshots unnoticed.
|
||||
- Crashing before restore swap removes staged restore volumes when safe.
|
||||
- Crashing after restore swap does not automatically destroy rollback copies.
|
||||
- Management can show cleanup-required states.
|
||||
|
||||
## 13. Improve health diagnostics in the UI
|
||||
|
||||
The agent health endpoint returns useful structured details, but the management UI currently summarizes this too aggressively. Operators need actionable health reasons without opening logs.
|
||||
|
||||
### Goal
|
||||
|
||||
Show per-node health detail in the UI with concrete failed checks and messages.
|
||||
|
||||
### Tasks
|
||||
|
||||
- Add a detailed health drawer or modal on the Nodes page.
|
||||
- Show command, ZFS pool, `/dev/zvol`, Restic repository, and credential check results.
|
||||
- Show last health timestamp and the management-side error if the agent is unreachable.
|
||||
- Distinguish unreachable, unauthorized, TLS failure, degraded health, and healthy states.
|
||||
- Avoid global UI timeout banners when only one enabled node is slow or unreachable.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- A missing command is visible by name in the UI.
|
||||
- Wrong Restic credentials are visible as a Restic health failure.
|
||||
- TLS or connectivity failures are distinguishable from degraded agent health.
|
||||
- [ ] Add tests.
|
||||
- [ ] Add CI pipeline.
|
||||
- [ ] Add `SECURITY.md`.
|
||||
- [ ] Document threat model and trusted network assumptions.
|
||||
- [ ] Add `LICENSE`.
|
||||
- [ ] Add `CONTRIBUTING.md`.
|
||||
- [ ] Add example `compose.yaml` or Ansible role for reproducible deployment.
|
||||
|
||||
Reference in New Issue
Block a user