# Issue Backlog Last reviewed against code: 2026-06-04. This is the single source of truth for product work, hardening, release tasks, and production blockers. The previous `docs/fixes-todo.md` list has been merged into this file. ## Current Status Implemented and verified in code: - Management persists accepted agent jobs in `job_history`. - Management polls agent jobs and updates final status, error, finish time, duration, current step, and parsed backup snapshot ID. - Operations page shows job history and audit log. - Node-agent health checks cover required commands, ZFS pool state, `/dev/zvol`, and Restic repository access. - Backup jobs parse the Restic snapshot ID, verify stored file size, and remove failed snapshots on verification errors. - VM restore no longer writes directly to the production ZVOL. It writes to a staged ZVOL first, then swaps volumes with `zfs rename`. - Agent jobs are persisted in SQLite. Active jobs are marked `failed` after an agent restart because subprocesses cannot survive restart. - Auth hardening landed: no default admin password, required agent token, CORS allowlist, `SameSite=Strict`, login brute-force protection, no frontend localStorage bearer token. - Settings secrets are write-only/masked in the UI and API. - Agent bearer-token comparison is timing-safe. - Agent directory was renamed from `backend/` to `agent/`. Known caveat: active agent jobs are persisted, but deeper cleanup recovery for partially changed host resources is still future work. ## Pre-Production Priorities 1. Harden backup verification and add tests for stream/pipeline failure cases. 2. Validate the staged restore workflow on a disposable Incus VM, including rollback scenarios. 3. Add cleanup and visibility for pre-restore/failed-restore ZVOLs. 4. Improve agent crash cleanup for partially changed ZFS/Incus resources. 5. Surface detailed node health diagnostics in the UI. 6. Keep the root-running agent tightly network-restricted. 7. Add automated tests and CI. ## P0 - Production Blockers ### 1. Harden Backup Verification Status: partially done. Goal: ensure a backup is marked `success` only when the expected source data was fully stored and verified. Tasks: - [x] Parse the created Restic snapshot ID after backup. - [x] Verify stored Restic file size against streamed source bytes or expected ZFS size. - [x] Remove failed Restic snapshots with `forget` and `prune` on verification errors. - [ ] Make stream/pipeline failure handling explicit for VM and container sources. - [ ] Avoid marking success if the source stream closes early but Restic exits successfully. - [ ] Add a post-backup Restic integrity-check strategy that is safe for large repositories. - [ ] Define an optional restore-probe workflow for critical VMs. - [ ] Add tests with mocked command failures, pipe failures, and short reads. Acceptance criteria: - [x] Successful backup jobs include a verifiable Restic snapshot ID in management history. - [x] Size mismatches fail the job. - [ ] Simulated source stream errors cannot produce a successful job. - [ ] A backup can be independently verified without trusting only the successful process exit. - [ ] Verification behavior is covered by automated tests. ### 2. Validate Staged Restore Workflow Status: implemented, not proven on real Incus/ZFS hardware in this repo session. Goal: prove restore safety before production use. Tasks: - [x] Require explicit typed confirmation including VM name. - [x] Validate snapshot ownership before restore. - [x] Replace direct `dd` to production ZVOL with a staged restore workflow. - [x] Log restore intent in audit log before dispatch. - [ ] Add restore preflight endpoint. - [ ] Validate node health before restore. - [ ] Show selected snapshot metadata before restore. - [ ] Show target VM status and disk size before restore. - [ ] Optionally offer “create backup before restore” when the VM is accessible. - [ ] Test staged restore on a disposable VM: backup, restore, boot, and confirm data. - [ ] Test failed staged restore paths: Restic dump failure before swap, rename failure after old volume rename, and VM start failure. Acceptance criteria: - [x] Restore stream failures happen on the staged volume, not the production disk. - [ ] UI displays a restore plan before final confirmation. - [ ] Restore is blocked when required preflight checks fail. - [ ] Staged restore behavior is validated on a real Incus/ZFS test node. ### 3. Add Pre-Restore Volume Cleanup Status: open. Current behavior: every successful VM restore keeps the old volume as `*.pre-restore-*`. This is good for rollback, but without retention or cleanup it can fill the ZFS pool. Tasks: - [ ] Track pre-restore and failed-restore volumes created by the agent. - [ ] Add a retention policy, for example keep the last N rollback volumes per VM. - [ ] Surface retained rollback volumes in Operations or VM detail. - [ ] Add explicit cleanup action with confirmation. - [ ] Add health warning when retained restore volumes consume significant pool space. Acceptance criteria: - [ ] Repeated restores cannot silently consume unbounded pool space. - [ ] Operators can see and clean retained restore volumes. ### 4. Harden Root-Running Agent Exposure Status: partially done. The agent runs with root-level host access because it needs Incus, ZFS, `/dev/zvol`, Restic, and device operations. A compromised agent is a host-level incident. Tasks: - [x] Require a non-empty `API_TOKEN` of at least 32 characters. - [x] Support `ALLOWED_MANAGEMENT_IPS`. - [x] Support HTTPS directly on the agent. - [x] Support private CA validation from management via `AGENT_CA_FILE`. - [ ] Require private management-to-agent connectivity in production docs, such as NetBird or a dedicated management network. - [ ] Prefer HTTPS agents with a private CA; document local HTTP only as development mode. - [ ] Add installer warning when `HTTPS_ENABLED=false` and no `ALLOWED_MANAGEMENT_IPS` is configured. - [ ] Add systemd hardening where compatible with Incus/ZFS access. - [ ] Replace `npm start` in systemd with direct `node src/index.js`. - [ ] Remove CORS from the agent; browsers should never call it directly. Acceptance criteria: - [ ] Production install docs do not expose the agent publicly by default. - [ ] Installer warns on insecure network exposure. - [ ] systemd unit has a documented minimum hardening baseline. ### 5. Encrypt or Protect Agent Tokens in Management Status: open. Current behavior: `nodes.token` is stored in plaintext in the management SQLite database. DB theft means all agent tokens are exposed. Tasks: - [ ] Decide between at-rest encryption with a dedicated management key or strict documented file/disk protection. - [ ] If encrypting, add key configuration and migration for existing tokens. - [ ] Ensure public node API responses never include tokens. - [ ] Document database file permissions and disk-encryption expectations. Acceptance criteria: - [ ] Stealing the management database alone is not enough to read every agent token, or the residual risk is explicitly documented and operationally mitigated. ## P1 - High Priority ### 6. Add Automated Tests and CI Status: open. Tasks: - [ ] Add a test runner for agent and management code. - [ ] Test validators for instance names, snapshot IDs, and ambiguous snapshot prefixes. - [ ] Test job locking, persistence, trimming, and restart behavior. - [ ] Test backup pipeline failure handling with mocked command/process failures. - [ ] Test restore staged-volume command sequencing with mocked ZFS/Incus/Restic commands. - [ ] Test management job polling updates `job_history` for success, failed, timeout, and missing-agent cases. - [ ] Add CI for install, tests, frontend build, and syntax checks. Acceptance criteria: - [ ] Pull requests run tests automatically. - [ ] Simulated backup stream failures fail the job. - [ ] Simulated restore failures leave the original ZVOL name restored in the command sequence. - [ ] Agent restart behavior is covered by tests. ### 7. Improve Agent Crash Cleanup and Resource Recovery Status: open. Agent jobs are persisted and active jobs are marked `failed` on restart, but a process crash can still leave host resources behind. Tasks: - [ ] Persist cleanup metadata for backup jobs: temporary Incus snapshot name, ZVOL, snapdev state, and backup type. - [ ] Persist cleanup metadata for restore jobs: staged ZVOL, backup ZVOL, failed ZVOL, and swap state. - [ ] On startup, scan failed active jobs and run safe cleanup actions. - [ ] Log cleanup results into the persisted job logs. - [ ] Avoid destructive cleanup when state is ambiguous; surface manual action instead. Acceptance criteria: - [ ] Crashing during backup streaming does not leave `snapdev=visible` or temporary snapshots unnoticed. - [ ] Crashing before restore swap removes staged restore volumes when safe. - [ ] Crashing after restore swap does not automatically destroy rollback copies. - [ ] Management can show cleanup-required states. ### 8. Improve Health Diagnostics in the UI Status: partially done on API, open in UI. Tasks: - [x] Agent health checks commands, ZFS pool, `/dev/zvol`, and Restic repository. - [x] Management records compact node health state. - [ ] Add a detailed health drawer or modal on the Nodes page. - [ ] Show command, ZFS pool, `/dev/zvol`, Restic repository, and credential check results. - [ ] Show last health timestamp and the management-side error if the agent is unreachable. - [ ] Distinguish unreachable, unauthorized, TLS failure, degraded health, and healthy states. - [ ] Avoid global UI timeout banners when only one enabled node is slow or unreachable. Acceptance criteria: - [ ] A missing command is visible by name in the UI. - [ ] Wrong Restic credentials are visible as a Restic health failure. - [ ] TLS or connectivity failures are distinguishable from degraded agent health. ### 9. Fix Snapshot Device Visibility Race Status: open. Current behavior: backup waits a fixed 2 seconds after setting `snapdev=visible`. Tasks: - [ ] Replace fixed sleep with polling `fs.access(snapshotDevice)` and timeout. - [ ] Run `udevadm trigger` and `udevadm settle`. - [ ] Surface timeout as a clear backup failure. Acceptance criteria: - [ ] Slow device-node creation does not fail randomly. ### 10. Make Snapshot-ID Prefix Matching Unambiguous Status: open. Current behavior: prefix matching uses `startsWith`; collisions choose the first match. Tasks: - [ ] Return conflict when a snapshot prefix matches more than one snapshot. - [ ] Prefer 12+ hex characters in the UI. - [ ] Add validator tests. Acceptance criteria: - [ ] Ambiguous snapshot prefixes cannot restore the wrong snapshot. ### 11. Settings and Environment Hardening Status: partially done. Tasks: - [x] Mask secret values in `GET /settings`. - [x] Treat empty secret field as “keep existing value”. - [ ] Improve `.env` escaping for `$`, backticks, and newlines. - [ ] Add tests for `readEnvSettings` and `writeEnvSettings`. - [ ] Document `.env` file permissions and enforce `chmod 600` in installer/deployment. - [ ] Keep or remove `SESSION_SECRET` deliberately: either use it for signing/encryption or remove dead config. - [ ] Set and document `SESSION_COOKIE_SECURE=true` for production. Acceptance criteria: - [ ] Secret settings cannot be exfiltrated through the UI/API. - [ ] Writing `.env` cannot create shell-expansion surprises. ### 12. Add RBAC Status: open. Current behavior: one user model, effectively all permissions. Tasks: - [ ] Add roles: `admin`, `operator`, `viewer`. - [ ] Restrict restore and node settings to `admin`. - [ ] Allow read-only dashboard access for `viewer`. - [ ] Audit role changes. Acceptance criteria: - [ ] Restore is admin-only. - [ ] Viewer cannot trigger backup, restore, node edits, or settings writes. ### 13. Session and Auth Cleanup Status: partially done. Tasks: - [x] Login brute-force protection. - [x] `SameSite=Strict`. - [ ] Server-side cleanup of expired sessions. - [ ] Session rotation after successful login. - [ ] Replace `Math.random()` in management user ID creation with `crypto.randomBytes`. - [ ] Consider argon2id or explicit scrypt parameters and migration path. Acceptance criteria: - [ ] Expired sessions do not accumulate unbounded in SQLite. - [ ] Session IDs are rotated after login. ### 14. Container Restore Decision Status: open. Current behavior: container backups can be created, but restore returns `501`. Tasks: - [ ] Implement safe container restore with `zfs receive`, or disable container backups in UI/API until restore exists. - [ ] Document the limitation prominently. - [ ] Add tests for unsupported restore behavior. Acceptance criteria: - [ ] Users cannot mistake an unrestorable container backup for complete protection. ## P2 - Product and Operations ### 15. Add Per-VM Backup Policy Status: partially done. Current state: schedules can be enabled per VM with interval and time of day. Retention is still global per agent. Tasks: - [ ] Add management database table for VM backup policies. - [ ] Support per-VM retention values: hourly, daily, weekly, monthly. - [x] Support per-VM schedule enablement, interval, and time. - [ ] Add optional policy flag: backup only when VM is running. - [ ] Update Scheduler UI to edit policies per node and VM. - [ ] Send policy retention to the agent backup request or apply it in management scheduling. Acceptance criteria: - [ ] Two VMs on the same node can have different schedules. - [ ] Two VMs on the same node can have different retention policies. - [ ] Disabled policies do not trigger backups. ### 16. Add Failure Notifications Status: open. Tasks: - [ ] Add notification settings in management. - [ ] Support webhook notifications first. - [ ] Include node, VM, job type, error, and timestamp. - [ ] Trigger notifications for failed scheduled backups. - [ ] Trigger notifications for failed manual backups/restores. - [ ] Add a “send test notification” action. Acceptance criteria: - [ ] A failed scheduled backup sends one notification. - [ ] A test notification can be triggered from the UI. - [ ] Notification failures are visible in Operations or audit logs. ### 17. Implement Real Snapshot File Browsing Status: open. Current snapshot browsing shows Restic contents, which for block-level backups is usually only `/vm.raw`. Tasks: - [ ] Design a safe read-only raw image inspection workflow. - [ ] Restore or mount raw image read-only in a temporary workspace. - [ ] Detect partitions and filesystems. - [ ] Browse directories through management UI. - [ ] Allow downloading a single file. - [ ] Ensure cleanup of mounts and temporary files. Acceptance criteria: - [ ] User can browse a Linux VM filesystem from a snapshot without restoring the VM. - [ ] Mounted/temporary resources are cleaned up after use. - [ ] Unsupported or unsafe disk images fail with a clear error. ### 18. Add Version Reporting Status: open. Tasks: - [ ] Add version field to management API. - [ ] Add version field to agent health response. - [ ] Show agent version in Nodes page. - [ ] Flag unsupported or outdated agents. - [ ] Document compatibility expectations. Acceptance criteria: - [ ] Nodes page displays agent version. - [ ] Management can identify incompatible agents. - [ ] Health output includes version information. ### 19. Containerized Management and UI Deployment Status: open. The node-agent remains a host-level systemd service. Management and frontend are good candidates for containers. Tasks: - [ ] Add a `management` container image. - [ ] Add a frontend image that serves the Vite build through a small static server or Nginx. - [ ] Provide `compose.yaml` with persistent SQLite volume for management. - [ ] Mount the agent CA certificate into the management container read-only. - [ ] Document `CORS_ORIGINS`, `SESSION_COOKIE_SECURE`, `DATABASE_PATH`, and reverse-proxy assumptions. - [ ] Decide whether frontend calls management through same-origin `/api` or separate API origin. - [ ] Add health checks for both containers. Acceptance criteria: - [ ] Management API and frontend can be started with Compose without installing Node.js on the management host. - [ ] Management database survives container recreation. - [ ] Cookie login works behind HTTPS. - [ ] Management can connect to HTTPS node-agents using the configured CA file. ### 20. Backup Scheduling and Quotas Status: open. Tasks: - [ ] Add scheduler jitter in management and agent schedulers. - [ ] Add per-VM and global backup concurrency limits. - [ ] Add disk-space/quota checks before backup and restore. - [ ] Add pre-backup VM state checks for live migration, existing snapshots, and permission problems. Acceptance criteria: - [ ] Many nodes do not all start scheduled backups at exactly the same time. - [ ] A full pool or too many concurrent streams blocks new jobs with a clear error. ## P3 - Cleanup and Refactoring ### 21. Systemd and Deployment Cleanup Tasks: - [ ] Add systemd hardening compatible with Incus/ZFS access. - [ ] Use direct `node src/index.js` in service files instead of `npm start`. - [ ] Make `SCHEDULES_PATH` explicitly configurable. - [ ] Installer should warn on insecure agent exposure. ### 22. Data Integrity and Schema Cleanup Tasks: - [ ] Add audit-log integrity, such as append-only behavior and optional hash chain. - [ ] Add `audit_events.details` size limit or JSON column handling. - [ ] Replace ad-hoc `addColumnIfMissing` with schema versioning and migrations. - [ ] Stream Restic `ls` instead of loading all output in RAM. ### 23. Code Cleanup Tasks: - [ ] Generalize `incus snapshot delete` retry instead of matching English stderr. - [ ] Deduplicate `formatBytes`. - [ ] Review `incus-backup-ui-plan.md` for stale paths, secrets, and obsolete implementation notes. - [ ] Remove tracked `.DS_Store` files. - [ ] Ensure `frontend/dist/` remains ignored and untracked. ## Done - [x] Default admin `admin/admin` fallback removed. - [x] Agent `API_TOKEN` required and minimum length enforced. - [x] HTTPS required for agent URLs in management, with dev-only insecure HTTP escape hatch. - [x] Management can trust internal agent CA through `AGENT_CA_FILE`. - [x] Agent supports direct HTTPS. - [x] CORS allowlist added for management. - [x] Auth cookie set to `SameSite=Strict`. - [x] Login brute-force protection added. - [x] Bearer token removed from frontend localStorage. - [x] Settings secrets masked/write-only. - [x] Agent token comparison made timing-safe. - [x] Agent job persistence added. - [x] Staged VM restore implemented. - [x] Node-agent renamed from `backend/` to `agent/`. - [x] Agent installer script added. - [x] Backup UI ETA/rate/bytes display added. ## OSS Release Requirements - [ ] Add tests. - [ ] Add CI pipeline. - [ ] Add `SECURITY.md`. - [ ] Document threat model and trusted network assumptions. - [ ] Add `LICENSE`. - [ ] Add `CONTRIBUTING.md`. - [ ] Add example `compose.yaml` or Ansible role for reproducible deployment.