495 lines
19 KiB
Markdown
495 lines
19 KiB
Markdown
# Issue Backlog
|
|
|
|
Last reviewed against code: 2026-06-04.
|
|
|
|
This is the single source of truth for product work, hardening, release tasks, and production blockers. The previous `docs/fixes-todo.md` list has been merged into this file.
|
|
|
|
## Current Status
|
|
|
|
Implemented and verified in code:
|
|
|
|
- Management persists accepted agent jobs in `job_history`.
|
|
- Management polls agent jobs and updates final status, error, finish time, duration, current step, and parsed backup snapshot ID.
|
|
- Operations page shows job history and audit log.
|
|
- Node-agent health checks cover required commands, ZFS pool state, `/dev/zvol`, and Restic repository access.
|
|
- Backup jobs parse the Restic snapshot ID, verify stored file size, and remove failed snapshots on verification errors.
|
|
- VM restore no longer writes directly to the production ZVOL. It writes to a staged ZVOL first, then swaps volumes with `zfs rename`.
|
|
- Agent jobs are persisted in SQLite. Active jobs are marked `failed` after an agent restart because subprocesses cannot survive restart.
|
|
- Auth hardening landed: no default admin password, required agent token, CORS allowlist, `SameSite=Strict`, login brute-force protection, no frontend localStorage bearer token.
|
|
- Settings secrets are write-only/masked in the UI and API.
|
|
- Agent bearer-token comparison is timing-safe.
|
|
- Agent directory was renamed from `backend/` to `agent/`.
|
|
|
|
Known caveat: active agent jobs are persisted, but deeper cleanup recovery for partially changed host resources is still future work.
|
|
|
|
## Pre-Production Priorities
|
|
|
|
1. Harden backup verification and add tests for stream/pipeline failure cases.
|
|
2. Validate the staged restore workflow on a disposable Incus VM, including rollback scenarios.
|
|
3. Add cleanup and visibility for pre-restore/failed-restore ZVOLs.
|
|
4. Improve agent crash cleanup for partially changed ZFS/Incus resources.
|
|
5. Surface detailed node health diagnostics in the UI.
|
|
6. Keep the root-running agent tightly network-restricted.
|
|
7. Add automated tests and CI.
|
|
|
|
## P0 - Production Blockers
|
|
|
|
### 1. Harden Backup Verification
|
|
|
|
Status: partially done.
|
|
|
|
Goal: ensure a backup is marked `success` only when the expected source data was fully stored and verified.
|
|
|
|
Tasks:
|
|
|
|
- [x] Parse the created Restic snapshot ID after backup.
|
|
- [x] Verify stored Restic file size against streamed source bytes or expected ZFS size.
|
|
- [x] Remove failed Restic snapshots with `forget` and `prune` on verification errors.
|
|
- [ ] Make stream/pipeline failure handling explicit for VM and container sources.
|
|
- [ ] Avoid marking success if the source stream closes early but Restic exits successfully.
|
|
- [ ] Add a post-backup Restic integrity-check strategy that is safe for large repositories.
|
|
- [ ] Define an optional restore-probe workflow for critical VMs.
|
|
- [ ] Add tests with mocked command failures, pipe failures, and short reads.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [x] Successful backup jobs include a verifiable Restic snapshot ID in management history.
|
|
- [x] Size mismatches fail the job.
|
|
- [ ] Simulated source stream errors cannot produce a successful job.
|
|
- [ ] A backup can be independently verified without trusting only the successful process exit.
|
|
- [ ] Verification behavior is covered by automated tests.
|
|
|
|
### 2. Validate Staged Restore Workflow
|
|
|
|
Status: implemented, not proven on real Incus/ZFS hardware in this repo session.
|
|
|
|
Goal: prove restore safety before production use.
|
|
|
|
Tasks:
|
|
|
|
- [x] Require explicit typed confirmation including VM name.
|
|
- [x] Validate snapshot ownership before restore.
|
|
- [x] Replace direct `dd` to production ZVOL with a staged restore workflow.
|
|
- [x] Log restore intent in audit log before dispatch.
|
|
- [ ] Add restore preflight endpoint.
|
|
- [ ] Validate node health before restore.
|
|
- [ ] Show selected snapshot metadata before restore.
|
|
- [ ] Show target VM status and disk size before restore.
|
|
- [ ] Optionally offer “create backup before restore” when the VM is accessible.
|
|
- [ ] Test staged restore on a disposable VM: backup, restore, boot, and confirm data.
|
|
- [ ] Test failed staged restore paths: Restic dump failure before swap, rename failure after old volume rename, and VM start failure.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [x] Restore stream failures happen on the staged volume, not the production disk.
|
|
- [ ] UI displays a restore plan before final confirmation.
|
|
- [ ] Restore is blocked when required preflight checks fail.
|
|
- [ ] Staged restore behavior is validated on a real Incus/ZFS test node.
|
|
|
|
### 3. Add Pre-Restore Volume Cleanup
|
|
|
|
Status: open.
|
|
|
|
Current behavior: every successful VM restore keeps the old volume as `*.pre-restore-*`. This is good for rollback, but without retention or cleanup it can fill the ZFS pool.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Track pre-restore and failed-restore volumes created by the agent.
|
|
- [ ] Add a retention policy, for example keep the last N rollback volumes per VM.
|
|
- [ ] Surface retained rollback volumes in Operations or VM detail.
|
|
- [ ] Add explicit cleanup action with confirmation.
|
|
- [ ] Add health warning when retained restore volumes consume significant pool space.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Repeated restores cannot silently consume unbounded pool space.
|
|
- [ ] Operators can see and clean retained restore volumes.
|
|
|
|
### 4. Harden Root-Running Agent Exposure
|
|
|
|
Status: partially done.
|
|
|
|
The agent runs with root-level host access because it needs Incus, ZFS, `/dev/zvol`, Restic, and device operations. A compromised agent is a host-level incident.
|
|
|
|
Tasks:
|
|
|
|
- [x] Require a non-empty `API_TOKEN` of at least 32 characters.
|
|
- [x] Support `ALLOWED_MANAGEMENT_IPS`.
|
|
- [x] Support HTTPS directly on the agent.
|
|
- [x] Support private CA validation from management via `AGENT_CA_FILE`.
|
|
- [ ] Require private management-to-agent connectivity in production docs, such as NetBird or a dedicated management network.
|
|
- [ ] Prefer HTTPS agents with a private CA; document local HTTP only as development mode.
|
|
- [ ] Add installer warning when `HTTPS_ENABLED=false` and no `ALLOWED_MANAGEMENT_IPS` is configured.
|
|
- [ ] Add systemd hardening where compatible with Incus/ZFS access.
|
|
- [ ] Replace `npm start` in systemd with direct `node src/index.js`.
|
|
- [ ] Remove CORS from the agent; browsers should never call it directly.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Production install docs do not expose the agent publicly by default.
|
|
- [ ] Installer warns on insecure network exposure.
|
|
- [ ] systemd unit has a documented minimum hardening baseline.
|
|
|
|
### 5. Encrypt or Protect Agent Tokens in Management
|
|
|
|
Status: open.
|
|
|
|
Current behavior: `nodes.token` is stored in plaintext in the management SQLite database. DB theft means all agent tokens are exposed.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Decide between at-rest encryption with a dedicated management key or strict documented file/disk protection.
|
|
- [ ] If encrypting, add key configuration and migration for existing tokens.
|
|
- [ ] Ensure public node API responses never include tokens.
|
|
- [ ] Document database file permissions and disk-encryption expectations.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Stealing the management database alone is not enough to read every agent token, or the residual risk is explicitly documented and operationally mitigated.
|
|
|
|
## P1 - High Priority
|
|
|
|
### 6. Add Automated Tests and CI
|
|
|
|
Status: open.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Add a test runner for agent and management code.
|
|
- [ ] Test validators for instance names, snapshot IDs, and ambiguous snapshot prefixes.
|
|
- [ ] Test job locking, persistence, trimming, and restart behavior.
|
|
- [ ] Test backup pipeline failure handling with mocked command/process failures.
|
|
- [ ] Test restore staged-volume command sequencing with mocked ZFS/Incus/Restic commands.
|
|
- [ ] Test management job polling updates `job_history` for success, failed, timeout, and missing-agent cases.
|
|
- [ ] Add CI for install, tests, frontend build, and syntax checks.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Pull requests run tests automatically.
|
|
- [ ] Simulated backup stream failures fail the job.
|
|
- [ ] Simulated restore failures leave the original ZVOL name restored in the command sequence.
|
|
- [ ] Agent restart behavior is covered by tests.
|
|
|
|
### 7. Improve Agent Crash Cleanup and Resource Recovery
|
|
|
|
Status: open.
|
|
|
|
Agent jobs are persisted and active jobs are marked `failed` on restart, but a process crash can still leave host resources behind.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Persist cleanup metadata for backup jobs: temporary Incus snapshot name, ZVOL, snapdev state, and backup type.
|
|
- [ ] Persist cleanup metadata for restore jobs: staged ZVOL, backup ZVOL, failed ZVOL, and swap state.
|
|
- [ ] On startup, scan failed active jobs and run safe cleanup actions.
|
|
- [ ] Log cleanup results into the persisted job logs.
|
|
- [ ] Avoid destructive cleanup when state is ambiguous; surface manual action instead.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Crashing during backup streaming does not leave `snapdev=visible` or temporary snapshots unnoticed.
|
|
- [ ] Crashing before restore swap removes staged restore volumes when safe.
|
|
- [ ] Crashing after restore swap does not automatically destroy rollback copies.
|
|
- [ ] Management can show cleanup-required states.
|
|
|
|
### 8. Improve Health Diagnostics in the UI
|
|
|
|
Status: partially done on API, open in UI.
|
|
|
|
Tasks:
|
|
|
|
- [x] Agent health checks commands, ZFS pool, `/dev/zvol`, and Restic repository.
|
|
- [x] Management records compact node health state.
|
|
- [ ] Add a detailed health drawer or modal on the Nodes page.
|
|
- [ ] Show command, ZFS pool, `/dev/zvol`, Restic repository, and credential check results.
|
|
- [ ] Show last health timestamp and the management-side error if the agent is unreachable.
|
|
- [ ] Distinguish unreachable, unauthorized, TLS failure, degraded health, and healthy states.
|
|
- [ ] Avoid global UI timeout banners when only one enabled node is slow or unreachable.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] A missing command is visible by name in the UI.
|
|
- [ ] Wrong Restic credentials are visible as a Restic health failure.
|
|
- [ ] TLS or connectivity failures are distinguishable from degraded agent health.
|
|
|
|
### 9. Fix Snapshot Device Visibility Race
|
|
|
|
Status: open.
|
|
|
|
Current behavior: backup waits a fixed 2 seconds after setting `snapdev=visible`.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Replace fixed sleep with polling `fs.access(snapshotDevice)` and timeout.
|
|
- [ ] Run `udevadm trigger` and `udevadm settle`.
|
|
- [ ] Surface timeout as a clear backup failure.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Slow device-node creation does not fail randomly.
|
|
|
|
### 10. Make Snapshot-ID Prefix Matching Unambiguous
|
|
|
|
Status: open.
|
|
|
|
Current behavior: prefix matching uses `startsWith`; collisions choose the first match.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Return conflict when a snapshot prefix matches more than one snapshot.
|
|
- [ ] Prefer 12+ hex characters in the UI.
|
|
- [ ] Add validator tests.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Ambiguous snapshot prefixes cannot restore the wrong snapshot.
|
|
|
|
### 11. Settings and Environment Hardening
|
|
|
|
Status: partially done.
|
|
|
|
Tasks:
|
|
|
|
- [x] Mask secret values in `GET /settings`.
|
|
- [x] Treat empty secret field as “keep existing value”.
|
|
- [ ] Improve `.env` escaping for `$`, backticks, and newlines.
|
|
- [ ] Add tests for `readEnvSettings` and `writeEnvSettings`.
|
|
- [ ] Document `.env` file permissions and enforce `chmod 600` in installer/deployment.
|
|
- [ ] Keep or remove `SESSION_SECRET` deliberately: either use it for signing/encryption or remove dead config.
|
|
- [ ] Set and document `SESSION_COOKIE_SECURE=true` for production.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Secret settings cannot be exfiltrated through the UI/API.
|
|
- [ ] Writing `.env` cannot create shell-expansion surprises.
|
|
|
|
### 12. Add RBAC
|
|
|
|
Status: open.
|
|
|
|
Current behavior: one user model, effectively all permissions.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Add roles: `admin`, `operator`, `viewer`.
|
|
- [ ] Restrict restore and node settings to `admin`.
|
|
- [ ] Allow read-only dashboard access for `viewer`.
|
|
- [ ] Audit role changes.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Restore is admin-only.
|
|
- [ ] Viewer cannot trigger backup, restore, node edits, or settings writes.
|
|
|
|
### 13. Session and Auth Cleanup
|
|
|
|
Status: partially done.
|
|
|
|
Tasks:
|
|
|
|
- [x] Login brute-force protection.
|
|
- [x] `SameSite=Strict`.
|
|
- [ ] Server-side cleanup of expired sessions.
|
|
- [ ] Session rotation after successful login.
|
|
- [ ] Replace `Math.random()` in management user ID creation with `crypto.randomBytes`.
|
|
- [ ] Consider argon2id or explicit scrypt parameters and migration path.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Expired sessions do not accumulate unbounded in SQLite.
|
|
- [ ] Session IDs are rotated after login.
|
|
|
|
### 14. Container Restore Decision
|
|
|
|
Status: open.
|
|
|
|
Current behavior: container backups can be created, but restore returns `501`.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Implement safe container restore with `zfs receive`, or disable container backups in UI/API until restore exists.
|
|
- [ ] Document the limitation prominently.
|
|
- [ ] Add tests for unsupported restore behavior.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Users cannot mistake an unrestorable container backup for complete protection.
|
|
|
|
## P2 - Product and Operations
|
|
|
|
### 15. Add Per-VM Backup Policy
|
|
|
|
Status: partially done.
|
|
|
|
Current state: schedules can be enabled per VM with interval and time of day. Retention is still global per agent.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Add management database table for VM backup policies.
|
|
- [ ] Support per-VM retention values: hourly, daily, weekly, monthly.
|
|
- [x] Support per-VM schedule enablement, interval, and time.
|
|
- [ ] Add optional policy flag: backup only when VM is running.
|
|
- [ ] Update Scheduler UI to edit policies per node and VM.
|
|
- [ ] Send policy retention to the agent backup request or apply it in management scheduling.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Two VMs on the same node can have different schedules.
|
|
- [ ] Two VMs on the same node can have different retention policies.
|
|
- [ ] Disabled policies do not trigger backups.
|
|
|
|
### 16. Add Failure Notifications
|
|
|
|
Status: open.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Add notification settings in management.
|
|
- [ ] Support webhook notifications first.
|
|
- [ ] Include node, VM, job type, error, and timestamp.
|
|
- [ ] Trigger notifications for failed scheduled backups.
|
|
- [ ] Trigger notifications for failed manual backups/restores.
|
|
- [ ] Add a “send test notification” action.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] A failed scheduled backup sends one notification.
|
|
- [ ] A test notification can be triggered from the UI.
|
|
- [ ] Notification failures are visible in Operations or audit logs.
|
|
|
|
### 17. Implement Real Snapshot File Browsing
|
|
|
|
Status: open.
|
|
|
|
Current snapshot browsing shows Restic contents, which for block-level backups is usually only `/vm.raw`.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Design a safe read-only raw image inspection workflow.
|
|
- [ ] Restore or mount raw image read-only in a temporary workspace.
|
|
- [ ] Detect partitions and filesystems.
|
|
- [ ] Browse directories through management UI.
|
|
- [ ] Allow downloading a single file.
|
|
- [ ] Ensure cleanup of mounts and temporary files.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] User can browse a Linux VM filesystem from a snapshot without restoring the VM.
|
|
- [ ] Mounted/temporary resources are cleaned up after use.
|
|
- [ ] Unsupported or unsafe disk images fail with a clear error.
|
|
|
|
### 18. Add Version Reporting
|
|
|
|
Status: open.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Add version field to management API.
|
|
- [ ] Add version field to agent health response.
|
|
- [ ] Show agent version in Nodes page.
|
|
- [ ] Flag unsupported or outdated agents.
|
|
- [ ] Document compatibility expectations.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Nodes page displays agent version.
|
|
- [ ] Management can identify incompatible agents.
|
|
- [ ] Health output includes version information.
|
|
|
|
### 19. Containerized Management and UI Deployment
|
|
|
|
Status: open.
|
|
|
|
The node-agent remains a host-level systemd service. Management and frontend are good candidates for containers.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Add a `management` container image.
|
|
- [ ] Add a frontend image that serves the Vite build through a small static server or Nginx.
|
|
- [ ] Provide `compose.yaml` with persistent SQLite volume for management.
|
|
- [ ] Mount the agent CA certificate into the management container read-only.
|
|
- [ ] Document `CORS_ORIGINS`, `SESSION_COOKIE_SECURE`, `DATABASE_PATH`, and reverse-proxy assumptions.
|
|
- [ ] Decide whether frontend calls management through same-origin `/api` or separate API origin.
|
|
- [ ] Add health checks for both containers.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Management API and frontend can be started with Compose without installing Node.js on the management host.
|
|
- [ ] Management database survives container recreation.
|
|
- [ ] Cookie login works behind HTTPS.
|
|
- [ ] Management can connect to HTTPS node-agents using the configured CA file.
|
|
|
|
### 20. Backup Scheduling and Quotas
|
|
|
|
Status: open.
|
|
|
|
Tasks:
|
|
|
|
- [ ] Add scheduler jitter in management and agent schedulers.
|
|
- [ ] Add per-VM and global backup concurrency limits.
|
|
- [ ] Add disk-space/quota checks before backup and restore.
|
|
- [ ] Add pre-backup VM state checks for live migration, existing snapshots, and permission problems.
|
|
|
|
Acceptance criteria:
|
|
|
|
- [ ] Many nodes do not all start scheduled backups at exactly the same time.
|
|
- [ ] A full pool or too many concurrent streams blocks new jobs with a clear error.
|
|
|
|
## P3 - Cleanup and Refactoring
|
|
|
|
### 21. Systemd and Deployment Cleanup
|
|
|
|
Tasks:
|
|
|
|
- [ ] Add systemd hardening compatible with Incus/ZFS access.
|
|
- [ ] Use direct `node src/index.js` in service files instead of `npm start`.
|
|
- [ ] Make `SCHEDULES_PATH` explicitly configurable.
|
|
- [ ] Installer should warn on insecure agent exposure.
|
|
|
|
### 22. Data Integrity and Schema Cleanup
|
|
|
|
Tasks:
|
|
|
|
- [ ] Add audit-log integrity, such as append-only behavior and optional hash chain.
|
|
- [ ] Add `audit_events.details` size limit or JSON column handling.
|
|
- [ ] Replace ad-hoc `addColumnIfMissing` with schema versioning and migrations.
|
|
- [ ] Stream Restic `ls` instead of loading all output in RAM.
|
|
|
|
### 23. Code Cleanup
|
|
|
|
Tasks:
|
|
|
|
- [ ] Generalize `incus snapshot delete` retry instead of matching English stderr.
|
|
- [ ] Deduplicate `formatBytes`.
|
|
- [ ] Review `incus-backup-ui-plan.md` for stale paths, secrets, and obsolete implementation notes.
|
|
- [ ] Remove tracked `.DS_Store` files.
|
|
- [ ] Ensure `frontend/dist/` remains ignored and untracked.
|
|
|
|
## Done
|
|
|
|
- [x] Default admin `admin/admin` fallback removed.
|
|
- [x] Agent `API_TOKEN` required and minimum length enforced.
|
|
- [x] HTTPS required for agent URLs in management, with dev-only insecure HTTP escape hatch.
|
|
- [x] Management can trust internal agent CA through `AGENT_CA_FILE`.
|
|
- [x] Agent supports direct HTTPS.
|
|
- [x] CORS allowlist added for management.
|
|
- [x] Auth cookie set to `SameSite=Strict`.
|
|
- [x] Login brute-force protection added.
|
|
- [x] Bearer token removed from frontend localStorage.
|
|
- [x] Settings secrets masked/write-only.
|
|
- [x] Agent token comparison made timing-safe.
|
|
- [x] Agent job persistence added.
|
|
- [x] Staged VM restore implemented.
|
|
- [x] Node-agent renamed from `backend/` to `agent/`.
|
|
- [x] Agent installer script added.
|
|
- [x] Backup UI ETA/rate/bytes display added.
|
|
|
|
## OSS Release Requirements
|
|
|
|
- [ ] Add tests.
|
|
- [ ] Add CI pipeline.
|
|
- [ ] Add `SECURITY.md`.
|
|
- [ ] Document threat model and trusted network assumptions.
|
|
- [ ] Add `LICENSE`.
|
|
- [ ] Add `CONTRIBUTING.md`.
|
|
- [ ] Add example `compose.yaml` or Ansible role for reproducible deployment.
|