incus-backup-ui/docs/issues.md

Issue Backlog

Last reviewed against code: 2026-06-04.

This is the single source of truth for product work, hardening, release tasks, and production blockers. The previous docs/fixes-todo.md list has been merged into this file.

Current Status

Implemented and verified in code:

• Management persists accepted agent jobs in job_history.
• Management polls agent jobs and updates final status, error, finish time, duration, current step, and parsed backup snapshot ID.
• Operations page shows job history and audit log.
• Node-agent health checks cover required commands, ZFS pool state, /dev/zvol, and Restic repository access.
• Backup jobs parse the Restic snapshot ID, verify stored file size, and remove failed snapshots on verification errors.
• VM restore no longer writes directly to the production ZVOL. It writes to a staged ZVOL first, then swaps volumes with zfs rename.
• Agent jobs are persisted in SQLite. Active jobs are marked failed after an agent restart because subprocesses cannot survive restart.
• Auth hardening landed: no default admin password, required agent token, CORS allowlist, SameSite=Strict, login brute-force protection, no frontend localStorage bearer token.
• Settings secrets are write-only/masked in the UI and API.
• Agent bearer-token comparison is timing-safe.
• Agent directory was renamed from backend/ to agent/.

Known caveat: active agent jobs are persisted, but deeper cleanup recovery for partially changed host resources is still future work.

Pre-Production Priorities

1. Harden backup verification and add tests for stream/pipeline failure cases.
2. Validate the staged restore workflow on a disposable Incus VM, including rollback scenarios.
3. Evaluate true incremental ZFS-send based backups for large VM disks.
4. Add cleanup and visibility for pre-restore/failed-restore ZVOLs.
5. Improve agent crash cleanup for partially changed ZFS/Incus resources.
6. Surface detailed node health diagnostics in the UI.
7. Keep the root-running agent tightly network-restricted.
8. Add automated tests and CI.

P0 - Production Blockers

1. Harden Backup Verification

Status: partially done.

Goal: ensure a backup is marked success only when the expected source data was fully stored and verified.

Tasks:

• Parse the created Restic snapshot ID after backup.
• Verify stored Restic file size against streamed source bytes or expected ZFS size.
• Remove failed Restic snapshots with forget and prune on verification errors.
• Make stream/pipeline failure handling explicit for VM and container sources.
• Avoid marking success if the source stream closes early but Restic exits successfully.
• Add a post-backup Restic integrity-check strategy that is safe for large repositories.
• Define an optional restore-probe workflow for critical VMs.
• Add tests with mocked command failures, pipe failures, and short reads.

Acceptance criteria:

• Successful backup jobs include a verifiable Restic snapshot ID in management history.
• Size mismatches fail the job.
• Simulated source stream errors cannot produce a successful job.
• A backup can be independently verified without trusting only the successful process exit.
• Verification behavior is covered by automated tests.

2. Validate Staged Restore Workflow

Status: implemented, not proven on real Incus/ZFS hardware in this repo session.

Goal: prove restore safety before production use.

Tasks:

• Require explicit typed confirmation including VM name.
• Validate snapshot ownership before restore.
• Replace direct dd to production ZVOL with a staged restore workflow.
• Log restore intent in audit log before dispatch.
• Add restore preflight endpoint.
• Validate node health before restore.
• Show selected snapshot metadata before restore.
• Show target VM status and disk size before restore.
• Optionally offer “create backup before restore” when the VM is accessible.
• Test staged restore on a disposable VM: backup, restore, boot, and confirm data.
• Test failed staged restore paths: Restic dump failure before swap, rename failure after old volume rename, and VM start failure.

Acceptance criteria:

• Restore stream failures happen on the staged volume, not the production disk.
• UI displays a restore plan before final confirmation.
• Restore is blocked when required preflight checks fail.
• Staged restore behavior is validated on a real Incus/ZFS test node.

3. Add Pre-Restore Volume Cleanup

Status: open.

Current behavior: every successful VM restore keeps the old volume as *.pre-restore-*. This is good for rollback, but without retention or cleanup it can fill the ZFS pool.

Tasks:

• Track pre-restore and failed-restore volumes created by the agent.
• Add a retention policy, for example keep the last N rollback volumes per VM.
• Surface retained rollback volumes in Operations or VM detail.
• Add explicit cleanup action with confirmation.
• Add health warning when retained restore volumes consume significant pool space.

Acceptance criteria:

• Repeated restores cannot silently consume unbounded pool space.
• Operators can see and clean retained restore volumes.

4. Harden Root-Running Agent Exposure

Status: partially done.

The agent runs with root-level host access because it needs Incus, ZFS, /dev/zvol, Restic, and device operations. A compromised agent is a host-level incident.

Tasks:

• Require a non-empty API_TOKEN of at least 32 characters.
• Support ALLOWED_MANAGEMENT_IPS.
• Support HTTPS directly on the agent.
• Support private CA validation from management via AGENT_CA_FILE.
• Require private management-to-agent connectivity in production docs, such as NetBird or a dedicated management network.
• Prefer HTTPS agents with a private CA; document local HTTP only as development mode.
• Add installer warning when HTTPS_ENABLED=false and no ALLOWED_MANAGEMENT_IPS is configured.
• Add systemd hardening where compatible with Incus/ZFS access.
• Replace npm start in systemd with direct node src/index.js.
• Remove CORS from the agent; browsers should never call it directly.

Acceptance criteria:

• Production install docs do not expose the agent publicly by default.
• Installer warns on insecure network exposure.
• systemd unit has a documented minimum hardening baseline.

5. Encrypt or Protect Agent Tokens in Management

Status: open.

Current behavior: nodes.token is stored in plaintext in the management SQLite database. DB theft means all agent tokens are exposed.

Tasks:

• Decide between at-rest encryption with a dedicated management key or strict documented file/disk protection.
• If encrypting, add key configuration and migration for existing tokens.
• Ensure public node API responses never include tokens.
• Document database file permissions and disk-encryption expectations.

Acceptance criteria:

• Stealing the management database alone is not enough to read every agent token, or the residual risk is explicitly documented and operationally mitigated.

P1 - High Priority

6. Evaluate True Incremental VM Backups

Status: open.

Current behavior: VM backups stream the full ZVOL as vm.raw through restic backup --stdin. Restic deduplicates storage, but the agent still has to read, chunk, and hash the complete virtual disk every run. A VM with no changed files can therefore still take a long time.

Goal: reduce backup duration for large mostly-unchanged VMs by reading only changed ZFS blocks after the first full backup.

Tasks:

• Evaluate ZFS snapshot-chain based incremental backups with zfs send -i previous current.
• Design snapshot naming and retention so required incremental bases are not deleted too early.
• Define restore behavior for full plus incremental send chains.
• Decide whether incremental streams should be stored in Restic, object storage directly, or another repository layout.
• Define compatibility behavior for existing .raw Restic snapshots.
• Compare operational tradeoffs: faster backups versus more complex restore and retention.
• Add UI wording that distinguishes Processed bytes from data actually uploaded/stored.

Acceptance criteria:

• A second backup of an unchanged large VM does not need to read the full ZVOL.
• Restore can reconstruct a VM from the chosen full/incremental chain.
• Retention cannot delete an incremental base required for restore.

7. Add Automated Tests and CI

Status: open.

Tasks:

• Add a test runner for agent and management code.
• Test validators for instance names, snapshot IDs, and ambiguous snapshot prefixes.
• Test job locking, persistence, trimming, and restart behavior.
• Test backup pipeline failure handling with mocked command/process failures.
• Test restore staged-volume command sequencing with mocked ZFS/Incus/Restic commands.
• Test management job polling updates job_history for success, failed, timeout, and missing-agent cases.
• Add CI for install, tests, frontend build, and syntax checks.

Acceptance criteria:

• Pull requests run tests automatically.
• Simulated backup stream failures fail the job.
• Simulated restore failures leave the original ZVOL name restored in the command sequence.
• Agent restart behavior is covered by tests.

8. Improve Agent Crash Cleanup and Resource Recovery

Status: open.

Agent jobs are persisted and active jobs are marked failed on restart, but a process crash can still leave host resources behind.

Tasks:

• Persist cleanup metadata for backup jobs: temporary Incus snapshot name, ZVOL, snapdev state, and backup type.
• Persist cleanup metadata for restore jobs: staged ZVOL, backup ZVOL, failed ZVOL, and swap state.
• On startup, scan failed active jobs and run safe cleanup actions.
• Log cleanup results into the persisted job logs.
• Avoid destructive cleanup when state is ambiguous; surface manual action instead.

Acceptance criteria:

• Crashing during backup streaming does not leave snapdev=visible or temporary snapshots unnoticed.
• Crashing before restore swap removes staged restore volumes when safe.
• Crashing after restore swap does not automatically destroy rollback copies.
• Management can show cleanup-required states.

9. Improve Health Diagnostics in the UI

Status: partially done on API, open in UI.

Tasks:

• Agent health checks commands, ZFS pool, /dev/zvol, and Restic repository.
• Management records compact node health state.
• Add a detailed health drawer or modal on the Nodes page.
• Show command, ZFS pool, /dev/zvol, Restic repository, and credential check results.
• Show last health timestamp and the management-side error if the agent is unreachable.
• Distinguish unreachable, unauthorized, TLS failure, degraded health, and healthy states.
• Avoid global UI timeout banners when only one enabled node is slow or unreachable.

Acceptance criteria:

• A missing command is visible by name in the UI.
• Wrong Restic credentials are visible as a Restic health failure.
• TLS or connectivity failures are distinguishable from degraded agent health.

10. Fix Snapshot Device Visibility Race

Status: open.

Current behavior: backup waits a fixed 2 seconds after setting snapdev=visible.

Tasks:

• Replace fixed sleep with polling fs.access(snapshotDevice) and timeout.
• Run udevadm trigger and udevadm settle.
• Surface timeout as a clear backup failure.

Acceptance criteria:

• Slow device-node creation does not fail randomly.

11. Make Snapshot-ID Prefix Matching Unambiguous

Status: open.

Current behavior: prefix matching uses startsWith; collisions choose the first match.

Tasks:

• Return conflict when a snapshot prefix matches more than one snapshot.
• Prefer 12+ hex characters in the UI.
• Add validator tests.

Acceptance criteria:

• Ambiguous snapshot prefixes cannot restore the wrong snapshot.

12. Settings and Environment Hardening

Status: partially done.

Tasks:

• Mask secret values in GET /settings.
• Treat empty secret field as “keep existing value”.
• Improve .env escaping for $, backticks, and newlines.
• Add tests for readEnvSettings and writeEnvSettings.
• Document .env file permissions and enforce chmod 600 in installer/deployment.
• Keep or remove SESSION_SECRET deliberately: either use it for signing/encryption or remove dead config.
• Set and document SESSION_COOKIE_SECURE=true for production.

Acceptance criteria:

• Secret settings cannot be exfiltrated through the UI/API.
• Writing .env cannot create shell-expansion surprises.

13. Add RBAC

Status: open.

Current behavior: one user model, effectively all permissions.

Tasks:

• Add roles: admin, operator, viewer.
• Restrict restore and node settings to admin.
• Allow read-only dashboard access for viewer.
• Audit role changes.

Acceptance criteria:

• Restore is admin-only.
• Viewer cannot trigger backup, restore, node edits, or settings writes.

14. Session and Auth Cleanup

Status: partially done.

Tasks:

• Login brute-force protection.
• SameSite=Strict.
• Server-side cleanup of expired sessions.
• Session rotation after successful login.
• Replace Math.random() in management user ID creation with crypto.randomBytes.
• Consider argon2id or explicit scrypt parameters and migration path.

Acceptance criteria:

• Expired sessions do not accumulate unbounded in SQLite.
• Session IDs are rotated after login.

15. Container Restore Decision

Status: open.

Current behavior: container backups can be created, but restore returns 501.

Tasks:

• Implement safe container restore with zfs receive, or disable container backups in UI/API until restore exists.
• Document the limitation prominently.
• Add tests for unsupported restore behavior.

Acceptance criteria:

• Users cannot mistake an unrestorable container backup for complete protection.

P2 - Product and Operations

16. Add Per-VM Backup Policy

Status: partially done.

Current state: schedules can be enabled per VM with interval and time of day. Retention is still global per agent.

Tasks:

• Add management database table for VM backup policies.
• Support per-VM retention values: hourly, daily, weekly, monthly.
• Support per-VM schedule enablement, interval, and time.
• Add optional policy flag: backup only when VM is running.
• Update Scheduler UI to edit policies per node and VM.
• Send policy retention to the agent backup request or apply it in management scheduling.

Acceptance criteria:

• Two VMs on the same node can have different schedules.
• Two VMs on the same node can have different retention policies.
• Disabled policies do not trigger backups.

17. Add Failure Notifications

Status: open.

Tasks:

• Add notification settings in management.
• Support webhook notifications first.
• Include node, VM, job type, error, and timestamp.
• Trigger notifications for failed scheduled backups.
• Trigger notifications for failed manual backups/restores.
• Add a “send test notification” action.

Acceptance criteria:

• A failed scheduled backup sends one notification.
• A test notification can be triggered from the UI.
• Notification failures are visible in Operations or audit logs.

18. Implement Real Snapshot File Browsing

Status: open.

Current snapshot browsing shows Restic contents, which for block-level backups is usually only /vm.raw.

Tasks:

• Design a safe read-only raw image inspection workflow.
• Restore or mount raw image read-only in a temporary workspace.
• Detect partitions and filesystems.
• Browse directories through management UI.
• Allow downloading a single file.
• Ensure cleanup of mounts and temporary files.

Acceptance criteria:

• User can browse a Linux VM filesystem from a snapshot without restoring the VM.
• Mounted/temporary resources are cleaned up after use.
• Unsupported or unsafe disk images fail with a clear error.

19. Add Version Reporting

Status: open.

Tasks:

• Add version field to management API.
• Add version field to agent health response.
• Show agent version in Nodes page.
• Flag unsupported or outdated agents.
• Document compatibility expectations.

Acceptance criteria:

• Nodes page displays agent version.
• Management can identify incompatible agents.
• Health output includes version information.

20. Containerized Management and UI Deployment

Status: open.

The node-agent remains a host-level systemd service. Management and frontend are good candidates for containers.

Tasks:

• Add a management container image.
• Add a frontend image that serves the Vite build through a small static server or Nginx.
• Provide compose.yaml with persistent SQLite volume for management.
• Mount the agent CA certificate into the management container read-only.
• Document CORS_ORIGINS, SESSION_COOKIE_SECURE, DATABASE_PATH, and reverse-proxy assumptions.
• Decide whether frontend calls management through same-origin /api or separate API origin.
• Add health checks for both containers.

Acceptance criteria:

• Management API and frontend can be started with Compose without installing Node.js on the management host.
• Management database survives container recreation.
• Cookie login works behind HTTPS.
• Management can connect to HTTPS node-agents using the configured CA file.

21. Backup Scheduling and Quotas

Status: open.

Tasks:

• Add scheduler jitter in management and agent schedulers.
• Add per-VM and global backup concurrency limits.
• Add disk-space/quota checks before backup and restore.
• Add pre-backup VM state checks for live migration, existing snapshots, and permission problems.

Acceptance criteria:

• Many nodes do not all start scheduled backups at exactly the same time.
• A full pool or too many concurrent streams blocks new jobs with a clear error.

P3 - Cleanup and Refactoring

22. Systemd and Deployment Cleanup

Tasks:

• Add systemd hardening compatible with Incus/ZFS access.
• Use direct node src/index.js in service files instead of npm start.
• Make SCHEDULES_PATH explicitly configurable.
• Installer should warn on insecure agent exposure.

23. Data Integrity and Schema Cleanup

Tasks:

• Add audit-log integrity, such as append-only behavior and optional hash chain.
• Add audit_events.details size limit or JSON column handling.
• Replace ad-hoc addColumnIfMissing with schema versioning and migrations.
• Stream Restic ls instead of loading all output in RAM.

24. Code Cleanup

Tasks:

• Generalize incus snapshot delete retry instead of matching English stderr.
• Deduplicate formatBytes.
• Review incus-backup-ui-plan.md for stale paths, secrets, and obsolete implementation notes.
• Remove tracked .DS_Store files.
• Ensure frontend/dist/ remains ignored and untracked.

Done

• Default admin admin/admin fallback removed.
• Agent API_TOKEN required and minimum length enforced.
• HTTPS required for agent URLs in management, with dev-only insecure HTTP escape hatch.
• Management can trust internal agent CA through AGENT_CA_FILE.
• Agent supports direct HTTPS.
• CORS allowlist added for management.
• Auth cookie set to SameSite=Strict.
• Login brute-force protection added.
• Bearer token removed from frontend localStorage.
• Settings secrets masked/write-only.
• Agent token comparison made timing-safe.
• Agent job persistence added.
• Staged VM restore implemented.
• Node-agent renamed from backend/ to agent/.
• Agent installer script added.
• Agent update/diagnostic CLI added as incubator.
• Backup UI ETA/rate/bytes display added.

OSS Release Requirements

• Add tests.
• Add CI pipeline.
• Add SECURITY.md.
• Document threat model and trusted network assumptions.
• Add LICENSE.
• Add CONTRIBUTING.md.
• Add example compose.yaml or Ansible role for reproducible deployment.