incus-backup-ui/docs/issues.md

Issue Backlog

Status snapshot

Last reviewed against code: 2026-06-04.

Several earlier backlog items have already landed in the codebase:

• Management persists accepted agent jobs in job_history.
• Management polls agent jobs and updates final status, error, finish time, duration, current step, and parsed backup snapshot ID.
• Operations page shows job history and audit log.
• Node-agent health checks cover required commands, ZFS pool state, /dev/zvol, and Restic repository access.
• Backup jobs verify the stored Restic file size and remove failed snapshots on verification errors.

Known caveat: agent-side jobs are persisted, but subprocesses cannot survive an agent restart. Active jobs are marked failed on startup; deeper cleanup recovery for partially changed host resources is still future work.

Current pre-production priorities:

1. Harden backup verification and add tests for stream/pipeline failure cases.
2. Validate the staged restore workflow on a disposable Incus VM, including rollback scenarios.
3. Improve agent crash cleanup for partially changed ZFS/Incus resources.
4. Surface detailed node health diagnostics in the UI.
5. Keep the root-running agent tightly network-restricted.

1. Persist final agent job status in management

Status: mostly done.

Management records when a backup or restore was accepted by an agent and now persists the final agent job result when the node-agent remains reachable long enough to be polled.

Goal

Persist reliable end-to-end job status in the management database.

Tasks

• Add polling for accepted agent jobs from management.
• Store final success or failed status in job_history.
• Store duration, finished timestamp, error message, and current step.
• Store agent job logs summary.
• Store created Restic snapshot ID for successful backup jobs when available.
• Surface final status in the Operations page.
• Persist recent agent-side jobs.
• Mark active agent jobs as failed after an agent restart so management can poll a final state.

Acceptance Criteria

• A backup started through management eventually shows success or failed while the agent remains reachable.
• A restore started through management eventually shows success or failed while the agent remains reachable.
• Management restart does not lose already persisted history.
• Agent restart does not lose the active job record; active work is marked failed because the subprocess cannot survive restart.

2. Expand node-agent health checks

Status: mostly done.

The health endpoint now provides deeper operational checks for backup readiness.

Goal

Make /api/health useful for diagnosing whether a node can actually run backup and restore operations.

Tasks

• Check that required commands exist: incus, zfs, zpool, restic, udevadm, dd.
• Check that configured ZFS pool exists.
• Check that /dev/zvol is accessible.
• Check Restic repository access.
• Check S3/Restic credentials by running a safe Restic command.
• Include ZFS pool capacity and free space.
• Return structured check names and messages.
• Surface detailed per-node health output in the management UI, not just a compact aggregate.

Acceptance Criteria

• [~] Management UI shows degraded node health with actionable check names. Detailed output exists through node health endpoints; UI can still be improved.
• A missing command, wrong pool, or wrong Restic credentials is visible in health output.

3. Harden backup verification

Status: partially done.

Backup jobs now verify the backed-up Restic file size after restic backup and attempt to remove failed snapshots. This reduces the risk of accepting a truncated backup, but the implementation still needs stronger stream failure handling and broader verification semantics.

Goal

Ensure a backup is marked success only when the expected source data was fully stored and verified.

Tasks

• Parse the created Restic snapshot ID after backup.
• Verify stored Restic file size against streamed source bytes or expected ZFS size.
• Remove failed Restic snapshots with forget and prune on verification errors.
• Make stream/pipeline failure handling explicit for all sources.
• Avoid marking success if source stream closes early but Restic exits successfully.
• Add a post-backup Restic integrity check strategy, such as targeted restic check/restic stats usage that is safe for large repositories.
• Define an optional restore-probe workflow for critical VMs.
• Add tests with mocked command failures and short reads.

Acceptance Criteria

• Successful backup jobs include a verifiable Restic snapshot ID in management history.
• Size mismatches fail the job.
• Simulated source stream errors cannot produce a successful job.
• A backup can be independently verified without trusting only the successful process exit.
• Verification behavior is covered by automated tests.

4. Add per-VM backup policy

Retention and schedule behavior is currently broad. Per-VM policies would make production usage more flexible.

Current state: schedules can be enabled per VM with interval and time of day. Retention is still global per agent.

Goal

Allow each VM to define its own backup policy.

Tasks

• Add management database table for VM backup policies.
• Support per-VM retention values: hourly, daily, weekly, monthly.
• Support per-VM schedule enablement, interval, and time.
• Add optional policy flag: backup only when VM is running.
• Update Scheduler UI to edit policies per node and VM.
• Send policy retention to the agent backup request or apply it in management scheduling.

Acceptance Criteria

• Two VMs on the same node can have different schedules.
• Two VMs on the same node can have different retention policies.
• Disabled policies do not trigger backups.

5. Improve restore safety workflow

Restore is destructive and should be guarded with a clearer preflight and confirmation flow.

Current state: VM restore requires typed confirmation, validates the snapshot against the VM, checks source/target size, writes the Restic dump to a staged ZFS volume, stops the VM, and swaps the staged volume into place with zfs rename. The previous production volume is kept as a rollback copy.

Goal

Reduce the risk of accidental or unsafe restores.

Tasks

• Add restore preflight endpoint.
• Validate node health before restore.
• Show selected snapshot metadata before restore.
• Show target VM status and disk size before restore.
• Require explicit typed confirmation including VM name.
• Optionally offer “create backup before restore” when the VM is accessible.
• Log restore intent in audit log before dispatch.
• Replace direct dd to production ZVOL with a staged restore workflow.
• Test staged restore on a disposable VM: backup, restore, boot, and confirm data.
• Test failed staged restore paths: Restic dump failure before swap, rename failure after old volume rename, and VM start failure.
• Define a safe container restore workflow with zfs receive or disable container restore surfaces entirely.

Acceptance Criteria

• UI displays a restore plan before the final confirmation.
• Restore is blocked when required preflight checks fail.
• Audit log records restore attempts and results.
• Restore stream failures happen on the staged volume, not the production disk.
• Staged restore behavior is validated on a real Incus/ZFS test node.

6. Add failure notifications

Operators need to know when scheduled backups or restores fail.

Goal

Send notifications for failed or degraded operations.

Tasks

• Add notification settings in management.
• Support webhook notifications first.
• Include node, VM, job type, error, and timestamp.
• Trigger notifications for failed scheduled backups.
• Trigger notifications for failed manual backups/restores.
• Add a “send test notification” action.

Acceptance Criteria

• A failed scheduled backup sends one notification.
• A test notification can be triggered from the UI.
• Notification failures are visible in Operations or audit logs.

7. Implement real snapshot file browsing

Current snapshot browsing shows Restic contents, which for block-level backups is usually only /vm.raw.

Goal

Allow browsing files inside a backed-up VM disk image.

Tasks

• Design a safe read-only raw image inspection workflow.
• Restore or mount raw image read-only in a temporary workspace.
• Detect partitions and filesystems.
• Browse directories through management UI.
• Allow downloading a single file.
• Ensure cleanup of mounts and temporary files.

Acceptance Criteria

• User can browse a Linux VM filesystem from a snapshot without restoring the VM.
• Mounted/temporary resources are cleaned up after use.
• The workflow refuses unsupported or unsafe disk images with a clear error.

8. Add agent and management version reporting

Multi-node setups need version visibility.

Goal

Show software version and compatibility state for management and each agent.

Tasks

• Add version field to management API.
• Add version field to agent health response.
• Show agent version in Nodes page.
• Flag unsupported or outdated agents.
• Document compatibility expectations.

Acceptance Criteria

• Nodes page displays agent version.
• Management can identify incompatible agents.
• Health output includes version information.

9. Containerized management and UI deployment

The node-agent must remain a host-level systemd service because it needs direct Incus, ZFS, /dev/zvol, Restic, and device access. Management and the frontend do not need those host privileges and are good candidates for container deployment.

Goal

Provide a production-ready Docker/Compose deployment for the management API and frontend while keeping node-agents installed as systemd services on Incus hosts.

Tasks

• Add a management container image.
• Add a frontend image that serves the Vite build through a small static server or Nginx.
• Provide compose.yaml with persistent SQLite volume for management.
• Mount the agent CA certificate into the management container as read-only.
• Document required CORS_ORIGINS, SESSION_COOKIE_SECURE, DATABASE_PATH, and reverse-proxy assumptions.
• Decide whether the frontend calls the management API through the same origin reverse proxy or a separate API origin.
• Add health checks for both containers.

Acceptance Criteria

• Management API and frontend can be started with Compose without installing Node.js on the management host.
• Management database survives container recreation.
• Cookie login works behind HTTPS.
• Management can connect to HTTPS node-agents using the configured CA file.

10. Add automated tests and CI

The project currently has no automated tests. This is the biggest engineering gap for a backup system because most dangerous failures happen in error paths, not in the happy path.

Goal

Catch regressions in validation, locking, backup verification, restore orchestration, authentication, and management polling before deployment.

Tasks

• Add a test runner for agent and management code.
• Test validators for instance names, snapshot IDs, and ambiguous snapshot prefixes.
• Test job locking, persistence, trimming, and restart behavior.
• Test backup pipeline failure handling with mocked command/process failures.
• Test restore staged-volume command sequencing with mocked ZFS/Incus/Restic commands.
• Test management job polling updates job_history correctly for success, failed, timeout, and missing-agent cases.
• Add CI for install, tests, frontend build, and syntax checks.

Acceptance Criteria

• Pull requests run tests automatically.
• Simulated backup stream failures fail the job.
• Simulated restore failures leave the original ZVOL name restored in the command sequence.
• Agent restart behavior is covered by tests.

11. Harden root-running agent exposure

The agent runs with root-level host access because it needs Incus, ZFS, /dev/zvol, Restic, and device operations. A compromised agent is therefore a host-level incident.

Goal

Reduce the network and systemd blast radius of the root-running node-agent.

Tasks

• Require private management-to-agent connectivity in production documentation, such as NetBird or a dedicated management network.
• Prefer HTTPS agents with a private CA; document local HTTP only as development mode.
• Enable and document ALLOWED_MANAGEMENT_IPS for production.
• Add systemd hardening where compatible with Incus/ZFS access.
• Add explicit installer warnings when HTTPS_ENABLED=false and no ALLOWED_MANAGEMENT_IPS is configured.
• Consider replacing npm start in systemd with direct node src/index.js.

Acceptance Criteria

• Production install docs do not expose the agent publicly by default.
• Installer warns on insecure network exposure.
• systemd unit has a documented minimum hardening baseline.

12. Improve agent crash cleanup and resource recovery

Agent jobs are persisted and active jobs are marked failed on restart, but a process crash can still leave host resources behind, such as temporary Incus snapshots, staged restore volumes, visible snapshot devices, or changed ZFS properties.

Goal

Make agent startup detect and clean up known leftover resources from interrupted jobs where doing so is safe.

Tasks

• Persist cleanup metadata for backup jobs: temporary Incus snapshot name, ZVOL, snapdev state, and backup type.
• Persist cleanup metadata for restore jobs: staged ZVOL, backup ZVOL, failed ZVOL, and swap state.
• On startup, scan failed active jobs and run safe cleanup actions.
• Log cleanup results into the persisted job logs.
• Avoid destructive cleanup when state is ambiguous; surface manual action instead.

Acceptance Criteria

• Crashing during backup streaming does not leave snapdev=visible or temporary snapshots unnoticed.
• Crashing before restore swap removes staged restore volumes when safe.
• Crashing after restore swap does not automatically destroy rollback copies.
• Management can show cleanup-required states.

13. Improve health diagnostics in the UI

The agent health endpoint returns useful structured details, but the management UI currently summarizes this too aggressively. Operators need actionable health reasons without opening logs.

Goal

Show per-node health detail in the UI with concrete failed checks and messages.

Tasks

• Add a detailed health drawer or modal on the Nodes page.
• Show command, ZFS pool, /dev/zvol, Restic repository, and credential check results.
• Show last health timestamp and the management-side error if the agent is unreachable.
• Distinguish unreachable, unauthorized, TLS failure, degraded health, and healthy states.
• Avoid global UI timeout banners when only one enabled node is slow or unreachable.

Acceptance Criteria

• A missing command is visible by name in the UI.
• Wrong Restic credentials are visible as a Restic health failure.
• TLS or connectivity failures are distinguishable from degraded agent health.