350 lines
15 KiB
Markdown
350 lines
15 KiB
Markdown
# Issue Backlog
|
|
|
|
## Status snapshot
|
|
|
|
Last reviewed against code: 2026-06-04.
|
|
|
|
Several earlier backlog items have already landed in the codebase:
|
|
|
|
- Management persists accepted agent jobs in `job_history`.
|
|
- Management polls agent jobs and updates final status, error, finish time, duration, current step, and parsed backup snapshot ID.
|
|
- Operations page shows job history and audit log.
|
|
- Node-agent health checks cover required commands, ZFS pool state, `/dev/zvol`, and Restic repository access.
|
|
- Backup jobs verify the stored Restic file size and remove failed snapshots on verification errors.
|
|
|
|
Known caveat: agent-side jobs are persisted, but subprocesses cannot survive an agent restart. Active jobs are marked `failed` on startup; deeper cleanup recovery for partially changed host resources is still future work.
|
|
|
|
Current pre-production priorities:
|
|
|
|
1. Harden backup verification and add tests for stream/pipeline failure cases.
|
|
2. Validate the staged restore workflow on a disposable Incus VM, including rollback scenarios.
|
|
3. Improve agent crash cleanup for partially changed ZFS/Incus resources.
|
|
4. Surface detailed node health diagnostics in the UI.
|
|
5. Keep the root-running agent tightly network-restricted.
|
|
|
|
## 1. Persist final agent job status in management
|
|
|
|
Status: mostly done.
|
|
|
|
Management records when a backup or restore was accepted by an agent and now persists the final agent job result when the node-agent remains reachable long enough to be polled.
|
|
|
|
### Goal
|
|
|
|
Persist reliable end-to-end job status in the management database.
|
|
|
|
### Tasks
|
|
|
|
- [x] Add polling for accepted agent jobs from management.
|
|
- [x] Store final `success` or `failed` status in `job_history`.
|
|
- [x] Store duration, finished timestamp, error message, and current step.
|
|
- [ ] Store agent job logs summary.
|
|
- [x] Store created Restic snapshot ID for successful backup jobs when available.
|
|
- [x] Surface final status in the Operations page.
|
|
- [x] Persist recent agent-side jobs.
|
|
- [x] Mark active agent jobs as `failed` after an agent restart so management can poll a final state.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- [x] A backup started through management eventually shows `success` or `failed` while the agent remains reachable.
|
|
- [x] A restore started through management eventually shows `success` or `failed` while the agent remains reachable.
|
|
- [x] Management restart does not lose already persisted history.
|
|
- [x] Agent restart does not lose the active job record; active work is marked `failed` because the subprocess cannot survive restart.
|
|
|
|
## 2. Expand node-agent health checks
|
|
|
|
Status: mostly done.
|
|
|
|
The health endpoint now provides deeper operational checks for backup readiness.
|
|
|
|
### Goal
|
|
|
|
Make `/api/health` useful for diagnosing whether a node can actually run backup and restore operations.
|
|
|
|
### Tasks
|
|
|
|
- [x] Check that required commands exist: `incus`, `zfs`, `zpool`, `restic`, `udevadm`, `dd`.
|
|
- [x] Check that configured ZFS pool exists.
|
|
- [x] Check that `/dev/zvol` is accessible.
|
|
- [x] Check Restic repository access.
|
|
- [x] Check S3/Restic credentials by running a safe Restic command.
|
|
- [x] Include ZFS pool capacity and free space.
|
|
- [x] Return structured check names and messages.
|
|
- [ ] Surface detailed per-node health output in the management UI, not just a compact aggregate.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- [~] Management UI shows degraded node health with actionable check names. Detailed output exists through node health endpoints; UI can still be improved.
|
|
- [x] A missing command, wrong pool, or wrong Restic credentials is visible in health output.
|
|
|
|
## 3. Harden backup verification
|
|
|
|
Status: partially done.
|
|
|
|
Backup jobs now verify the backed-up Restic file size after `restic backup` and attempt to remove failed snapshots. This reduces the risk of accepting a truncated backup, but the implementation still needs stronger stream failure handling and broader verification semantics.
|
|
|
|
### Goal
|
|
|
|
Ensure a backup is marked `success` only when the expected source data was fully stored and verified.
|
|
|
|
### Tasks
|
|
|
|
- [x] Parse the created Restic snapshot ID after backup.
|
|
- [x] Verify stored Restic file size against streamed source bytes or expected ZFS size.
|
|
- [x] Remove failed Restic snapshots with `forget` and `prune` on verification errors.
|
|
- [ ] Make stream/pipeline failure handling explicit for all sources.
|
|
- [ ] Avoid marking success if source stream closes early but Restic exits successfully.
|
|
- [ ] Add a post-backup Restic integrity check strategy, such as targeted `restic check`/`restic stats` usage that is safe for large repositories.
|
|
- [ ] Define an optional restore-probe workflow for critical VMs.
|
|
- [ ] Add tests with mocked command failures and short reads.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- [x] Successful backup jobs include a verifiable Restic snapshot ID in management history.
|
|
- [x] Size mismatches fail the job.
|
|
- [ ] Simulated source stream errors cannot produce a successful job.
|
|
- [ ] A backup can be independently verified without trusting only the successful process exit.
|
|
- [ ] Verification behavior is covered by automated tests.
|
|
|
|
## 4. Add per-VM backup policy
|
|
|
|
Retention and schedule behavior is currently broad. Per-VM policies would make production usage more flexible.
|
|
|
|
Current state: schedules can be enabled per VM with interval and time of day. Retention is still global per agent.
|
|
|
|
### Goal
|
|
|
|
Allow each VM to define its own backup policy.
|
|
|
|
### Tasks
|
|
|
|
- Add management database table for VM backup policies.
|
|
- Support per-VM retention values: hourly, daily, weekly, monthly.
|
|
- [x] Support per-VM schedule enablement, interval, and time.
|
|
- Add optional policy flag: backup only when VM is running.
|
|
- Update Scheduler UI to edit policies per node and VM.
|
|
- Send policy retention to the agent backup request or apply it in management scheduling.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Two VMs on the same node can have different schedules.
|
|
- Two VMs on the same node can have different retention policies.
|
|
- Disabled policies do not trigger backups.
|
|
|
|
## 5. Improve restore safety workflow
|
|
|
|
Restore is destructive and should be guarded with a clearer preflight and confirmation flow.
|
|
|
|
Current state: VM restore requires typed confirmation, validates the snapshot against the VM, checks source/target size, writes the Restic dump to a staged ZFS volume, stops the VM, and swaps the staged volume into place with `zfs rename`. The previous production volume is kept as a rollback copy.
|
|
|
|
### Goal
|
|
|
|
Reduce the risk of accidental or unsafe restores.
|
|
|
|
### Tasks
|
|
|
|
- Add restore preflight endpoint.
|
|
- Validate node health before restore.
|
|
- Show selected snapshot metadata before restore.
|
|
- Show target VM status and disk size before restore.
|
|
- [x] Require explicit typed confirmation including VM name.
|
|
- Optionally offer “create backup before restore” when the VM is accessible.
|
|
- [x] Log restore intent in audit log before dispatch.
|
|
- [x] Replace direct `dd` to production ZVOL with a staged restore workflow.
|
|
- [ ] Test staged restore on a disposable VM: backup, restore, boot, and confirm data.
|
|
- [ ] Test failed staged restore paths: Restic dump failure before swap, rename failure after old volume rename, and VM start failure.
|
|
- [ ] Define a safe container restore workflow with `zfs receive` or disable container restore surfaces entirely.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- UI displays a restore plan before the final confirmation.
|
|
- Restore is blocked when required preflight checks fail.
|
|
- Audit log records restore attempts and results.
|
|
- [x] Restore stream failures happen on the staged volume, not the production disk.
|
|
- [ ] Staged restore behavior is validated on a real Incus/ZFS test node.
|
|
|
|
## 6. Add failure notifications
|
|
|
|
Operators need to know when scheduled backups or restores fail.
|
|
|
|
### Goal
|
|
|
|
Send notifications for failed or degraded operations.
|
|
|
|
### Tasks
|
|
|
|
- Add notification settings in management.
|
|
- Support webhook notifications first.
|
|
- Include node, VM, job type, error, and timestamp.
|
|
- Trigger notifications for failed scheduled backups.
|
|
- Trigger notifications for failed manual backups/restores.
|
|
- Add a “send test notification” action.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- A failed scheduled backup sends one notification.
|
|
- A test notification can be triggered from the UI.
|
|
- Notification failures are visible in Operations or audit logs.
|
|
|
|
## 7. Implement real snapshot file browsing
|
|
|
|
Current snapshot browsing shows Restic contents, which for block-level backups is usually only `/vm.raw`.
|
|
|
|
### Goal
|
|
|
|
Allow browsing files inside a backed-up VM disk image.
|
|
|
|
### Tasks
|
|
|
|
- Design a safe read-only raw image inspection workflow.
|
|
- Restore or mount raw image read-only in a temporary workspace.
|
|
- Detect partitions and filesystems.
|
|
- Browse directories through management UI.
|
|
- Allow downloading a single file.
|
|
- Ensure cleanup of mounts and temporary files.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- User can browse a Linux VM filesystem from a snapshot without restoring the VM.
|
|
- Mounted/temporary resources are cleaned up after use.
|
|
- The workflow refuses unsupported or unsafe disk images with a clear error.
|
|
|
|
## 8. Add agent and management version reporting
|
|
|
|
Multi-node setups need version visibility.
|
|
|
|
### Goal
|
|
|
|
Show software version and compatibility state for management and each agent.
|
|
|
|
### Tasks
|
|
|
|
- Add version field to management API.
|
|
- Add version field to agent health response.
|
|
- Show agent version in Nodes page.
|
|
- Flag unsupported or outdated agents.
|
|
- Document compatibility expectations.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Nodes page displays agent version.
|
|
- Management can identify incompatible agents.
|
|
- Health output includes version information.
|
|
|
|
## 9. Containerized management and UI deployment
|
|
|
|
The node-agent must remain a host-level systemd service because it needs direct Incus, ZFS, `/dev/zvol`, Restic, and device access. Management and the frontend do not need those host privileges and are good candidates for container deployment.
|
|
|
|
### Goal
|
|
|
|
Provide a production-ready Docker/Compose deployment for the management API and frontend while keeping node-agents installed as systemd services on Incus hosts.
|
|
|
|
### Tasks
|
|
|
|
- Add a `management` container image.
|
|
- Add a frontend image that serves the Vite build through a small static server or Nginx.
|
|
- Provide `compose.yaml` with persistent SQLite volume for management.
|
|
- Mount the agent CA certificate into the management container as read-only.
|
|
- Document required `CORS_ORIGINS`, `SESSION_COOKIE_SECURE`, `DATABASE_PATH`, and reverse-proxy assumptions.
|
|
- Decide whether the frontend calls the management API through the same origin reverse proxy or a separate API origin.
|
|
- Add health checks for both containers.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Management API and frontend can be started with Compose without installing Node.js on the management host.
|
|
- Management database survives container recreation.
|
|
- Cookie login works behind HTTPS.
|
|
- Management can connect to HTTPS node-agents using the configured CA file.
|
|
|
|
## 10. Add automated tests and CI
|
|
|
|
The project currently has no automated tests. This is the biggest engineering gap for a backup system because most dangerous failures happen in error paths, not in the happy path.
|
|
|
|
### Goal
|
|
|
|
Catch regressions in validation, locking, backup verification, restore orchestration, authentication, and management polling before deployment.
|
|
|
|
### Tasks
|
|
|
|
- Add a test runner for agent and management code.
|
|
- Test validators for instance names, snapshot IDs, and ambiguous snapshot prefixes.
|
|
- Test job locking, persistence, trimming, and restart behavior.
|
|
- Test backup pipeline failure handling with mocked command/process failures.
|
|
- Test restore staged-volume command sequencing with mocked ZFS/Incus/Restic commands.
|
|
- Test management job polling updates `job_history` correctly for success, failed, timeout, and missing-agent cases.
|
|
- Add CI for install, tests, frontend build, and syntax checks.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Pull requests run tests automatically.
|
|
- Simulated backup stream failures fail the job.
|
|
- Simulated restore failures leave the original ZVOL name restored in the command sequence.
|
|
- Agent restart behavior is covered by tests.
|
|
|
|
## 11. Harden root-running agent exposure
|
|
|
|
The agent runs with root-level host access because it needs Incus, ZFS, `/dev/zvol`, Restic, and device operations. A compromised agent is therefore a host-level incident.
|
|
|
|
### Goal
|
|
|
|
Reduce the network and systemd blast radius of the root-running node-agent.
|
|
|
|
### Tasks
|
|
|
|
- Require private management-to-agent connectivity in production documentation, such as NetBird or a dedicated management network.
|
|
- Prefer HTTPS agents with a private CA; document local HTTP only as development mode.
|
|
- Enable and document `ALLOWED_MANAGEMENT_IPS` for production.
|
|
- Add systemd hardening where compatible with Incus/ZFS access.
|
|
- Add explicit installer warnings when `HTTPS_ENABLED=false` and no `ALLOWED_MANAGEMENT_IPS` is configured.
|
|
- Consider replacing `npm start` in systemd with direct `node src/index.js`.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Production install docs do not expose the agent publicly by default.
|
|
- Installer warns on insecure network exposure.
|
|
- systemd unit has a documented minimum hardening baseline.
|
|
|
|
## 12. Improve agent crash cleanup and resource recovery
|
|
|
|
Agent jobs are persisted and active jobs are marked `failed` on restart, but a process crash can still leave host resources behind, such as temporary Incus snapshots, staged restore volumes, visible snapshot devices, or changed ZFS properties.
|
|
|
|
### Goal
|
|
|
|
Make agent startup detect and clean up known leftover resources from interrupted jobs where doing so is safe.
|
|
|
|
### Tasks
|
|
|
|
- Persist cleanup metadata for backup jobs: temporary Incus snapshot name, ZVOL, snapdev state, and backup type.
|
|
- Persist cleanup metadata for restore jobs: staged ZVOL, backup ZVOL, failed ZVOL, and swap state.
|
|
- On startup, scan failed active jobs and run safe cleanup actions.
|
|
- Log cleanup results into the persisted job logs.
|
|
- Avoid destructive cleanup when state is ambiguous; surface manual action instead.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Crashing during backup streaming does not leave `snapdev=visible` or temporary snapshots unnoticed.
|
|
- Crashing before restore swap removes staged restore volumes when safe.
|
|
- Crashing after restore swap does not automatically destroy rollback copies.
|
|
- Management can show cleanup-required states.
|
|
|
|
## 13. Improve health diagnostics in the UI
|
|
|
|
The agent health endpoint returns useful structured details, but the management UI currently summarizes this too aggressively. Operators need actionable health reasons without opening logs.
|
|
|
|
### Goal
|
|
|
|
Show per-node health detail in the UI with concrete failed checks and messages.
|
|
|
|
### Tasks
|
|
|
|
- Add a detailed health drawer or modal on the Nodes page.
|
|
- Show command, ZFS pool, `/dev/zvol`, Restic repository, and credential check results.
|
|
- Show last health timestamp and the management-side error if the agent is unreachable.
|
|
- Distinguish unreachable, unauthorized, TLS failure, degraded health, and healthy states.
|
|
- Avoid global UI timeout banners when only one enabled node is slow or unreachable.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- A missing command is visible by name in the UI.
|
|
- Wrong Restic credentials are visible as a Restic health failure.
|
|
- TLS or connectivity failures are distinguishable from degraded agent health.
|