243 lines
9.7 KiB
Markdown
243 lines
9.7 KiB
Markdown
# Issue Backlog
|
|
|
|
## Status snapshot
|
|
|
|
Last reviewed against code: 2026-06-04.
|
|
|
|
Several earlier backlog items have already landed in the codebase:
|
|
|
|
- Management persists accepted agent jobs in `job_history`.
|
|
- Management polls agent jobs and updates final status, error, finish time, duration, current step, and parsed backup snapshot ID.
|
|
- Operations page shows job history and audit log.
|
|
- Node-agent health checks cover required commands, ZFS pool state, `/dev/zvol`, and Restic repository access.
|
|
- Backup jobs verify the stored Restic file size and remove failed snapshots on verification errors.
|
|
|
|
Known caveat: agent-side jobs are persisted, but subprocesses cannot survive an agent restart. Active jobs are marked `failed` on startup; deeper cleanup recovery for partially changed host resources is still future work.
|
|
|
|
## 1. Persist final agent job status in management
|
|
|
|
Status: mostly done.
|
|
|
|
Management records when a backup or restore was accepted by an agent and now persists the final agent job result when the node-agent remains reachable long enough to be polled.
|
|
|
|
### Goal
|
|
|
|
Persist reliable end-to-end job status in the management database.
|
|
|
|
### Tasks
|
|
|
|
- [x] Add polling for accepted agent jobs from management.
|
|
- [x] Store final `success` or `failed` status in `job_history`.
|
|
- [x] Store duration, finished timestamp, error message, and current step.
|
|
- [ ] Store agent job logs summary.
|
|
- [x] Store created Restic snapshot ID for successful backup jobs when available.
|
|
- [x] Surface final status in the Operations page.
|
|
- [x] Persist recent agent-side jobs.
|
|
- [x] Mark active agent jobs as `failed` after an agent restart so management can poll a final state.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- [x] A backup started through management eventually shows `success` or `failed` while the agent remains reachable.
|
|
- [x] A restore started through management eventually shows `success` or `failed` while the agent remains reachable.
|
|
- [x] Management restart does not lose already persisted history.
|
|
- [x] Agent restart does not lose the active job record; active work is marked `failed` because the subprocess cannot survive restart.
|
|
|
|
## 2. Expand node-agent health checks
|
|
|
|
Status: mostly done.
|
|
|
|
The health endpoint now provides deeper operational checks for backup readiness.
|
|
|
|
### Goal
|
|
|
|
Make `/api/health` useful for diagnosing whether a node can actually run backup and restore operations.
|
|
|
|
### Tasks
|
|
|
|
- [x] Check that required commands exist: `incus`, `zfs`, `zpool`, `restic`, `udevadm`, `dd`.
|
|
- [x] Check that configured ZFS pool exists.
|
|
- [x] Check that `/dev/zvol` is accessible.
|
|
- [x] Check Restic repository access.
|
|
- [x] Check S3/Restic credentials by running a safe Restic command.
|
|
- [x] Include ZFS pool capacity and free space.
|
|
- [x] Return structured check names and messages.
|
|
- [ ] Surface detailed per-node health output in the management UI, not just a compact aggregate.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- [~] Management UI shows degraded node health with actionable check names. Detailed output exists through node health endpoints; UI can still be improved.
|
|
- [x] A missing command, wrong pool, or wrong Restic credentials is visible in health output.
|
|
|
|
## 3. Harden backup verification
|
|
|
|
Status: partially done.
|
|
|
|
Backup jobs now verify the backed-up Restic file size after `restic backup` and attempt to remove failed snapshots. This reduces the risk of accepting a truncated backup, but the implementation still needs stronger stream failure handling and broader verification semantics.
|
|
|
|
### Goal
|
|
|
|
Ensure a backup is marked `success` only when the expected source data was fully stored and verified.
|
|
|
|
### Tasks
|
|
|
|
- [x] Parse the created Restic snapshot ID after backup.
|
|
- [x] Verify stored Restic file size against streamed source bytes or expected ZFS size.
|
|
- [x] Remove failed Restic snapshots with `forget` and `prune` on verification errors.
|
|
- [ ] Make stream/pipeline failure handling explicit for all sources.
|
|
- [ ] Avoid marking success if source stream closes early but Restic exits successfully.
|
|
- [ ] Add tests with mocked command failures and short reads.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- [x] Successful backup jobs include a verifiable Restic snapshot ID in management history.
|
|
- [x] Size mismatches fail the job.
|
|
- [ ] Simulated source stream errors cannot produce a successful job.
|
|
- [ ] Verification behavior is covered by automated tests.
|
|
|
|
## 4. Add per-VM backup policy
|
|
|
|
Retention and schedule behavior is currently broad. Per-VM policies would make production usage more flexible.
|
|
|
|
Current state: schedules can be enabled per VM with interval and time of day. Retention is still global per agent.
|
|
|
|
### Goal
|
|
|
|
Allow each VM to define its own backup policy.
|
|
|
|
### Tasks
|
|
|
|
- Add management database table for VM backup policies.
|
|
- Support per-VM retention values: hourly, daily, weekly, monthly.
|
|
- [x] Support per-VM schedule enablement, interval, and time.
|
|
- Add optional policy flag: backup only when VM is running.
|
|
- Update Scheduler UI to edit policies per node and VM.
|
|
- Send policy retention to the agent backup request or apply it in management scheduling.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Two VMs on the same node can have different schedules.
|
|
- Two VMs on the same node can have different retention policies.
|
|
- Disabled policies do not trigger backups.
|
|
|
|
## 5. Improve restore safety workflow
|
|
|
|
Restore is destructive and should be guarded with a clearer preflight and confirmation flow.
|
|
|
|
Current state: VM restore requires typed confirmation, validates the snapshot against the VM, checks source/target size, writes the Restic dump to a staged ZFS volume, stops the VM, and swaps the staged volume into place with `zfs rename`. The previous production volume is kept as a rollback copy.
|
|
|
|
### Goal
|
|
|
|
Reduce the risk of accidental or unsafe restores.
|
|
|
|
### Tasks
|
|
|
|
- Add restore preflight endpoint.
|
|
- Validate node health before restore.
|
|
- Show selected snapshot metadata before restore.
|
|
- Show target VM status and disk size before restore.
|
|
- [x] Require explicit typed confirmation including VM name.
|
|
- Optionally offer “create backup before restore” when the VM is accessible.
|
|
- [x] Log restore intent in audit log before dispatch.
|
|
- [x] Replace direct `dd` to production ZVOL with a staged restore workflow.
|
|
- [ ] Define a safe container restore workflow with `zfs receive` or disable container restore surfaces entirely.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- UI displays a restore plan before the final confirmation.
|
|
- Restore is blocked when required preflight checks fail.
|
|
- Audit log records restore attempts and results.
|
|
- [x] Restore stream failures happen on the staged volume, not the production disk.
|
|
|
|
## 6. Add failure notifications
|
|
|
|
Operators need to know when scheduled backups or restores fail.
|
|
|
|
### Goal
|
|
|
|
Send notifications for failed or degraded operations.
|
|
|
|
### Tasks
|
|
|
|
- Add notification settings in management.
|
|
- Support webhook notifications first.
|
|
- Include node, VM, job type, error, and timestamp.
|
|
- Trigger notifications for failed scheduled backups.
|
|
- Trigger notifications for failed manual backups/restores.
|
|
- Add a “send test notification” action.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- A failed scheduled backup sends one notification.
|
|
- A test notification can be triggered from the UI.
|
|
- Notification failures are visible in Operations or audit logs.
|
|
|
|
## 7. Implement real snapshot file browsing
|
|
|
|
Current snapshot browsing shows Restic contents, which for block-level backups is usually only `/vm.raw`.
|
|
|
|
### Goal
|
|
|
|
Allow browsing files inside a backed-up VM disk image.
|
|
|
|
### Tasks
|
|
|
|
- Design a safe read-only raw image inspection workflow.
|
|
- Restore or mount raw image read-only in a temporary workspace.
|
|
- Detect partitions and filesystems.
|
|
- Browse directories through management UI.
|
|
- Allow downloading a single file.
|
|
- Ensure cleanup of mounts and temporary files.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- User can browse a Linux VM filesystem from a snapshot without restoring the VM.
|
|
- Mounted/temporary resources are cleaned up after use.
|
|
- The workflow refuses unsupported or unsafe disk images with a clear error.
|
|
|
|
## 8. Add agent and management version reporting
|
|
|
|
Multi-node setups need version visibility.
|
|
|
|
### Goal
|
|
|
|
Show software version and compatibility state for management and each agent.
|
|
|
|
### Tasks
|
|
|
|
- Add version field to management API.
|
|
- Add version field to agent health response.
|
|
- Show agent version in Nodes page.
|
|
- Flag unsupported or outdated agents.
|
|
- Document compatibility expectations.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Nodes page displays agent version.
|
|
- Management can identify incompatible agents.
|
|
- Health output includes version information.
|
|
|
|
## 9. Containerized management and UI deployment
|
|
|
|
The node-agent must remain a host-level systemd service because it needs direct Incus, ZFS, `/dev/zvol`, Restic, and device access. Management and the frontend do not need those host privileges and are good candidates for container deployment.
|
|
|
|
### Goal
|
|
|
|
Provide a production-ready Docker/Compose deployment for the management API and frontend while keeping node-agents installed as systemd services on Incus hosts.
|
|
|
|
### Tasks
|
|
|
|
- Add a `management` container image.
|
|
- Add a frontend image that serves the Vite build through a small static server or Nginx.
|
|
- Provide `compose.yaml` with persistent SQLite volume for management.
|
|
- Mount the agent CA certificate into the management container as read-only.
|
|
- Document required `CORS_ORIGINS`, `SESSION_COOKIE_SECURE`, `DATABASE_PATH`, and reverse-proxy assumptions.
|
|
- Decide whether the frontend calls the management API through the same origin reverse proxy or a separate API origin.
|
|
- Add health checks for both containers.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Management API and frontend can be started with Compose without installing Node.js on the management host.
|
|
- Management database survives container recreation.
|
|
- Cookie login works behind HTTPS.
|
|
- Management can connect to HTTPS node-agents using the configured CA file.
|