philipp/incus-backup-ui
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

merged issues and todo.

Browse Source 
removed todo
This commit is contained in:
Philipp
2026-06-04 15:51:59 +02:00
parent 8a80978aef
commit 507945fd87
2 changed files with 406 additions and 489 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/issues.md
+406 -261
View File
@@ -1,103 +1,57 @@
# Issue Backlog
## Status snapshot
Last reviewed against code: 2026-06-04.
Several earlier backlog items have already landed in the codebase:
This is the single source of truth for product work, hardening, release tasks, and production blockers. The previous `docs/fixes-todo.md` list has been merged into this file.
## Current Status
Implemented and verified in code:
- Management persists accepted agent jobs in `job_history`.
- Management polls agent jobs and updates final status, error, finish time, duration, current step, and parsed backup snapshot ID.
- Operations page shows job history and audit log.
- Node-agent health checks cover required commands, ZFS pool state, `/dev/zvol`, and Restic repository access.
- Backup jobs verify the stored Restic file size and remove failed snapshots on verification errors.
- Backup jobs parse the Restic snapshot ID, verify stored file size, and remove failed snapshots on verification errors.
- VM restore no longer writes directly to the production ZVOL. It writes to a staged ZVOL first, then swaps volumes with `zfs rename`.
- Agent jobs are persisted in SQLite. Active jobs are marked `failed` after an agent restart because subprocesses cannot survive restart.
- Auth hardening landed: no default admin password, required agent token, CORS allowlist, `SameSite=Strict`, login brute-force protection, no frontend localStorage bearer token.
- Settings secrets are write-only/masked in the UI and API.
- Agent bearer-token comparison is timing-safe.
- Agent directory was renamed from `backend/` to `agent/`.
Known caveat: agent-side jobs are persisted, but subprocesses cannot survive an agent restart. Active jobs are marked `failed` on startup; deeper cleanup recovery for partially changed host resources is still future work.
Known caveat: active agent jobs are persisted, but deeper cleanup recovery for partially changed host resources is still future work.
Current pre-production priorities:
## Pre-Production Priorities
1. Harden backup verification and add tests for stream/pipeline failure cases.
2. Validate the staged restore workflow on a disposable Incus VM, including rollback scenarios.
3. Improve agent crash cleanup for partially changed ZFS/Incus resources.
4. Surface detailed node health diagnostics in the UI.
5. Keep the root-running agent tightly network-restricted.
3. Add cleanup and visibility for pre-restore/failed-restore ZVOLs.
4. Improve agent crash cleanup for partially changed ZFS/Incus resources.
5. Surface detailed node health diagnostics in the UI.
6. Keep the root-running agent tightly network-restricted.
7. Add automated tests and CI.
## 1. Persist final agent job status in management
## P0 - Production Blockers
Status: mostly done.
Management records when a backup or restore was accepted by an agent and now persists the final agent job result when the node-agent remains reachable long enough to be polled.
### Goal
Persist reliable end-to-end job status in the management database.
### Tasks
- [x] Add polling for accepted agent jobs from management.
- [x] Store final `success` or `failed` status in `job_history`.
- [x] Store duration, finished timestamp, error message, and current step.
- [ ] Store agent job logs summary.
- [x] Store created Restic snapshot ID for successful backup jobs when available.
- [x] Surface final status in the Operations page.
- [x] Persist recent agent-side jobs.
- [x] Mark active agent jobs as `failed` after an agent restart so management can poll a final state.
### Acceptance Criteria
- [x] A backup started through management eventually shows `success` or `failed` while the agent remains reachable.
- [x] A restore started through management eventually shows `success` or `failed` while the agent remains reachable.
- [x] Management restart does not lose already persisted history.
- [x] Agent restart does not lose the active job record; active work is marked `failed` because the subprocess cannot survive restart.
## 2. Expand node-agent health checks
Status: mostly done.
The health endpoint now provides deeper operational checks for backup readiness.
### Goal
Make `/api/health` useful for diagnosing whether a node can actually run backup and restore operations.
### Tasks
- [x] Check that required commands exist: `incus`, `zfs`, `zpool`, `restic`, `udevadm`, `dd`.
- [x] Check that configured ZFS pool exists.
- [x] Check that `/dev/zvol` is accessible.
- [x] Check Restic repository access.
- [x] Check S3/Restic credentials by running a safe Restic command.
- [x] Include ZFS pool capacity and free space.
- [x] Return structured check names and messages.
- [ ] Surface detailed per-node health output in the management UI, not just a compact aggregate.
### Acceptance Criteria
- [~] Management UI shows degraded node health with actionable check names. Detailed output exists through node health endpoints; UI can still be improved.
- [x] A missing command, wrong pool, or wrong Restic credentials is visible in health output.
## 3. Harden backup verification
### 1. Harden Backup Verification
Status: partially done.
Backup jobs now verify the backed-up Restic file size after `restic backup` and attempt to remove failed snapshots. This reduces the risk of accepting a truncated backup, but the implementation still needs stronger stream failure handling and broader verification semantics.
Goal: ensure a backup is marked `success` only when the expected source data was fully stored and verified.
### Goal
Ensure a backup is marked `success` only when the expected source data was fully stored and verified.
### Tasks
Tasks:
- [x] Parse the created Restic snapshot ID after backup.
- [x] Verify stored Restic file size against streamed source bytes or expected ZFS size.
- [x] Remove failed Restic snapshots with `forget` and `prune` on verification errors.
- [ ] Make stream/pipeline failure handling explicit for all sources.
- [ ] Avoid marking success if source stream closes early but Restic exits successfully.
- [ ] Add a post-backup Restic integrity check strategy, such as targeted `restic check`/`restic stats` usage that is safe for large repositories.
- [ ] Make stream/pipeline failure handling explicit for VM and container sources.
- [ ] Avoid marking success if the source stream closes early but Restic exits successfully.
- [ ] Add a post-backup Restic integrity-check strategy that is safe for large repositories.
- [ ] Define an optional restore-probe workflow for critical VMs.
- [ ] Add tests with mocked command failures and short reads.
- [ ] Add tests with mocked command failures, pipe failures, and short reads.
### Acceptance Criteria
Acceptance criteria:
- [x] Successful backup jobs include a verifiable Restic snapshot ID in management history.
- [x] Size mismatches fail the job.
@@ -105,245 +59,436 @@ Ensure a backup is marked `success` only when the expected source data was fully
- [ ] A backup can be independently verified without trusting only the successful process exit.
- [ ] Verification behavior is covered by automated tests.
## 4. Add per-VM backup policy
### 2. Validate Staged Restore Workflow
Retention and schedule behavior is currently broad. Per-VM policies would make production usage more flexible.
Status: implemented, not proven on real Incus/ZFS hardware in this repo session.
Goal: prove restore safety before production use.
Tasks:
- [x] Require explicit typed confirmation including VM name.
- [x] Validate snapshot ownership before restore.
- [x] Replace direct `dd` to production ZVOL with a staged restore workflow.
- [x] Log restore intent in audit log before dispatch.
- [ ] Add restore preflight endpoint.
- [ ] Validate node health before restore.
- [ ] Show selected snapshot metadata before restore.
- [ ] Show target VM status and disk size before restore.
- [ ] Optionally offer “create backup before restore” when the VM is accessible.
- [ ] Test staged restore on a disposable VM: backup, restore, boot, and confirm data.
- [ ] Test failed staged restore paths: Restic dump failure before swap, rename failure after old volume rename, and VM start failure.
Acceptance criteria:
- [x] Restore stream failures happen on the staged volume, not the production disk.
- [ ] UI displays a restore plan before final confirmation.
- [ ] Restore is blocked when required preflight checks fail.
- [ ] Staged restore behavior is validated on a real Incus/ZFS test node.
### 3. Add Pre-Restore Volume Cleanup
Status: open.
Current behavior: every successful VM restore keeps the old volume as `*.pre-restore-*`. This is good for rollback, but without retention or cleanup it can fill the ZFS pool.
Tasks:
- [ ] Track pre-restore and failed-restore volumes created by the agent.
- [ ] Add a retention policy, for example keep the last N rollback volumes per VM.
- [ ] Surface retained rollback volumes in Operations or VM detail.
- [ ] Add explicit cleanup action with confirmation.
- [ ] Add health warning when retained restore volumes consume significant pool space.
Acceptance criteria:
- [ ] Repeated restores cannot silently consume unbounded pool space.
- [ ] Operators can see and clean retained restore volumes.
### 4. Harden Root-Running Agent Exposure
Status: partially done.
The agent runs with root-level host access because it needs Incus, ZFS, `/dev/zvol`, Restic, and device operations. A compromised agent is a host-level incident.
Tasks:
- [x] Require a non-empty `API_TOKEN` of at least 32 characters.
- [x] Support `ALLOWED_MANAGEMENT_IPS`.
- [x] Support HTTPS directly on the agent.
- [x] Support private CA validation from management via `AGENT_CA_FILE`.
- [ ] Require private management-to-agent connectivity in production docs, such as NetBird or a dedicated management network.
- [ ] Prefer HTTPS agents with a private CA; document local HTTP only as development mode.
- [ ] Add installer warning when `HTTPS_ENABLED=false` and no `ALLOWED_MANAGEMENT_IPS` is configured.
- [ ] Add systemd hardening where compatible with Incus/ZFS access.
- [ ] Replace `npm start` in systemd with direct `node src/index.js`.
- [ ] Remove CORS from the agent; browsers should never call it directly.
Acceptance criteria:
- [ ] Production install docs do not expose the agent publicly by default.
- [ ] Installer warns on insecure network exposure.
- [ ] systemd unit has a documented minimum hardening baseline.
### 5. Encrypt or Protect Agent Tokens in Management
Status: open.
Current behavior: `nodes.token` is stored in plaintext in the management SQLite database. DB theft means all agent tokens are exposed.
Tasks:
- [ ] Decide between at-rest encryption with a dedicated management key or strict documented file/disk protection.
- [ ] If encrypting, add key configuration and migration for existing tokens.
- [ ] Ensure public node API responses never include tokens.
- [ ] Document database file permissions and disk-encryption expectations.
Acceptance criteria:
- [ ] Stealing the management database alone is not enough to read every agent token, or the residual risk is explicitly documented and operationally mitigated.
## P1 - High Priority
### 6. Add Automated Tests and CI
Status: open.
Tasks:
- [ ] Add a test runner for agent and management code.
- [ ] Test validators for instance names, snapshot IDs, and ambiguous snapshot prefixes.
- [ ] Test job locking, persistence, trimming, and restart behavior.
- [ ] Test backup pipeline failure handling with mocked command/process failures.
- [ ] Test restore staged-volume command sequencing with mocked ZFS/Incus/Restic commands.
- [ ] Test management job polling updates `job_history` for success, failed, timeout, and missing-agent cases.
- [ ] Add CI for install, tests, frontend build, and syntax checks.
Acceptance criteria:
- [ ] Pull requests run tests automatically.
- [ ] Simulated backup stream failures fail the job.
- [ ] Simulated restore failures leave the original ZVOL name restored in the command sequence.
- [ ] Agent restart behavior is covered by tests.
### 7. Improve Agent Crash Cleanup and Resource Recovery
Status: open.
Agent jobs are persisted and active jobs are marked `failed` on restart, but a process crash can still leave host resources behind.
Tasks:
- [ ] Persist cleanup metadata for backup jobs: temporary Incus snapshot name, ZVOL, snapdev state, and backup type.
- [ ] Persist cleanup metadata for restore jobs: staged ZVOL, backup ZVOL, failed ZVOL, and swap state.
- [ ] On startup, scan failed active jobs and run safe cleanup actions.
- [ ] Log cleanup results into the persisted job logs.
- [ ] Avoid destructive cleanup when state is ambiguous; surface manual action instead.
Acceptance criteria:
- [ ] Crashing during backup streaming does not leave `snapdev=visible` or temporary snapshots unnoticed.
- [ ] Crashing before restore swap removes staged restore volumes when safe.
- [ ] Crashing after restore swap does not automatically destroy rollback copies.
- [ ] Management can show cleanup-required states.
### 8. Improve Health Diagnostics in the UI
Status: partially done on API, open in UI.
Tasks:
- [x] Agent health checks commands, ZFS pool, `/dev/zvol`, and Restic repository.
- [x] Management records compact node health state.
- [ ] Add a detailed health drawer or modal on the Nodes page.
- [ ] Show command, ZFS pool, `/dev/zvol`, Restic repository, and credential check results.
- [ ] Show last health timestamp and the management-side error if the agent is unreachable.
- [ ] Distinguish unreachable, unauthorized, TLS failure, degraded health, and healthy states.
- [ ] Avoid global UI timeout banners when only one enabled node is slow or unreachable.
Acceptance criteria:
- [ ] A missing command is visible by name in the UI.
- [ ] Wrong Restic credentials are visible as a Restic health failure.
- [ ] TLS or connectivity failures are distinguishable from degraded agent health.
### 9. Fix Snapshot Device Visibility Race
Status: open.
Current behavior: backup waits a fixed 2 seconds after setting `snapdev=visible`.
Tasks:
- [ ] Replace fixed sleep with polling `fs.access(snapshotDevice)` and timeout.
- [ ] Run `udevadm trigger` and `udevadm settle`.
- [ ] Surface timeout as a clear backup failure.
Acceptance criteria:
- [ ] Slow device-node creation does not fail randomly.
### 10. Make Snapshot-ID Prefix Matching Unambiguous
Status: open.
Current behavior: prefix matching uses `startsWith`; collisions choose the first match.
Tasks:
- [ ] Return conflict when a snapshot prefix matches more than one snapshot.
- [ ] Prefer 12+ hex characters in the UI.
- [ ] Add validator tests.
Acceptance criteria:
- [ ] Ambiguous snapshot prefixes cannot restore the wrong snapshot.
### 11. Settings and Environment Hardening
Status: partially done.
Tasks:
- [x] Mask secret values in `GET /settings`.
- [x] Treat empty secret field as “keep existing value”.
- [ ] Improve `.env` escaping for `$`, backticks, and newlines.
- [ ] Add tests for `readEnvSettings` and `writeEnvSettings`.
- [ ] Document `.env` file permissions and enforce `chmod 600` in installer/deployment.
- [ ] Keep or remove `SESSION_SECRET` deliberately: either use it for signing/encryption or remove dead config.
- [ ] Set and document `SESSION_COOKIE_SECURE=true` for production.
Acceptance criteria:
- [ ] Secret settings cannot be exfiltrated through the UI/API.
- [ ] Writing `.env` cannot create shell-expansion surprises.
### 12. Add RBAC
Status: open.
Current behavior: one user model, effectively all permissions.
Tasks:
- [ ] Add roles: `admin`, `operator`, `viewer`.
- [ ] Restrict restore and node settings to `admin`.
- [ ] Allow read-only dashboard access for `viewer`.
- [ ] Audit role changes.
Acceptance criteria:
- [ ] Restore is admin-only.
- [ ] Viewer cannot trigger backup, restore, node edits, or settings writes.
### 13. Session and Auth Cleanup
Status: partially done.
Tasks:
- [x] Login brute-force protection.
- [x] `SameSite=Strict`.
- [ ] Server-side cleanup of expired sessions.
- [ ] Session rotation after successful login.
- [ ] Replace `Math.random()` in management user ID creation with `crypto.randomBytes`.
- [ ] Consider argon2id or explicit scrypt parameters and migration path.
Acceptance criteria:
- [ ] Expired sessions do not accumulate unbounded in SQLite.
- [ ] Session IDs are rotated after login.
### 14. Container Restore Decision
Status: open.
Current behavior: container backups can be created, but restore returns `501`.
Tasks:
- [ ] Implement safe container restore with `zfs receive`, or disable container backups in UI/API until restore exists.
- [ ] Document the limitation prominently.
- [ ] Add tests for unsupported restore behavior.
Acceptance criteria:
- [ ] Users cannot mistake an unrestorable container backup for complete protection.
## P2 - Product and Operations
### 15. Add Per-VM Backup Policy
Status: partially done.
Current state: schedules can be enabled per VM with interval and time of day. Retention is still global per agent.
### Goal
Tasks:
Allow each VM to define its own backup policy.
### Tasks
- Add management database table for VM backup policies.
- Support per-VM retention values: hourly, daily, weekly, monthly.
- [ ] Add management database table for VM backup policies.
- [ ] Support per-VM retention values: hourly, daily, weekly, monthly.
- [x] Support per-VM schedule enablement, interval, and time.
- Add optional policy flag: backup only when VM is running.
- Update Scheduler UI to edit policies per node and VM.
- Send policy retention to the agent backup request or apply it in management scheduling.
- [ ] Add optional policy flag: backup only when VM is running.
- [ ] Update Scheduler UI to edit policies per node and VM.
- [ ] Send policy retention to the agent backup request or apply it in management scheduling.
### Acceptance Criteria
Acceptance criteria:
- Two VMs on the same node can have different schedules.
- Two VMs on the same node can have different retention policies.
- Disabled policies do not trigger backups.
- [ ] Two VMs on the same node can have different schedules.
- [ ] Two VMs on the same node can have different retention policies.
- [ ] Disabled policies do not trigger backups.
## 5. Improve restore safety workflow
### 16. Add Failure Notifications
Restore is destructive and should be guarded with a clearer preflight and confirmation flow.
Status: open.
Current state: VM restore requires typed confirmation, validates the snapshot against the VM, checks source/target size, writes the Restic dump to a staged ZFS volume, stops the VM, and swaps the staged volume into place with `zfs rename`. The previous production volume is kept as a rollback copy.
Tasks:
### Goal
- [ ] Add notification settings in management.
- [ ] Support webhook notifications first.
- [ ] Include node, VM, job type, error, and timestamp.
- [ ] Trigger notifications for failed scheduled backups.
- [ ] Trigger notifications for failed manual backups/restores.
- [ ] Add a “send test notification” action.
Reduce the risk of accidental or unsafe restores.
Acceptance criteria:
### Tasks
- [ ] A failed scheduled backup sends one notification.
- [ ] A test notification can be triggered from the UI.
- [ ] Notification failures are visible in Operations or audit logs.
- Add restore preflight endpoint.
- Validate node health before restore.
- Show selected snapshot metadata before restore.
- Show target VM status and disk size before restore.
- [x] Require explicit typed confirmation including VM name.
- Optionally offer “create backup before restore” when the VM is accessible.
- [x] Log restore intent in audit log before dispatch.
- [x] Replace direct `dd` to production ZVOL with a staged restore workflow.
- [ ] Test staged restore on a disposable VM: backup, restore, boot, and confirm data.
- [ ] Test failed staged restore paths: Restic dump failure before swap, rename failure after old volume rename, and VM start failure.
- [ ] Define a safe container restore workflow with `zfs receive` or disable container restore surfaces entirely.
### 17. Implement Real Snapshot File Browsing
### Acceptance Criteria
- UI displays a restore plan before the final confirmation.
- Restore is blocked when required preflight checks fail.
- Audit log records restore attempts and results.
- [x] Restore stream failures happen on the staged volume, not the production disk.
- [ ] Staged restore behavior is validated on a real Incus/ZFS test node.
## 6. Add failure notifications
Operators need to know when scheduled backups or restores fail.
### Goal
Send notifications for failed or degraded operations.
### Tasks
- Add notification settings in management.
- Support webhook notifications first.
- Include node, VM, job type, error, and timestamp.
- Trigger notifications for failed scheduled backups.
- Trigger notifications for failed manual backups/restores.
- Add a “send test notification” action.
### Acceptance Criteria
- A failed scheduled backup sends one notification.
- A test notification can be triggered from the UI.
- Notification failures are visible in Operations or audit logs.
## 7. Implement real snapshot file browsing
Status: open.
Current snapshot browsing shows Restic contents, which for block-level backups is usually only `/vm.raw`.
### Goal
Tasks:
Allow browsing files inside a backed-up VM disk image.
- [ ] Design a safe read-only raw image inspection workflow.
- [ ] Restore or mount raw image read-only in a temporary workspace.
- [ ] Detect partitions and filesystems.
- [ ] Browse directories through management UI.
- [ ] Allow downloading a single file.
- [ ] Ensure cleanup of mounts and temporary files.
### Tasks
Acceptance criteria:
- Design a safe read-only raw image inspection workflow.
- Restore or mount raw image read-only in a temporary workspace.
- Detect partitions and filesystems.
- Browse directories through management UI.
- Allow downloading a single file.
- Ensure cleanup of mounts and temporary files.
- [ ] User can browse a Linux VM filesystem from a snapshot without restoring the VM.
- [ ] Mounted/temporary resources are cleaned up after use.
- [ ] Unsupported or unsafe disk images fail with a clear error.
### Acceptance Criteria
### 18. Add Version Reporting
- User can browse a Linux VM filesystem from a snapshot without restoring the VM.
- Mounted/temporary resources are cleaned up after use.
- The workflow refuses unsupported or unsafe disk images with a clear error.
Status: open.
## 8. Add agent and management version reporting
Tasks:
Multi-node setups need version visibility.
- [ ] Add version field to management API.
- [ ] Add version field to agent health response.
- [ ] Show agent version in Nodes page.
- [ ] Flag unsupported or outdated agents.
- [ ] Document compatibility expectations.
### Goal
Acceptance criteria:
Show software version and compatibility state for management and each agent.
- [ ] Nodes page displays agent version.
- [ ] Management can identify incompatible agents.
- [ ] Health output includes version information.
### Tasks
### 19. Containerized Management and UI Deployment
- Add version field to management API.
- Add version field to agent health response.
- Show agent version in Nodes page.
- Flag unsupported or outdated agents.
- Document compatibility expectations.
Status: open.
### Acceptance Criteria
The node-agent remains a host-level systemd service. Management and frontend are good candidates for containers.
- Nodes page displays agent version.
- Management can identify incompatible agents.
- Health output includes version information.
Tasks:
## 9. Containerized management and UI deployment
- [ ] Add a `management` container image.
- [ ] Add a frontend image that serves the Vite build through a small static server or Nginx.
- [ ] Provide `compose.yaml` with persistent SQLite volume for management.
- [ ] Mount the agent CA certificate into the management container read-only.
- [ ] Document `CORS_ORIGINS`, `SESSION_COOKIE_SECURE`, `DATABASE_PATH`, and reverse-proxy assumptions.
- [ ] Decide whether frontend calls management through same-origin `/api` or separate API origin.
- [ ] Add health checks for both containers.
The node-agent must remain a host-level systemd service because it needs direct Incus, ZFS, `/dev/zvol`, Restic, and device access. Management and the frontend do not need those host privileges and are good candidates for container deployment.
Acceptance criteria:
### Goal
- [ ] Management API and frontend can be started with Compose without installing Node.js on the management host.
- [ ] Management database survives container recreation.
- [ ] Cookie login works behind HTTPS.
- [ ] Management can connect to HTTPS node-agents using the configured CA file.
Provide a production-ready Docker/Compose deployment for the management API and frontend while keeping node-agents installed as systemd services on Incus hosts.
### 20. Backup Scheduling and Quotas
### Tasks
Status: open.
- Add a `management` container image.
- Add a frontend image that serves the Vite build through a small static server or Nginx.
- Provide `compose.yaml` with persistent SQLite volume for management.
- Mount the agent CA certificate into the management container as read-only.
- Document required `CORS_ORIGINS`, `SESSION_COOKIE_SECURE`, `DATABASE_PATH`, and reverse-proxy assumptions.
- Decide whether the frontend calls the management API through the same origin reverse proxy or a separate API origin.
- Add health checks for both containers.
Tasks:
### Acceptance Criteria
- [ ] Add scheduler jitter in management and agent schedulers.
- [ ] Add per-VM and global backup concurrency limits.
- [ ] Add disk-space/quota checks before backup and restore.
- [ ] Add pre-backup VM state checks for live migration, existing snapshots, and permission problems.
- Management API and frontend can be started with Compose without installing Node.js on the management host.
- Management database survives container recreation.
- Cookie login works behind HTTPS.
- Management can connect to HTTPS node-agents using the configured CA file.
Acceptance criteria:
## 10. Add automated tests and CI
- [ ] Many nodes do not all start scheduled backups at exactly the same time.
- [ ] A full pool or too many concurrent streams blocks new jobs with a clear error.
The project currently has no automated tests. This is the biggest engineering gap for a backup system because most dangerous failures happen in error paths, not in the happy path.
## P3 - Cleanup and Refactoring
### Goal
### 21. Systemd and Deployment Cleanup
Catch regressions in validation, locking, backup verification, restore orchestration, authentication, and management polling before deployment.
Tasks:
### Tasks
- [ ] Add systemd hardening compatible with Incus/ZFS access.
- [ ] Use direct `node src/index.js` in service files instead of `npm start`.
- [ ] Make `SCHEDULES_PATH` explicitly configurable.
- [ ] Installer should warn on insecure agent exposure.
- Add a test runner for agent and management code.
- Test validators for instance names, snapshot IDs, and ambiguous snapshot prefixes.
- Test job locking, persistence, trimming, and restart behavior.
- Test backup pipeline failure handling with mocked command/process failures.
- Test restore staged-volume command sequencing with mocked ZFS/Incus/Restic commands.
- Test management job polling updates `job_history` correctly for success, failed, timeout, and missing-agent cases.
- Add CI for install, tests, frontend build, and syntax checks.
### 22. Data Integrity and Schema Cleanup
### Acceptance Criteria
Tasks:
- Pull requests run tests automatically.
- Simulated backup stream failures fail the job.
- Simulated restore failures leave the original ZVOL name restored in the command sequence.
- Agent restart behavior is covered by tests.
- [ ] Add audit-log integrity, such as append-only behavior and optional hash chain.
- [ ] Add `audit_events.details` size limit or JSON column handling.
- [ ] Replace ad-hoc `addColumnIfMissing` with schema versioning and migrations.
- [ ] Stream Restic `ls` instead of loading all output in RAM.
## 11. Harden root-running agent exposure
### 23. Code Cleanup
The agent runs with root-level host access because it needs Incus, ZFS, `/dev/zvol`, Restic, and device operations. A compromised agent is therefore a host-level incident.
Tasks:
### Goal
- [ ] Generalize `incus snapshot delete` retry instead of matching English stderr.
- [ ] Deduplicate `formatBytes`.
- [ ] Review `incus-backup-ui-plan.md` for stale paths, secrets, and obsolete implementation notes.
- [ ] Remove tracked `.DS_Store` files.
- [ ] Ensure `frontend/dist/` remains ignored and untracked.
Reduce the network and systemd blast radius of the root-running node-agent.
## Done
### Tasks
- [x] Default admin `admin/admin` fallback removed.
- [x] Agent `API_TOKEN` required and minimum length enforced.
- [x] HTTPS required for agent URLs in management, with dev-only insecure HTTP escape hatch.
- [x] Management can trust internal agent CA through `AGENT_CA_FILE`.
- [x] Agent supports direct HTTPS.
- [x] CORS allowlist added for management.
- [x] Auth cookie set to `SameSite=Strict`.
- [x] Login brute-force protection added.
- [x] Bearer token removed from frontend localStorage.
- [x] Settings secrets masked/write-only.
- [x] Agent token comparison made timing-safe.
- [x] Agent job persistence added.
- [x] Staged VM restore implemented.
- [x] Node-agent renamed from `backend/` to `agent/`.
- [x] Agent installer script added.
- [x] Backup UI ETA/rate/bytes display added.
- Require private management-to-agent connectivity in production documentation, such as NetBird or a dedicated management network.
- Prefer HTTPS agents with a private CA; document local HTTP only as development mode.
- Enable and document `ALLOWED_MANAGEMENT_IPS` for production.
- Add systemd hardening where compatible with Incus/ZFS access.
- Add explicit installer warnings when `HTTPS_ENABLED=false` and no `ALLOWED_MANAGEMENT_IPS` is configured.
- Consider replacing `npm start` in systemd with direct `node src/index.js`.
## OSS Release Requirements
### Acceptance Criteria
- Production install docs do not expose the agent publicly by default.
- Installer warns on insecure network exposure.
- systemd unit has a documented minimum hardening baseline.
## 12. Improve agent crash cleanup and resource recovery
Agent jobs are persisted and active jobs are marked `failed` on restart, but a process crash can still leave host resources behind, such as temporary Incus snapshots, staged restore volumes, visible snapshot devices, or changed ZFS properties.
### Goal
Make agent startup detect and clean up known leftover resources from interrupted jobs where doing so is safe.
### Tasks
- Persist cleanup metadata for backup jobs: temporary Incus snapshot name, ZVOL, snapdev state, and backup type.
- Persist cleanup metadata for restore jobs: staged ZVOL, backup ZVOL, failed ZVOL, and swap state.
- On startup, scan failed active jobs and run safe cleanup actions.
- Log cleanup results into the persisted job logs.
- Avoid destructive cleanup when state is ambiguous; surface manual action instead.
### Acceptance Criteria
- Crashing during backup streaming does not leave `snapdev=visible` or temporary snapshots unnoticed.
- Crashing before restore swap removes staged restore volumes when safe.
- Crashing after restore swap does not automatically destroy rollback copies.
- Management can show cleanup-required states.
## 13. Improve health diagnostics in the UI
The agent health endpoint returns useful structured details, but the management UI currently summarizes this too aggressively. Operators need actionable health reasons without opening logs.
### Goal
Show per-node health detail in the UI with concrete failed checks and messages.
### Tasks
- Add a detailed health drawer or modal on the Nodes page.
- Show command, ZFS pool, `/dev/zvol`, Restic repository, and credential check results.
- Show last health timestamp and the management-side error if the agent is unreachable.
- Distinguish unreachable, unauthorized, TLS failure, degraded health, and healthy states.
- Avoid global UI timeout banners when only one enabled node is slow or unreachable.
### Acceptance Criteria
- A missing command is visible by name in the UI.
- Wrong Restic credentials are visible as a Restic health failure.
- TLS or connectivity failures are distinguishable from degraded agent health.
- [ ] Add tests.
- [ ] Add CI pipeline.
- [ ] Add `SECURITY.md`.
- [ ] Document threat model and trusted network assumptions.
- [ ] Add `LICENSE`.
- [ ] Add `CONTRIBUTING.md`.
- [ ] Add example `compose.yaml` or Ansible role for reproducible deployment.