incus-backup-ui/incus-backup-ui-plan.md

Project: Incus-Restic Backup Control Plane (ZFS Block-Level)

Overview

Build a dark-mode backup operations dashboard (React frontend + Node.js agent) to manage incremental ZFS block-level backups of Incus VMs via Restic to S3 storage.

The UI should be visually inspired by Zerobyte: calm dark operator interface, compact status panels, clear backup lifecycle visibility, and strong restore safeguards. This is an application dashboard, not a marketing landing page.

The agent is a safe CLI orchestration layer that executes native host commands (incus, zfs, restic, udevadm, dd) and exposes structured REST APIs to the frontend.

Core product goals:

• Show Incus VM backup health at a glance.
• Trigger and monitor ZFS block-level Restic backups.
• List Restic snapshots per VM.
• Restore a VM from a selected snapshot with explicit destructive confirmation.
• Provide job status, logs, cleanup behavior, and per-VM locking for operational safety.

---

Tech Stack

Layer	Technology
Frontend	React (Vite), Tailwind CSS, Lucide Icons, Axios
Agent	Node.js, Express
Communication	REST API
Execution model	Background jobs with per-VM locks
Host requirement	Agent must run as root/sudo on the Incus host (needs /dev/zvol/ and Incus Unix socket access)

---

Repository Structure

/
├── agent/
│   ├── .env
│   ├── package.json
│   └── src/
│       ├── index.js        # Express server entry point
│       ├── config.js       # environment validation
│       ├── executor.js     # spawnCommand() and streaming helpers
│       ├── jobs.js         # in-memory job store and per-VM locks
│       ├── validators.js   # VM/snapshot input validation
│       └── routes/
│           ├── vms.js
│           ├── snapshots.js
│           ├── backup.js
│           ├── restore.js
│           ├── jobs.js
│           └── health.js
└── frontend/
    ├── vite.config.js
    ├── tailwind.config.js
    └── src/
        ├── main.jsx
        ├── api.js          # Axios instance
        ├── App.jsx
        └── components/
            ├── Dashboard.jsx
            ├── VMDetail.jsx
            ├── SnapshotTable.jsx
            ├── JobStatusPanel.jsx
            ├── StatusCard.jsx
            └── RestoreModal.jsx

---

Environment Variables (agent/.env)

# Restic & S3 Config
AWS_ACCESS_KEY_ID="your_s3_key"
AWS_SECRET_ACCESS_KEY="your_s3_secret"
RESTIC_REPOSITORY="s3:https://s3.eu-central-1.amazonaws.com/bucket-name"
RESTIC_PASSWORD="restic_encryption_password"

# Incus & ZFS Config
ZFS_POOL_NAME="incus-pool"
PORT=3000

# Optional UI/API guard for first MVP deployment
API_TOKEN="change-me"

Note: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, RESTIC_REPOSITORY, and RESTIC_PASSWORD must be passed as environment variables to every restic subprocess call.

---

Agent Implementation Principles

Safe Command Execution

• Use Node.js child_process.spawn(command, args, options) for all host commands.
• Do not interpolate user-controlled values into shell strings.
• Avoid exec and shell: true.
• Always inject required env vars (AWS credentials, Restic config) into the child process environment.
• Return { stdout, stderr, exitCode }.
• On non-zero exit code, throw an error with the stderr content.
• Stream long-running command output into the job log.
• Implement pipelines such as restic dump | dd with connected Node streams, not by passing a shell pipeline string to sh.

Validation

• vmName must be validated against incus list --format json before use.
• snapshotId must be either "latest" or match an existing Restic snapshot for that VM.
• Restore requests must include explicit confirmation:

{
  "snapshotId": "a1b2c3d4",
  "confirmVmName": "my-vm"
}

The agent must reject restore requests where confirmVmName !== vmName.

Job Model

Backup and restore operations are long-running and must run as background jobs.

Job shape:

{
  "id": "job_abc123",
  "type": "backup",
  "vmName": "my-vm",
  "status": "queued|running|success|failed",
  "startedAt": "2026-05-20T12:00:00Z",
  "finishedAt": null,
  "currentStep": "Creating Incus snapshot",
  "logs": ["..."],
  "error": null
}

Rules:

• Only one backup or restore job may run per VM at a time.
• Use an in-memory job store for MVP.
• Return 409 Conflict if a job is already running for the same VM.
• Keep recent completed jobs in memory for UI visibility.
• Restore jobs must never be automatically retried.

API Endpoints

---

GET /api/health

Purpose: Show agent and host command readiness.

Checks:

• Required environment variables are present.
• incus, zfs, and restic are executable.
• Restic repository can be reached with restic snapshots --json or a lightweight equivalent.

Response Example:

{
  "ok": true,
  "checks": {
    "config": "ok",
    "incus": "ok",
    "zfs": "ok",
    "restic": "ok"
  }
}

---

GET /api/vms

Purpose: List all Incus VMs on the host.

CLI Command:

incus list --format json

Logic:

1. Parse the JSON array returned by Incus.
2. Filter for entries where type === "virtual-machine".
3. Enrich with latest known job status if available.
4. Return array of VM summary objects.

Response Example:

[
  {
    "name": "my-vm",
    "status": "Running",
    "activeJob": null,
    "lastJobStatus": "success"
  }
]

---

GET /api/snapshots/:vmName

Purpose: List all Restic backup snapshots for a specific VM.

CLI Command:

restic snapshots --json --tag <vmName>

Logic:

1. Parse the JSON array.
2. Sort descending by time field.
3. Return the array as-is (frontend uses id, time, tags).

Response Example:

[
  { "id": "a1b2c3d4", "time": "2024-07-10T02:00:00Z", "tags": ["my-vm"] }
]

---

GET /api/jobs

Purpose: List recent backup and restore jobs.

Response Example:

[
  {
    "id": "job_abc123",
    "type": "backup",
    "vmName": "my-vm",
    "status": "running",
    "currentStep": "Streaming block device to Restic"
  }
]

---

GET /api/jobs/:jobId

Purpose: Retrieve detailed job status and logs.

---

POST /api/backup/:vmName

Purpose: Trigger a ZFS block-level backup to S3 via Restic.

Request Body: (none)

Response: Start a background job.

{
  "jobId": "job_abc123",
  "message": "Backup job started."
}

Sequential Shell Steps — execute in order, abort on failure:

# Step 1: Create a temporary Incus snapshot
incus snapshot create <vmName> s3-backup-<timestamp>

# Step 2: Make the ZFS snapshot device visible
zfs set snapdev=visible <ZFS_POOL_NAME>/virtual-machines/<vmName>.block

# Step 3: Wait for device to appear
sleep 2

# Step 4: Stream the block device into Restic (pipe)
restic backup --stdin \
  --stdin-filename "<vmName>.raw" \
  --tag "<vmName>" \
  < /dev/zvol/<ZFS_POOL_NAME>/virtual-machines/<vmName>.block@snapshot-s3-backup-<timestamp>

# Step 5: Hide the snapshot device again
zfs set snapdev=hidden <ZFS_POOL_NAME>/virtual-machines/<vmName>.block

# Step 6: Delete the temporary Incus snapshot
incus snapshot delete <vmName> s3-backup-<timestamp>

# Step 7: Apply retention policy
restic forget --tag "<vmName>" --keep-daily 7 --prune

Error Handling / Cleanup: If any step (1–4) fails, the agent must still execute the following cleanup commands before marking the job as failed:

• zfs set snapdev=hidden <ZFS_POOL_NAME>/virtual-machines/<vmName>.block
• incus snapshot delete <vmName> s3-backup-<timestamp> (ignore errors if snapshot doesn't exist)

The job record must move to success or failed and preserve logs for the frontend.

---

POST /api/restore/:vmName

Purpose: Restore a VM's disk from a Restic snapshot.

Request Body:

{
  "snapshotId": "a1b2c3d4",
  "confirmVmName": "my-vm"
}

Pass "latest" as snapshotId to restore the most recent snapshot.

Response: Start a background job.

{
  "jobId": "job_def456",
  "message": "Restore job started."
}

Sequential Shell Steps — execute in order:

# Step 1: Stop the VM (ignore error if already stopped)
incus stop <vmName> --force || true

# Step 2: Set ZFS volume to raw device mode
zfs set volmode=dev <ZFS_POOL_NAME>/virtual-machines/<vmName>.block

# Step 3: Trigger udev and wait for device node
udevadm trigger && udevadm settle && sleep 2

# Step 4: Stream Restic snapshot data into the block device
restic dump <snapshotId> <vmName>.raw \
  | dd of=/dev/zvol/<ZFS_POOL_NAME>/virtual-machines/<vmName>.block \
       bs=4M conv=sparse status=none

# Step 5: Restore ZFS volume mode
zfs set volmode=none <ZFS_POOL_NAME>/virtual-machines/<vmName>.block

# Step 6: Start the VM
incus start <vmName>

Restore Error Handling / Cleanup:

• If restore fails after volmode=dev, always attempt zfs set volmode=none <ZFS_POOL_NAME>/virtual-machines/<vmName>.block.
• Do not automatically start the VM if the disk write failed.
• Preserve detailed job logs and expose the failure in the UI.

---

Frontend Implementation

Configuration

• Tailwind dark theme base: bg-zinc-950, text-zinc-100.
• Create an Axios instance pointing to http://localhost:3000/api.
• Display agent errors as both toast notifications and persistent job errors where relevant.
• Use Lucide icons for actions and status indicators.

Visual Direction

The interface should follow the Zerobyte-inspired operator dashboard direction:

• Dark, quiet, dense UI.
• Subtle panel borders: border-zinc-800.
• Panels: bg-zinc-900 or bg-zinc-900/70.
• Muted secondary text: text-zinc-400.
• Status colors:
  • Running/success: emerald or green.
  • Failed/destructive: red.
  • Warning/attention: amber.
  • Active/running job: cyan or blue.
• Avoid marketing hero sections, oversized decorative cards, gradient blobs, and visual noise.
• First screen must be the actual dashboard.
• Cards are for repeated status panels or VM rows only; do not nest cards inside cards.

View A: Dashboard

Top Status Strip:

• Total VMs.
• Running VMs.
• Active jobs.
• Last successful backup.
• Repository health.

VM Backup Table:

• Dense table layout, optimized for scanning.
• Per VM row:
  • VM Name
  • Status indicator: green dot = Running, red dot = Stopped
  • Latest snapshot time
  • Snapshot count
  • Last job status
  • Active job progress, if any
  • Button: Manage Backups → navigates to View B for that VM.
• Data sources:
  • GET /api/health
  • GET /api/vms
  • GET /api/jobs

View B: VM Detail & Snapshot List

Header:

• VM name as page title.
• Incus status badge.
• Latest snapshot timestamp.
• Primary button: Create Backup
  • On click: calls POST /api/backup/:vmName.
  • Receives jobId.
  • Shows job status and logs via GET /api/jobs/:jobId.

Job Status Panel:

• Shows active or latest job.
• Fields:
  • Status
  • Current step
  • Started/finished time
  • Log output
  • Error details
• This panel is persistent; do not rely on toast messages for long-running operations.

Snapshot Table:

• Data source: GET /api/snapshots/:vmName
• Columns:

Column	Value
ID	First 8 characters of snapshot hash
Date & Time	Formatted time field
Tags	Comma-separated tag list
Actions	"Restore" button

Restore Button Behavior:

1. User clicks "Restore".

2. A red warning modal appears with the text:

   ⚠️ Warning: The VM will be stopped and the current disk will be irreversibly overwritten with the state from snapshot [snapshot-id]. Do you want to continue?

3. Modal has two buttons: "Cancel" and "Confirm Restore" (red, destructive style).

4. Modal requires explicit VM name confirmation.

5. Only on "Confirm Restore": call POST /api/restore/:vmName with { "snapshotId": "<id>", "confirmVmName": "<vmName>" }.

6. Show restore job status in the persistent job panel.

---

Error Handling Summary

Layer	Requirement
Agent command execution	Use spawn with argument arrays; no shell interpolation for user input
Agent validation	Validate VM names against Incus and snapshot IDs against Restic
Agent jobs	Backup/restore run as jobs with status, logs, timestamps, and errors
Agent locking	Only one active backup/restore job per VM
Agent backup cleanup	On failure in backup steps 1–4, always hide snapdev and delete the temp snapshot
Agent restore cleanup	If restore fails after volmode=dev, attempt to restore volmode=none
Agent errors	Return structured { "error": "..." }; job failures must also be visible via /api/jobs/:jobId
Frontend errors	Show immediate toast plus persistent job error/details
Frontend restore action	Always show destructive confirmation modal and require VM-name confirmation

---

MVP Implementation Order

1. Scaffold agent and frontend folders.
2. Implement agent config validation and safe command executor.
3. Implement GET /api/health.
4. Implement GET /api/vms.
5. Implement GET /api/snapshots/:vmName.
6. Implement in-memory job store and per-VM lock.
7. Implement backup job flow with cleanup.
8. Build dashboard UI in the Zerobyte-inspired operator style.
9. Build VM detail UI with snapshot table and job status panel.
10. Implement restore job flow with defensive validation and cleanup.
11. Wire restore modal with explicit destructive confirmation.
12. Add deployment notes for running agent as root/sudo on the Incus host.

---

Setup Instructions

Agent

cd agent
npm init -y
npm install express cors dotenv
# Copy .env and fill in values
node src/index.js

Frontend

cd frontend
npm create vite@latest . -- --template react
npm install
npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init -p
npm install axios lucide-react
npm run dev

In tailwind.config.js, set darkMode: 'class' and use bg-zinc-950 as the default body background.