# Project: Incus-Restic Backup Control Plane (ZFS Block-Level)

## Overview

Build a dark-mode backup operations dashboard (React frontend + Node.js backend) to manage incremental ZFS block-level backups of Incus VMs via Restic to S3 storage.

The UI should be visually inspired by [Zerobyte](https://zerobyte.app/): calm dark operator interface, compact status panels, clear backup lifecycle visibility, and strong restore safeguards. This is an application dashboard, not a marketing landing page.

The backend is a **safe CLI orchestration layer** that executes native host commands (`incus`, `zfs`, `restic`, `udevadm`, `dd`) and exposes structured REST APIs to the frontend.

Core product goals:

- Show Incus VM backup health at a glance.
- Trigger and monitor ZFS block-level Restic backups.
- List Restic snapshots per VM.
- Restore a VM from a selected snapshot with explicit destructive confirmation.
- Provide job status, logs, cleanup behavior, and per-VM locking for operational safety.

---

## Tech Stack

| Layer | Technology |
|---|---|
| Frontend | React (Vite), Tailwind CSS, Lucide Icons, Axios |
| Backend | Node.js, Express |
| Communication | REST API |
| Execution model | Background jobs with per-VM locks |
| Host requirement | Backend must run as root/sudo on the Incus host (needs `/dev/zvol/` and Incus Unix socket access) |

---

## Repository Structure

```
/
├── backend/
│   ├── .env
│   ├── package.json
│   └── src/
│       ├── index.js        # Express server entry point
│       ├── config.js       # environment validation
│       ├── executor.js     # spawnCommand() and streaming helpers
│       ├── jobs.js         # in-memory job store and per-VM locks
│       ├── validators.js   # VM/snapshot input validation
│       └── routes/
│           ├── vms.js
│           ├── snapshots.js
│           ├── backup.js
│           ├── restore.js
│           ├── jobs.js
│           └── health.js
└── frontend/
    ├── vite.config.js
    ├── tailwind.config.js
    └── src/
        ├── main.jsx
        ├── api.js          # Axios instance
        ├── App.jsx
        └── components/
            ├── Dashboard.jsx
            ├── VMDetail.jsx
            ├── SnapshotTable.jsx
            ├── JobStatusPanel.jsx
            ├── StatusCard.jsx
            └── RestoreModal.jsx
```

---

## Environment Variables (`backend/.env`)

```env
# Restic & S3 Config
AWS_ACCESS_KEY_ID="your_s3_key"
AWS_SECRET_ACCESS_KEY="your_s3_secret"
RESTIC_REPOSITORY="s3:https://s3.eu-central-1.amazonaws.com/bucket-name"
RESTIC_PASSWORD="restic_encryption_password"

# Incus & ZFS Config
ZFS_POOL_NAME="incus-pool"
PORT=3000

# Optional UI/API guard for first MVP deployment
API_TOKEN="change-me"
```

> **Note:** `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `RESTIC_REPOSITORY`, and `RESTIC_PASSWORD` must be passed as environment variables to every `restic` subprocess call.

---

## Backend Implementation Principles

### Safe Command Execution

- Use Node.js `child_process.spawn(command, args, options)` for all host commands.
- Do **not** interpolate user-controlled values into shell strings.
- Avoid `exec` and `shell: true`.
- Always inject required env vars (AWS credentials, Restic config) into the child process environment.
- Return `{ stdout, stderr, exitCode }`.
- On non-zero exit code, throw an error with the stderr content.
- Stream long-running command output into the job log.
- Implement pipelines such as `restic dump | dd` with connected Node streams, not by passing a shell pipeline string to `sh`.

### Validation

- `vmName` must be validated against `incus list --format json` before use.
- `snapshotId` must be either `"latest"` or match an existing Restic snapshot for that VM.
- Restore requests must include explicit confirmation:

```json
{
  "snapshotId": "a1b2c3d4",
  "confirmVmName": "my-vm"
}
```

The backend must reject restore requests where `confirmVmName !== vmName`.

### Job Model

Backup and restore operations are long-running and must run as background jobs.

Job shape:

```json
{
  "id": "job_abc123",
  "type": "backup",
  "vmName": "my-vm",
  "status": "queued|running|success|failed",
  "startedAt": "2026-05-20T12:00:00Z",
  "finishedAt": null,
  "currentStep": "Creating Incus snapshot",
  "logs": ["..."],
  "error": null
}
```

Rules:

- Only one backup or restore job may run per VM at a time.
- Use an in-memory job store for MVP.
- Return `409 Conflict` if a job is already running for the same VM.
- Keep recent completed jobs in memory for UI visibility.
- Restore jobs must never be automatically retried.

### API Endpoints

---

#### `GET /api/health`

**Purpose:** Show backend and host command readiness.

**Checks:**
- Required environment variables are present.
- `incus`, `zfs`, and `restic` are executable.
- Restic repository can be reached with `restic snapshots --json` or a lightweight equivalent.

**Response Example:**
```json
{
  "ok": true,
  "checks": {
    "config": "ok",
    "incus": "ok",
    "zfs": "ok",
    "restic": "ok"
  }
}
```

---

#### `GET /api/vms`

**Purpose:** List all Incus VMs on the host.

**CLI Command:**
```bash
incus list --format json
```

**Logic:**
1. Parse the JSON array returned by Incus.
2. Filter for entries where `type === "virtual-machine"`.
3. Enrich with latest known job status if available.
4. Return array of VM summary objects.

**Response Example:**
```json
[
  {
    "name": "my-vm",
    "status": "Running",
    "activeJob": null,
    "lastJobStatus": "success"
  }
]
```

---

#### `GET /api/snapshots/:vmName`

**Purpose:** List all Restic backup snapshots for a specific VM.

**CLI Command:**
```bash
restic snapshots --json --tag <vmName>
```

**Logic:**
1. Parse the JSON array.
2. Sort descending by `time` field.
3. Return the array as-is (frontend uses `id`, `time`, `tags`).

**Response Example:**
```json
[
  { "id": "a1b2c3d4", "time": "2024-07-10T02:00:00Z", "tags": ["my-vm"] }
]
```

---

#### `GET /api/jobs`

**Purpose:** List recent backup and restore jobs.

**Response Example:**
```json
[
  {
    "id": "job_abc123",
    "type": "backup",
    "vmName": "my-vm",
    "status": "running",
    "currentStep": "Streaming block device to Restic"
  }
]
```

---

#### `GET /api/jobs/:jobId`

**Purpose:** Retrieve detailed job status and logs.

---

#### `POST /api/backup/:vmName`

**Purpose:** Trigger a ZFS block-level backup to S3 via Restic.

**Request Body:** _(none)_

**Response:** Start a background job.

```json
{
  "jobId": "job_abc123",
  "message": "Backup job started."
}
```

**Sequential Shell Steps — execute in order, abort on failure:**

```bash
# Step 1: Create a temporary Incus snapshot
incus snapshot create <vmName> s3-backup-<timestamp>

# Step 2: Make the ZFS snapshot device visible
zfs set snapdev=visible <ZFS_POOL_NAME>/virtual-machines/<vmName>.block

# Step 3: Wait for device to appear
sleep 2

# Step 4: Stream the block device into Restic (pipe)
restic backup --stdin \
  --stdin-filename "<vmName>.raw" \
  --tag "<vmName>" \
  < /dev/zvol/<ZFS_POOL_NAME>/virtual-machines/<vmName>.block@snapshot-s3-backup-<timestamp>

# Step 5: Hide the snapshot device again
zfs set snapdev=hidden <ZFS_POOL_NAME>/virtual-machines/<vmName>.block

# Step 6: Delete the temporary Incus snapshot
incus snapshot delete <vmName> s3-backup-<timestamp>

# Step 7: Apply retention policy
restic forget --tag "<vmName>" --keep-daily 7 --prune
```

**Error Handling / Cleanup:**
If any step (1–4) fails, the backend **must** still execute the following cleanup commands before marking the job as failed:
- `zfs set snapdev=hidden <ZFS_POOL_NAME>/virtual-machines/<vmName>.block`
- `incus snapshot delete <vmName> s3-backup-<timestamp>` (ignore errors if snapshot doesn't exist)

The job record must move to `success` or `failed` and preserve logs for the frontend.

---

#### `POST /api/restore/:vmName`

**Purpose:** Restore a VM's disk from a Restic snapshot.

**Request Body:**
```json
{
  "snapshotId": "a1b2c3d4",
  "confirmVmName": "my-vm"
}
```
> Pass `"latest"` as `snapshotId` to restore the most recent snapshot.

**Response:** Start a background job.

```json
{
  "jobId": "job_def456",
  "message": "Restore job started."
}
```

**Sequential Shell Steps — execute in order:**

```bash
# Step 1: Stop the VM (ignore error if already stopped)
incus stop <vmName> --force || true

# Step 2: Set ZFS volume to raw device mode
zfs set volmode=dev <ZFS_POOL_NAME>/virtual-machines/<vmName>.block

# Step 3: Trigger udev and wait for device node
udevadm trigger && udevadm settle && sleep 2

# Step 4: Stream Restic snapshot data into the block device
restic dump <snapshotId> <vmName>.raw \
  | dd of=/dev/zvol/<ZFS_POOL_NAME>/virtual-machines/<vmName>.block \
       bs=4M conv=sparse status=none

# Step 5: Restore ZFS volume mode
zfs set volmode=none <ZFS_POOL_NAME>/virtual-machines/<vmName>.block

# Step 6: Start the VM
incus start <vmName>
```

**Restore Error Handling / Cleanup:**

- If restore fails after `volmode=dev`, always attempt `zfs set volmode=none <ZFS_POOL_NAME>/virtual-machines/<vmName>.block`.
- Do not automatically start the VM if the disk write failed.
- Preserve detailed job logs and expose the failure in the UI.

---

## Frontend Implementation

### Configuration

- Tailwind dark theme base: `bg-zinc-950`, `text-zinc-100`.
- Create an Axios instance pointing to `http://localhost:3000/api`.
- Display backend errors as both toast notifications and persistent job errors where relevant.
- Use Lucide icons for actions and status indicators.

### Visual Direction

The interface should follow the Zerobyte-inspired operator dashboard direction:

- Dark, quiet, dense UI.
- Subtle panel borders: `border-zinc-800`.
- Panels: `bg-zinc-900` or `bg-zinc-900/70`.
- Muted secondary text: `text-zinc-400`.
- Status colors:
  - Running/success: emerald or green.
  - Failed/destructive: red.
  - Warning/attention: amber.
  - Active/running job: cyan or blue.
- Avoid marketing hero sections, oversized decorative cards, gradient blobs, and visual noise.
- First screen must be the actual dashboard.
- Cards are for repeated status panels or VM rows only; do not nest cards inside cards.

### View A: Dashboard

**Top Status Strip:**

- Total VMs.
- Running VMs.
- Active jobs.
- Last successful backup.
- Repository health.

**VM Backup Table:**

- Dense table layout, optimized for scanning.
- Per VM row:
  - VM Name
  - Status indicator: green dot = `Running`, red dot = `Stopped`
  - Latest snapshot time
  - Snapshot count
  - Last job status
  - Active job progress, if any
  - Button: **Manage Backups** → navigates to View B for that VM.
- Data sources:
  - `GET /api/health`
  - `GET /api/vms`
  - `GET /api/jobs`

### View B: VM Detail & Snapshot List

**Header:**
- VM name as page title.
- Incus status badge.
- Latest snapshot timestamp.
- Primary button: **Create Backup**
  - On click: calls `POST /api/backup/:vmName`.
  - Receives `jobId`.
  - Shows job status and logs via `GET /api/jobs/:jobId`.

**Job Status Panel:**

- Shows active or latest job.
- Fields:
  - Status
  - Current step
  - Started/finished time
  - Log output
  - Error details
- This panel is persistent; do not rely on toast messages for long-running operations.

**Snapshot Table:**
- Data source: `GET /api/snapshots/:vmName`
- Columns:

| Column | Value |
|---|---|
| ID | First 8 characters of snapshot hash |
| Date & Time | Formatted `time` field |
| Tags | Comma-separated tag list |
| Actions | "Restore" button |

**Restore Button Behavior:**
1. User clicks **"Restore"**.
2. A **red warning modal** appears with the text:

   > ⚠️ **Warning:** The VM will be stopped and the current disk will be **irreversibly overwritten** with the state from snapshot `[snapshot-id]`. Do you want to continue?

3. Modal has two buttons: **"Cancel"** and **"Confirm Restore"** (red, destructive style).
4. Modal requires explicit VM name confirmation.
5. Only on **"Confirm Restore"**: call `POST /api/restore/:vmName` with `{ "snapshotId": "<id>", "confirmVmName": "<vmName>" }`.
6. Show restore job status in the persistent job panel.

---

## Error Handling Summary

| Layer | Requirement |
|---|---|
| Backend command execution | Use `spawn` with argument arrays; no shell interpolation for user input |
| Backend validation | Validate VM names against Incus and snapshot IDs against Restic |
| Backend jobs | Backup/restore run as jobs with status, logs, timestamps, and errors |
| Backend locking | Only one active backup/restore job per VM |
| Backend backup cleanup | On failure in backup steps 1–4, always hide snapdev and delete the temp snapshot |
| Backend restore cleanup | If restore fails after `volmode=dev`, attempt to restore `volmode=none` |
| Backend errors | Return structured `{ "error": "..." }`; job failures must also be visible via `/api/jobs/:jobId` |
| Frontend errors | Show immediate toast plus persistent job error/details |
| Frontend restore action | Always show destructive confirmation modal and require VM-name confirmation |

---

## MVP Implementation Order

1. Scaffold backend and frontend folders.
2. Implement backend config validation and safe command executor.
3. Implement `GET /api/health`.
4. Implement `GET /api/vms`.
5. Implement `GET /api/snapshots/:vmName`.
6. Implement in-memory job store and per-VM lock.
7. Implement backup job flow with cleanup.
8. Build dashboard UI in the Zerobyte-inspired operator style.
9. Build VM detail UI with snapshot table and job status panel.
10. Implement restore job flow with defensive validation and cleanup.
11. Wire restore modal with explicit destructive confirmation.
12. Add deployment notes for running backend as root/sudo on the Incus host.

---

## Setup Instructions

### Backend

```bash
cd backend
npm init -y
npm install express cors dotenv
# Copy .env and fill in values
node src/index.js
```

### Frontend

```bash
cd frontend
npm create vite@latest . -- --template react
npm install
npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init -p
npm install axios lucide-react
npm run dev
```

In `tailwind.config.js`, set `darkMode: 'class'` and use `bg-zinc-950` as the default body background.