feat(distlock): integrate distributed lease management into Machine API

* feat: support stdin_open and tty

This can be useful to leave a container running without specifying a
command like `sleep`, and probably in other situations as well.

Signed-off-by: Miek Gieben <miek@miek.nl>

* Complete full spec test case

Signed-off-by: Miek Gieben <miek@miek.nl>

* fix container log streaming for containers with TTY

---------

Signed-off-by: Miek Gieben <miek@miek.nl>
Co-authored-by: Pasha Sviderski <me@psviderski.name>

.github/workflows/nightly-build.yml

@@ -57,7 +57,8 @@ jobs:
 
           The commands below show how to install the nightly version of uncloud.
 
-          > **Warning**: These are nightly development builds and may be unstable.
+          > [!WARNING]
+          > These are nightly development builds and may be unstable.
 
           **Install uncloud CLI:**
 

README.md

@@ -154,7 +154,6 @@ Here is a diagram of an Uncloud multi-provider cluster of 3 machines:
 ```bash
 $ uc machine init --name oracle-vm ubuntu@152.67.101.197
 
-Downloading Uncloud install script: https://raw.githubusercontent.com/psviderski/uncloud/refs/heads/main/scripts/install.sh
 ⏳ Running Uncloud install script...
 ✓ Docker is already installed.
 ⏳ Installing Docker...
@@ -169,9 +168,6 @@ Downloading Uncloud install script: https://raw.githubusercontent.com/psviderski
 ✓ uncloud-uninstall script installed: /usr/local/bin/uncloud-uninstall
 ✓ Systemd unit file created: /etc/systemd/system/uncloud.service
 Created symlink /etc/systemd/system/multi-user.target.wants/uncloud.service → /etc/systemd/system/uncloud.service.
-⏳ Downloading uncloud-corrosion binary: https://github.com/psviderski/corrosion/releases/latest/download/corrosion-aarch64-unknown-linux-gnu.tar.gz
-✓ uncloud-corrosion binary installed: /usr/local/bin/uncloud-corrosion
-✓ Systemd unit file created: /etc/systemd/system/uncloud-corrosion.service
 ⏳ Starting Uncloud machine daemon (uncloud.service)...
 ✓ Uncloud machine daemon started.
 ✓ Uncloud installed on the machine successfully! 🎉
@@ -190,21 +186,20 @@ DNS records updated to use only the internet-reachable machines running caddy se
   *.xuw3xd.cluster.uncloud.run  A → 152.67.101.197
 ```
 
-1. The CLI SSHs into the machine and installs Docker, the `uncloudd` machine daemon and
-   [corrosion](https://github.com/superfly/corrosion) service, managed by systemd.
+1. The CLI SSHs into the machine and installs Docker and the `uncloudd` machine daemon, managed by systemd.
 2. Generates a unique WireGuard key pair, allocates a dedicated subnet `10.210.0.0/24` for the machine and its
    containers, and configures `uncloudd` accordingly. All subsequent communication happens with `uncloudd`
    through its gRPC API over SSH.
-3. Configures and starts `corrosion`, a CRDT-based distributed SQLite database to share cluster state between machines.
+3. Configures and starts [corrosion](https://github.com/superfly/corrosion), a CRDT-based distributed SQLite database to
+   share cluster state between machines, as a Docker container managed by `uncloudd`.
 4. Creates a Docker bridge network connected to the WireGuard interface.
-5. This machine becomes an entry point for the newly created cluster which is stored in the cluster config under
-   `~/.config/uncloud` on your local machine.
+5. This machine becomes an entry point for the newly created cluster. Its address is stored as a connection info in the
+   cluster config at `~/.config/uncloud/config.yaml` on your local machine.
 
 **When you add another machine:**
 
 ```bash
 $ uc machine add --name hetzner-server root@5.223.45.199
-Downloading Uncloud install script: https://raw.githubusercontent.com/psviderski/uncloud/refs/heads/main/scripts/install.sh
 ⏳ Running Uncloud install script...
 ✓ Docker is already installed.
 ✓ Linux user and group 'uncloud' created.
@@ -215,9 +210,6 @@ Downloading Uncloud install script: https://raw.githubusercontent.com/psviderski
 ✓ uncloud-uninstall script installed: /usr/local/bin/uncloud-uninstall
 ✓ Systemd unit file created: /etc/systemd/system/uncloud.service
 Created symlink /etc/systemd/system/multi-user.target.wants/uncloud.service → /etc/systemd/system/uncloud.service.
-⏳ Downloading uncloud-corrosion binary: https://github.com/psviderski/corrosion/releases/latest/download/corrosion-x86_64-unknown-linux-gnu.tar.gz
-✓ uncloud-corrosion binary installed: /usr/local/bin/uncloud-corrosion
-✓ Systemd unit file created: /etc/systemd/system/uncloud-corrosion.service
 ⏳ Starting Uncloud machine daemon (uncloud.service)...
 ✓ Uncloud machine daemon started.
 ✓ Uncloud installed on the machine successfully! 🎉
@@ -246,9 +238,10 @@ hetzner-server   Up      10.210.1.1/24   5.223.45.199     5.223.45.199:51820, [2
 3. Registers the second machine in the cluster state and exchanges WireGuard keys with the first machine.
 4. Both machines establish a WireGuard tunnel between each other, allowing Docker containers connected to the bridge
    network to communicate directly across machines.
-5. Configures and starts `corrosion` on the second machine to sync the cluster state.
+5. Configures and starts the `uncloud-corrosion` container on the second machine to sync the cluster state.
 6. The second machine is added as an alternative entry point in the cluster config.
-7. If one of the machines goes offline, the other machine can still serve cluster operations.
+7. If one of the machines goes offline, the other machine can still be used by `uc` to connect and run cluster
+   operations.
 
 If one more machine is added, the process repeats with a new subnet. The new machine needs to establish a WireGuard
 connection with only one of the existing machines. Other machines will learn about it through the shared cluster state

cmd/uc/machine/add.go

@@ -47,9 +47,12 @@ Connection methods:
   [ssh://]user@host   - Use system 'ssh' command with full SSH config support (default, no prefix required)
   ssh+go://user@host  - Use Go's built-in SSH library`,
 		Args: cobra.ExactArgs(1),
-		RunE: func(cmd *cobra.Command, args []string) error {
+		PreRunE: func(cmd *cobra.Command, args []string) error {
 			cli.BindEnvToFlag(cmd, "yes", "UNCLOUD_AUTO_CONFIRM")
-
+			cli.BindEnvToFlag(cmd, "version", "UNCLOUD_DAEMON_VERSION")
+			return nil
+		},
+		RunE: func(cmd *cobra.Command, args []string) error {
 			uncli := cmd.Context().Value("cli").(*cli.CLI)
 
 			// Determine connection mode and strip scheme.
@@ -97,7 +100,7 @@ Connection methods:
 	)
 	cmd.Flags().StringVar(
 		&opts.version, "version", "latest",
-		"Version of the Uncloud daemon to install on the machine.",
+		"Version of the Uncloud daemon to install on the machine. [$UNCLOUD_DAEMON_VERSION]",
 	)
 	cmd.Flags().StringSliceVar(
 		&opts.wgEndpoints, "wg-endpoint", nil,

cmd/uc/machine/flags_test.go

@@ -0,0 +1,72 @@
+package machine
+
+import (
+	"testing"
+
+	"github.com/spf13/cobra"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// runVersionFlagEnvTest builds the given command for each case, stubs out RunE so only flag
+// parsing and PreRunE run (no SSH calls), and asserts the resolved --version flag value.
+// Note: cannot use t.Parallel() because subtests use t.Setenv().
+func runVersionFlagEnvTest(t *testing.T, newCommand func() *cobra.Command) {
+	t.Helper()
+
+	tests := []struct {
+		name string
+		args []string
+		env  string
+		want string
+	}{
+		{
+			name: "env var sets version when flag not passed",
+			args: []string{"root@localhost"},
+			env:  "v1.2.3",
+			want: "v1.2.3",
+		},
+		{
+			name: "explicit flag wins over env var",
+			args: []string{"root@localhost", "--version", "v9.9.9"},
+			env:  "v1.2.3",
+			want: "v9.9.9",
+		},
+		{
+			name: "defaults to latest when neither flag nor env var set",
+			args: []string{"root@localhost"},
+			want: "latest",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if tt.env != "" {
+				t.Setenv("UNCLOUD_DAEMON_VERSION", tt.env)
+			}
+
+			cmd := newCommand()
+			cmd.RunE = func(*cobra.Command, []string) error {
+				return nil
+			}
+			cmd.SetArgs(tt.args)
+			require.NoError(t, cmd.Execute())
+
+			version, err := cmd.Flags().GetString("version")
+			require.NoError(t, err)
+			assert.Equal(t, tt.want, version)
+		})
+	}
+}
+
+// TestInitCommandVersionFlagEnvVar verifies that the UNCLOUD_DAEMON_VERSION environment variable
+// is bound to the --version flag of 'machine init', and that an explicit flag wins.
+func TestInitCommandVersionFlagEnvVar(t *testing.T) {
+	runVersionFlagEnvTest(t, NewInitCommand)
+}
+
+// TestAddCommandVersionFlagEnvVar verifies that the UNCLOUD_DAEMON_VERSION environment variable
+// is bound to the --version flag of 'machine add', and that an explicit flag wins.
+func TestAddCommandVersionFlagEnvVar(t *testing.T) {
+	runVersionFlagEnvTest(t, NewAddCommand)
+}

cmd/uc/machine/init.go

@@ -65,9 +65,12 @@ Connection methods:
   uc machine init root@<your-server-ip> --no-caddy --no-dns`,
 		// TODO: support initialising a cluster on the local machine.
 		Args: cobra.MaximumNArgs(1),
-		RunE: func(cmd *cobra.Command, args []string) error {
+		PreRunE: func(cmd *cobra.Command, args []string) error {
 			cli.BindEnvToFlag(cmd, "yes", "UNCLOUD_AUTO_CONFIRM")
-
+			cli.BindEnvToFlag(cmd, "version", "UNCLOUD_DAEMON_VERSION")
+			return nil
+		},
+		RunE: func(cmd *cobra.Command, args []string) error {
 			uncli := cmd.Context().Value("cli").(*cli.CLI)
 
 			var remoteMachine *cli.RemoteMachine
@@ -135,7 +138,7 @@ Connection methods:
 	)
 	cmd.Flags().StringVar(
 		&opts.version, "version", "latest",
-		"Version of the Uncloud daemon to install on the machine.",
+		"Version of the Uncloud daemon to install on the machine. [$UNCLOUD_DAEMON_VERSION]",
 	)
 	cmd.Flags().StringSliceVar(
 		&opts.wgEndpoints, "wg-endpoint", nil,

cmd/uc/main.go

@@ -24,6 +24,7 @@ import (
 	"github.com/psviderski/uncloud/internal/machine"
 	"github.com/psviderski/uncloud/internal/version"
 	"github.com/spf13/cobra"
+	"github.com/spf13/pflag"
 )
 
 type globalOptions struct {
@@ -42,6 +43,13 @@ func main() {
 		SilenceUsage:  true,
 		SilenceErrors: true,
 		PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
+			// Shell completion runs through the hidden __complete command which has flag parsing disabled,
+			// so the global flags from the completed command line are never parsed. Apply them manually to make
+			// completion work with --connect, --context, and --uncloud-config.
+			if cmd.Name() == cobra.ShellCompRequestCmd {
+				applyGlobalFlagsFromCompletionArgs(cmd.Root().PersistentFlags(), os.Args[1:])
+			}
+
 			cli.BindEnvToFlag(cmd, "connect", "UNCLOUD_CONNECT")
 			cli.BindEnvToFlag(cmd, "context", "UNCLOUD_CONTEXT")
 			cli.BindEnvToFlag(cmd, "uncloud-config", "UNCLOUD_CONFIG")
@@ -159,3 +167,27 @@ func main() {
 		cobra.CheckErr(err)
 	}
 }
+
+// applyGlobalFlagsFromCompletionArgs parses the global flags from the raw arguments of a __complete command and applies
+// the ones found to flags. The trailing word being completed, unknown flags, and positional arguments are ignored.
+func applyGlobalFlagsFromCompletionArgs(flags *pflag.FlagSet, args []string) {
+	// The shell always passes the word being completed as the last argument, even if it's empty.
+	// Exclude it from parsing as its value may not be complete yet.
+	if len(args) == 0 {
+		return
+	}
+	args = args[:len(args)-1]
+
+	fset := pflag.NewFlagSet("global", pflag.ContinueOnError)
+	fset.ParseErrorsAllowlist.UnknownFlags = true
+	fset.String("connect", "", "")
+	fset.StringP("context", "c", "", "")
+	fset.String("uncloud-config", "", "")
+	// Parsing an incomplete command line may fail, apply the flags parsed so far anyway.
+	_ = fset.Parse(args)
+
+	fset.Visit(func(f *pflag.Flag) {
+		// Setting the flag marks it as changed so it takes precedence over environment variables.
+		_ = flags.Set(f.Name, f.Value.String())
+	})
+}

cmd/uc/main_test.go

@@ -0,0 +1,116 @@
+package main
+
+import (
+	"slices"
+	"testing"
+
+	"github.com/spf13/pflag"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestApplyGlobalFlagsFromCompletionArgs(t *testing.T) {
+	defaultConfigPath := "~/.config/uncloud/config.yaml"
+
+	tests := []struct {
+		name           string
+		args           []string
+		wantConnect    string
+		wantContext    string
+		wantConfigPath string
+		// Flag names expected to be marked as changed on the target flag set.
+		wantChanged []string
+	}{
+		{
+			name: "no flags",
+			args: []string{"__complete", "inspect", ""},
+		},
+		{
+			name:        "connect with space",
+			args:        []string{"__complete", "--connect", "ssh://user@host", "inspect", ""},
+			wantConnect: "ssh://user@host",
+			wantChanged: []string{"connect"},
+		},
+		{
+			name:        "connect with equals",
+			args:        []string{"__complete", "--connect=tcp://127.0.0.1:51000", "inspect", ""},
+			wantConnect: "tcp://127.0.0.1:51000",
+			wantChanged: []string{"connect"},
+		},
+		{
+			name:        "context shorthand",
+			args:        []string{"__complete", "-c", "prod", "inspect", ""},
+			wantContext: "prod",
+			wantChanged: []string{"context"},
+		},
+		{
+			name:           "all flags",
+			args:           []string{"__complete", "--connect", "user@host", "-c", "prod", "--uncloud-config", "/tmp/uncloud.yaml", "inspect", ""},
+			wantConnect:    "user@host",
+			wantContext:    "prod",
+			wantConfigPath: "/tmp/uncloud.yaml",
+			wantChanged:    []string{"connect", "context", "uncloud-config"},
+		},
+		{
+			name:        "unknown flags are ignored",
+			args:        []string{"__complete", "--quiet", "-n", "5", "--connect", "user@host", "logs", ""},
+			wantConnect: "user@host",
+			wantChanged: []string{"connect"},
+		},
+		{
+			name: "flags after double dash are ignored",
+			args: []string{"__complete", "exec", "svc", "--", "sh", "--connect", "user@host"},
+		},
+		{
+			name:        "flags before double dash are applied",
+			args:        []string{"__complete", "--connect", "user@host", "exec", "svc", "--", "sh", "-c", "env"},
+			wantConnect: "user@host",
+			wantChanged: []string{"connect"},
+		},
+		{
+			name:        "partial flag name being completed is excluded",
+			args:        []string{"__complete", "--connect", "user@host", "inspect", "--context"},
+			wantConnect: "user@host",
+			wantChanged: []string{"connect"},
+		},
+		{
+			name: "partial flag value being completed is excluded",
+			args: []string{"__complete", "--uncloud-config", "/tmp/"},
+		},
+		{
+			name: "partial connect value being completed is excluded",
+			args: []string{"__complete", "--connect", "tcp://127.0.0.1:5"},
+		},
+		{
+			name:           "completed flag value with partial command word",
+			args:           []string{"__complete", "--uncloud-config", "/tmp/uncloud.yaml", "insp"},
+			wantConfigPath: "/tmp/uncloud.yaml",
+			wantChanged:    []string{"uncloud-config"},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			// Mirror the global persistent flags defined on the root command.
+			var opts globalOptions
+			flags := pflag.NewFlagSet("test", pflag.ContinueOnError)
+			flags.StringVar(&opts.connect, "connect", "", "")
+			flags.StringVarP(&opts.context, "context", "c", "", "")
+			flags.StringVar(&opts.configPath, "uncloud-config", defaultConfigPath, "")
+
+			applyGlobalFlagsFromCompletionArgs(flags, tt.args)
+
+			assert.Equal(t, tt.wantConnect, opts.connect)
+			assert.Equal(t, tt.wantContext, opts.context)
+			wantConfigPath := tt.wantConfigPath
+			if wantConfigPath == "" {
+				wantConfigPath = defaultConfigPath
+			}
+			assert.Equal(t, wantConfigPath, opts.configPath)
+
+			for _, name := range []string{"connect", "context", "uncloud-config"} {
+				assert.Equal(t, slices.Contains(tt.wantChanged, name), flags.Changed(name),
+					"changed status of flag '%s'", name)
+			}
+		})
+	}
+}

cmd/uc/proxy.go

@@ -132,35 +132,34 @@ func runProxy(ctx context.Context, uncli *cli.CLI, opts proxyOptions) error {
 	// endpoint and shuffles the data, *it* will actually experience errors.
 	remoteAddr := net.JoinHostPort(ip.String(), strconv.Itoa(opts.remotePort))
 
-	ctx, cancel := context.WithCancel(ctx)
-	defer cancel()
-
 	p := &proxy.Proxy{
 		Listener:    listener,
 		RemoteAddr:  remoteAddr,
 		DialContext: dialer.DialContext,
 		OnError: func(err error) {
-			fmt.Printf("Failed to proxy to '%s': %v\n", remoteAddr, err)
-			cancel()
-		},
+			if proxy.IsConnectionClosedError(err) {
+				return
+			}
+			// A more actionable error instead of the cryptic [ssh -W] command error.
+			if strings.Contains(err.Error(), "Session open refused by peer") {
+				fmt.Printf("Could not connect to '%s': connection refused. "+
+					"Check that the service is running and listening on port %d inside the container.\n",
+					remoteAddr, opts.remotePort)
+				return
 			}
 
-	// Run the proxy in the background and signal when it has fully shut down.
-	done := make(chan struct{})
-	go func() {
-		p.Run(ctx)
-		close(done)
-	}()
+			fmt.Printf("Failed to proxy a connection to '%s': %v\n", remoteAddr, err)
+		},
+	}
 
 	// Prefix the local address with the scheme for common HTTP ports so it becomes control-clickable in most
 	// terminals. We assume plain HTTP since TLS is typically terminated by Caddy in front of the service.
 	fmt.Printf("%s%s → %s (%s%s%s)\n", schemeForPort(opts.remotePort), p.Listener.Addr().String(),
 		remoteAddr, opts.service, tui.Faint.Render("/"), containerID)
 
-	<-ctx.Done()
-	// Wait for the proxy to drain in-flight connections and shut down gracefully.
-	<-done
-
+	if err = p.Run(ctx); err != nil {
+		return fmt.Errorf("run proxy to '%s': %w", remoteAddr, err)
+	}
 	return nil
 }
 

cmd/uc/ps.go

@@ -187,7 +187,7 @@ func printContainers(containers []containerInfo) error {
 }
 
 func collectContainers(ctx context.Context, cli *client.Client) ([]containerInfo, error) {
-	listCtx := cli.ProxyMachinesContext(ctx, nil)
+	listCtx := client.ProxyMachinesContext(ctx, nil)
 
 	// List all service containers across all machines in the cluster.
 	machineContainers, err := cli.Docker.ListServiceContainers(

cmd/uc/service/scale.go

@@ -30,9 +30,11 @@ func NewScaleCommand(groupID string) *cobra.Command {
 		Short: "Scale a replicated service by changing the number of replicas.",
 		Long:  "Scale a replicated service by changing the number of replicas.",
 		Args:  cobra.ExactArgs(2),
-		RunE: func(cmd *cobra.Command, args []string) error {
+		PreRunE: func(cmd *cobra.Command, args []string) error {
 			cli.BindEnvToFlag(cmd, "yes", "UNCLOUD_AUTO_CONFIRM")
-
+			return nil
+		},
+		RunE: func(cmd *cobra.Command, args []string) error {
 			uncli := cmd.Context().Value("cli").(*cli.CLI)
 
 			opts.service = args[0]

experiment/go.mod

@@ -124,8 +124,8 @@ require (
 	golang.org/x/exp v0.0.0-20250408133849-7e4ce0ab07d0 // indirect
 	golang.org/x/mod v0.27.0 // indirect
 	golang.org/x/net v0.43.0 // indirect
-	golang.org/x/sync v0.19.0 // indirect
-	golang.org/x/sys v0.42.0 // indirect
+	golang.org/x/sync v0.20.0 // indirect
+	golang.org/x/sys v0.45.0 // indirect
 	golang.org/x/text v0.28.0 // indirect
 	golang.org/x/tools v0.36.0 // indirect
 	golang.zx2c4.com/wireguard v0.0.0-20231211153847-12269c276173 // indirect

experiment/go.sum

@@ -660,8 +660,8 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
-golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
+golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
+golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
 golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -694,8 +694,8 @@ golang.org/x/sys v0.0.0-20221010170243-090e33056c14/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.42.0 h1:omrd2nAlyT5ESRdCLYdm3+fMfNFE/+Rf4bDIQImRJeo=
-golang.org/x/sys v0.42.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
+golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc=

internal/cli/completion/context.go

@@ -11,6 +11,11 @@ import (
 )
 
 func Contexts(ctx context.Context, uncli *cli.CLI, args []string, toComplete string) ([]cobra.Completion, cobra.ShellCompDirective) {
+	// There are no contexts to complete when the CLI uses a direct machine connection (--connect) without a config.
+	if uncli.Config == nil {
+		return nil, cobra.ShellCompDirectiveNoFileComp
+	}
+
 	contexts := slices.Sorted(maps.Keys(uncli.Config.Contexts))
 
 	names := []cobra.Completion{}
@@ -21,7 +26,6 @@ func Contexts(ctx context.Context, uncli *cli.CLI, args []string, toComplete str
 		if strings.HasPrefix(context, toComplete) {
 			names = append(names, context)
 		}
-		names = append(names, context)
 	}
 
 	return names, cobra.ShellCompDirectiveNoFileComp

internal/cli/completion/machine.go

@@ -10,7 +10,8 @@ import (
 )
 
 func Machines(ctx context.Context, uncli *cli.CLI, args []string, toComplete string) ([]cobra.Completion, cobra.ShellCompDirective) {
-	client, err := uncli.ConnectCluster(ctx)
+	// Disable the connection progress output to not interfere with the shell completion output.
+	client, err := uncli.ConnectClusterWithOptions(ctx, cli.ConnectOptions{})
 	if err != nil {
 		return nil, cobra.ShellCompDirectiveError
 	}

internal/cli/completion/service.go

@@ -11,7 +11,8 @@ import (
 )
 
 func Services(ctx context.Context, uncli *cli.CLI, args []string, toComplete string) ([]cobra.Completion, cobra.ShellCompDirective) {
-	client, err := uncli.ConnectCluster(ctx)
+	// Disable the connection progress output to not interfere with the shell completion output.
+	client, err := uncli.ConnectClusterWithOptions(ctx, cli.ConnectOptions{})
 	if err != nil {
 		return nil, cobra.ShellCompDirectiveError
 	}

internal/cli/completion/volume.go

@@ -11,7 +11,8 @@ import (
 )
 
 func Volumes(ctx context.Context, uncli *cli.CLI, args []string, toComplete string) ([]cobra.Completion, cobra.ShellCompDirective) {
-	client, err := uncli.ConnectCluster(ctx)
+	// Disable the connection progress output to not interfere with the shell completion output.
+	client, err := uncli.ConnectClusterWithOptions(ctx, cli.ConnectOptions{})
 	if err != nil {
 		return nil, cobra.ShellCompDirectiveError
 	}

internal/machine/corroservice/docker.go

@@ -9,6 +9,7 @@ import (
 	"time"
 
 	"github.com/containerd/errdefs"
+	systemd "github.com/coreos/go-systemd/daemon"
 	"github.com/docker/docker/api/types/container"
 	"github.com/docker/docker/api/types/image"
 	"github.com/docker/docker/api/types/mount"
@@ -22,6 +23,13 @@ const (
 	Image = "ghcr.io/unlabs-dev/corrosion:2026.6.15"
 	// ContainerName is the name of the managed Corrosion container.
 	ContainerName = "uncloud-corrosion"
+
+	// systemdStartTimeoutExtension is the duration of each systemd service start timeout extension that prevents
+	// a slow image pull from exceeding the start timeout (TimeoutStartSec). Extensions are a no-op if Corrosion
+	// is not starting during a systemd service startup.
+	systemdStartTimeoutExtension = 30 * time.Second
+	// systemdStartTimeoutExtendMax caps the total duration the start timeout can be extended for.
+	systemdStartTimeoutExtendMax = 5 * time.Minute
 )
 
 type DockerService struct {
@@ -175,8 +183,14 @@ func (s *DockerService) createAndStart(ctx context.Context) error {
 		}
 		defer respBody.Close()
 
+		// The pull on the first daemon start may take longer than the systemd service start timeout (TimeoutStartSec)
+		// on a slow connection. Keep extending the timeout while the pull is in progress so systemd doesn't kill
+		// the daemon before it reports readiness. This is a no-op if not running under systemd.
+		stopExtending := extendSystemdStartTimeout(ctx)
 		// Wait for pull to complete.
-		if _, err := io.Copy(io.Discard, respBody); err != nil {
+		_, err = io.Copy(io.Discard, respBody)
+		stopExtending()
+		if err != nil {
 			return fmt.Errorf("read pull response: %w", err)
 		}
 		slog.Info("Docker image pulled.", "image", s.Image, "duration", time.Since(start).String())
@@ -192,3 +206,40 @@ func (s *DockerService) createAndStart(ctx context.Context) error {
 	}
 	return nil
 }
+
+// extendSystemdStartTimeout periodically asks systemd to extend the service start timeout, for up to
+// systemdStartTimeoutExtendMax. The returned function stops the extensions. It is a no-op when the service
+// is not running under systemd (NOTIFY_SOCKET is not set).
+func extendSystemdStartTimeout(ctx context.Context) (stop context.CancelFunc) {
+	ctx, cancel := context.WithCancel(ctx)
+
+	go func() {
+		// Send extensions more frequently than they expire so a single missed tick doesn't time out the start.
+		ticker := time.NewTicker(systemdStartTimeoutExtension / 3)
+		defer ticker.Stop()
+		deadline := time.Now().Add(systemdStartTimeoutExtendMax)
+
+		for {
+			select {
+			case <-ctx.Done():
+				return
+			case <-ticker.C:
+				if time.Now().After(deadline) {
+					slog.Warn("Stopped extending the systemd service start timeout: corrosion service is taking "+
+						"too long to start.", "timeout", systemdStartTimeoutExtendMax)
+					return
+				}
+				msg := fmt.Sprintf("EXTEND_TIMEOUT_USEC=%d", systemdStartTimeoutExtension.Microseconds())
+				if _, err := systemd.SdNotify(false, msg); err != nil {
+					slog.Warn("Failed to extend the systemd service start timeout when starting corrosion service.",
+						"err", err)
+					return
+				}
+				slog.Info("Extended systemd service start timeout while corrosion service is starting.",
+					"duration", systemdStartTimeoutExtension)
+			}
+		}
+	}()
+
+	return cancel
+}

internal/machine/docker/server.go

@@ -563,6 +563,8 @@ func (s *Server) CreateServiceContainer(
 			api.LabelManaged:     "",
 		},
 		User:      spec.Container.User,
+		Tty:       spec.Container.Tty,
+		OpenStdin: spec.Container.OpenStdin,
 	}
 	if spec.Mode == "" {
 		config.Labels[api.LabelServiceMode] = api.ServiceModeReplicated

internal/machine/docker/service.go

@@ -1,12 +1,14 @@
 package docker
 
 import (
+	"bufio"
 	"bytes"
 	"context"
 	"database/sql"
 	"encoding/json"
 	"errors"
 	"fmt"
+	"io"
 	"log/slog"
 	"strconv"
 	"strings"
@@ -233,11 +235,17 @@ func (s *Service) ListImages(ctx context.Context, opts image.ListOptions) (Image
 	return imagesResp, nil
 }
 
-// ContainerLogs streams logs from a container and returns demultiplexed entries via a channel.
+// ContainerLogs streams logs from a container and returns entries via a channel.
 // The channel is closed when streaming completes or context is cancelled.
 func (s *Service) ContainerLogs(
 	ctx context.Context, containerID string, opts api.ServiceLogsOptions,
 ) (<-chan api.LogEntry, error) {
+	ctr, err := s.Client.ContainerInspect(ctx, containerID)
+	if err != nil {
+		return nil, fmt.Errorf("inspect container '%s': %w", containerID, err)
+	}
+	isTTY := ctr.Config != nil && ctr.Config.Tty
+
 	dockerOpts := container.LogsOptions{
 		ShowStdout: true,
 		ShowStderr: true,
@@ -257,25 +265,31 @@ func (s *Service) ContainerLogs(
 	stdoutWriter := &logsChannelWriter{ctx: ctx, ch: outCh, isStderr: false}
 	stderrWriter := &logsChannelWriter{ctx: ctx, ch: outCh, isStderr: true}
 
-	// Wrap the context in a cancellable one to unblock the second goroutine below when StdCopy completes.
+	// Wrap the context in a cancellable one to unblock the second goroutine when log copying completes.
 	ctx, cancel := context.WithCancel(ctx)
 
-	// Run StdCopy in a goroutine to be able to handle context cancellation.
+	// Copy logs in a goroutine to be able to handle context cancellation.
 	go func() {
 		defer close(outCh)
 		defer cancel()
 
-		// StdCopy is blocking and will return when the reader is closed in another goroutine below or on error.
-		if _, err := stdcopy.StdCopy(stdoutWriter, stderrWriter, reader); err != nil {
+		// Docker returns raw stdout for TTY containers and multiplexed stdout/stderr otherwise.
+		var err error
+		if isTTY {
+			_, err = copyRawContainerLogs(stdoutWriter, reader)
+		} else {
+			_, err = stdcopy.StdCopy(stdoutWriter, stderrWriter, reader)
+		}
+		if err != nil {
 			// Send error as the last entry.
 			select {
-			case outCh <- api.LogEntry{Err: fmt.Errorf("demultiplex container logs: %w", err)}:
+			case outCh <- api.LogEntry{Err: fmt.Errorf("copy container logs: %w", err)}:
 			case <-ctx.Done():
 			}
 		}
 	}()
 
-	// Close the reader when the context is done to cancel StdCopy if it's still running.
+	// Close the reader when the context is done to cancel log copying if it's still running.
 	go func() {
 		<-ctx.Done()
 		reader.Close()
@@ -284,7 +298,32 @@ func (s *Service) ContainerLogs(
 	return outCh, nil
 }
 
-// logsChannelWriter is a writer for stdcopy.StdCopy that sends demultiplexed container logs to a channel.
+// copyRawContainerLogs copies a raw TTY log stream one line at a time so each write produces one log entry.
+func copyRawContainerLogs(dst io.Writer, src io.Reader) (written int64, _ error) {
+	reader := bufio.NewReader(src)
+	for {
+		line, readErr := reader.ReadBytes('\n')
+		if len(line) > 0 {
+			n, writeErr := dst.Write(line)
+			written += int64(n)
+			if writeErr != nil {
+				return written, writeErr
+			}
+			if n != len(line) {
+				return written, io.ErrShortWrite
+			}
+		}
+
+		if readErr != nil {
+			if errors.Is(readErr, io.EOF) {
+				return written, nil
+			}
+			return written, readErr
+		}
+	}
+}
+
+// logsChannelWriter sends container log writes to a channel.
 type logsChannelWriter struct {
 	ctx      context.Context
 	ch       chan<- api.LogEntry

internal/machine/docker/service_test.go

@@ -0,0 +1,122 @@
+package docker
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"strings"
+	"testing"
+
+	"github.com/docker/docker/api/types/container"
+	dockerclient "github.com/docker/docker/client"
+	"github.com/docker/docker/pkg/stdcopy"
+	"github.com/psviderski/uncloud/pkg/api"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestServiceContainerLogs(t *testing.T) {
+	t.Parallel()
+
+	const (
+		containerID = "container-id"
+		firstLog    = "2025-01-01T00:00:00.000000000Z first message\n"
+		secondLog   = "2025-01-01T00:00:01.000000000Z second message\n"
+	)
+
+	var multiplexedLogs bytes.Buffer
+	_, err := stdcopy.NewStdWriter(&multiplexedLogs, stdcopy.Stdout).Write([]byte(firstLog))
+	require.NoError(t, err)
+	_, err = stdcopy.NewStdWriter(&multiplexedLogs, stdcopy.Stderr).Write([]byte(secondLog))
+	require.NoError(t, err)
+
+	tests := []struct {
+		name     string
+		tty      bool
+		logs     []byte
+		streams  []api.LogStreamType
+		messages []string
+	}{
+		{
+			name:     "TTY raw stream",
+			tty:      true,
+			logs:     []byte(firstLog + secondLog),
+			streams:  []api.LogStreamType{api.LogStreamStdout, api.LogStreamStdout},
+			messages: []string{"first message\n", "second message\n"},
+		},
+		{
+			name:     "non-TTY multiplexed stream",
+			logs:     multiplexedLogs.Bytes(),
+			streams:  []api.LogStreamType{api.LogStreamStdout, api.LogStreamStderr},
+			messages: []string{"first message\n", "second message\n"},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			dockerClient := newLogsTestClient(t, tt.tty, tt.logs)
+			service := NewService(dockerClient, nil)
+
+			logsCh, err := service.ContainerLogs(context.Background(), containerID, api.ServiceLogsOptions{})
+			require.NoError(t, err)
+
+			var entries []api.LogEntry
+			for entry := range logsCh {
+				require.NoError(t, entry.Err)
+				entries = append(entries, entry)
+			}
+
+			require.Len(t, entries, len(tt.messages))
+			for i := range entries {
+				assert.Equal(t, tt.streams[i], entries[i].Stream)
+				assert.Equal(t, tt.messages[i], string(entries[i].Message))
+				assert.False(t, entries[i].Timestamp.IsZero())
+			}
+		})
+	}
+}
+
+func newLogsTestClient(t *testing.T, tty bool, logs []byte) *dockerclient.Client {
+	t.Helper()
+
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case strings.HasSuffix(r.URL.Path, "/containers/container-id/json"):
+			w.Header().Set("Content-Type", "application/json")
+			if err := json.NewEncoder(w).Encode(container.InspectResponse{
+				ContainerJSONBase: &container.ContainerJSONBase{ID: "container-id"},
+				Config:            &container.Config{Tty: tty},
+			}); err != nil {
+				t.Errorf("encode inspect response: %v", err)
+			}
+		case strings.HasSuffix(r.URL.Path, "/containers/container-id/logs"):
+			w.Header().Set("Content-Type", "application/vnd.docker.raw-stream")
+			if _, err := w.Write(logs); err != nil {
+				t.Errorf("write logs response: %v", err)
+			}
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	t.Cleanup(server.Close)
+
+	serverURL, err := url.Parse(server.URL)
+	require.NoError(t, err)
+
+	dockerClient, err := dockerclient.NewClientWithOpts(
+		dockerclient.WithHost("tcp://"+serverURL.Host),
+		dockerclient.WithHTTPClient(server.Client()),
+		dockerclient.WithVersion("1.48"),
+	)
+	require.NoError(t, err)
+	t.Cleanup(func() {
+		require.NoError(t, dockerClient.Close())
+	})
+
+	return dockerClient
+}

internal/machine/machine.go

@@ -42,6 +42,8 @@ import (
 	"github.com/psviderski/uncloud/internal/secret"
 	"github.com/psviderski/uncloud/internal/version"
 	"github.com/psviderski/uncloud/pkg/api"
+	"github.com/psviderski/uncloud/pkg/distlock"
+	distlockgrpc "github.com/psviderski/uncloud/pkg/distlock/grpc"
 	"github.com/psviderski/unregistry"
 	"github.com/siderolabs/grpc-proxy/proxy"
 	"golang.org/x/sync/errgroup"
@@ -305,7 +307,8 @@ func NewMachine(config *Config) (*Machine, error) {
 		WaitForNetworkReady: m.WaitForNetworkReady,
 	})
 	caddyServer := caddyconfig.NewServer(caddyconfig.NewService(config.CaddyConfigDir))
-	m.localMachineServer = newGRPCServer(m, c, m.dockerServer, caddyServer)
+	leaseServer := distlockgrpc.NewServer(distlock.NewMemoryStore())
+	m.localMachineServer = newGRPCServer(m, c, m.dockerServer, caddyServer, leaseServer)
 
 	if m.Initialised() {
 		close(m.initialised)
@@ -314,12 +317,19 @@ func NewMachine(config *Config) (*Machine, error) {
 	return m, nil
 }
 
-func newGRPCServer(m pb.MachineServer, c pb.ClusterServer, d pb.DockerServer, caddy pb.CaddyServer) *grpc.Server {
+func newGRPCServer(
+	m pb.MachineServer,
+	c pb.ClusterServer,
+	d pb.DockerServer,
+	caddy pb.CaddyServer,
+	lease distlockgrpc.LeaseServer,
+) *grpc.Server {
 	s := grpc.NewServer()
 	pb.RegisterMachineServer(s, m)
 	pb.RegisterClusterServer(s, c)
 	pb.RegisterDockerServer(s, d)
 	pb.RegisterCaddyServer(s, caddy)
+	distlockgrpc.RegisterLeaseServer(s, lease)
 	return s
 }
 

internal/proxy/proxy.go

@@ -2,11 +2,12 @@ package proxy
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"io"
 	"net"
-	"os"
 	"sync"
+	"syscall"
 	"time"
 )
 
@@ -15,58 +16,53 @@ type Proxy struct {
 	Listener    net.Listener
 	RemoteAddr  string
 	DialContext func(ctx context.Context, network, address string) (net.Conn, error)
+	// OnError is called for errors that occur during proxying individual connections. It may be called concurrently
+	// for different connections.
 	OnError     func(error)
 	activeConns sync.WaitGroup
 }
 
-// deadliner is an interface for listeners that support setting deadlines.
-type deadliner interface {
-	SetDeadline(t time.Time) error
-}
-
 // halfCloser is an interface for connections that support half-close.
 type halfCloser interface {
 	CloseWrite() error
 }
 
-// Run starts the proxy and runs until the context is canceled.
-func (p *Proxy) Run(ctx context.Context) {
+// IsConnectionClosedError reports whether err indicates that a connection was closed or aborted by either peer.
+// Callers can use it to ignore routine connection shutdown or broken pipe errors reported to Proxy.OnError.
+func IsConnectionClosedError(err error) bool {
+	return errors.Is(err, net.ErrClosed) || errors.Is(err, io.ErrClosedPipe) ||
+		errors.Is(err, syscall.EPIPE) || errors.Is(err, syscall.ECONNRESET)
+}
+
+// Run starts the proxy and runs until the context is canceled or the listener fails. It returns nil when the context
+// is canceled. Errors handling individual connections are reported to OnError and do not stop the proxy.
+func (p *Proxy) Run(ctx context.Context) error {
 	if p.DialContext == nil {
 		p.DialContext = (&net.Dialer{}).DialContext
 	}
 
+	ctx, cancel := context.WithCancel(ctx)
+	defer cancel()
 	defer p.Listener.Close()
 
-	// Handle incoming connections until context is canceled.
-Loop:
+	// Closing the listener unblocks Accept when the context is canceled. This works for both TCP and Unix listeners
+	// and avoids polling with listener deadlines.
+	stopClose := context.AfterFunc(ctx, func() {
+		p.Listener.Close()
+	})
+	defer stopClose()
+
+	var runErr error
 	for {
-		select {
-		case <-ctx.Done():
-			break Loop
-		default:
-		}
-
-		// Set a deadline on the listener if supported to check context periodically.
-		if dl, ok := p.Listener.(deadliner); ok {
-			dl.SetDeadline(time.Now().Add(1 * time.Second))
-		}
-
 		conn, err := p.Listener.Accept()
 		if err != nil {
-			if os.IsTimeout(err) {
-				// Just a timeout, continue to check context and accept again.
-				continue
+			if ctx.Err() != nil {
+				break
 			}
 
-			select {
-			case <-ctx.Done():
-				break Loop
-			default:
-				if p.OnError != nil {
-					p.OnError(fmt.Errorf("accept local connection: %w", err))
-				}
-				continue
-			}
+			runErr = fmt.Errorf("accept local connection: %w", err)
+			cancel()
+			break
 		}
 
 		p.activeConns.Add(1)
@@ -75,6 +71,7 @@ Loop:
 
 	// Wait for all connections to finish.
 	p.activeConns.Wait()
+	return runErr
 }
 
 func (p *Proxy) handleConnection(ctx context.Context, localConn net.Conn) {
@@ -87,46 +84,62 @@ func (p *Proxy) handleConnection(ctx context.Context, localConn net.Conn) {
 
 	remoteConn, err := p.DialContext(dialCtx, "tcp", p.RemoteAddr)
 	if err != nil {
-		if p.OnError != nil {
+		if ctx.Err() == nil && p.OnError != nil {
 			p.OnError(fmt.Errorf("connect remote address '%s': %w", p.RemoteAddr, err))
 		}
 		return
 	}
 	defer remoteConn.Close()
 
-	// Bidirectional copy with proper half-close handling.
+	// Closing both connections aborts both copies after cancellation or a copy error. A clean EOF still uses
+	// half-close so the other direction can finish sending any remaining data.
+	closeConnections := func() {
+		localConn.Close()
+		remoteConn.Close()
+	}
+	stopClose := context.AfterFunc(ctx, closeConnections)
+	defer stopClose()
+
 	done := make(chan error, 2)
 
 	go func() {
 		_, err := io.Copy(remoteConn, localConn)
+		if err != nil {
+			done <- err
+			closeConnections()
+			return
+		}
 		// Close write half of remote connection if supported.
 		if hc, ok := remoteConn.(halfCloser); ok {
 			hc.CloseWrite()
 		}
-		done <- err
+		done <- nil
 	}()
 
 	go func() {
 		_, err := io.Copy(localConn, remoteConn)
+		if err != nil {
+			done <- err
+			closeConnections()
+			return
+		}
 		// Close write half of local connection if supported.
 		if hc, ok := localConn.(halfCloser); ok {
 			hc.CloseWrite()
 		}
-		done <- err
+		done <- nil
 	}()
 
-	// Wait for both copies to complete or context cancel.
+	// Wait for both copies to complete. The first error is the original failure because a copy reports it before
+	// closing the connections to unblock the other copy.
+	var copyErr error
 	for range 2 {
-		select {
-		case <-ctx.Done():
-			// Close connections to abort ongoing copies.
-			localConn.Close()
-			remoteConn.Close()
-			return
-		case err = <-done:
-			if err != nil && p.OnError != nil {
-				p.OnError(fmt.Errorf("data copy: %w", err))
+		if err = <-done; err != nil && copyErr == nil {
+			copyErr = err
 		}
 	}
+
+	if copyErr != nil && ctx.Err() == nil && p.OnError != nil {
+		p.OnError(fmt.Errorf("data copy: %w", copyErr))
 	}
 }

internal/proxy/proxy_test.go

@@ -0,0 +1,188 @@
+package proxy
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"sync"
+	"sync/atomic"
+	"syscall"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestIsConnectionClosedError(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name string
+		err  error
+		want bool
+	}{
+		{name: "closed network connection", err: net.ErrClosed, want: true},
+		{name: "closed pipe", err: io.ErrClosedPipe, want: true},
+		{name: "broken pipe", err: syscall.EPIPE, want: true},
+		{name: "connection reset", err: syscall.ECONNRESET, want: true},
+		{name: "wrapped connection error", err: fmt.Errorf("copy data: %w", syscall.EPIPE), want: true},
+		{name: "other error", err: errors.New("copy failed"), want: false},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			require.Equal(t, tt.want, IsConnectionClosedError(tt.err))
+		})
+	}
+}
+
+func TestRunContinuesAfterClosedConnectionError(t *testing.T) {
+	t.Parallel()
+
+	listener := newTestListener()
+
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	closedErrCh := make(chan error, 1)
+	unexpectedErrCh := make(chan error, 1)
+	var dialCount atomic.Int32
+
+	p := &Proxy{
+		Listener:   listener,
+		RemoteAddr: "remote:80",
+		DialContext: func(context.Context, string, string) (net.Conn, error) {
+			if dialCount.Add(1) == 1 {
+				return readErrorConn{err: syscall.EPIPE}, nil
+			}
+
+			proxyConn, upstreamConn := net.Pipe()
+			go func() {
+				defer upstreamConn.Close()
+				_, _ = upstreamConn.Write([]byte("ok"))
+			}()
+			return proxyConn, nil
+		},
+		OnError: func(err error) {
+			if IsConnectionClosedError(err) {
+				closedErrCh <- err
+				return
+			}
+			unexpectedErrCh <- err
+		},
+	}
+
+	runErrCh := make(chan error, 1)
+	go func() {
+		runErrCh <- p.Run(ctx)
+	}()
+
+	firstConn := listener.connect()
+	defer firstConn.Close()
+
+	select {
+	case connErr := <-closedErrCh:
+		require.ErrorIs(t, connErr, syscall.EPIPE)
+	case <-time.After(time.Second):
+		t.Fatal("timed out waiting for closed connection error")
+	}
+
+	secondConn := listener.connect()
+	defer secondConn.Close()
+	require.NoError(t, secondConn.SetReadDeadline(time.Now().Add(time.Second)))
+
+	got := make([]byte, 2)
+	_, err := io.ReadFull(secondConn, got)
+	require.NoError(t, err)
+	require.Equal(t, "ok", string(got))
+
+	select {
+	case unexpectedErr := <-unexpectedErrCh:
+		t.Fatalf("unexpected connection error: %v", unexpectedErr)
+	default:
+	}
+
+	cancel()
+	select {
+	case runErr := <-runErrCh:
+		require.NoError(t, runErr)
+	case <-time.After(time.Second):
+		t.Fatal("timed out waiting for proxy to stop")
+	}
+}
+
+func TestRunReturnsListenerError(t *testing.T) {
+	t.Parallel()
+
+	listenerErr := errors.New("listener failed")
+	listener := errorListener{err: listenerErr}
+
+	p := &Proxy{Listener: listener}
+	err := p.Run(context.Background())
+	require.Error(t, err)
+	require.ErrorContains(t, err, "accept local connection")
+	require.ErrorIs(t, err, listenerErr)
+}
+
+type testListener struct {
+	conns     chan net.Conn
+	closed    chan struct{}
+	closeOnce sync.Once
+}
+
+func newTestListener() *testListener {
+	return &testListener{
+		conns:  make(chan net.Conn),
+		closed: make(chan struct{}),
+	}
+}
+
+func (l *testListener) connect() net.Conn {
+	clientConn, proxyConn := net.Pipe()
+	l.conns <- proxyConn
+	return clientConn
+}
+
+func (l *testListener) Accept() (net.Conn, error) {
+	select {
+	case conn := <-l.conns:
+		return conn, nil
+	case <-l.closed:
+		return nil, net.ErrClosed
+	}
+}
+
+func (l *testListener) Close() error {
+	l.closeOnce.Do(func() {
+		close(l.closed)
+	})
+	return nil
+}
+
+func (l *testListener) Addr() net.Addr {
+	return &net.TCPAddr{}
+}
+
+type errorListener struct {
+	err error
+}
+
+func (l errorListener) Accept() (net.Conn, error) { return nil, l.err }
+func (errorListener) Close() error                { return nil }
+func (errorListener) Addr() net.Addr              { return &net.TCPAddr{} }
+
+// readErrorConn fails reads immediately so tests can deterministically exercise a proxy copy failure.
+type readErrorConn struct {
+	err error
+}
+
+func (c readErrorConn) Read([]byte) (int, error)       { return 0, c.err }
+func (readErrorConn) Write(p []byte) (int, error)      { return len(p), nil }
+func (readErrorConn) Close() error                     { return nil }
+func (readErrorConn) LocalAddr() net.Addr              { return &net.TCPAddr{} }
+func (readErrorConn) RemoteAddr() net.Addr             { return &net.TCPAddr{} }
+func (readErrorConn) SetDeadline(time.Time) error      { return nil }
+func (readErrorConn) SetReadDeadline(time.Time) error  { return nil }
+func (readErrorConn) SetWriteDeadline(time.Time) error { return nil }

misc/user_guide.md

@@ -22,7 +22,7 @@ Uncloud stores its configuration in `~/.config/uncloud/config.yaml`. If you wish
 
 Begin by initialising the first node in your cluster with `uc machine init [USER@HOST:PORT]`. If you do not have a need for Caddy reverse proxy, you may disable this feature with `--no-caddy`. If you want to avoid using uncloud's managed DNS service, add the `--no-dns` flag.
 
-This command will idempotently install Docker, uncloudd, uncloud-corrosion. If Caddy is enabled, it will set up a reverse proxy. If Uncloud DNS is enabled, it will create a DNS A record for the machine's public IP address under `*.[CLUSTER ID].uncld.dev`.
+This command will idempotently install Docker and uncloudd. If Caddy is enabled, it will set up a reverse proxy. If Uncloud DNS is enabled, it will create a DNS A record for the machine's public IP address under `*.[CLUSTER ID].uncld.dev`.
 
 If you wish to uninstall Uncloud and its components, run `uncloud-uninstall`.
 

mise.toml

@@ -44,7 +44,7 @@ run = "mise lock --platform linux-x64,macos-arm64"
 description = "Regenerate gRPC API code from .proto files"
 run = """
 protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative \
-  --proto_path=. --proto_path=internal/machine/api/vendor internal/machine/api/pb/*.proto
+  --proto_path=. --proto_path=internal/machine/api/vendor internal/machine/api/pb/*.proto pkg/distlock/grpc/*.proto
 """
 
 [tasks.uc]

pkg/api/service.go

@@ -265,6 +265,11 @@ type ContainerSpec struct {
 	LogDriver *LogDriver
 	// PidMode sets the PID namespace mode for the container. Currently only "" or "host" is supported.
 	PidMode string
+	// Tty allocates a pseudo-TTY and connects the container's standard streams to it.
+	// Standard output and standard error share one stream.
+	Tty bool
+	// OpenStdin allocates standard input and keeps it open.
+	OpenStdin bool
 	// Privileged gives extended privileges to the container. This is a security risk and should be used with caution.
 	Privileged bool
 	// PullPolicy determines when to pull the image from the registry or use the image already available in the cluster.

pkg/client/client.go

@@ -10,6 +10,7 @@ import (
 	"github.com/psviderski/uncloud/internal/machine/api/pb"
 	"github.com/psviderski/uncloud/internal/machine/docker"
 	"github.com/psviderski/uncloud/pkg/api"
+	distlockgrpc "github.com/psviderski/uncloud/pkg/distlock/grpc"
 	"golang.org/x/net/proxy"
 	"google.golang.org/grpc"
 	"google.golang.org/grpc/metadata"
@@ -28,6 +29,7 @@ type Client struct {
 	// Docker is a namespaced client for the Docker service to distinguish Uncloud-specific service container operations
 	// from generic Docker operations.
 	Docker *docker.Client
+	leases distlockgrpc.LeaseClient
 }
 
 var _ api.Client = (*Client)(nil)
@@ -57,6 +59,7 @@ func New(ctx context.Context, connector Connector) (*Client, error) {
 	c.ClusterClient = pb.NewClusterClient(c.conn)
 	c.Caddy = pb.NewCaddyClient(c.conn)
 	c.Docker = docker.NewClient(c.conn)
+	c.leases = distlockgrpc.NewLeaseClient(c.conn)
 
 	return c, nil
 }
@@ -78,8 +81,8 @@ func (cli *Client) progressOut() *streams.Out {
 // ProxyMachinesContext returns a new context that proxies gRPC requests to the specified machines.
 // If namesOrIDs is nil or empty, all machines are included.
 // This triggers One2Many proxying, which always injects metadata into the response.
-func (cli *Client) ProxyMachinesContext(ctx context.Context, namesOrIDs []string) context.Context {
-	md := metadata.New(nil)
+func ProxyMachinesContext(ctx context.Context, namesOrIDs []string) context.Context {
+	md := outgoingMetadataWithoutProxyTargets(ctx)
 	if len(namesOrIDs) == 0 {
 		md.Append("machines", "*")
 	} else {
@@ -92,7 +95,26 @@ func (cli *Client) ProxyMachinesContext(ctx context.Context, namesOrIDs []string
 // ProxySingleMachineContext returns a new context that proxies gRPC requests to a single specified machine.
 // This triggers One2One proxying, which does NOT inject metadata into the response.
 // Use this for requests that expect a single response message without metadata wrapper.
-func (cli *Client) ProxySingleMachineContext(ctx context.Context, nameOrID string) context.Context {
-	md := metadata.Pairs("machine", nameOrID)
+func ProxySingleMachineContext(ctx context.Context, nameOrID string) context.Context {
+	md := outgoingMetadataWithoutProxyTargets(ctx)
+	md.Set("machine", nameOrID)
 	return metadata.NewOutgoingContext(ctx, md)
 }
+
+func outgoingMetadataWithoutProxyTargets(ctx context.Context) metadata.MD {
+	md, _ := metadata.FromOutgoingContext(ctx)
+	md = md.Copy()
+	md.Delete("machine")
+	md.Delete("machines")
+	return md
+}
+
+// ProxyMachinesContext returns a new context that proxies gRPC requests to the specified machines.
+func (cli *Client) ProxyMachinesContext(ctx context.Context, namesOrIDs []string) context.Context {
+	return ProxyMachinesContext(ctx, namesOrIDs)
+}
+
+// ProxySingleMachineContext returns a new context that proxies gRPC requests to a single specified machine.
+func (cli *Client) ProxySingleMachineContext(ctx context.Context, nameOrID string) context.Context {
+	return ProxySingleMachineContext(ctx, nameOrID)
+}

pkg/client/client_test.go

@@ -0,0 +1,79 @@
+package client
+
+import (
+	"context"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+	"google.golang.org/grpc/metadata"
+)
+
+func TestProxySingleMachineContext(t *testing.T) {
+	original := metadata.Pairs(
+		"authorization", "token",
+		"machine", "old-machine",
+		"machines", "old-machine-a",
+		"machines", "old-machine-b",
+	)
+	ctx := metadata.NewOutgoingContext(context.Background(), original)
+
+	proxyCtx := ProxySingleMachineContext(ctx, "new-machine")
+
+	md, ok := metadata.FromOutgoingContext(proxyCtx)
+	require.True(t, ok)
+	require.Equal(t, metadata.Pairs(
+		"authorization", "token",
+		"machine", "new-machine",
+	), md)
+	require.Equal(t, metadata.Pairs(
+		"authorization", "token",
+		"machine", "old-machine",
+		"machines", "old-machine-a",
+		"machines", "old-machine-b",
+	), original)
+}
+
+func TestProxyMachinesContext(t *testing.T) {
+	tests := []struct {
+		name     string
+		machines []string
+		want     []string
+	}{
+		{
+			name:     "specified machines",
+			machines: []string{"machine-a", "machine-b"},
+			want:     []string{"machine-a", "machine-b"},
+		},
+		{
+			name: "all machines",
+			want: []string{"*"},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			original := metadata.Pairs(
+				"authorization", "token",
+				"machine", "old-machine",
+				"machines", "old-machine-a",
+				"machines", "old-machine-b",
+			)
+			ctx := metadata.NewOutgoingContext(context.Background(), original)
+
+			proxyCtx := ProxyMachinesContext(ctx, tt.machines)
+
+			md, ok := metadata.FromOutgoingContext(proxyCtx)
+			require.True(t, ok)
+			require.Equal(t, metadata.MD{
+				"authorization": {"token"},
+				"machines":      tt.want,
+			}, md)
+			require.Equal(t, metadata.Pairs(
+				"authorization", "token",
+				"machine", "old-machine",
+				"machines", "old-machine-a",
+				"machines", "old-machine-b",
+			), original)
+		})
+	}
+}

pkg/client/compose/service.go

@@ -54,6 +54,8 @@ func ServiceSpecFromCompose(project *types.Project, serviceName string) (api.Ser
 			Image:       service.Image,
 			Init:        service.Init,
 			PidMode:     service.Pid,
+			Tty:         service.Tty,
+			OpenStdin:   service.StdinOpen,
 			Privileged:  service.Privileged,
 			PullPolicy:  pullPolicy,
 			Resources:   resourcesFromCompose(service),

pkg/client/compose/service_test.go

@@ -128,6 +128,8 @@ func TestServiceSpecFromCompose(t *testing.T) {
 							},
 						},
 						PidMode:    "host",
+						Tty:        true,
+						OpenStdin:  true,
 						Privileged: true,
 						PullPolicy: api.PullPolicyAlways,
 						Resources: api.ContainerResources{

pkg/client/compose/testdata/compose-full-spec.yaml

@@ -7,6 +7,8 @@ services:
     command: ["nginx", "updated", "command"]
     cpus: 0.5
     pid: host
+    tty: true
+    stdin_open: true
     deploy:
       update_config:
         order: stop-first

pkg/client/container.go

@@ -74,7 +74,7 @@ func (cli *Client) createServiceContainerWithPull(
 	resp.Name = containerName
 
 	// Proxy Docker gRPC requests to the selected machine.
-	ctx = cli.ProxySingleMachineContext(ctx, machine.Machine.Id)
+	ctx = ProxySingleMachineContext(ctx, machine.Machine.Id)
 
 	pw := progress.ContextWriter(ctx)
 	eventID := cliprogress.NewContainerEventID(ctx, containerName, machine.Machine.Name)
@@ -277,7 +277,7 @@ func (cli *Client) resolveContainerOperation(
 
 	eventID := cliprogress.ContainerEventID(ctx, ctr.Container.ServiceSpec.Name, ctr.Container.ID, ctr.MachineName)
 	return containerOperationContext{
-		ctx:         cli.ProxySingleMachineContext(ctx, ctr.MachineID),
+		ctx:         ProxySingleMachineContext(ctx, ctr.MachineID),
 		containerID: ctr.Container.ID,
 		eventID:     eventID,
 	}, nil
@@ -375,7 +375,7 @@ func (cli *Client) ExecContainer(
 	}
 
 	// Proxy Docker gRPC requests to the machine hosting the container
-	ctx = cli.ProxySingleMachineContext(ctx, machine.Machine.Id)
+	ctx = ProxySingleMachineContext(ctx, machine.Machine.Id)
 
 	// Execute the command in the container
 	exitCode, err := cli.Docker.ExecContainer(ctx, machinedocker.ExecConfig{
@@ -452,7 +452,7 @@ func (cli *Client) WaitContainerHealthy(
 	}
 
 	// For containers with a health check, wait until Docker reports healthy or unhealthy.
-	mctx := cli.ProxySingleMachineContext(ctx, machine.Machine.Id)
+	mctx := ProxySingleMachineContext(ctx, machine.Machine.Id)
 	mctx, cancel := context.WithTimeout(mctx, healthcheckTimeout(mc.Container.Config.Healthcheck))
 	defer cancel()
 	ticker := time.NewTicker(1 * time.Second)

pkg/client/deploy/container_test.go

@@ -66,6 +66,26 @@ func TestEvalContainerSpecChange_ContainerPidMode(t *testing.T) {
 	assert.Equal(t, ContainerNeedsRecreate, EvalContainerSpecChange(newSpec, currentSpec))
 }
 
+func TestEvalContainerSpecChange_ContainerTty(t *testing.T) {
+	t.Parallel()
+
+	currentSpec := api.ServiceSpec{
+		Container: api.ContainerSpec{
+			Image: "nginx:latest",
+		},
+	}
+	newSpec := api.ServiceSpec{
+		Container: api.ContainerSpec{
+			Image:     "nginx:latest",
+			Tty:       true,
+			OpenStdin: true,
+		},
+	}
+
+	assert.Equal(t, ContainerNeedsRecreate, EvalContainerSpecChange(currentSpec, newSpec))
+	assert.Equal(t, ContainerNeedsRecreate, EvalContainerSpecChange(newSpec, currentSpec))
+}
+
 func TestEvalContainerSpecChange_ContainerResources(t *testing.T) {
 	t.Parallel()
 

pkg/client/image.go

@@ -59,7 +59,7 @@ func (cli *Client) InspectRemoteImage(ctx context.Context, id string) ([]api.Mac
 // it lists images on all machines.
 func (cli *Client) ListImages(ctx context.Context, filter api.ImageFilter) ([]api.MachineImages, error) {
 	// Broadcast the image list request to the specified machines or all machines if none specified.
-	listCtx := cli.ProxyMachinesContext(ctx, filter.Machines)
+	listCtx := ProxyMachinesContext(ctx, filter.Machines)
 
 	opts := image.ListOptions{Manifests: true}
 	if filter.Name != "" {
@@ -252,13 +252,19 @@ func (cli *Client) pushImageToMachine(
 	// The proxy runs in a goroutine. Capture the first error in a channel
 	// so we can surface it alongside the push error if push fails.
 	proxyErrCh := make(chan error, 1)
-	onProxyError := func(err error) {
+	recordProxyError := func(err error) {
 		select {
 		case proxyErrCh <- fmt.Errorf("proxy to unregistry: %w", err):
 		default:
 		}
 		pw.Event(progress.NewEvent(proxyEventID, progress.Error, err.Error()))
 	}
+	onProxyError := func(err error) {
+		if proxy.IsConnectionClosedError(err) {
+			return
+		}
+		recordProxyError(err)
+	}
 
 	// socketPath is set for plain rootless Docker (not running inside a VM): the Go proxy listens on a unix
 	// socket that is bind-mounted into the socat container, bypassing slirp4netns network routing entirely.
@@ -315,7 +321,11 @@ func (cli *Client) pushImageToMachine(
 	}
 	defer cleanup()
 
-	go unregProxy.Run(proxyCtx)
+	go func() {
+		if err := unregProxy.Run(proxyCtx); err != nil {
+			recordProxyError(err)
+		}
+	}()
 
 	if dockerEnv.Virtualised {
 		// VM-based Docker (Docker Desktop, Rancher Desktop, etc.): run a socat container inside the VM

pkg/client/locker.go

@@ -0,0 +1,95 @@
+package client
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	"github.com/psviderski/uncloud/pkg/distlock"
+	distlockgrpc "github.com/psviderski/uncloud/pkg/distlock/grpc"
+	"google.golang.org/protobuf/types/known/durationpb"
+)
+
+// NewLocker creates a distlock.Locker that acquires automatically renewed distributed leases over the machines
+// in the cluster. The Locker uses the client's connection, so callers must release its active leases before closing
+// the client.
+func (cli *Client) NewLocker(config distlock.Config) (*distlock.Locker, error) {
+	return distlock.New(&lockCluster{client: cli}, config)
+}
+
+type lockCluster struct {
+	client *Client
+}
+
+var _ distlock.Cluster = (*lockCluster)(nil)
+
+// Nodes returns a point-in-time snapshot of the registered machines in the cluster, including temporarily unavailable
+// ones. The Locker calls Nodes at the start of each Acquire and retains the returned snapshot across acquisition
+// retries and for the lifetime of any acquired lease.
+//
+// Adding or removing machines, combined with eventual replication of the machine list, can cause different
+// acquisitions to use different snapshots while active leases continue using older ones. This adapter does not
+// version snapshots or coordinate membership transitions. If old and new snapshots allow disjoint quorums, two
+// clients can acquire leases for the same resource. Membership changes must preserve quorum overlap while leases from
+// older snapshots may remain valid.
+func (c *lockCluster) Nodes(ctx context.Context) ([]distlock.Node, error) {
+	machines, err := c.client.ListMachines(ctx, nil)
+	if err != nil {
+		return nil, fmt.Errorf("list machines: %w", err)
+	}
+
+	nodes := make([]distlock.Node, 0, len(machines))
+	for _, m := range machines {
+		nodes = append(nodes, &lockNode{
+			id:     m.Machine.Id,
+			leases: c.client.leases,
+		})
+	}
+	return nodes, nil
+}
+
+type lockNode struct {
+	id     string
+	leases distlockgrpc.LeaseClient
+}
+
+var _ distlock.Node = (*lockNode)(nil)
+
+func (n *lockNode) Acquire(
+	ctx context.Context, resource string, token []byte, ttl time.Duration,
+) (bool, error) {
+	resp, err := n.leases.Acquire(ProxySingleMachineContext(ctx, n.id), &distlockgrpc.AcquireLeaseRequest{
+		Resource: resource,
+		Token:    token,
+		Ttl:      durationpb.New(ttl),
+	})
+	if err != nil {
+		return false, err
+	}
+	return resp.Acquired, nil
+}
+
+func (n *lockNode) Renew(
+	ctx context.Context, resource string, token []byte, ttl time.Duration,
+) (bool, error) {
+	resp, err := n.leases.Renew(ProxySingleMachineContext(ctx, n.id), &distlockgrpc.RenewLeaseRequest{
+		Resource: resource,
+		Token:    token,
+		Ttl:      durationpb.New(ttl),
+	})
+	if err != nil {
+		return false, err
+	}
+	return resp.Renewed, nil
+}
+
+func (n *lockNode) Release(ctx context.Context, resource string, token []byte) (bool, error) {
+	resp, err := n.leases.Release(ProxySingleMachineContext(ctx, n.id), &distlockgrpc.ReleaseLeaseRequest{
+		Resource: resource,
+		Token:    token,
+	})
+	if err != nil {
+		return false, err
+	}
+	return resp.Released, nil
+}

pkg/client/logs.go

@@ -102,7 +102,7 @@ func (cli *Client) ServiceLogs(
 func (cli *Client) ContainerLogs(
 	ctx context.Context, machineNameOrID string, containerID string, opts api.ServiceLogsOptions,
 ) (<-chan api.LogEntry, error) {
-	proxyCtx := cli.ProxySingleMachineContext(ctx, machineNameOrID)
+	proxyCtx := ProxySingleMachineContext(ctx, machineNameOrID)
 
 	req := &pb.LogsRequest{
 		Id:     containerID,
@@ -198,7 +198,7 @@ func (cli *Client) MachineLogs(
 func (cli *Client) systemServiceLogs(
 	ctx context.Context, machineID, service string, opts api.ServiceLogsOptions,
 ) (<-chan api.LogEntry, error) {
-	proxyCtx := cli.ProxySingleMachineContext(ctx, machineID)
+	proxyCtx := ProxySingleMachineContext(ctx, machineID)
 
 	req := &pb.LogsRequest{
 		Id:     service,

pkg/client/machine.go

@@ -78,7 +78,7 @@ func (cli *Client) ListMachines(ctx context.Context, filter *api.MachineFilter)
 func (cli *Client) UpdateMachine(
 	ctx context.Context, nameOrID string, req *pb.UpdateMachineRequest,
 ) (*pb.MachineInfo, error) {
-	ctx = cli.ProxySingleMachineContext(ctx, nameOrID)
+	ctx = ProxySingleMachineContext(ctx, nameOrID)
 	resp, err := cli.MachineClient.UpdateMachine(ctx, req)
 	if err != nil {
 		if s, ok := status.FromError(err); ok && s.Code() == codes.NotFound {

pkg/client/volume.go

@@ -27,7 +27,7 @@ func (cli *Client) CreateVolume(
 		return resp, fmt.Errorf("inspect machine '%s': %w", machineNameOrID, err)
 	}
 	// Proxy Docker gRPC requests to the selected machine.
-	ctx = cli.ProxySingleMachineContext(ctx, machine.Machine.Id)
+	ctx = ProxySingleMachineContext(ctx, machine.Machine.Id)
 
 	pw := progress.ContextWriter(ctx)
 	eventID := cliprogress.VolumeEventID(opts.Name, machine.Machine.Name)
@@ -56,7 +56,7 @@ func (cli *Client) ListVolumes(ctx context.Context, filter *api.VolumeFilter) ([
 		proxyMachines = filter.Machines
 	}
 
-	listCtx := cli.ProxyMachinesContext(ctx, proxyMachines)
+	listCtx := ProxyMachinesContext(ctx, proxyMachines)
 	machineVolumes, err := cli.Docker.ListVolumes(listCtx, volume.ListOptions{})
 	if err != nil {
 		return nil, err
@@ -107,7 +107,7 @@ func (cli *Client) RemoveVolume(ctx context.Context, machineNameOrID, volumeName
 		return fmt.Errorf("inspect machine '%s': %w", machineNameOrID, err)
 	}
 	// Proxy Docker gRPC requests to the selected machine.
-	ctx = cli.ProxySingleMachineContext(ctx, machine.Machine.Id)
+	ctx = ProxySingleMachineContext(ctx, machine.Machine.Id)
 
 	pw := progress.ContextWriter(ctx)
 	eventID := cliprogress.VolumeEventID(volumeName, machine.Machine.Name)

pkg/distlock/doc.go

@@ -0,0 +1,12 @@
+// Package distlock provides distributed, automatically renewed leases across independent nodes.
+//
+// Its quorum and lease semantics are based on the Redlock algorithm described at
+// https://redis.io/docs/latest/develop/clients/patterns/distributed-locks/. The core package is independent of storage
+// and network transport and does not require Redis. Applications that communicate with remote nodes over gRPC can use
+// the grpc subpackage to expose and call node-local lease operations.
+//
+// A Cluster must return every node in the lock group, including temporarily unavailable nodes, because every node
+// counts toward quorum. Changing the node set is unsafe if a new quorum can be disjoint from an earlier quorum while
+// leases acquired from the earlier node set may still be valid. Callers must stop protected work when the context
+// returned by Lease.Context is done.
+package distlock

pkg/distlock/grpc/lease.pb.go

@@ -0,0 +1,547 @@
+// Code generated by protoc-gen-go. DO NOT EDIT.
+// versions:
+// 	protoc-gen-go v1.34.2
+// 	protoc        v5.27.3
+// source: pkg/distlock/grpc/lease.proto
+
+package grpc
+
+import (
+	protoreflect "google.golang.org/protobuf/reflect/protoreflect"
+	protoimpl "google.golang.org/protobuf/runtime/protoimpl"
+	durationpb "google.golang.org/protobuf/types/known/durationpb"
+	reflect "reflect"
+	sync "sync"
+)
+
+const (
+	// Verify that this generated code is sufficiently up-to-date.
+	_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
+	// Verify that runtime/protoimpl is sufficiently up-to-date.
+	_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
+)
+
+type AcquireLeaseRequest struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Resource string `protobuf:"bytes,1,opt,name=resource,proto3" json:"resource,omitempty"`
+	// Token uniquely identifies the lease owner.
+	Token []byte `protobuf:"bytes,2,opt,name=token,proto3" json:"token,omitempty"`
+	// TTL sets how long the lease remains valid without renewal.
+	Ttl *durationpb.Duration `protobuf:"bytes,3,opt,name=ttl,proto3" json:"ttl,omitempty"`
+}
+
+func (x *AcquireLeaseRequest) Reset() {
+	*x = AcquireLeaseRequest{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_pkg_distlock_grpc_lease_proto_msgTypes[0]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *AcquireLeaseRequest) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*AcquireLeaseRequest) ProtoMessage() {}
+
+func (x *AcquireLeaseRequest) ProtoReflect() protoreflect.Message {
+	mi := &file_pkg_distlock_grpc_lease_proto_msgTypes[0]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use AcquireLeaseRequest.ProtoReflect.Descriptor instead.
+func (*AcquireLeaseRequest) Descriptor() ([]byte, []int) {
+	return file_pkg_distlock_grpc_lease_proto_rawDescGZIP(), []int{0}
+}
+
+func (x *AcquireLeaseRequest) GetResource() string {
+	if x != nil {
+		return x.Resource
+	}
+	return ""
+}
+
+func (x *AcquireLeaseRequest) GetToken() []byte {
+	if x != nil {
+		return x.Token
+	}
+	return nil
+}
+
+func (x *AcquireLeaseRequest) GetTtl() *durationpb.Duration {
+	if x != nil {
+		return x.Ttl
+	}
+	return nil
+}
+
+type AcquireLeaseResponse struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	// Acquired is true when this request successfully created the lease.
+	Acquired bool `protobuf:"varint,1,opt,name=acquired,proto3" json:"acquired,omitempty"`
+}
+
+func (x *AcquireLeaseResponse) Reset() {
+	*x = AcquireLeaseResponse{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_pkg_distlock_grpc_lease_proto_msgTypes[1]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *AcquireLeaseResponse) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*AcquireLeaseResponse) ProtoMessage() {}
+
+func (x *AcquireLeaseResponse) ProtoReflect() protoreflect.Message {
+	mi := &file_pkg_distlock_grpc_lease_proto_msgTypes[1]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use AcquireLeaseResponse.ProtoReflect.Descriptor instead.
+func (*AcquireLeaseResponse) Descriptor() ([]byte, []int) {
+	return file_pkg_distlock_grpc_lease_proto_rawDescGZIP(), []int{1}
+}
+
+func (x *AcquireLeaseResponse) GetAcquired() bool {
+	if x != nil {
+		return x.Acquired
+	}
+	return false
+}
+
+type RenewLeaseRequest struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Resource string `protobuf:"bytes,1,opt,name=resource,proto3" json:"resource,omitempty"`
+	// Token identifies the owner of the existing lease.
+	Token []byte `protobuf:"bytes,2,opt,name=token,proto3" json:"token,omitempty"`
+	// TTL sets how long the renewed lease remains valid.
+	Ttl *durationpb.Duration `protobuf:"bytes,3,opt,name=ttl,proto3" json:"ttl,omitempty"`
+}
+
+func (x *RenewLeaseRequest) Reset() {
+	*x = RenewLeaseRequest{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_pkg_distlock_grpc_lease_proto_msgTypes[2]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *RenewLeaseRequest) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*RenewLeaseRequest) ProtoMessage() {}
+
+func (x *RenewLeaseRequest) ProtoReflect() protoreflect.Message {
+	mi := &file_pkg_distlock_grpc_lease_proto_msgTypes[2]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use RenewLeaseRequest.ProtoReflect.Descriptor instead.
+func (*RenewLeaseRequest) Descriptor() ([]byte, []int) {
+	return file_pkg_distlock_grpc_lease_proto_rawDescGZIP(), []int{2}
+}
+
+func (x *RenewLeaseRequest) GetResource() string {
+	if x != nil {
+		return x.Resource
+	}
+	return ""
+}
+
+func (x *RenewLeaseRequest) GetToken() []byte {
+	if x != nil {
+		return x.Token
+	}
+	return nil
+}
+
+func (x *RenewLeaseRequest) GetTtl() *durationpb.Duration {
+	if x != nil {
+		return x.Ttl
+	}
+	return nil
+}
+
+type RenewLeaseResponse struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	// Renewed is true when an unexpired lease matched the ownership token and was successfully renewed.
+	Renewed bool `protobuf:"varint,1,opt,name=renewed,proto3" json:"renewed,omitempty"`
+}
+
+func (x *RenewLeaseResponse) Reset() {
+	*x = RenewLeaseResponse{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_pkg_distlock_grpc_lease_proto_msgTypes[3]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *RenewLeaseResponse) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*RenewLeaseResponse) ProtoMessage() {}
+
+func (x *RenewLeaseResponse) ProtoReflect() protoreflect.Message {
+	mi := &file_pkg_distlock_grpc_lease_proto_msgTypes[3]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use RenewLeaseResponse.ProtoReflect.Descriptor instead.
+func (*RenewLeaseResponse) Descriptor() ([]byte, []int) {
+	return file_pkg_distlock_grpc_lease_proto_rawDescGZIP(), []int{3}
+}
+
+func (x *RenewLeaseResponse) GetRenewed() bool {
+	if x != nil {
+		return x.Renewed
+	}
+	return false
+}
+
+type ReleaseLeaseRequest struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Resource string `protobuf:"bytes,1,opt,name=resource,proto3" json:"resource,omitempty"`
+	// Token identifies the owner of the existing lease.
+	Token []byte `protobuf:"bytes,2,opt,name=token,proto3" json:"token,omitempty"`
+}
+
+func (x *ReleaseLeaseRequest) Reset() {
+	*x = ReleaseLeaseRequest{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_pkg_distlock_grpc_lease_proto_msgTypes[4]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *ReleaseLeaseRequest) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ReleaseLeaseRequest) ProtoMessage() {}
+
+func (x *ReleaseLeaseRequest) ProtoReflect() protoreflect.Message {
+	mi := &file_pkg_distlock_grpc_lease_proto_msgTypes[4]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ReleaseLeaseRequest.ProtoReflect.Descriptor instead.
+func (*ReleaseLeaseRequest) Descriptor() ([]byte, []int) {
+	return file_pkg_distlock_grpc_lease_proto_rawDescGZIP(), []int{4}
+}
+
+func (x *ReleaseLeaseRequest) GetResource() string {
+	if x != nil {
+		return x.Resource
+	}
+	return ""
+}
+
+func (x *ReleaseLeaseRequest) GetToken() []byte {
+	if x != nil {
+		return x.Token
+	}
+	return nil
+}
+
+type ReleaseLeaseResponse struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	// Released is true when an unexpired lease existed, matched the ownership token, and was successfully released.
+	Released bool `protobuf:"varint,1,opt,name=released,proto3" json:"released,omitempty"`
+}
+
+func (x *ReleaseLeaseResponse) Reset() {
+	*x = ReleaseLeaseResponse{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_pkg_distlock_grpc_lease_proto_msgTypes[5]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *ReleaseLeaseResponse) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ReleaseLeaseResponse) ProtoMessage() {}
+
+func (x *ReleaseLeaseResponse) ProtoReflect() protoreflect.Message {
+	mi := &file_pkg_distlock_grpc_lease_proto_msgTypes[5]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ReleaseLeaseResponse.ProtoReflect.Descriptor instead.
+func (*ReleaseLeaseResponse) Descriptor() ([]byte, []int) {
+	return file_pkg_distlock_grpc_lease_proto_rawDescGZIP(), []int{5}
+}
+
+func (x *ReleaseLeaseResponse) GetReleased() bool {
+	if x != nil {
+		return x.Released
+	}
+	return false
+}
+
+var File_pkg_distlock_grpc_lease_proto protoreflect.FileDescriptor
+
+var file_pkg_distlock_grpc_lease_proto_rawDesc = []byte{
+	0x0a, 0x1d, 0x70, 0x6b, 0x67, 0x2f, 0x64, 0x69, 0x73, 0x74, 0x6c, 0x6f, 0x63, 0x6b, 0x2f, 0x67,
+	0x72, 0x70, 0x63, 0x2f, 0x6c, 0x65, 0x61, 0x73, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12,
+	0x0b, 0x64, 0x69, 0x73, 0x74, 0x6c, 0x6f, 0x63, 0x6b, 0x2e, 0x76, 0x31, 0x1a, 0x1e, 0x67, 0x6f,
+	0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2f, 0x64, 0x75,
+	0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x74, 0x0a, 0x13,
+	0x41, 0x63, 0x71, 0x75, 0x69, 0x72, 0x65, 0x4c, 0x65, 0x61, 0x73, 0x65, 0x52, 0x65, 0x71, 0x75,
+	0x65, 0x73, 0x74, 0x12, 0x1a, 0x0a, 0x08, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18,
+	0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12,
+	0x14, 0x0a, 0x05, 0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x05,
+	0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x12, 0x2b, 0x0a, 0x03, 0x74, 0x74, 0x6c, 0x18, 0x03, 0x20, 0x01,
+	0x28, 0x0b, 0x32, 0x19, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74,
+	0x6f, 0x62, 0x75, 0x66, 0x2e, 0x44, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x03, 0x74,
+	0x74, 0x6c, 0x22, 0x32, 0x0a, 0x14, 0x41, 0x63, 0x71, 0x75, 0x69, 0x72, 0x65, 0x4c, 0x65, 0x61,
+	0x73, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1a, 0x0a, 0x08, 0x61, 0x63,
+	0x71, 0x75, 0x69, 0x72, 0x65, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x08, 0x52, 0x08, 0x61, 0x63,
+	0x71, 0x75, 0x69, 0x72, 0x65, 0x64, 0x22, 0x72, 0x0a, 0x11, 0x52, 0x65, 0x6e, 0x65, 0x77, 0x4c,
+	0x65, 0x61, 0x73, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x1a, 0x0a, 0x08, 0x72,
+	0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x72,
+	0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x14, 0x0a, 0x05, 0x74, 0x6f, 0x6b, 0x65, 0x6e,
+	0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x05, 0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x12, 0x2b, 0x0a,
+	0x03, 0x74, 0x74, 0x6c, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x67, 0x6f, 0x6f,
+	0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x44, 0x75, 0x72,
+	0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x03, 0x74, 0x74, 0x6c, 0x22, 0x2e, 0x0a, 0x12, 0x52, 0x65,
+	0x6e, 0x65, 0x77, 0x4c, 0x65, 0x61, 0x73, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65,
+	0x12, 0x18, 0x0a, 0x07, 0x72, 0x65, 0x6e, 0x65, 0x77, 0x65, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28,
+	0x08, 0x52, 0x07, 0x72, 0x65, 0x6e, 0x65, 0x77, 0x65, 0x64, 0x22, 0x47, 0x0a, 0x13, 0x52, 0x65,
+	0x6c, 0x65, 0x61, 0x73, 0x65, 0x4c, 0x65, 0x61, 0x73, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73,
+	0x74, 0x12, 0x1a, 0x0a, 0x08, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x01, 0x20,
+	0x01, 0x28, 0x09, 0x52, 0x08, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x14, 0x0a,
+	0x05, 0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x05, 0x74, 0x6f,
+	0x6b, 0x65, 0x6e, 0x22, 0x32, 0x0a, 0x14, 0x52, 0x65, 0x6c, 0x65, 0x61, 0x73, 0x65, 0x4c, 0x65,
+	0x61, 0x73, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1a, 0x0a, 0x08, 0x72,
+	0x65, 0x6c, 0x65, 0x61, 0x73, 0x65, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x08, 0x52, 0x08, 0x72,
+	0x65, 0x6c, 0x65, 0x61, 0x73, 0x65, 0x64, 0x32, 0xf1, 0x01, 0x0a, 0x05, 0x4c, 0x65, 0x61, 0x73,
+	0x65, 0x12, 0x4e, 0x0a, 0x07, 0x41, 0x63, 0x71, 0x75, 0x69, 0x72, 0x65, 0x12, 0x20, 0x2e, 0x64,
+	0x69, 0x73, 0x74, 0x6c, 0x6f, 0x63, 0x6b, 0x2e, 0x76, 0x31, 0x2e, 0x41, 0x63, 0x71, 0x75, 0x69,
+	0x72, 0x65, 0x4c, 0x65, 0x61, 0x73, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x21,
+	0x2e, 0x64, 0x69, 0x73, 0x74, 0x6c, 0x6f, 0x63, 0x6b, 0x2e, 0x76, 0x31, 0x2e, 0x41, 0x63, 0x71,
+	0x75, 0x69, 0x72, 0x65, 0x4c, 0x65, 0x61, 0x73, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73,
+	0x65, 0x12, 0x48, 0x0a, 0x05, 0x52, 0x65, 0x6e, 0x65, 0x77, 0x12, 0x1e, 0x2e, 0x64, 0x69, 0x73,
+	0x74, 0x6c, 0x6f, 0x63, 0x6b, 0x2e, 0x76, 0x31, 0x2e, 0x52, 0x65, 0x6e, 0x65, 0x77, 0x4c, 0x65,
+	0x61, 0x73, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1f, 0x2e, 0x64, 0x69, 0x73,
+	0x74, 0x6c, 0x6f, 0x63, 0x6b, 0x2e, 0x76, 0x31, 0x2e, 0x52, 0x65, 0x6e, 0x65, 0x77, 0x4c, 0x65,
+	0x61, 0x73, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x4e, 0x0a, 0x07, 0x52,
+	0x65, 0x6c, 0x65, 0x61, 0x73, 0x65, 0x12, 0x20, 0x2e, 0x64, 0x69, 0x73, 0x74, 0x6c, 0x6f, 0x63,
+	0x6b, 0x2e, 0x76, 0x31, 0x2e, 0x52, 0x65, 0x6c, 0x65, 0x61, 0x73, 0x65, 0x4c, 0x65, 0x61, 0x73,
+	0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x21, 0x2e, 0x64, 0x69, 0x73, 0x74, 0x6c,
+	0x6f, 0x63, 0x6b, 0x2e, 0x76, 0x31, 0x2e, 0x52, 0x65, 0x6c, 0x65, 0x61, 0x73, 0x65, 0x4c, 0x65,
+	0x61, 0x73, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x42, 0x31, 0x5a, 0x2f, 0x67,
+	0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x70, 0x73, 0x76, 0x69, 0x64, 0x65,
+	0x72, 0x73, 0x6b, 0x69, 0x2f, 0x75, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x70, 0x6b, 0x67,
+	0x2f, 0x64, 0x69, 0x73, 0x74, 0x6c, 0x6f, 0x63, 0x6b, 0x2f, 0x67, 0x72, 0x70, 0x63, 0x62, 0x06,
+	0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
+}
+
+var (
+	file_pkg_distlock_grpc_lease_proto_rawDescOnce sync.Once
+	file_pkg_distlock_grpc_lease_proto_rawDescData = file_pkg_distlock_grpc_lease_proto_rawDesc
+)
+
+func file_pkg_distlock_grpc_lease_proto_rawDescGZIP() []byte {
+	file_pkg_distlock_grpc_lease_proto_rawDescOnce.Do(func() {
+		file_pkg_distlock_grpc_lease_proto_rawDescData = protoimpl.X.CompressGZIP(file_pkg_distlock_grpc_lease_proto_rawDescData)
+	})
+	return file_pkg_distlock_grpc_lease_proto_rawDescData
+}
+
+var file_pkg_distlock_grpc_lease_proto_msgTypes = make([]protoimpl.MessageInfo, 6)
+var file_pkg_distlock_grpc_lease_proto_goTypes = []any{
+	(*AcquireLeaseRequest)(nil),  // 0: distlock.v1.AcquireLeaseRequest
+	(*AcquireLeaseResponse)(nil), // 1: distlock.v1.AcquireLeaseResponse
+	(*RenewLeaseRequest)(nil),    // 2: distlock.v1.RenewLeaseRequest
+	(*RenewLeaseResponse)(nil),   // 3: distlock.v1.RenewLeaseResponse
+	(*ReleaseLeaseRequest)(nil),  // 4: distlock.v1.ReleaseLeaseRequest
+	(*ReleaseLeaseResponse)(nil), // 5: distlock.v1.ReleaseLeaseResponse
+	(*durationpb.Duration)(nil),  // 6: google.protobuf.Duration
+}
+var file_pkg_distlock_grpc_lease_proto_depIdxs = []int32{
+	6, // 0: distlock.v1.AcquireLeaseRequest.ttl:type_name -> google.protobuf.Duration
+	6, // 1: distlock.v1.RenewLeaseRequest.ttl:type_name -> google.protobuf.Duration
+	0, // 2: distlock.v1.Lease.Acquire:input_type -> distlock.v1.AcquireLeaseRequest
+	2, // 3: distlock.v1.Lease.Renew:input_type -> distlock.v1.RenewLeaseRequest
+	4, // 4: distlock.v1.Lease.Release:input_type -> distlock.v1.ReleaseLeaseRequest
+	1, // 5: distlock.v1.Lease.Acquire:output_type -> distlock.v1.AcquireLeaseResponse
+	3, // 6: distlock.v1.Lease.Renew:output_type -> distlock.v1.RenewLeaseResponse
+	5, // 7: distlock.v1.Lease.Release:output_type -> distlock.v1.ReleaseLeaseResponse
+	5, // [5:8] is the sub-list for method output_type
+	2, // [2:5] is the sub-list for method input_type
+	2, // [2:2] is the sub-list for extension type_name
+	2, // [2:2] is the sub-list for extension extendee
+	0, // [0:2] is the sub-list for field type_name
+}
+
+func init() { file_pkg_distlock_grpc_lease_proto_init() }
+func file_pkg_distlock_grpc_lease_proto_init() {
+	if File_pkg_distlock_grpc_lease_proto != nil {
+		return
+	}
+	if !protoimpl.UnsafeEnabled {
+		file_pkg_distlock_grpc_lease_proto_msgTypes[0].Exporter = func(v any, i int) any {
+			switch v := v.(*AcquireLeaseRequest); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_pkg_distlock_grpc_lease_proto_msgTypes[1].Exporter = func(v any, i int) any {
+			switch v := v.(*AcquireLeaseResponse); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_pkg_distlock_grpc_lease_proto_msgTypes[2].Exporter = func(v any, i int) any {
+			switch v := v.(*RenewLeaseRequest); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_pkg_distlock_grpc_lease_proto_msgTypes[3].Exporter = func(v any, i int) any {
+			switch v := v.(*RenewLeaseResponse); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_pkg_distlock_grpc_lease_proto_msgTypes[4].Exporter = func(v any, i int) any {
+			switch v := v.(*ReleaseLeaseRequest); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_pkg_distlock_grpc_lease_proto_msgTypes[5].Exporter = func(v any, i int) any {
+			switch v := v.(*ReleaseLeaseResponse); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+	}
+	type x struct{}
+	out := protoimpl.TypeBuilder{
+		File: protoimpl.DescBuilder{
+			GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
+			RawDescriptor: file_pkg_distlock_grpc_lease_proto_rawDesc,
+			NumEnums:      0,
+			NumMessages:   6,
+			NumExtensions: 0,
+			NumServices:   1,
+		},
+		GoTypes:           file_pkg_distlock_grpc_lease_proto_goTypes,
+		DependencyIndexes: file_pkg_distlock_grpc_lease_proto_depIdxs,
+		MessageInfos:      file_pkg_distlock_grpc_lease_proto_msgTypes,
+	}.Build()
+	File_pkg_distlock_grpc_lease_proto = out.File
+	file_pkg_distlock_grpc_lease_proto_rawDesc = nil
+	file_pkg_distlock_grpc_lease_proto_goTypes = nil
+	file_pkg_distlock_grpc_lease_proto_depIdxs = nil
+}

pkg/distlock/grpc/lease.proto

@@ -0,0 +1,54 @@
+syntax = "proto3";
+
+package distlock.v1;
+
+option go_package = "github.com/psviderski/uncloud/pkg/distlock/grpc";
+
+import "google/protobuf/duration.proto";
+
+// Lease provides atomic operations for time-bound ownership of resources on one node.
+service Lease {
+  // Acquire creates a lease when the resource has no unexpired lease.
+  rpc Acquire(AcquireLeaseRequest) returns (AcquireLeaseResponse);
+  // Renew extends an unexpired lease when its ownership token matches.
+  rpc Renew(RenewLeaseRequest) returns (RenewLeaseResponse);
+  // Release removes an unexpired lease when its ownership token matches.
+  rpc Release(ReleaseLeaseRequest) returns (ReleaseLeaseResponse);
+}
+
+message AcquireLeaseRequest {
+  string resource = 1;
+  // Token uniquely identifies the lease owner.
+  bytes token = 2;
+  // TTL sets how long the lease remains valid without renewal.
+  google.protobuf.Duration ttl = 3;
+}
+
+message AcquireLeaseResponse {
+  // Acquired is true when this request successfully created the lease.
+  bool acquired = 1;
+}
+
+message RenewLeaseRequest {
+  string resource = 1;
+  // Token identifies the owner of the existing lease.
+  bytes token = 2;
+  // TTL sets how long the renewed lease remains valid.
+  google.protobuf.Duration ttl = 3;
+}
+
+message RenewLeaseResponse {
+  // Renewed is true when an unexpired lease matched the ownership token and was successfully renewed.
+  bool renewed = 1;
+}
+
+message ReleaseLeaseRequest {
+  string resource = 1;
+  // Token identifies the owner of the existing lease.
+  bytes token = 2;
+}
+
+message ReleaseLeaseResponse {
+  // Released is true when an unexpired lease existed, matched the ownership token, and was successfully released.
+  bool released = 1;
+}

pkg/distlock/grpc/lease_grpc.pb.go

@@ -0,0 +1,207 @@
+// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
+// versions:
+// - protoc-gen-go-grpc v1.5.1
+// - protoc             v5.27.3
+// source: pkg/distlock/grpc/lease.proto
+
+package grpc
+
+import (
+	context "context"
+	grpc "google.golang.org/grpc"
+	codes "google.golang.org/grpc/codes"
+	status "google.golang.org/grpc/status"
+)
+
+// This is a compile-time assertion to ensure that this generated file
+// is compatible with the grpc package it is being compiled against.
+// Requires gRPC-Go v1.64.0 or later.
+const _ = grpc.SupportPackageIsVersion9
+
+const (
+	Lease_Acquire_FullMethodName = "/distlock.v1.Lease/Acquire"
+	Lease_Renew_FullMethodName   = "/distlock.v1.Lease/Renew"
+	Lease_Release_FullMethodName = "/distlock.v1.Lease/Release"
+)
+
+// LeaseClient is the client API for Lease service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+//
+// Lease provides atomic operations for time-bound ownership of resources on one node.
+type LeaseClient interface {
+	// Acquire creates a lease when the resource has no unexpired lease.
+	Acquire(ctx context.Context, in *AcquireLeaseRequest, opts ...grpc.CallOption) (*AcquireLeaseResponse, error)
+	// Renew extends an unexpired lease when its ownership token matches.
+	Renew(ctx context.Context, in *RenewLeaseRequest, opts ...grpc.CallOption) (*RenewLeaseResponse, error)
+	// Release removes an unexpired lease when its ownership token matches.
+	Release(ctx context.Context, in *ReleaseLeaseRequest, opts ...grpc.CallOption) (*ReleaseLeaseResponse, error)
+}
+
+type leaseClient struct {
+	cc grpc.ClientConnInterface
+}
+
+func NewLeaseClient(cc grpc.ClientConnInterface) LeaseClient {
+	return &leaseClient{cc}
+}
+
+func (c *leaseClient) Acquire(ctx context.Context, in *AcquireLeaseRequest, opts ...grpc.CallOption) (*AcquireLeaseResponse, error) {
+	cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+	out := new(AcquireLeaseResponse)
+	err := c.cc.Invoke(ctx, Lease_Acquire_FullMethodName, in, out, cOpts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+func (c *leaseClient) Renew(ctx context.Context, in *RenewLeaseRequest, opts ...grpc.CallOption) (*RenewLeaseResponse, error) {
+	cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+	out := new(RenewLeaseResponse)
+	err := c.cc.Invoke(ctx, Lease_Renew_FullMethodName, in, out, cOpts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+func (c *leaseClient) Release(ctx context.Context, in *ReleaseLeaseRequest, opts ...grpc.CallOption) (*ReleaseLeaseResponse, error) {
+	cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+	out := new(ReleaseLeaseResponse)
+	err := c.cc.Invoke(ctx, Lease_Release_FullMethodName, in, out, cOpts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// LeaseServer is the server API for Lease service.
+// All implementations must embed UnimplementedLeaseServer
+// for forward compatibility.
+//
+// Lease provides atomic operations for time-bound ownership of resources on one node.
+type LeaseServer interface {
+	// Acquire creates a lease when the resource has no unexpired lease.
+	Acquire(context.Context, *AcquireLeaseRequest) (*AcquireLeaseResponse, error)
+	// Renew extends an unexpired lease when its ownership token matches.
+	Renew(context.Context, *RenewLeaseRequest) (*RenewLeaseResponse, error)
+	// Release removes an unexpired lease when its ownership token matches.
+	Release(context.Context, *ReleaseLeaseRequest) (*ReleaseLeaseResponse, error)
+	mustEmbedUnimplementedLeaseServer()
+}
+
+// UnimplementedLeaseServer must be embedded to have
+// forward compatible implementations.
+//
+// NOTE: this should be embedded by value instead of pointer to avoid a nil
+// pointer dereference when methods are called.
+type UnimplementedLeaseServer struct{}
+
+func (UnimplementedLeaseServer) Acquire(context.Context, *AcquireLeaseRequest) (*AcquireLeaseResponse, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method Acquire not implemented")
+}
+func (UnimplementedLeaseServer) Renew(context.Context, *RenewLeaseRequest) (*RenewLeaseResponse, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method Renew not implemented")
+}
+func (UnimplementedLeaseServer) Release(context.Context, *ReleaseLeaseRequest) (*ReleaseLeaseResponse, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method Release not implemented")
+}
+func (UnimplementedLeaseServer) mustEmbedUnimplementedLeaseServer() {}
+func (UnimplementedLeaseServer) testEmbeddedByValue()               {}
+
+// UnsafeLeaseServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to LeaseServer will
+// result in compilation errors.
+type UnsafeLeaseServer interface {
+	mustEmbedUnimplementedLeaseServer()
+}
+
+func RegisterLeaseServer(s grpc.ServiceRegistrar, srv LeaseServer) {
+	// If the following call pancis, it indicates UnimplementedLeaseServer was
+	// embedded by pointer and is nil.  This will cause panics if an
+	// unimplemented method is ever invoked, so we test this at initialization
+	// time to prevent it from happening at runtime later due to I/O.
+	if t, ok := srv.(interface{ testEmbeddedByValue() }); ok {
+		t.testEmbeddedByValue()
+	}
+	s.RegisterService(&Lease_ServiceDesc, srv)
+}
+
+func _Lease_Acquire_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(AcquireLeaseRequest)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(LeaseServer).Acquire(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: Lease_Acquire_FullMethodName,
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(LeaseServer).Acquire(ctx, req.(*AcquireLeaseRequest))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+func _Lease_Renew_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(RenewLeaseRequest)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(LeaseServer).Renew(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: Lease_Renew_FullMethodName,
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(LeaseServer).Renew(ctx, req.(*RenewLeaseRequest))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+func _Lease_Release_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(ReleaseLeaseRequest)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(LeaseServer).Release(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: Lease_Release_FullMethodName,
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(LeaseServer).Release(ctx, req.(*ReleaseLeaseRequest))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+// Lease_ServiceDesc is the grpc.ServiceDesc for Lease service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var Lease_ServiceDesc = grpc.ServiceDesc{
+	ServiceName: "distlock.v1.Lease",
+	HandlerType: (*LeaseServer)(nil),
+	Methods: []grpc.MethodDesc{
+		{
+			MethodName: "Acquire",
+			Handler:    _Lease_Acquire_Handler,
+		},
+		{
+			MethodName: "Renew",
+			Handler:    _Lease_Renew_Handler,
+		},
+		{
+			MethodName: "Release",
+			Handler:    _Lease_Release_Handler,
+		},
+	},
+	Streams:  []grpc.StreamDesc{},
+	Metadata: "pkg/distlock/grpc/lease.proto",
+}

pkg/distlock/grpc/server.go

@@ -0,0 +1,99 @@
+// Package grpc provides a gRPC transport for distlock node-local lease operations.
+package grpc
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	"github.com/psviderski/uncloud/pkg/distlock"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+	"google.golang.org/protobuf/types/known/durationpb"
+)
+
+// Server adapts a node-local distlock.Store to the Lease gRPC service.
+type Server struct {
+	UnimplementedLeaseServer
+	store distlock.Store
+}
+
+// NewServer creates a node-local lease server.
+func NewServer(store distlock.Store) *Server {
+	return &Server{store: store}
+}
+
+// Acquire creates a lease when the resource has no unexpired lease.
+func (s *Server) Acquire(ctx context.Context, req *AcquireLeaseRequest) (*AcquireLeaseResponse, error) {
+	ttl, err := validateLeaseRequest(req.Resource, req.Token, req.Ttl)
+	if err != nil {
+		return nil, status.Error(codes.InvalidArgument, err.Error())
+	}
+
+	acquired, err := s.store.Acquire(ctx, req.Resource, req.Token, ttl)
+	if err != nil {
+		return nil, storeStatusError(ctx, "acquire lease", err)
+	}
+	return &AcquireLeaseResponse{Acquired: acquired}, nil
+}
+
+// Renew extends an unexpired lease when its ownership token matches.
+func (s *Server) Renew(ctx context.Context, req *RenewLeaseRequest) (*RenewLeaseResponse, error) {
+	ttl, err := validateLeaseRequest(req.Resource, req.Token, req.Ttl)
+	if err != nil {
+		return nil, status.Error(codes.InvalidArgument, err.Error())
+	}
+
+	renewed, err := s.store.Renew(ctx, req.Resource, req.Token, ttl)
+	if err != nil {
+		return nil, storeStatusError(ctx, "renew lease", err)
+	}
+	return &RenewLeaseResponse{Renewed: renewed}, nil
+}
+
+// Release removes an unexpired lease when its ownership token matches.
+func (s *Server) Release(ctx context.Context, req *ReleaseLeaseRequest) (*ReleaseLeaseResponse, error) {
+	if err := validateResourceToken(req.Resource, req.Token); err != nil {
+		return nil, status.Error(codes.InvalidArgument, err.Error())
+	}
+
+	released, err := s.store.Release(ctx, req.Resource, req.Token)
+	if err != nil {
+		return nil, storeStatusError(ctx, "release lease", err)
+	}
+	return &ReleaseLeaseResponse{Released: released}, nil
+}
+
+func validateLeaseRequest(resource string, token []byte, ttl *durationpb.Duration) (time.Duration, error) {
+	if err := validateResourceToken(resource, token); err != nil {
+		return 0, err
+	}
+	if ttl == nil {
+		return 0, fmt.Errorf("TTL is not set")
+	}
+	if err := ttl.CheckValid(); err != nil {
+		return 0, fmt.Errorf("invalid TTL: %w", err)
+	}
+	duration := ttl.AsDuration()
+	if duration <= 0 {
+		return 0, fmt.Errorf("TTL must be positive")
+	}
+	return duration, nil
+}
+
+func validateResourceToken(resource string, token []byte) error {
+	if resource == "" {
+		return fmt.Errorf("resource is empty")
+	}
+	if len(token) == 0 {
+		return fmt.Errorf("token is empty")
+	}
+	return nil
+}
+
+func storeStatusError(ctx context.Context, operation string, err error) error {
+	if ctxErr := ctx.Err(); ctxErr != nil {
+		return status.FromContextError(ctxErr).Err()
+	}
+	return status.Error(codes.Internal, fmt.Sprintf("%s: %v", operation, err))
+}

pkg/distlock/locker.go

@@ -0,0 +1,393 @@
+package distlock
+
+import (
+	"context"
+	"crypto/rand"
+	"errors"
+	"fmt"
+	"math"
+	"slices"
+	"sync"
+	"time"
+
+	"github.com/cenkalti/backoff/v4"
+)
+
+const (
+	DefaultLeaseDuration      = 10 * time.Second
+	DefaultClockDriftFactor   = 0.01
+	DefaultMaxNodeCallTimeout = 5 * time.Second
+)
+
+var (
+	// ErrLeaseLost is the cancellation cause when automatic renewal can no longer maintain a lease.
+	ErrLeaseLost = errors.New("distributed lease lost")
+	// ErrLeaseReleased is the cancellation cause of an explicitly released lease.
+	ErrLeaseReleased = errors.New("distributed lease released")
+)
+
+// Config configures a Locker.
+type Config struct {
+	// LeaseDuration is the TTL used for acquisitions and renewals. The default is DefaultLeaseDuration.
+	LeaseDuration time.Duration
+	// ClockDriftFactor is the fraction of LeaseDuration reserved for differences in clock rates between the Locker and
+	// nodes. The default is DefaultClockDriftFactor.
+	ClockDriftFactor float64
+	// NodeCallTimeout sets the context timeout for an Acquire, Renew, or Release call to one node.
+	// The default is the smaller of DefaultMaxNodeCallTimeout and one third of the lease duration.
+	NodeCallTimeout time.Duration
+	// NewBackOff creates independent retry policies for acquisitions and renewal cycles. The default is an exponential
+	// backoff starting at 100ms and capped at 1s. A policy should not impose its own elapsed-time limit because the
+	// acquisition context and current lease validity already bound retries.
+	NewBackOff func() backoff.BackOff
+}
+
+func (c Config) clockDrift() time.Duration {
+	return time.Duration(math.Ceil(float64(c.LeaseDuration) * c.ClockDriftFactor))
+}
+
+// Locker acquires automatically renewed distributed leases over a Cluster.
+type Locker struct {
+	config  Config
+	cluster Cluster
+}
+
+// New creates a Locker over cluster.
+func New(cluster Cluster, config Config) (*Locker, error) {
+	if cluster == nil {
+		return nil, fmt.Errorf("cluster is nil")
+	}
+
+	if config.LeaseDuration == 0 {
+		config.LeaseDuration = DefaultLeaseDuration
+	}
+	if config.LeaseDuration < 0 {
+		return nil, fmt.Errorf("lease duration must be positive")
+	}
+	if config.ClockDriftFactor == 0 {
+		config.ClockDriftFactor = DefaultClockDriftFactor
+	}
+	if config.ClockDriftFactor <= 0 || config.ClockDriftFactor >= 1 {
+		return nil, fmt.Errorf("clock drift factor must be greater than 0 and less than 1")
+	}
+	if config.NodeCallTimeout == 0 {
+		config.NodeCallTimeout = min(DefaultMaxNodeCallTimeout, config.LeaseDuration/3)
+	}
+	if config.NodeCallTimeout < 0 {
+		return nil, fmt.Errorf("node call timeout must be positive")
+	}
+	if config.NewBackOff == nil {
+		config.NewBackOff = defaultBackOff
+	}
+
+	return &Locker{config: config, cluster: cluster}, nil
+}
+
+func defaultBackOff() backoff.BackOff {
+	return backoff.NewExponentialBackOff(
+		backoff.WithInitialInterval(100*time.Millisecond),
+		backoff.WithMaxInterval(time.Second),
+		backoff.WithMaxElapsedTime(0),
+	)
+}
+
+// Acquire waits until it acquires a lease for resource or ctx ends.
+func (l *Locker) Acquire(ctx context.Context, resource string) (*Lease, error) {
+	if resource == "" {
+		return nil, fmt.Errorf("resource is empty")
+	}
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+
+	nodes, err := l.cluster.Nodes(ctx)
+	if err != nil {
+		return nil, fmt.Errorf("get cluster nodes: %w", err)
+	}
+	if len(nodes) == 0 {
+		return nil, fmt.Errorf("cluster has no nodes")
+	}
+	nodes = slices.Clone(nodes)
+
+	boff := backoff.WithContext(l.config.NewBackOff(), ctx)
+	var lease *Lease
+	err = backoff.Retry(func() error {
+		token, tokenErr := newOwnershipToken()
+		if tokenErr != nil {
+			return backoff.Permanent(fmt.Errorf("generate lease token: %w", tokenErr))
+		}
+
+		candidate := newLease(l, nodes, resource, token)
+		if err := candidate.acquire(ctx); err != nil {
+			return err
+		}
+
+		lease = candidate
+		return nil
+	}, boff)
+	if err != nil {
+		return nil, fmt.Errorf("acquire distributed lease for %q: %w", resource, err)
+	}
+
+	return lease, nil
+}
+
+// newOwnershipToken generates a unique 128-bit random ownership token for a lease.
+func newOwnershipToken() ([]byte, error) {
+	token := make([]byte, 16)
+	if _, err := rand.Read(token); err != nil {
+		return nil, err
+	}
+	return token, nil
+}
+
+// Lease is an automatically renewed distributed lease.
+type Lease struct {
+	config   Config
+	nodes    []Node
+	resource string
+	token    []byte
+
+	ctx    context.Context
+	cancel context.CancelCauseFunc
+	// done is closed when the renewal goroutine exits.
+	done chan struct{}
+
+	// operationMu prevents acquisition, renewal, and release operations for the lease from overlapping.
+	operationMu sync.Mutex
+	quorum      int
+}
+
+func newLease(locker *Locker, nodes []Node, resource string, token []byte) *Lease {
+	ctx, cancel := context.WithCancelCause(context.Background())
+	return &Lease{
+		config:   locker.config,
+		nodes:    nodes,
+		resource: resource,
+		token:    token,
+		ctx:      ctx,
+		cancel:   cancel,
+		done:     make(chan struct{}),
+		quorum:   len(nodes)/2 + 1,
+	}
+}
+
+// Context returns a context that is cancelled when the lease is lost or explicitly released.
+func (l *Lease) Context() context.Context {
+	return l.ctx
+}
+
+// Release stops automatic renewal and attempts to remove the lease from every node in its acquisition snapshot.
+func (l *Lease) Release(ctx context.Context) error {
+	l.cancel(ErrLeaseReleased)
+	select {
+	case <-l.done:
+	case <-ctx.Done():
+		return ctx.Err()
+	}
+
+	if err := l.release(ctx); err != nil {
+		return fmt.Errorf("release distributed lease for %q: %w", l.resource, err)
+	}
+	return nil
+}
+
+type nodeResult struct {
+	success bool
+	err     error
+}
+
+func collectNodeResults(results <-chan nodeResult) (successes int, err error) {
+	var errs []error
+	for result := range results {
+		if result.err != nil {
+			errs = append(errs, result.err)
+		} else if result.success {
+			successes++
+		}
+	}
+	return successes, errors.Join(errs...)
+}
+
+func (l *Lease) executeNodes(ctx context.Context, fn func(context.Context, Node) (bool, error)) <-chan nodeResult {
+	results := make(chan nodeResult, len(l.nodes))
+	var wg sync.WaitGroup
+	for _, node := range l.nodes {
+		wg.Go(func() {
+			callCtx, cancel := context.WithTimeout(ctx, l.config.NodeCallTimeout)
+			defer cancel()
+
+			success, err := fn(callCtx, node)
+			results <- nodeResult{success: success, err: err}
+		})
+	}
+	go func() {
+		wg.Wait()
+		close(results)
+	}()
+	return results
+}
+
+// acquire makes one attempt to obtain the lease from a quorum of nodes and starts renewal on success.
+func (l *Lease) acquire(ctx context.Context) error {
+	startedAt := time.Now()
+	validUntil := startedAt.Add(l.config.LeaseDuration - l.config.clockDrift())
+	resultCh := make(chan error, 1)
+	// Aggregate asynchronously so acquire can return at quorum while this goroutine drains the remaining results and
+	// holds operationMu until every node call has finished.
+	go func() {
+		l.operationMu.Lock()
+		defer l.operationMu.Unlock()
+
+		// Use the lease context so cancelling the context passed to Acquire after it succeeds does not stop node calls
+		// still pending after quorum. On failure, acquire cancels the lease context below. The validity deadline and
+		// per-node NodeCallTimeout bound these calls.
+		operationCtx, cancel := context.WithDeadline(l.ctx, validUntil)
+		defer cancel()
+		results := l.executeNodes(operationCtx, func(ctx context.Context, node Node) (bool, error) {
+			return node.Acquire(ctx, l.resource, l.token, l.config.LeaseDuration)
+		})
+
+		successes := 0
+		errs := make([]error, 0, len(l.nodes))
+		reported := false
+		for result := range results {
+			if result.err != nil {
+				errs = append(errs, result.err)
+			} else if result.success {
+				successes++
+			}
+			if !reported && successes >= l.quorum {
+				resultCh <- nil
+				reported = true
+			}
+		}
+		if !reported {
+			quorumErr := fmt.Errorf("lease acquired on %d of %d nodes, need at least %d",
+				successes, len(l.nodes), l.quorum)
+			resultCh <- errors.Join(quorumErr, errors.Join(errs...))
+		}
+	}()
+
+	var acquireErr error
+	select {
+	case acquireErr = <-resultCh:
+	case <-ctx.Done():
+		acquireErr = ctx.Err()
+	}
+
+	if acquireErr == nil {
+		if !time.Now().Before(validUntil) {
+			acquireErr = fmt.Errorf("lease validity expired during acquisition")
+		} else {
+			go l.runRenew(validUntil)
+			return nil
+		}
+	}
+
+	// Cancel any node Acquire calls still in progress. release waits for them to finish before removing partial leases,
+	// so no node can create this lease after cleanup has run.
+	l.cancel(acquireErr)
+	_ = l.release(context.WithoutCancel(ctx))
+	return acquireErr
+}
+
+// release waits for any in-progress lease operation, removes the lease from every node, and returns any errors.
+func (l *Lease) release(ctx context.Context) error {
+	l.operationMu.Lock()
+	defer l.operationMu.Unlock()
+	if err := ctx.Err(); err != nil {
+		return err
+	}
+
+	results := l.executeNodes(ctx, func(ctx context.Context, node Node) (bool, error) {
+		return node.Release(ctx, l.resource, l.token)
+	})
+	_, err := collectNodeResults(results)
+	return err
+}
+
+// runRenew periodically renews the lease until it is released or lost.
+func (l *Lease) runRenew(validUntil time.Time) {
+	defer close(l.done)
+
+	for {
+		remaining := time.Until(validUntil)
+		if remaining <= 0 {
+			break
+		}
+
+		// Start renewal with two thirds of the current validity remaining
+		// to leave time for slow node calls and retries.
+		timer := time.NewTimer(remaining / 3)
+		select {
+		case <-l.ctx.Done():
+			timer.Stop()
+			return
+		case <-timer.C:
+		}
+
+		renewedUntil, err := l.renew(validUntil)
+		if err != nil {
+			if l.ctx.Err() != nil {
+				return
+			}
+			break
+		}
+		validUntil = renewedUntil
+	}
+
+	l.cancel(ErrLeaseLost)
+	_ = l.release(context.Background())
+}
+
+// renew retries node renewals until a quorum succeeds, the configured backoff stops, or the current lease validity
+// ends. It returns the new validity deadline after reaching quorum.
+func (l *Lease) renew(currentValidUntil time.Time) (time.Time, error) {
+	ctx, cancel := context.WithDeadline(l.ctx, currentValidUntil)
+	defer cancel()
+
+	var validUntil time.Time
+	resultCh := make(chan error, 1)
+	// Coordinate in the background so renew can report expiry even if a node call does not return after cancellation.
+	// Keep operationMu held until every call finishes so cleanup cannot race a pending renewal.
+	go func() {
+		l.operationMu.Lock()
+		defer l.operationMu.Unlock()
+
+		if err := ctx.Err(); err != nil {
+			resultCh <- err
+			return
+		}
+
+		boff := backoff.WithContext(l.config.NewBackOff(), ctx)
+		resultCh <- backoff.Retry(func() error {
+			startedAt := time.Now()
+			results := l.executeNodes(ctx, func(ctx context.Context, node Node) (bool, error) {
+				return node.Renew(ctx, l.resource, l.token, l.config.LeaseDuration)
+			})
+			successes, nodeErr := collectNodeResults(results)
+			if ctx.Err() != nil {
+				return ctx.Err()
+			}
+			if !time.Now().Before(currentValidUntil) {
+				return backoff.Permanent(fmt.Errorf("renewal attempt took longer than its validity window"))
+			}
+
+			if successes >= l.quorum {
+				validUntil = startedAt.Add(l.config.LeaseDuration - l.config.clockDrift())
+				return nil
+			}
+			quorumErr := fmt.Errorf("lease renewed on %d of %d nodes, need at least %d",
+				successes, len(l.nodes), l.quorum)
+			return errors.Join(quorumErr, nodeErr)
+		}, boff)
+	}()
+
+	select {
+	case err := <-resultCh:
+		return validUntil, err
+	case <-ctx.Done():
+		return time.Time{}, ctx.Err()
+	}
+}

pkg/distlock/locker_test.go

@@ -0,0 +1,445 @@
+package distlock_test
+
+import (
+	"context"
+	"errors"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/cenkalti/backoff/v4"
+	"github.com/psviderski/uncloud/pkg/distlock"
+	"github.com/stretchr/testify/require"
+)
+
+var errMemoryNodeUnavailable = errors.New("memory node unavailable")
+
+type memoryCluster struct {
+	nodes []*memoryNode
+}
+
+func newMemoryCluster(size int) *memoryCluster {
+	cluster := &memoryCluster{nodes: make([]*memoryNode, size)}
+	for i := range cluster.nodes {
+		cluster.nodes[i] = &memoryNode{store: distlock.NewMemoryStore(), available: true}
+	}
+	return cluster
+}
+
+func (c *memoryCluster) Nodes(ctx context.Context) ([]distlock.Node, error) {
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+
+	nodes := make([]distlock.Node, len(c.nodes))
+	for i, node := range c.nodes {
+		nodes[i] = node
+	}
+	return nodes, nil
+}
+
+type memoryNode struct {
+	mu                  sync.Mutex
+	store               distlock.Store
+	available           bool
+	unavailableObserved chan struct{}
+}
+
+func (n *memoryNode) setAvailable(available bool) {
+	if !available {
+		n.makeUnavailable()
+		return
+	}
+
+	n.mu.Lock()
+	defer n.mu.Unlock()
+	n.available = true
+}
+
+func (n *memoryNode) makeUnavailable() <-chan struct{} {
+	n.mu.Lock()
+	defer n.mu.Unlock()
+	n.available = false
+	n.unavailableObserved = make(chan struct{})
+	return n.unavailableObserved
+}
+
+func (n *memoryNode) restart() {
+	n.mu.Lock()
+	defer n.mu.Unlock()
+	n.store = distlock.NewMemoryStore()
+}
+
+func (n *memoryNode) currentStore(ctx context.Context) (distlock.Store, error) {
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+
+	n.mu.Lock()
+	defer n.mu.Unlock()
+	if !n.available {
+		if n.unavailableObserved != nil {
+			close(n.unavailableObserved)
+			n.unavailableObserved = nil
+		}
+		return nil, errMemoryNodeUnavailable
+	}
+	return n.store, nil
+}
+
+func (n *memoryNode) Acquire(
+	ctx context.Context, resource string, token []byte, ttl time.Duration,
+) (bool, error) {
+	store, err := n.currentStore(ctx)
+	if err != nil {
+		return false, err
+	}
+	return store.Acquire(ctx, resource, token, ttl)
+}
+
+func (n *memoryNode) Renew(
+	ctx context.Context, resource string, token []byte, ttl time.Duration,
+) (bool, error) {
+	store, err := n.currentStore(ctx)
+	if err != nil {
+		return false, err
+	}
+	return store.Renew(ctx, resource, token, ttl)
+}
+
+func (n *memoryNode) Release(ctx context.Context, resource string, token []byte) (bool, error) {
+	store, err := n.currentStore(ctx)
+	if err != nil {
+		return false, err
+	}
+	return store.Release(ctx, resource, token)
+}
+
+func retryBackOff() backoff.BackOff {
+	return backoff.NewConstantBackOff(5 * time.Millisecond)
+}
+
+func oneAttemptBackOff() backoff.BackOff {
+	return &backoff.StopBackOff{}
+}
+
+func newTestLocker(
+	t *testing.T, cluster distlock.Cluster, leaseDuration time.Duration, newBackOff func() backoff.BackOff,
+) *distlock.Locker {
+	t.Helper()
+	locker, err := distlock.New(cluster, distlock.Config{
+		LeaseDuration:   leaseDuration,
+		NodeCallTimeout: leaseDuration / 3,
+		NewBackOff:      newBackOff,
+	})
+	require.NoError(t, err)
+	return locker
+}
+
+func acquireLease(t *testing.T, locker *distlock.Locker, resource string) *distlock.Lease {
+	t.Helper()
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+	defer cancel()
+	lease, err := locker.Acquire(ctx, resource)
+	require.NoError(t, err)
+	return lease
+}
+
+func releaseLease(t *testing.T, lease *distlock.Lease) {
+	t.Helper()
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+	defer cancel()
+	require.NoError(t, lease.Release(ctx))
+}
+
+func requireNoReceive[T any](t *testing.T, ch <-chan T, timeout time.Duration, message string) {
+	t.Helper()
+	select {
+	case <-ch:
+		require.FailNow(t, message)
+	case <-time.After(timeout):
+	}
+}
+
+func requireReceive[T any](t *testing.T, ch <-chan T, timeout time.Duration, message string) T {
+	t.Helper()
+	select {
+	case value := <-ch:
+		return value
+	case <-time.After(timeout):
+		require.FailNow(t, message)
+		var zero T
+		return zero
+	}
+}
+
+func requireContextActive(t *testing.T, ctx context.Context, message string) {
+	t.Helper()
+	select {
+	case <-ctx.Done():
+		require.FailNow(t, message, "cause: %v", context.Cause(ctx))
+	default:
+	}
+}
+
+func requireSignal(t *testing.T, ch <-chan struct{}, timeout time.Duration, message string) {
+	t.Helper()
+	select {
+	case <-ch:
+	case <-time.After(timeout):
+		require.FailNow(t, message)
+	}
+}
+
+func TestLockerAcquireQuorumBoundaries(t *testing.T) {
+	tests := []struct {
+		name        string
+		nodes       int
+		unavailable int
+		wantAcquire bool
+	}{
+		{name: "two nodes at quorum", nodes: 2, wantAcquire: true},
+		{name: "two nodes below quorum", nodes: 2, unavailable: 1, wantAcquire: false},
+		{name: "three nodes at quorum", nodes: 3, unavailable: 1, wantAcquire: true},
+		{name: "three nodes below quorum", nodes: 3, unavailable: 2, wantAcquire: false},
+		{name: "four nodes at quorum", nodes: 4, unavailable: 1, wantAcquire: true},
+		{name: "four nodes below quorum", nodes: 4, unavailable: 2, wantAcquire: false},
+		{name: "five nodes at quorum", nodes: 5, unavailable: 2, wantAcquire: true},
+		{name: "five nodes below quorum", nodes: 5, unavailable: 3, wantAcquire: false},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			cluster := newMemoryCluster(tt.nodes)
+			unavailableObserved := make([]<-chan struct{}, 0, tt.unavailable)
+			for i := range tt.unavailable {
+				unavailableObserved = append(unavailableObserved, cluster.nodes[tt.nodes-1-i].makeUnavailable())
+			}
+			locker := newTestLocker(t, cluster, 300*time.Millisecond, oneAttemptBackOff)
+
+			acquireCtx, cancelAcquire := context.WithTimeout(context.Background(), 2*time.Second)
+			lease, err := locker.Acquire(acquireCtx, "resource")
+			cancelAcquire()
+			for _, observed := range unavailableObserved {
+				requireSignal(t, observed, time.Second, "unavailable node did not receive acquisition")
+			}
+			for i := range tt.unavailable {
+				cluster.nodes[tt.nodes-1-i].setAvailable(true)
+			}
+
+			if tt.wantAcquire {
+				require.NoError(t, err)
+				require.NotNil(t, lease)
+				releaseLease(t, lease)
+			} else {
+				if lease != nil {
+					releaseLease(t, lease)
+				}
+				require.Error(t, err)
+				require.Nil(t, lease)
+			}
+		})
+	}
+}
+
+func TestLockerSingleNodeLifecycle(t *testing.T) {
+	const leaseDuration = 300 * time.Millisecond
+	cluster := newMemoryCluster(1)
+	locker := newTestLocker(t, cluster, leaseDuration, oneAttemptBackOff)
+
+	lease := acquireLease(t, locker, "resource")
+	defer func() {
+		if lease.Context().Err() != nil {
+			return
+		}
+		ctx, cancel := context.WithTimeout(context.Background(), time.Second)
+		defer cancel()
+		_ = lease.Release(ctx)
+	}()
+
+	time.Sleep(leaseDuration + 100*time.Millisecond)
+	requireContextActive(t, lease.Context(), "single-node lease was not renewed")
+	releaseLease(t, lease)
+
+	secondLease := acquireLease(t, locker, "resource")
+	releaseLease(t, secondLease)
+}
+
+func TestLockerFailedAcquireCleansUpPartialLease(t *testing.T) {
+	cluster := newMemoryCluster(3)
+	cluster.nodes[1].setAvailable(false)
+	cluster.nodes[2].setAvailable(false)
+	locker := newTestLocker(t, cluster, 300*time.Millisecond, oneAttemptBackOff)
+
+	lease, err := locker.Acquire(context.Background(), "resource")
+	require.Error(t, err)
+	require.Nil(t, lease)
+
+	cluster.nodes[1].setAvailable(true)
+	cluster.nodes[2].setAvailable(true)
+	lease = acquireLease(t, locker, "resource")
+	releaseLease(t, lease)
+}
+
+func TestLockerAcquiresIndependentResourcesConcurrently(t *testing.T) {
+	cluster := newMemoryCluster(3)
+	locker := newTestLocker(t, cluster, 300*time.Millisecond, oneAttemptBackOff)
+	resources := []string{"database", "deployment", "network", "volume"}
+
+	type acquireResult struct {
+		resource string
+		lease    *distlock.Lease
+		err      error
+	}
+	resultCh := make(chan acquireResult, len(resources))
+	start := make(chan struct{})
+	acquireCtx, cancelAcquire := context.WithTimeout(context.Background(), 2*time.Second)
+	defer cancelAcquire()
+	for _, resource := range resources {
+		go func() {
+			<-start
+			lease, err := locker.Acquire(acquireCtx, resource)
+			resultCh <- acquireResult{resource: resource, lease: lease, err: err}
+		}()
+	}
+	close(start)
+
+	results := make([]acquireResult, 0, len(resources))
+	leases := make([]*distlock.Lease, 0, len(resources))
+	for range resources {
+		result := requireReceive(t, resultCh, 2*time.Second, "concurrent acquisition did not finish")
+		results = append(results, result)
+		if result.lease != nil {
+			leases = append(leases, result.lease)
+		}
+	}
+	defer func() {
+		for _, lease := range leases {
+			releaseLease(t, lease)
+		}
+	}()
+
+	for _, result := range results {
+		require.NoErrorf(t, result.err, "acquire %q", result.resource)
+		require.NotNilf(t, result.lease, "acquire %q", result.resource)
+		requireContextActive(t, result.lease.Context(), "independent lease was lost")
+	}
+}
+
+func TestLockerExcludesCompetingLeaseUntilRelease(t *testing.T) {
+	cluster := newMemoryCluster(3)
+	firstLocker := newTestLocker(t, cluster, 300*time.Millisecond, retryBackOff)
+	secondLocker := newTestLocker(t, cluster, 300*time.Millisecond, retryBackOff)
+
+	firstLease := acquireLease(t, firstLocker, "resource")
+
+	type acquireResult struct {
+		lease *distlock.Lease
+		err   error
+	}
+	resultCh := make(chan acquireResult, 1)
+	acquireCtx, cancelAcquire := context.WithTimeout(context.Background(), time.Second)
+	defer cancelAcquire()
+	go func() {
+		lease, acquireErr := secondLocker.Acquire(acquireCtx, "resource")
+		resultCh <- acquireResult{lease: lease, err: acquireErr}
+	}()
+
+	requireNoReceive(t, resultCh, 50*time.Millisecond, "competing acquisition returned before release")
+	releaseLease(t, firstLease)
+	require.ErrorIs(t, context.Cause(firstLease.Context()), distlock.ErrLeaseReleased)
+
+	result := requireReceive(t, resultCh, time.Second, "competing acquisition did not finish after release")
+	require.NoError(t, result.err)
+	require.NotNil(t, result.lease)
+	releaseLease(t, result.lease)
+}
+
+func TestLockerContendingAcquireRespectsContext(t *testing.T) {
+	cluster := newMemoryCluster(3)
+	firstLocker := newTestLocker(t, cluster, 300*time.Millisecond, retryBackOff)
+	secondLocker := newTestLocker(t, cluster, 300*time.Millisecond, retryBackOff)
+
+	firstLease := acquireLease(t, firstLocker, "resource")
+	t.Cleanup(func() {
+		if firstLease.Context().Err() != nil {
+			return
+		}
+		ctx, cancel := context.WithTimeout(context.Background(), time.Second)
+		defer cancel()
+		_ = firstLease.Release(ctx)
+	})
+
+	acquireCtx, cancelAcquire := context.WithTimeout(context.Background(), 75*time.Millisecond)
+	defer cancelAcquire()
+	competingLease, err := secondLocker.Acquire(acquireCtx, "resource")
+	require.ErrorIs(t, err, context.DeadlineExceeded)
+	require.Nil(t, competingLease)
+	requireContextActive(t, firstLease.Context(), "holding lease was affected by a competing acquisition")
+
+	releaseLease(t, firstLease)
+	secondLease := acquireLease(t, secondLocker, "resource")
+	releaseLease(t, secondLease)
+}
+
+func TestLockerAutomaticallyRenewsLease(t *testing.T) {
+	const leaseDuration = 300 * time.Millisecond
+	cluster := newMemoryCluster(3)
+	locker := newTestLocker(t, cluster, leaseDuration, retryBackOff)
+	competingLocker := newTestLocker(t, cluster, leaseDuration, oneAttemptBackOff)
+
+	lease := acquireLease(t, locker, "resource")
+	defer releaseLease(t, lease)
+
+	time.Sleep(leaseDuration + 100*time.Millisecond)
+	requireContextActive(t, lease.Context(), "lease was lost instead of renewed")
+
+	competingLease, err := competingLocker.Acquire(context.Background(), "resource")
+	require.Error(t, err)
+	require.Nil(t, competingLease)
+}
+
+func TestLockerRenewsLeaseWithMinorityUnavailable(t *testing.T) {
+	const leaseDuration = 300 * time.Millisecond
+	cluster := newMemoryCluster(3)
+	locker := newTestLocker(t, cluster, leaseDuration, oneAttemptBackOff)
+	competingLocker := newTestLocker(t, cluster, leaseDuration, oneAttemptBackOff)
+
+	lease := acquireLease(t, locker, "resource")
+	unavailableObserved := cluster.nodes[2].makeUnavailable()
+	originalValidityElapsed := time.NewTimer(leaseDuration + 100*time.Millisecond)
+	defer originalValidityElapsed.Stop()
+	defer func() {
+		cluster.nodes[2].setAvailable(true)
+		releaseLease(t, lease)
+	}()
+
+	requireSignal(t, unavailableObserved, time.Second, "unavailable node did not receive renewal")
+	<-originalValidityElapsed.C
+	requireContextActive(t, lease.Context(), "lease was lost after a minority node became unavailable")
+
+	competingLease, err := competingLocker.Acquire(context.Background(), "resource")
+	require.Error(t, err)
+	require.Nil(t, competingLease)
+}
+
+func TestLockerLosesLeaseWhenNodesRestart(t *testing.T) {
+	const leaseDuration = 300 * time.Millisecond
+	cluster := newMemoryCluster(3)
+	locker := newTestLocker(t, cluster, leaseDuration, oneAttemptBackOff)
+
+	lease := acquireLease(t, locker, "resource")
+	for _, node := range cluster.nodes {
+		node.restart()
+	}
+
+	select {
+	case <-lease.Context().Done():
+		require.ErrorIs(t, context.Cause(lease.Context()), distlock.ErrLeaseLost)
+	case <-time.After(time.Second):
+		require.FailNow(t, "lease was not lost after its node state disappeared")
+	}
+
+	releaseLease(t, lease)
+	require.ErrorIs(t, context.Cause(lease.Context()), distlock.ErrLeaseLost)
+}

pkg/distlock/memory.go

@@ -0,0 +1,139 @@
+package distlock
+
+import (
+	"bytes"
+	"context"
+	"fmt"
+	"sync"
+	"time"
+)
+
+type memoryLease struct {
+	token     []byte
+	expiresAt time.Time
+}
+
+// MemoryStore stores leases in memory. Its contents are lost when the process exits.
+type MemoryStore struct {
+	mu     sync.Mutex
+	leases map[string]memoryLease
+	now    func() time.Time
+}
+
+// NewMemoryStore creates an empty in-memory lease store.
+func NewMemoryStore() *MemoryStore {
+	return &MemoryStore{
+		leases: make(map[string]memoryLease),
+		now:    time.Now,
+	}
+}
+
+// Acquire creates a lease when the resource does not have an unexpired lease.
+func (s *MemoryStore) Acquire(
+	ctx context.Context, resource string, token []byte, ttl time.Duration,
+) (bool, error) {
+	if err := validateStoreInput(ctx, resource, token, ttl); err != nil {
+		return false, err
+	}
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if err := ctx.Err(); err != nil {
+		return false, err
+	}
+	now := s.now()
+
+	if lease, ok := s.leases[resource]; ok && now.Before(lease.expiresAt) {
+		return false, nil
+	}
+
+	s.leases[resource] = memoryLease{
+		token:     bytes.Clone(token),
+		expiresAt: now.Add(ttl),
+	}
+	return true, nil
+}
+
+// Renew extends an unexpired lease when its ownership token matches.
+func (s *MemoryStore) Renew(
+	ctx context.Context, resource string, token []byte, ttl time.Duration,
+) (bool, error) {
+	if err := validateStoreInput(ctx, resource, token, ttl); err != nil {
+		return false, err
+	}
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if err := ctx.Err(); err != nil {
+		return false, err
+	}
+	now := s.now()
+
+	lease, ok := s.leases[resource]
+	if !ok {
+		return false, nil
+	}
+	if !now.Before(lease.expiresAt) {
+		delete(s.leases, resource)
+		return false, nil
+	}
+	if !bytes.Equal(lease.token, token) {
+		return false, nil
+	}
+
+	lease.expiresAt = now.Add(ttl)
+	s.leases[resource] = lease
+	return true, nil
+}
+
+// Release removes an unexpired lease when its ownership token matches.
+func (s *MemoryStore) Release(ctx context.Context, resource string, token []byte) (bool, error) {
+	if err := validateStoreResourceToken(ctx, resource, token); err != nil {
+		return false, err
+	}
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if err := ctx.Err(); err != nil {
+		return false, err
+	}
+	now := s.now()
+
+	lease, ok := s.leases[resource]
+	if !ok {
+		return false, nil
+	}
+	if !now.Before(lease.expiresAt) {
+		delete(s.leases, resource)
+		return false, nil
+	}
+	if !bytes.Equal(lease.token, token) {
+		return false, nil
+	}
+
+	delete(s.leases, resource)
+	return true, nil
+}
+
+func validateStoreInput(ctx context.Context, resource string, token []byte, ttl time.Duration) error {
+	if err := validateStoreResourceToken(ctx, resource, token); err != nil {
+		return err
+	}
+	if ttl <= 0 {
+		return fmt.Errorf("TTL must be positive")
+	}
+	return nil
+}
+
+func validateStoreResourceToken(ctx context.Context, resource string, token []byte) error {
+	if err := ctx.Err(); err != nil {
+		return err
+	}
+	if resource == "" {
+		return fmt.Errorf("resource is empty")
+	}
+	if len(token) == 0 {
+		return fmt.Errorf("token is empty")
+	}
+	return nil
+}

pkg/distlock/memory_test.go

@@ -0,0 +1,243 @@
+package distlock
+
+import (
+	"context"
+	"fmt"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestMemoryStoreAcquire(t *testing.T) {
+	store := newTestMemoryStore()
+	ctx := context.Background()
+
+	acquired, err := store.Acquire(ctx, "resource-a", []byte("owner-a"), time.Minute)
+	require.NoError(t, err)
+	require.True(t, acquired)
+
+	acquired, err = store.Acquire(ctx, "resource-a", []byte("owner-b"), time.Minute)
+	require.NoError(t, err)
+	require.False(t, acquired)
+
+	acquired, err = store.Acquire(ctx, "resource-b", []byte("owner-b"), time.Minute)
+	require.NoError(t, err)
+	require.True(t, acquired)
+}
+
+func TestMemoryStoreExpirationAndStaleOwner(t *testing.T) {
+	now := time.Date(2026, time.August, 26, 12, 0, 0, 0, time.UTC)
+	store := newTestMemoryStore()
+	store.now = func() time.Time { return now }
+	ctx := context.Background()
+	oldToken := []byte("old-owner")
+	newToken := []byte("new-owner")
+
+	acquired, err := store.Acquire(ctx, "resource", oldToken, time.Minute)
+	require.NoError(t, err)
+	require.True(t, acquired)
+
+	// A lease is expired at its expiration time, not only after it.
+	now = now.Add(time.Minute)
+	acquired, err = store.Acquire(ctx, "resource", newToken, time.Minute)
+	require.NoError(t, err)
+	require.True(t, acquired)
+
+	released, err := store.Release(ctx, "resource", oldToken)
+	require.NoError(t, err)
+	require.False(t, released)
+
+	renewed, err := store.Renew(ctx, "resource", oldToken, time.Minute)
+	require.NoError(t, err)
+	require.False(t, renewed)
+
+	released, err = store.Release(ctx, "resource", newToken)
+	require.NoError(t, err)
+	require.True(t, released)
+}
+
+func TestMemoryStoreRenew(t *testing.T) {
+	now := time.Date(2026, time.August, 26, 12, 0, 0, 0, time.UTC)
+	store := newTestMemoryStore()
+	store.now = func() time.Time { return now }
+	ctx := context.Background()
+	token := []byte("owner")
+
+	acquired, err := store.Acquire(ctx, "resource", token, time.Minute)
+	require.NoError(t, err)
+	require.True(t, acquired)
+
+	now = now.Add(30 * time.Second)
+	renewed, err := store.Renew(ctx, "resource", []byte("another-owner"), time.Minute)
+	require.NoError(t, err)
+	require.False(t, renewed)
+
+	renewed, err = store.Renew(ctx, "resource", token, time.Minute)
+	require.NoError(t, err)
+	require.True(t, renewed)
+
+	// The renewal extends the lease from the renewal time.
+	now = now.Add(30 * time.Second)
+	acquired, err = store.Acquire(ctx, "resource", []byte("another-owner"), time.Minute)
+	require.NoError(t, err)
+	require.False(t, acquired)
+
+	now = now.Add(30 * time.Second)
+	renewed, err = store.Renew(ctx, "resource", token, time.Minute)
+	require.NoError(t, err)
+	require.False(t, renewed)
+}
+
+func TestMemoryStoreRelease(t *testing.T) {
+	store := newTestMemoryStore()
+	ctx := context.Background()
+	token := []byte("owner")
+
+	released, err := store.Release(ctx, "resource", token)
+	require.NoError(t, err)
+	require.False(t, released)
+
+	acquired, err := store.Acquire(ctx, "resource", token, time.Minute)
+	require.NoError(t, err)
+	require.True(t, acquired)
+
+	released, err = store.Release(ctx, "resource", []byte("another-owner"))
+	require.NoError(t, err)
+	require.False(t, released)
+
+	released, err = store.Release(ctx, "resource", token)
+	require.NoError(t, err)
+	require.True(t, released)
+
+	released, err = store.Release(ctx, "resource", token)
+	require.NoError(t, err)
+	require.False(t, released)
+}
+
+func TestMemoryStoreCopiesToken(t *testing.T) {
+	store := newTestMemoryStore()
+	ctx := context.Background()
+	token := []byte("owner")
+	originalToken := append([]byte(nil), token...)
+
+	acquired, err := store.Acquire(ctx, "resource", token, time.Minute)
+	require.NoError(t, err)
+	require.True(t, acquired)
+
+	token[0] = 'x'
+	released, err := store.Release(ctx, "resource", originalToken)
+	require.NoError(t, err)
+	require.True(t, released)
+}
+
+func TestMemoryStoreValidation(t *testing.T) {
+	tests := []struct {
+		name string
+		run  func(*MemoryStore) error
+	}{
+		{
+			name: "acquire with empty resource",
+			run: func(store *MemoryStore) error {
+				_, err := store.Acquire(context.Background(), "", []byte("owner"), time.Minute)
+				return err
+			},
+		},
+		{
+			name: "acquire with empty token",
+			run: func(store *MemoryStore) error {
+				_, err := store.Acquire(context.Background(), "resource", nil, time.Minute)
+				return err
+			},
+		},
+		{
+			name: "acquire with zero TTL",
+			run: func(store *MemoryStore) error {
+				_, err := store.Acquire(context.Background(), "resource", []byte("owner"), 0)
+				return err
+			},
+		},
+		{
+			name: "renew with negative TTL",
+			run: func(store *MemoryStore) error {
+				_, err := store.Renew(context.Background(), "resource", []byte("owner"), -time.Second)
+				return err
+			},
+		},
+		{
+			name: "release with empty token",
+			run: func(store *MemoryStore) error {
+				_, err := store.Release(context.Background(), "resource", nil)
+				return err
+			},
+		},
+		{
+			name: "cancelled context",
+			run: func(store *MemoryStore) error {
+				ctx, cancel := context.WithCancel(context.Background())
+				cancel()
+				_, err := store.Acquire(ctx, "resource", []byte("owner"), time.Minute)
+				return err
+			},
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			store := newTestMemoryStore()
+			require.Error(t, test.run(store))
+
+			// Invalid operations must not create or replace a lease.
+			acquired, err := store.Acquire(context.Background(), "resource", []byte("valid-owner"), time.Minute)
+			require.NoError(t, err)
+			require.True(t, acquired)
+		})
+	}
+}
+
+func TestMemoryStoreConcurrentAcquire(t *testing.T) {
+	const attempts = 100
+
+	store := newTestMemoryStore()
+	ctx := context.Background()
+	start := make(chan struct{})
+	errCh := make(chan error, attempts)
+	var acquired atomic.Int64
+	var wg sync.WaitGroup
+
+	for i := range attempts {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			<-start
+
+			ok, err := store.Acquire(ctx, "resource", []byte(fmt.Sprintf("owner-%d", i)), time.Minute)
+			if err != nil {
+				errCh <- err
+				return
+			}
+			if ok {
+				acquired.Add(1)
+			}
+		}()
+	}
+
+	close(start)
+	wg.Wait()
+	close(errCh)
+
+	for err := range errCh {
+		require.NoError(t, err)
+	}
+	require.EqualValues(t, 1, acquired.Load())
+}
+
+func newTestMemoryStore() *MemoryStore {
+	store := NewMemoryStore()
+	store.now = func() time.Time {
+		return time.Date(2026, time.August, 26, 12, 0, 0, 0, time.UTC)
+	}
+	return store
+}

pkg/distlock/node.go

@@ -0,0 +1,29 @@
+package distlock
+
+import (
+	"context"
+	"time"
+)
+
+// Cluster provides point-in-time snapshots of the independent nodes that participate in distributed leases.
+// Implementations must be safe for concurrent use.
+type Cluster interface {
+	// Nodes returns each node counted toward quorum exactly once, including nodes that are temporarily unavailable.
+	// The Locker may keep and use the returned nodes throughout acquisition and until any acquired lease is released.
+	Nodes(ctx context.Context) ([]Node, error)
+}
+
+// Node provides lease operations on a single cluster node.
+//
+// Each operation performs one attempt. A true result means the operation took effect. A false result with no error
+// means the node responded but its lease state rejected the operation. If err is non-nil, the result is unknown and
+// the boolean result must be ignored. Implementations must return promptly when ctx is cancelled, not modify token,
+// and be safe for concurrent use.
+type Node interface {
+	// Acquire creates a lease when resource does not have an unexpired lease.
+	Acquire(ctx context.Context, resource string, token []byte, ttl time.Duration) (bool, error)
+	// Renew extends an unexpired lease when its ownership token matches.
+	Renew(ctx context.Context, resource string, token []byte, ttl time.Duration) (bool, error)
+	// Release removes an unexpired lease when its ownership token matches.
+	Release(ctx context.Context, resource string, token []byte) (bool, error)
+}

pkg/distlock/store.go

@@ -0,0 +1,16 @@
+package distlock
+
+import (
+	"context"
+	"time"
+)
+
+// Store holds node-local lease state and provides atomic operations over it.
+type Store interface {
+	// Acquire creates a lease when the resource does not have an unexpired lease.
+	Acquire(ctx context.Context, resource string, token []byte, ttl time.Duration) (bool, error)
+	// Renew extends an unexpired lease when its ownership token matches.
+	Renew(ctx context.Context, resource string, token []byte, ttl time.Duration) (bool, error)
+	// Release removes a lease when its ownership token matches.
+	Release(ctx context.Context, resource string, token []byte) (bool, error)
+}

scripts/install.sh

@@ -220,7 +220,7 @@ install_uncloud_binaries() {
                 # 0.20.0~nightly-abc < 0.20.0 < 0.21.0~nightly-def.
                 # latest_version is always a clean stable tag from releases/latest, so the substitution is one-sided.
                 local newest
-                newest=$(printf '%s\n%s\n' "${installed_version//-/~}" "${latest_version}" | sort -V | tail -n1)
+                newest=$(printf '%s\n%s\n' "${installed_version//-/\~}" "${latest_version}" | sort -V | tail -n1)
                 if [ "${newest}" = "${latest_version}" ]; then
                     log "⏳ Upgrading uncloudd ${installed_version} → ${latest_version}..."
                     uncloudd_url="${UNCLOUD_GITHUB_URL}/releases/download/v${latest_version}/${uncloudd_archive_name}"
@@ -287,7 +287,7 @@ Wants=network-online.target
 [Service]
 Type=notify
 ExecStart=${INSTALL_BIN_DIR}/uncloudd
-TimeoutStartSec=15
+TimeoutStartSec=20
 Restart=always
 RestartSec=2
 

website/Caddyfile

@@ -1,9 +1,32 @@
 {
     admin off
+    servers {
+        # Trust the Uncloud ingress Caddy (cluster network) so the X-Forwarded-For chain it forwards
+        # is preserved and the real client IP is logged in the client_ip field.
+        trusted_proxies static 10.210.0.0/16
+    }
 }
 
 :8000 {
+    # Reverse proxy PostHog through our own domain so ad blockers don't drop analytics requests.
+    # Static assets (e.g. array.js) are served from a separate PostHog assets host.
+    handle_path /phproxy/static/* {
+        rewrite * /static{uri}
+        reverse_proxy https://us-assets.i.posthog.com {
+            header_up Host us-assets.i.posthog.com
+        }
+    }
+    handle_path /phproxy/* {
+        reverse_proxy https://us.i.posthog.com {
+            header_up Host us.i.posthog.com
+        }
+    }
+
     root * /usr/share/caddy
+
+    # Serve clean URLs for landing pages, e.g. /hub -> hub.html.
+    try_files {path}.html
+
     file_server
     log
 }

website/Dockerfile

@@ -28,4 +28,4 @@ COPY --from=prod /opt/docusaurus/build /usr/share/caddy
 # Copy the landing page assets.
 COPY landing/images /usr/share/caddy/images
 COPY landing/js /usr/share/caddy/js
-COPY landing/index.html landing/style.css /usr/share/caddy/
+COPY landing/index.html landing/hub.html landing/style.css /usr/share/caddy/

website/docs/1-overview.md

@@ -14,33 +14,41 @@ Uncloud covers all the essentials for operating apps in production without overw
 traditional container orchestrators like Kubernetes or Swarm:
 
 * Initial machine and network setup
+* Building images and pushing them directly to your machines without a registry
 * Zero-downtime rolling deployments
 * Health checks and automatic restarts
-* Automatic HTTPS and reverse proxy configuration
 * Scaling services across multiple machines
 * Cross-machine service communication without exposing ports to the internet
 * DNS-based service discovery
+* Automatic HTTPS and reverse proxy configuration
 * Load balancing
 * Persistent storage
 
 ## Use cases
 
-Some of the common use cases Uncloud is a great fit for:
+Uncloud is a great fit for anything from production workloads to a single-server homelab:
 
-- **Self-hosting and Homelabs**: Run your self-hosted apps on your own hardware. Start with a single machine and add
-  more as your needs grow.
+- **Production web apps and SaaS**: Run your product on VMs from any cloud provider or your own servers with
+  zero-downtime rolling deployments, health checks, and automatic HTTPS. Spread replicas across multiple machines to
+  keep your app available even when a machine goes down.
 - **Outgrowing Docker Compose**: Level up your Docker Compose setup with zero-downtime deployments, replicas across
   multiple machines for improved reliability, cross-machine service communication, automated reverse proxy management,
   and more using the same Compose file.
-- **Small to medium web applications**: Deploy your SaaS product, websites, or personal projects with redundancy across
-  multiple machines for better reliability and your peace of mind.
-- **Hybrid setups (cloud + on-prem)**: Combine cloud VMs with on-premise for cost savings and data sovereignty — all
-  managed through the same interface.
+- **Moving off a cloud PaaS or Kubernetes**: Get a Heroku-like deployment workflow on your own servers, without the high
+  PaaS costs or the complexity of Kubernetes.
+- **Migrating from Docker Swarm**: Swarm has been in maintenance mode for years. Uncloud offers an actively developed
+  alternative that keeps the familiar Compose format and drops the manager quorum. You also get secure WireGuard
+  networking across machines, image push without a registry, and automatic reverse proxy management out of the box.
+- **Hybrid setups (cloud + on-prem)**: Combine cloud VMs with on-premise servers and distribute workloads for cost
+  savings and data sovereignty. For example, keep your database on your own hardware and scale web replicas out to cloud
+  VMs. Manage everything together through the same interface.
 - **Agencies and freelancers**: Host multiple client projects with proper isolation on shared infrastructure, optimising
   costs and resources.
 - **Edge computing**: Deploy applications closer to your users for lower latency and better performance.
-- **Dev/staging environments**: Spin up additional environments for development and testing that mirror production
-  reusing the same Compose configuration.
+- **Self-hosting and homelabs**: Run your self-hosted apps on your own hardware. Start with a single machine and add
+  more as your needs grow.
+- **Dev/staging environments**: Spin up additional environments for development and testing that mirror production,
+  using the same Compose configuration.
 
 ## What makes Uncloud special
 
@@ -69,10 +77,10 @@ Talos [KubeSpan](https://www.talos.dev/v1.10/talos-guides/network/kubespan/).
 
 ### Managed DNS service (optional)
 
-Uncloud can provide **managed DNS records** like `<service-name>.<cluster-id>.uncld.dev` for your public
-services through free [Uncloud DNS](https://github.com/psviderski/uncloud-dns) service. You can deploy a service and
-instantly access it from anywhere with a proper DNS name and HTTPS without any manual DNS configuration. This makes
-self-hosting much more accessible and simplifies the process of adding your own domain later.
+Uncloud can provide **managed DNS records** like `<service-name>.<cluster-id>.uncld.dev` for your public services
+through free [Uncloud DNS](https://github.com/psviderski/uncloud-dns) service. You can deploy a service and instantly
+access it from anywhere with a proper DNS name and HTTPS without any manual DNS configuration. This makes self-hosting
+much more accessible and simplifies the process of adding your own domain later.
 
 ### No complex orchestration
 

website/docs/2-getting-started/1-install-cli.md

@@ -57,6 +57,12 @@ curl -fsS https://get.uncloud.run/install.sh | VERSION=nightly sh
 
 More information about nightly builds of the CLI and daemon can be found [here](https://github.com/psviderski/uncloud/releases/tag/nightly).
 
+## mise
+You can install uncloud with [mise](https://mise.jdx.dev/):
+```
+mise use github:psviderski/uncloud[exe=uc]
+```
+
 ## GitHub download (macOS, Linux)
 
 You can manually download and use a pre-built binary from the
@@ -99,13 +105,15 @@ Follow the same steps to upgrade to the latest version in the future.
 ## Debian
 
 On a Debian system, you can install Uncloud CLI from an unofficial
-[repository](https://debian.griffo.io/) maintained by
+[repository](https://deb.griffo.io/) maintained by
 [@dariogriffo](https://github.com/dariogriffo):
 
 ```shell
-curl -sS https://debian.griffo.io/EA0F721D231FDD3A0A17B9AC7808B4DD62C41256.asc | sudo gpg --dearmor --yes -o /etc/apt/trusted.gpg.d/debian.griffo.io.gpg
-echo "deb https://debian.griffo.io/apt $(lsb_release -sc 2>/dev/null) main" | sudo tee /etc/apt/sources.list.d/debian.griffo.io.list
-apt install -y uncloud
+sudo install -d -m 0755 /etc/apt/keyrings
+curl -fsSL https://deb.griffo.io/EA0F721D231FDD3A0A17B9AC7808B4DD62C41256.asc | sudo gpg --dearmor --yes -o /etc/apt/keyrings/deb.griffo.io.gpg
+echo "deb [signed-by=/etc/apt/keyrings/deb.griffo.io.gpg] https://deb.griffo.io/apt $(lsb_release -sc 2>/dev/null) main" | sudo tee /etc/apt/sources.list.d/deb.griffo.io.list
+sudo apt update
+sudo apt install -y uncloud
 ```
 
 Alternatively, you can download `.deb` packages directly from the repository

website/docs/2-getting-started/2-deploy-demo-app.md

@@ -1,15 +1,19 @@
 # Deploy demo app
 
-In this guide, we'll deploy [Excalidraw](https://excalidraw.com) — a popular sketching and diagramming tool — to your
+In this guide, we'll deploy [Excalidraw](https://excalidraw.com), a popular sketching and diagramming tool, to your
 Linux server. You'll learn the **basics of Uncloud** and see how simple it is to **run web apps** on your own
 infrastructure with secure internet access.
 
 :::info NOTE
-To give you a chance to play with Uncloud without even leaving your browser or needing your own servers, we're providing interactive tutorials and playgrounds on the [iximiuz Labs](https://labs.iximiuz.com/) platform.
 
-You can follow [this tutorial](https://labs.iximiuz.com/tutorials/uncloud-create-cluster-ebebf72b) which walks you through creating a new cluster with two machines and then deploying a simple web service to it.
+To give you a chance to play with Uncloud without even leaving your browser or needing your own servers, we're providing
+interactive tutorials and playgrounds on the [iximiuz Labs](https://labs.iximiuz.com/) platform.
 
-You can also launch the [Uncloud playground](https://labs.iximiuz.com/playgrounds/uncloud-cluster-64523f7c) where you can play with an already initialized Uncloud cluster.
+You can follow [this tutorial](https://labs.iximiuz.com/tutorials/uncloud-create-cluster-ebebf72b) which walks you
+through creating a new cluster with two machines and then deploying a simple web service to it.
+
+You can also launch the [Uncloud playground](https://labs.iximiuz.com/playgrounds/uncloud-cluster-64523f7c) where you
+can play with an already initialised Uncloud cluster.
 :::
 
 ## Prerequisites
@@ -53,8 +57,7 @@ This command will:
 - Install the Uncloud daemon on your server
 - Create a Docker network for Uncloud-managed containers
 - Deploy [Caddy](https://caddyserver.com/) as your reverse proxy listening on host ports 80 and 443
-- Reserve a free `xxxxxx.uncld.dev` subdomain via the Uncloud managed DNS service and point it to your
-  server's IP
+- Reserve a free `xxxxxx.uncld.dev` subdomain via the Uncloud managed DNS service and point it to your server's IP
 
 All in about a minute!
 
@@ -63,7 +66,6 @@ All in about a minute!
 
 ```
 $ uc machine init root@157.180.72.195
-Downloading Uncloud install script: https://raw.githubusercontent.com/psviderski/uncloud/refs/heads/main/scripts/install.sh
 ⏳ Running Uncloud install script...
 ⏳ Installing Docker...
 # Executing docker install script, commit: 53a22f61c0628e58e1d6680b49e82993d304b449
@@ -142,9 +144,6 @@ WARNING: Access to the remote API on a privileged Docker daemon is equivalent
 ✓ uncloud-uninstall script installed: /usr/local/bin/uncloud-uninstall
 ✓ Systemd unit file created: /etc/systemd/system/uncloud.service
 Created symlink /etc/systemd/system/multi-user.target.wants/uncloud.service → /etc/systemd/system/uncloud.service.
-⏳ Downloading uncloud-corrosion binary: https://github.com/psviderski/corrosion/releases/latest/download/corrosion-x86_64-unknown-linux-gnu.tar.gz
-✓ uncloud-corrosion binary installed: /usr/local/bin/uncloud-corrosion
-✓ Systemd unit file created: /etc/systemd/system/uncloud-corrosion.service
 ⏳ Starting Uncloud machine daemon (uncloud.service)...
 ✓ Uncloud machine daemon started.
 ✓ Uncloud installed on the machine successfully! 🎉
@@ -152,17 +151,17 @@ Cluster initialised with machine 'machine-dc3c' and saved as context 'default' i
 Current cluster context is now 'default'.
 Waiting for the machine to be ready...
 
-Reserved cluster domain: 7za6s7.uncld.dev
-[+] Deploying service caddy 7/2
- ✔ Container caddy-d7uk on machine-dc3c     Started                                            6.1s
-   ✔ Image caddy:2.10.0 on machine-dc3c       Pulled                                           3.7s
+Reserved cluster domain: sh8hsb.uncld.dev
+[+] Deploying service caddy 2/2
+ ✔ Container caddy-d7uk on machine-dc3c     Running                                           11.1s
+   ✔ Image caddy:2.11.4 on machine-dc3c     Pulled                                             3.7s
 
 Updating cluster domain records in Uncloud DNS to point to machines running caddy service...
 [+] Verifying internet access to caddy service 1/1
  ✔ Machine machine-dc3c (157.180.72.195)  Reachable                                            0.7s
 
 DNS records updated to use only the internet-reachable machines running caddy service:
-  *.7za6s7.uncld.dev  A → 157.180.72.195
+  *.sh8hsb.uncld.dev  A → 157.180.72.195
 ```
 
 </details>
@@ -181,11 +180,11 @@ You'll see the progress of the deployment and the public URL where you can acces
 
 ```
 [+] Running service excalidraw (replicated mode) 2/2
- ✔ Container excalidraw-azpc on machine-dc3c  Started                       8.9s
+ ✔ Container excalidraw-azpc on machine-dc3c  Healthy                      37.1s
    ✔ Image excalidraw/excalidraw on machine-dc3c  Pulled                    4.7s
 
 excalidraw endpoints:
- • https://excalidraw.7za6s7.uncld.dev → :80
+ • https://excalidraw.sh8hsb.uncld.dev → :80
 ```
 
 ## Verify your deployment
@@ -197,12 +196,12 @@ uc inspect excalidraw
 ```
 
 ```
-ID:    4d2de1600b6ada221a03896cd388836c
+Service ID: 4d2de1600b6ada221a03896cd388836c
 Name:       excalidraw
 Mode:       replicated
 
-CONTAINER ID   IMAGE                   CREATED              STATUS                        MACHINE
-fde7ac7f11ad   excalidraw/excalidraw   About a minute ago   Up About a minute (healthy)   machine-dc3c
+CONTAINER ID   IMAGE                          CREATED              STATUS                        IP ADDRESS   MACHINE
+fde7ac7f11ad   excalidraw/excalidraw:latest   About a minute ago   Up About a minute (healthy)   10.210.0.3   machine-dc3c
 ```
 
 In this example, the service has one container running on the machine `machine-dc3c` (our server). The container is up
@@ -215,9 +214,9 @@ uc ls
 ```
 
 ```
-NAME         MODE         REPLICAS   ENDPOINTS
-caddy        global       1
-excalidraw   replicated   1          https://excalidraw.7za6s7.uncld.dev → :80
+NAME         MODE         REPLICAS   IMAGE                          ENDPOINTS
+caddy        global       1          caddy:2.11.4
+excalidraw   replicated   1          excalidraw/excalidraw:latest   https://excalidraw.sh8hsb.uncld.dev → :80
 ```
 
 You can see `caddy` service listed here. That's your reverse proxy, running as a regular Uncloud service.
@@ -227,7 +226,7 @@ You can see `caddy` service listed here. That's your reverse proxy, running as a
 Open your browser and navigate to the URL shown in the endpoints. It may take a moment for Caddy to obtain a TLS
 certificate from Let's Encrypt. If it doesn't load immediately, wait a few seconds and try again.
 
-![Excalidraw running on Uncloud](./img/excalidraw-browser.png)
+![Excalidraw running on Uncloud](https://media.uncloud.run/docs/excalidraw-browser.webp)
 
 You now have:
 
@@ -235,6 +234,25 @@ You now have:
 - A **public URL** with **automatic HTTPS** you can share with your team and friends
 - **Full control over your data** — no analytics or tracking
 
+## View service logs
+
+Want to see what's happening inside your service? Use the `uc logs` command to view logs from the service container:
+
+```shell
+uc logs excalidraw
+```
+
+```
+Jul 14 10:53:05.910 machine-dc3c excalidraw/fde7a ::1 - - [14/Jul/2026:00:53:05 +0000] "GET / HTTP/1.1" 200 6843 "-" "Wget" "-"
+Jul 14 10:53:31.456 machine-dc3c excalidraw/fde7a 10.210.0.2 - - [14/Jul/2026:00:53:31 +0000] "GET /sw.js HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:150.0) Gecko/20100101 Firefox/150.0"
+```
+
+Add the `-f` flag to stream new logs in real-time. Press `Ctrl+C` to stop:
+
+```shell
+uc logs excalidraw -f
+```
+
 ## Convert to Docker Compose format
 
 Uncloud supports the [Compose file format](https://docs.docker.com/reference/compose-file/) for defining services. This
@@ -253,9 +271,9 @@ services:
 
 :::info note
 
-The `x-ports` key is an Uncloud-specific extension to the Compose file format. It allows you to specify ports that
-should be published as HTTP(S) endpoints. Uncloud automatically configures the reverse proxy (Caddy) to route traffic to
-these ports.
+The [`x-ports`](../8-compose-file-reference/2-extensions.md#x-ports) key is an Uncloud-specific extension to the Compose
+file format. It allows you to specify ports that should be published as HTTP(S) endpoints. Uncloud automatically
+configures the reverse proxy (Caddy) to route traffic to these ports.
 
 :::
 
@@ -274,25 +292,27 @@ We've successfully converted our deployment created with `uc run` to a Compose f
 
 ## Use your own domain
 
-Want to use your own domain, for example, `excalidraw.example.com` instead of `excalidraw.7za6s7.uncld.dev`?
+Want to use your own domain, for example, `excalidraw.example.com` instead of `excalidraw.sh8hsb.uncld.dev`?
 
 Add a CNAME record `excalidraw.example.com` in your DNS provider (Cloudflare, Namecheap, etc.) pointing to
-`excalidraw.7za6s7.uncld.dev`. Alternatively, you can add an A record pointing to your server's IP.
+`excalidraw.sh8hsb.uncld.dev`. Alternatively, you can add an A record pointing to your server's IP.
 
 :::info note
 
 These instructions set up your own domain **in addition to** the Uncloud managed DNS name
-`excalidraw.7za6s7.uncld.dev`.
+`excalidraw.sh8hsb.uncld.dev`.
 
 If you want to avoid the managed service altogether, add `--no-dns` to your `uc machine init` command, and point an A
-DNS record to your server(s)'s IP(s).
+DNS record to your servers' IPs.
 
 :::
 
 Then update the published port `80/https` in `compose.yaml` to use your domain:
 
 ```yaml title="compose.yaml"
-...
+services:
+  excalidraw:
+    image: excalidraw/excalidraw
     x-ports:
       - excalidraw.example.com:80/https
 ```
@@ -304,25 +324,32 @@ uc deploy
 ```
 
 ```
-Deployment plan:
-- Deploy service [name=excalidraw]
-  - machine-dc3c: Run container [image=excalidraw/excalidraw]
-  - machine-dc3c: Remove container [name=excalidraw-azpc]
+Deployment plan
 
-Do you want to continue?
+context: default
 
-Choose [y/N]: y
-Chose: Yes!
+~ update service excalidraw
+  │   image: excalidraw/excalidraw:latest
+  │
+  ╰── +/- replace container excalidraw/fde7ac7f11ad on machine-dc3c
 
-[+] Deploying services 2/2
- ✔ Container excalidraw-0z12 on machine-dc3c  Started                       3.5s
- ✔ Container excalidraw-azpc on machine-dc3c  Removed                       3.4s
+──────────────────────────────────────────
+1 replace (start-first) · across 1 machine
+
+Proceed with deployment to default? [y/N] y
+
+[+] Deploying to default 2/2
+ ✔ Container excalidraw-0z12 on machine-dc3c          Healthy              30.6s
+ ✔ Container excalidraw/fde7ac7f11ad on machine-dc3c  Removed               0.4s
 ```
 
-Notice how Uncloud performed a **zero-downtime deployment** — it started the new container with the updated
-configuration before removing the old one. Your service stayed available throughout the update.
+Uncloud prints a deployment plan and asks for confirmation before making any changes. The plan says it will replace the
+running container with a new one using the
+[`start-first` order](../4-guides/1-deployments/4-rolling-deployments.md#update-order). This means Uncloud starts the
+new container with the updated configuration, waits for it to become healthy, and only then removes the old one. Your
+service stays available throughout the update. That's a **zero-downtime deployment**.
 
-Give it a moment for Caddy to obtain a TLS certificate, then visit https://excalidraw.example.com.
+Give it a moment for Caddy to obtain a TLS certificate, then visit https://excalidraw.example.com (use your own domain).
 
 ## Clean up
 
@@ -365,68 +392,73 @@ The following actions will be performed:
 - Remove Uncloud WireGuard interface
 Do you want to proceed with uninstallation? [y/N] y
 ⏳ Stopping systemd services...
-Removed "/etc/systemd/system/multi-user.target.wants/uncloud.service".
-The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=,
-Also=, or Alias= settings in the [Install] section, and DefaultInstance= for
-template units). This means they are not meant to be enabled or disabled using systemctl.
-
-Possible reasons for having these kinds of units are:
-• A unit may be statically enabled by being symlinked from another unit's
-  .wants/, .requires/, or .upholds/ directory.
-• A unit's purpose may be to act as a helper for some other unit which has
-  a requirement dependency on it.
-• A unit may be started when needed via activation (socket, path, timer,
-  D-Bus, udev, scripted systemctl call, ...).
-• In case of template units, the unit is meant to be enabled with some
-  instance name specified.
+Removed /etc/systemd/system/multi-user.target.wants/uncloud.service.
 ✓ Systemd services stopped.
 ⏳ Removing systemd service files...
 removed '/etc/systemd/system/uncloud.service'
-removed '/etc/systemd/system/uncloud-corrosion.service'
 ✓ Systemd service files removed.
 ⏳ Removing binaries...
 removed '/usr/local/bin/uncloudd'
-removed '/usr/local/bin/uncloud-corrosion'
 ✓ Binaries removed.
+⏳ Removing uncloudd-managed corrosion Docker container...
+uncloud-corrosion
+✓ uncloud-corrosion container removed.
 ⏳ Removing data and run directories...
-removed '/var/lib/uncloud/machine.db-wal'
-removed '/var/lib/uncloud/caddy/caddy/autosave.json'
-removed directory '/var/lib/uncloud/caddy/caddy'
-removed '/var/lib/uncloud/caddy/caddy.json'
-removed directory '/var/lib/uncloud/caddy'
-removed '/var/lib/uncloud/machine.json'
 removed '/var/lib/uncloud/machine.db-shm'
-removed '/var/lib/uncloud/corrosion/admin.sock'
-removed '/var/lib/uncloud/corrosion/config.toml'
-removed '/var/lib/uncloud/corrosion/subscriptions/b4e825113f1143e5b27715b62193a9f8/sub.sqlite-wal'
-removed '/var/lib/uncloud/corrosion/subscriptions/b4e825113f1143e5b27715b62193a9f8/sub.sqlite-shm'
-removed '/var/lib/uncloud/corrosion/subscriptions/b4e825113f1143e5b27715b62193a9f8/sub.sqlite'
-removed directory '/var/lib/uncloud/corrosion/subscriptions/b4e825113f1143e5b27715b62193a9f8'
-removed '/var/lib/uncloud/corrosion/subscriptions/5e04cbb20a2743c382cfbd4949922351/sub.sqlite'
-removed directory '/var/lib/uncloud/corrosion/subscriptions/5e04cbb20a2743c382cfbd4949922351'
-removed directory '/var/lib/uncloud/corrosion/subscriptions'
-removed '/var/lib/uncloud/corrosion/schema.sql'
-removed '/var/lib/uncloud/corrosion/store.db'
-removed directory '/var/lib/uncloud/corrosion'
 removed '/var/lib/uncloud/machine.db'
+removed '/var/lib/uncloud/corrosion/store.db'
+removed '/var/lib/uncloud/corrosion/subscriptions/754e24df40f8476389cf6dbfa7b542c8/sub.sqlite'
+removed directory '/var/lib/uncloud/corrosion/subscriptions/754e24df40f8476389cf6dbfa7b542c8'
+removed '/var/lib/uncloud/corrosion/subscriptions/125e6ada8eec4f3cad192e1890db55c2/sub.sqlite'
+removed directory '/var/lib/uncloud/corrosion/subscriptions/125e6ada8eec4f3cad192e1890db55c2'
+removed directory '/var/lib/uncloud/corrosion/subscriptions'
+removed '/var/lib/uncloud/corrosion/config.toml'
+removed '/var/lib/uncloud/corrosion/schema.sql'
+removed directory '/var/lib/uncloud/corrosion'
+removed '/var/lib/uncloud/machine.json'
+removed '/var/lib/uncloud/caddy/caddy.json'
+removed '/var/lib/uncloud/caddy/caddy/autosave.json'
+removed '/var/lib/uncloud/caddy/caddy/last_clean.json'
+removed directory '/var/lib/uncloud/caddy/caddy/locks'
+removed '/var/lib/uncloud/caddy/caddy/instance.uuid'
+removed '/var/lib/uncloud/caddy/caddy/acme/acme-staging-v02.api.letsencrypt.org-directory/users/default/default.json'
+removed '/var/lib/uncloud/caddy/caddy/acme/acme-staging-v02.api.letsencrypt.org-directory/users/default/default.key'
+removed directory '/var/lib/uncloud/caddy/caddy/acme/acme-staging-v02.api.letsencrypt.org-directory/users/default'
+removed directory '/var/lib/uncloud/caddy/caddy/acme/acme-staging-v02.api.letsencrypt.org-directory/users'
+removed directory '/var/lib/uncloud/caddy/caddy/acme/acme-staging-v02.api.letsencrypt.org-directory'
+removed '/var/lib/uncloud/caddy/caddy/acme/acme-v02.api.letsencrypt.org-directory/users/default/default.json'
+removed '/var/lib/uncloud/caddy/caddy/acme/acme-v02.api.letsencrypt.org-directory/users/default/default.key'
+removed directory '/var/lib/uncloud/caddy/caddy/acme/acme-v02.api.letsencrypt.org-directory/users/default'
+removed directory '/var/lib/uncloud/caddy/caddy/acme/acme-v02.api.letsencrypt.org-directory/users'
+removed directory '/var/lib/uncloud/caddy/caddy/acme/acme-v02.api.letsencrypt.org-directory/challenge_tokens'
+removed directory '/var/lib/uncloud/caddy/caddy/acme/acme-v02.api.letsencrypt.org-directory'
+removed directory '/var/lib/uncloud/caddy/caddy/acme'
+removed '/var/lib/uncloud/caddy/caddy/certificates/acme-v02.api.letsencrypt.org-directory/excalidraw.sh8hsb.uncld.dev/excalidraw.sh8hsb.uncld.dev.key'
+removed '/var/lib/uncloud/caddy/caddy/certificates/acme-v02.api.letsencrypt.org-directory/excalidraw.sh8hsb.uncld.dev/excalidraw.sh8hsb.uncld.dev.crt'
+removed '/var/lib/uncloud/caddy/caddy/certificates/acme-v02.api.letsencrypt.org-directory/excalidraw.sh8hsb.uncld.dev/excalidraw.sh8hsb.uncld.dev.json'
+removed directory '/var/lib/uncloud/caddy/caddy/certificates/acme-v02.api.letsencrypt.org-directory/excalidraw.sh8hsb.uncld.dev'
+removed directory '/var/lib/uncloud/caddy/caddy/certificates/acme-v02.api.letsencrypt.org-directory'
+removed directory '/var/lib/uncloud/caddy/caddy/certificates'
+removed directory '/var/lib/uncloud/caddy/caddy'
+removed '/var/lib/uncloud/caddy/Caddyfile'
+removed directory '/var/lib/uncloud/caddy'
+removed '/var/lib/uncloud/machine.db-wal'
 removed directory '/var/lib/uncloud'
+removed '/run/uncloud/caddy/admin.sock'
+removed directory '/run/uncloud/caddy'
+removed '/run/uncloud/corrosion/admin.sock'
+removed directory '/run/uncloud/corrosion'
 removed directory '/run/uncloud'
 ✓ Data and run directories removed.
 ⏳ Removing Linux user and group...
 ✓ Linux user 'uncloud' removed.
 Linux group 'uncloud' does not exist or was already removed.
 ⏳ Looking for Docker containers and network created by Uncloud...
-Found 4 Uncloud managed containers.
+Found 1 Uncloud managed containers.
 ⏳ Stopping Uncloud managed containers...
-20613f6046d0
-1f1a65b78e93
-4300bde4a2b0
-053fdd57ec56
+b2eb9968e468
 ⏳ Removing Uncloud managed containers...
-20613f6046d0
-1f1a65b78e93
-4300bde4a2b0
-053fdd57ec56
+b2eb9968e468
 ✓ Uncloud managed containers stopped and removed.
 ⏳ Removing Docker network uncloud...
 uncloud

website/docs/3-concepts/1-clusters/1-connecting.md

@@ -1,6 +1,6 @@
 # Connecting to a cluster
 
-`uc` only needs to reach one machine to work with the entire cluster. That machine acts as an **entry point** and
+`uc` only needs to reach **one machine** to work with the entire cluster. That machine acts as an **entry point** and
 forwards requests to other machines as needed.
 
 `uc` stores **cluster contexts** and **connection details** in a [configuration file](../../7-cli-config-reference.md)
@@ -46,6 +46,27 @@ When you run a `uc` command, it determines which cluster to connect to using thi
 Once the context is resolved, `uc` tries each connection in the context's `connections` list in order until one
 succeeds.
 
+## User permissions on the machine
+
+When `uc` connects to a machine over SSH, it communicates with the Uncloud daemon through the Unix socket
+`/run/uncloud/uncloud.sock` on that machine. The daemon restricts access to the socket to the `root` user and members
+of the `uncloud` Linux group. This means your SSH user must be either `root` or a member of the `uncloud` group.
+
+In most cases you don't need to set this up manually. When you initialise or add a machine with a non-root user,
+`uc machine init` and `uc machine add` automatically add that user to the `uncloud` group during installation.
+
+If you want to connect with a different non-root user later, add them to the group on the machine:
+
+```shell
+sudo usermod -aG uncloud <username>
+```
+
+The group change only applies to new SSH sessions. If `uc` still fails with a permission denied error after adding the
+user, close any long-running SSH connections to the machine (for example, SSH ControlMaster sessions) and try again.
+
+The same requirement applies when running `uc` locally on a cluster machine with a `unix://` connection. The local user
+must be `root` or a member of the `uncloud` group.
+
 ## Global flags and environment variables
 
 These flags are available on every `uc` command. They can also be set with an environment variable. The flag takes

website/docs/3-concepts/2-ingress/2-publishing-services.md

@@ -17,7 +17,7 @@ uc run -p app.example.com:8000/https app:latest
 
 ```
 [+] Running service app-mwng (replicated mode) 1/1
- ✔ Container app-mwng-6lub on machine-fnr9  Started
+ ✔ Container app-mwng-6lub on machine-fnr9  Running
 
 app-mwng endpoints:
  • https://app.example.com → :8000

website/docs/8-compose-file-reference/1-support-matrix.md

@@ -45,9 +45,11 @@ If you rely on a specific Compose feature that is not supported by Uncloud, plea
 | `secrets`                        | ⚠️ Limited         | Reference secrets in `environment`, see [Secrets](../3-concepts/8-secrets.md). File mounts not supported                                   |
 | `security_opt`                   | ❌ Not supported    |                                                                                                                                            |
 | `shm_size`                       | ✅ Supported        | Shared memory size                                                                                                                         |
+| `stdin_open`                     | ✅ Supported        | Allocate standard input and keep it open                                                                                                   |
 | `stop_grace_period`              | ✅ Supported        | Time to wait after SIGTERM before SIGKILL                                                                                                  |
 | `storage_opt`                    | ❌ Not supported    |                                                                                                                                            |
 | `sysctls`                        | ✅ Supported        | Namespaced kernel parameters                                                                                                               |
+| `tty`                            | ✅ Supported        | Allocate a pseudo-TTY and connect the container's standard streams to it                                                                   |
 | `ulimits`                        | ✅ Supported        | Resource limits                                                                                                                            |
 | `user`                           | ✅ Supported        | Set container user                                                                                                                         |
 | `volumes`                        | ✅ Supported        | Named volumes, bind mounts, tmpfs                                                                                                          |

website/docs/9-cli-reference/uc_machine_add.md

@@ -26,7 +26,7 @@ uc machine add [USER@]HOST[:PORT] [flags]
       --no-install            Skip installation of Docker and the Uncloud daemon on the machine. Assumes they're already installed and running.
       --public-ip string      Public IP address of the machine for ingress configuration. Use 'auto' for automatic detection, blank '' or 'none' to disable ingress on this machine, or specify an IP address. (default "auto")
   -i, --ssh-key string        Path to SSH private key for remote login (if not already added to SSH agent). (default "~/.ssh/id_ed25519")
-      --version string        Version of the Uncloud daemon to install on the machine. (default "latest")
+      --version string        Version of the Uncloud daemon to install on the machine. [$UNCLOUD_DAEMON_VERSION] (default "latest")
       --wg-endpoint strings   WireGuard endpoint address that other machines in the cluster should use to establish WireGuard connections
                               to this machine. This doesn't change the address/port WireGuard listens on the machine.
                               Format: IP, IP:PORT, IPv6, or [IPv6]:PORT. Default port is the value of --wg-port if omitted.

website/docs/9-cli-reference/uc_machine_init.md

@@ -48,7 +48,7 @@ uc machine init [schema://]USER@HOST[:PORT] [flags]
       --no-install            Skip installation of Docker and the Uncloud daemon on the machine. Assumes they're already installed and running.
       --public-ip string      Public IP address of the machine for ingress configuration. Use 'auto' for automatic detection, blank '' or 'none' to disable ingress on this machine, or specify an IP address. (default "auto")
   -i, --ssh-key string        Path to SSH private key for remote login (if not already added to SSH agent). (default "~/.ssh/id_ed25519")
-      --version string        Version of the Uncloud daemon to install on the machine. (default "latest")
+      --version string        Version of the Uncloud daemon to install on the machine. [$UNCLOUD_DAEMON_VERSION] (default "latest")
       --wg-endpoint strings   WireGuard endpoint address that other machines in the cluster should use to establish WireGuard connections
                               to this machine. This doesn't change the address/port WireGuard listens on the machine.
                               Format: IP, IP:PORT, IPv6, or [IPv6]:PORT. Default port is the value of --wg-port if omitted.

website/docusaurus.config.js

@@ -12,7 +12,19 @@ import {themes as prismThemes} from 'prism-react-renderer';
 const config = {
   title: 'Uncloud',
   tagline: 'Self-host and scale web apps without Kubernetes complexity',
-  favicon: 'img/favicon.png',
+  // Use the SVG logo as the primary favicon with a PNG fallback to match the landing pages.
+  favicon: 'img/logo.svg',
+  headTags: [
+    {
+      tagName: 'link',
+      attributes: {
+        rel: 'alternate icon',
+        type: 'image/png',
+        href: '/img/favicon.png',
+        sizes: '96x96',
+      },
+    },
+  ],
 
   // Set the production url of your site here
   url: 'https://uncloud.run',

website/landing/css/style.css

@@ -1,4 +1,4 @@
-@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500&family=Inter+Tight:ital,wght@0,500;0,600;0,700;1,700&display=fallback');
+@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500&family=Inter+Tight:ital,wght@0,500;0,600;0,700;1,700&family=Caveat:wght@600&display=fallback');
 
 @import 'tailwindcss/base';
 @import 'tailwindcss/components';
@@ -13,3 +13,47 @@
 [x-cloak=""] {
     display: none;
 }
+
+/* Smooth in-page anchor scrolling, disabled for users who prefer reduced motion. */
+html {
+    scroll-behavior: smooth;
+}
+
+@media (prefers-reduced-motion: reduce) {
+    html {
+        scroll-behavior: auto;
+    }
+}
+
+.text-balance {
+    text-wrap: balance;
+}
+/* Handwritten margin notes. */
+.hw-note {
+    font-family: 'Caveat', cursive;
+    font-weight: 600;
+    font-size: 1.5rem;
+    line-height: 1.2;
+}
+
+/* A hand-swiped stroke with uneven rounded edges. */
+.marker-red {
+    position: relative;
+    white-space: nowrap;
+    color: #fff;
+    isolation: isolate;
+}
+
+.marker-red::before {
+    content: "";
+    position: absolute;
+    z-index: -1;
+    inset: 0.08em -0.18em -0.02em -0.14em;
+    background: linear-gradient(100deg,
+    rgba(220, 38, 38, 0.82) 0%,
+    rgba(239, 68, 68, 0.95) 28%,
+    rgba(225, 29, 72, 0.9) 72%,
+    rgba(220, 38, 38, 0.88) 100%);
+    border-radius: 0.3em 0.5em 0.4em 0.6em;
+    transform: skew(-10deg) rotate(-0.5deg);
+}

website/landing/index.html

@@ -3,22 +3,22 @@
 
 <head>
     <meta charset="utf-8">
-    <title>Self-host and scale web apps without Kubernetes complexity | Uncloud</title>
+    <title>Uncloud | Multi-node Docker Compose for production</title>
     <meta name="description"
-          content="Take your Docker Compose apps to production with zero-downtime deployments, automatic HTTPS, and cross-machine scaling. Self-hosting made reliable without the complexity.">
+          content="Deploy and scale Docker Compose apps across cloud VMs and your own servers. A simpler alternative to Kubernetes, Swarm, Nomad, and Kamal.">
     <meta name="viewport" content="width=device-width,initial-scale=1">
+    <link rel="canonical" href="https://uncloud.run">
     <link href="./style.css" rel="stylesheet">
     <link rel="apple-touch-icon" href="./images/apple-touch-icon.png" sizes="180x180"/>
     <link rel="icon" type="image/svg+xml" href="./images/logo.svg"/>
     <link rel="alternate icon" type="image/png" href="./images/favicon.png" sizes="96x96"/>
 
     <!-- OpenGraph tags for social media -->
-    <meta property="og:title"
-          content="Self-host and scale web apps without Kubernetes complexity | Uncloud"/>
+    <meta property="og:title" content="Multi-node Docker Compose for production"/>
     <meta property="og:description"
-          content="Take your Docker Compose apps to production with zero-downtime deployments, automatic HTTPS, and cross-machine scaling. Self-hosting made reliable without the complexity."/>
-    <meta property="og:image" content="https://uncloud.run/images/logo-wide.png"/>
-    <meta property="og:image:alt" content="Uncloud logo - Self-host web apps with ease"/>
+          content="Deploy and scale Docker Compose apps across cloud VMs and your own servers. A simpler alternative to Kubernetes, Swarm, Nomad, and Kamal."/>
+    <meta property="og:image" content="https://uncloud.run/images/og.png"/>
+    <meta property="og:image:alt" content="Uncloud: multi-node Docker Compose for production"/>
     <meta property="og:url" content="https://uncloud.run"/>
     <meta property="og:type" content="website"/>
     <meta property="og:site_name" content="Uncloud"/>
@@ -26,23 +26,13 @@
     <meta property="twitter:domain" content="uncloud.run">
     <meta property="twitter:url" content="https://uncloud.run">
     <meta name="twitter:card" content="summary_large_image">
-    <meta name="twitter:title"
-          content="Self-host and scale web apps without Kubernetes complexity | Uncloud">
+    <meta name="twitter:title" content="Multi-node Docker Compose for production">
     <meta name="twitter:description"
-          content="Take your Docker Compose apps to production with zero-downtime deployments, automatic HTTPS, and cross-machine scaling. Self-hosting made reliable without the complexity.">
-    <meta name="twitter:image" content="https://uncloud.run/images/logo-wide.png">
-    <meta name="twitter:image:alt" content="Uncloud logo - Self-host web apps with ease">
-    <style>
-        /* Prevent all inline SVGs from appearing full screen */
-        svg {
-            max-width: 24px;
-            max-height: 24px;
-        }
+          content="Deploy and scale Docker Compose apps across cloud VMs and your own servers. A simpler alternative to Kubernetes, Swarm, Nomad, and Kamal.">
+    <meta name="twitter:image" content="https://uncloud.run/images/og.png">
+    <meta name="twitter:image:alt" content="Uncloud: multi-node Docker Compose for production">
 
-        .text-balance {
-            text-wrap: balance;
-        }
-    </style>
+    <script src="./js/posthog.js" defer></script>
 </head>
 
 <body class="font-inter antialiased bg-white text-zinc-900 tracking-tight">
@@ -67,6 +57,15 @@
                 <nav class="flex grow">
                     <!-- Left side navigation -->
                     <ul class="flex items-center">
+                        <li>
+                            <a class="text-sm font-medium text-zinc-500 hover:text-zinc-900 px-2 sm:px-5 py-2 transition inline-flex items-center"
+                               href="/hub">
+                                Hub
+                                <span class="inline-block ml-1 -translate-y-1.5 text-[10px] font-semibold uppercase tracking-wide text-violet-700 bg-violet-100 px-1.5 py-0.5 rounded">
+                                    New
+                                </span>
+                            </a>
+                        </li>
                         <li>
                             <a class="text-sm font-medium text-zinc-500 hover:text-zinc-900 px-2 sm:px-5 py-2 transition"
                                href="/docs">Docs</a>
@@ -75,7 +74,8 @@
                             <a class="text-sm font-medium text-zinc-500 hover:text-zinc-900 px-2 sm:px-5 py-2 transition"
                                href="/blog">Blog</a>
                         </li>
-                        <li>
+                        <!-- Hidden on small screens to keep the nav from overflowing at 390px. -->
+                        <li class="hidden md:block">
                             <a class="text-sm font-medium text-zinc-500 hover:text-zinc-900 px-2 sm:px-5 py-2 transition"
                                href="https://psviderski.substack.com/" target="_blank">Newsletter</a>
                         </li>
@@ -140,7 +140,7 @@
                             <div class="max-w-xs mx-auto sm:max-w-none sm:inline-flex sm:justify-center space-y-4 sm:space-y-0 sm:space-x-4">
                                 <div>
                                     <a class="btn text-zinc-100 bg-zinc-900 hover:bg-zinc-800 w-full shadow flex items-center justify-center"
-                                       href="https://github.com/psviderski/uncloud/?tab=readme-ov-file#-quick-start">
+                                       href="/docs/getting-started/deploy-demo-app">
                                         <svg class="w-5 h-5 mr-2" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"
                                              fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"
                                              stroke-linejoin="round">
@@ -213,7 +213,7 @@
                         <a href="https://uncloud.run/discord" target="_blank" rel="noopener"
                            class="group block relative text-center md:px-5 after:hidden md:after:block after:absolute after:right-0 after:top-1/2 after:-translate-y-1/2 after:w-px after:h-8 after:border-l after:border-zinc-300 after:border-dashed last:after:hidden">
                             <h4 class="font-inter-tight text-2xl md:text-3xl font-bold tabular-nums mb-2"><span
-                                    x-data="counter(380)" x-text="counterValue">380</span></h4>
+                                    x-data="counter(390)" x-text="counterValue">390</span>+</h4>
                             <p class="text-zinc-500 transition-colors group-hover:text-zinc-800">Fans on
                                 Discord</p>
                         </a>
@@ -714,11 +714,47 @@ prod-3  201.45.91.123:51820  1m56s ago  11ms   5.12MB    9.34MB
             </div>
         </section>
 
+        <!-- Uncloud Hub -->
+        <section class="relative bg-violet-50 border-y border-violet-100">
+            <div class="py-12 md:py-20">
+                <div class="max-w-6xl mx-auto px-4 sm:px-6">
+                    <div class="grid lg:grid-cols-2 gap-10 lg:gap-16 items-center">
+                        <div class="lg:max-w-md lg:col-start-2 lg:row-start-1">
+                            <div class="font-mono text-xs font-medium uppercase tracking-[0.18em] text-violet-600 mb-4">
+                                Coming soon
+                            </div>
+                            <h2 class="font-inter-tight text-3xl md:text-4xl font-bold text-zinc-900 mb-4">
+                                <span class="text-violet-600">Uncloud Hub</span>: see what's happening in
+                                production
+                            </h2>
+                            <p class="text-lg text-zinc-500 mb-6">
+                                A hosted dashboard and observability stack for your Uncloud clusters. Connect a
+                                cluster and get metrics, logs, and alerts for every service and machine, without
+                                building and monitoring the monitoring yourself.
+                            </p>
+                            <div class="flex flex-wrap items-center gap-x-6 gap-y-4">
+                                <a class="btn text-zinc-100 bg-zinc-900 hover:bg-zinc-800 shadow"
+                                   href="/hub">Learn about Hub</a>
+                                <a class="text-sm font-medium text-violet-700 hover:text-violet-900 transition"
+                                   href="/hub#early-access">Get early access &rarr;</a>
+                            </div>
+                        </div>
+
+                        <div class="lg:col-start-1 lg:row-start-1 rounded-xl overflow-hidden border border-zinc-200 shadow-2xl shadow-zinc-950/10">
+                            <img src="./images/hub-hero.webp" width="2560" height="1520"
+                                 alt="Uncloud Hub cluster overview dashboard, design preview"
+                                 class="w-full block" loading="lazy">
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </section>
+
         <!-- FAQ -->
         <section id="faq" class="relative bg-zinc-50">
             <div class="py-12 md:py-20">
                 <div class="max-w-6xl mx-auto px-4 sm:px-6">
-                    <div class="max-w-3xl mx-auto">
+                    <div class="max-w-2xl mx-auto">
                         <div class="text-center pb-8 md:pb-12">
                             <h2 class="font-inter-tight text-3xl md:text-4xl font-bold text-zinc-900 mb-4">
                                 Frequently asked questions
@@ -731,17 +767,17 @@ prod-3  201.45.91.123:51820  1m56s ago  11ms   5.12MB    9.34MB
                         </div>
 
                         <div class="divide-y divide-zinc-200 border-y border-zinc-200">
-                            <details class="group py-5">
+                            <details class="group">
                                 <summary
-                                        class="flex items-center justify-between gap-4 cursor-pointer list-none [&::-webkit-details-marker]:hidden font-inter-tight text-lg font-semibold text-zinc-900">
+                                        class="flex items-center justify-between gap-4 py-5 cursor-pointer list-none [&::-webkit-details-marker]:hidden font-inter-tight font-semibold text-zinc-900">
                                     <span>How is this different from Docker Swarm?</span>
-                                    <svg class="w-4 h-4 shrink-0 text-zinc-400 transition-transform duration-200 group-open:rotate-45"
+                                    <svg class="w-5 h-5 shrink-0 text-zinc-400 transition-transform duration-200 group-open:rotate-45"
                                          fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24"
                                          aria-hidden="true">
                                         <path stroke-linecap="round" stroke-linejoin="round" d="M12 5v14m-7-7h14"/>
                                     </svg>
                                 </summary>
-                                <p class="mt-3 text-zinc-500">
+                                <p class="text-zinc-500 leading-relaxed pb-5">
                                     Swarm needs a Raft quorum of manager nodes and has been in maintenance mode for
                                     years. Uncloud has no control plane at all. Machines sync state peer to peer, and
                                     any subset keeps working. You also get WireGuard networking across clouds and even
@@ -750,17 +786,17 @@ prod-3  201.45.91.123:51820  1m56s ago  11ms   5.12MB    9.34MB
                                 </p>
                             </details>
 
-                            <details class="group py-5">
+                            <details class="group">
                                 <summary
-                                        class="flex items-center justify-between gap-4 cursor-pointer list-none [&::-webkit-details-marker]:hidden font-inter-tight text-lg font-semibold text-zinc-900">
+                                        class="flex items-center justify-between gap-4 py-5 cursor-pointer list-none [&::-webkit-details-marker]:hidden font-inter-tight font-semibold text-zinc-900">
                                     <span>What happens when a machine goes down?</span>
-                                    <svg class="w-4 h-4 shrink-0 text-zinc-400 transition-transform duration-200 group-open:rotate-45"
+                                    <svg class="w-5 h-5 shrink-0 text-zinc-400 transition-transform duration-200 group-open:rotate-45"
                                          fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24"
                                          aria-hidden="true">
                                         <path stroke-linecap="round" stroke-linejoin="round" d="M12 5v14m-7-7h14"/>
                                     </svg>
                                 </summary>
-                                <p class="mt-3 text-zinc-500">
+                                <p class="text-zinc-500 leading-relaxed pb-5">
                                     The rest of the cluster keeps working. There's no control plane to lose, and
                                     healthy replicas on other machines keep serving traffic. Uncloud deliberately
                                     doesn't auto-reschedule containers. You decide when to replace the machine and
@@ -768,23 +804,66 @@ prod-3  201.45.91.123:51820  1m56s ago  11ms   5.12MB    9.34MB
                                 </p>
                             </details>
 
-                            <details class="group py-5">
+                            <details class="group">
                                 <summary
-                                        class="flex items-center justify-between gap-4 cursor-pointer list-none [&::-webkit-details-marker]:hidden font-inter-tight text-lg font-semibold text-zinc-900">
+                                        class="flex items-center justify-between gap-4 py-5 cursor-pointer list-none [&::-webkit-details-marker]:hidden font-inter-tight font-semibold text-zinc-900">
                                     <span>Will it replace my CI/CD?</span>
-                                    <svg class="w-4 h-4 shrink-0 text-zinc-400 transition-transform duration-200 group-open:rotate-45"
+                                    <svg class="w-5 h-5 shrink-0 text-zinc-400 transition-transform duration-200 group-open:rotate-45"
                                          fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24"
                                          aria-hidden="true">
                                         <path stroke-linecap="round" stroke-linejoin="round" d="M12 5v14m-7-7h14"/>
                                     </svg>
                                 </summary>
-                                <p class="mt-3 text-zinc-500">
+                                <p class="text-zinc-500 leading-relaxed pb-5">
                                     No, and that's deliberate. <code>uc</code>
                                     is just a CLI that connects to your cluster over SSH. Run <code>uc deploy</code>
                                     locally or in GitHub Actions to integrate with existing workflows.
                                     There are no agents to install and no pipelines to migrate.
                                 </p>
                             </details>
+
+                            <details class="group">
+                                <summary
+                                        class="flex items-center justify-between gap-4 py-5 cursor-pointer list-none [&::-webkit-details-marker]:hidden font-inter-tight font-semibold text-zinc-900">
+                                    <span>Do I need Hub to use Uncloud?</span>
+                                    <svg class="w-5 h-5 shrink-0 text-zinc-400 transition-transform duration-200 group-open:rotate-45"
+                                         fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24"
+                                         aria-hidden="true">
+                                        <path stroke-linecap="round" stroke-linejoin="round" d="M12 5v14m-7-7h14"/>
+                                    </svg>
+                                </summary>
+                                <p class="text-zinc-500 leading-relaxed pb-5">
+                                    No, Uncloud is open source and fully functional on its own without Hub. You can use
+                                    the CLI to manage your clusters and deploy your apps across servers from Compose
+                                    files. You keep full control over your servers. Hub adds a web UI and an
+                                    observability stack on top so you don't have to build and manage one yourself.
+                                </p>
+                            </details>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </section>
+
+        <!-- Newsletter subscription -->
+        <section class="relative bg-white">
+            <div class="py-12 md:py-20">
+                <div class="max-w-6xl mx-auto px-4 sm:px-6">
+                    <div class="max-w-2xl mx-auto text-center">
+                        <h2 class="font-inter-tight text-3xl md:text-4xl font-bold text-zinc-900 mb-4">
+                            Follow the development journey
+                        </h2>
+                        <p class="text-lg text-zinc-500">
+                            Subscribe to get early insights into new features.
+                            See <a class="font-medium text-zinc-600 underline decoration-zinc-300 underline-offset-2 hover:text-zinc-900 hover:decoration-zinc-400 transition-colors"
+                                   href="https://psviderski.substack.com/" target="_blank" rel="noopener">previous
+                            newsletters</a>.
+                        </p>
+                        <div class="mt-6 md:mt-8 flex justify-center">
+                            <iframe src="https://psviderski.substack.com/embed?transparent=true" width="480"
+                                    height="160" title="Subscribe to the Uncloud newsletter" class="w-full max-w-[480px]"
+                                    style="border:none; background:transparent;" frameborder="0"
+                                    scrolling="no"></iframe>
                         </div>
                     </div>
                 </div>
@@ -829,21 +908,6 @@ prod-3  201.45.91.123:51820  1m56s ago  11ms   5.12MB    9.34MB
                         </div>
                     </div>
 
-                    <!-- Newsletter subscription -->
-                    <div class="mt-12 md:mt-16">
-                        <p class="text-zinc-400 mb-4">
-                            Subscribe to follow the development journey and get early insights into new features.<br>
-                            See <a href="https://psviderski.substack.com/" target="_blank" rel="noopener"
-                                   class="text-zinc-200 underline decoration-zinc-500 underline-offset-2 hover:text-zinc-100 hover:decoration-zinc-300 transition-colors">previous
-                            newsletters</a>.
-                        </p>
-                        <div class="flex justify-center">
-                            <iframe src="https://psviderski.substack.com/embed?transparent=true" width="480"
-                                    height="150" title="Subscribe to the Uncloud newsletter"
-                                    style="border:none; background:transparent;" frameborder="0"
-                                    scrolling="no"></iframe>
-                        </div>
-                    </div>
                 </div>
             </div>
         </section>
@@ -859,7 +923,7 @@ prod-3  201.45.91.123:51820  1m56s ago  11ms   5.12MB    9.34MB
                 <div class="sm:col-span-6 md:col-span-6 lg:col-span-8 max-md:order-1 flex flex-col">
                     <div class="mb-4">
                         <!-- Logo -->
-                        <a class="flex items-center" href="index.html">
+                        <a class="flex items-center" href="/">
                             <img class="w-8 h-8 rounded shadow-sm shadow-zinc-950/20" src="./images/logo.svg"
                                  alt="Logo">
                             <span class="ml-2 font-semibold text-lg text-zinc-900">uncloud</span>

website/landing/js/posthog.js

@@ -0,0 +1,15 @@
+// PostHog analytics and form processing for the landing pages.
+// The project API key is public and safe to expose in client-side code.
+const POSTHOG_API_KEY = 'phc_nsuhPtAsiYAFiSYmc2KwA5Homz6miXWjf3Hy4J4H3QMV';
+
+(function () {
+    // Official PostHog JS snippet from Project settings - General - HTML snippet.
+    !function(t,e){var o,n,p,r;e.__SV||(window.posthog && window.posthog.__loaded)||(window.posthog=e,e._i=[],e.init=function(i,s,a){function g(t,e){var o=e.split(".");2==o.length&&(t=t[o[0]],e=o[1]),t[e]=function(){t.push([e].concat(Array.prototype.slice.call(arguments,0)))}}(p=t.createElement("script")).type="text/javascript",p.crossOrigin="anonymous",p.async=!0,p.src=s.api_host.replace(".i.posthog.com","-assets.i.posthog.com")+"/static/array.js",(r=t.getElementsByTagName("script")[0]).parentNode.insertBefore(p,r);var u=e;for(void 0!==a?u=e[a]=[]:a="posthog",u.people=u.people||[],u.toString=function(t){var e="posthog";return"posthog"!==a&&(e+="."+a),t||(e+=" (stub)"),e},u.people.toString=function(){return u.toString(1)+".people (stub)"},o="ki Ci init qi Hi pr Bi zi Di capture calculateEventProperties Qi register register_once register_for_session unregister unregister_for_session Ki getFeatureFlag getFeatureFlagPayload getFeatureFlagResult getAllFeatureFlags isFeatureEnabled reloadFeatureFlags updateFlags updateEarlyAccessFeatureEnrollment getEarlyAccessFeatures on onFeatureFlags onSurveysLoaded onSessionId getSurveys getActiveMatchingSurveys renderSurvey displaySurvey cancelPendingSurvey canRenderSurvey canRenderSurveyAsync Xi identify setPersonProperties unsetPersonProperties group resetGroups setPersonPropertiesForFlags resetPersonPropertiesForFlags setGroupPropertiesForFlags resetGroupPropertiesForFlags reset shutdown setIdentity clearIdentity get_distinct_id getGroups get_session_id get_session_replay_url alias set_config startSessionRecording stopSessionRecording sessionRecordingStarted captureException addExceptionStep captureLog startExceptionAutocapture stopExceptionAutocapture loadToolbar get_property getSessionProperty Ji Gi createPersonProfile setInternalOrTestUser Yi Ai rn opt_in_capturing opt_out_capturing has_opted_in_capturing has_opted_out_capturing get_explicit_consent_status is_capturing clear_opt_in_out_capturing Vi debug mr it getPageViewId captureTraceFeedback captureTraceMetric Oi".split(" "),n=0;n<o.length;n++)g(u,o[n]);e._i.push([i,s,a])},e.__SV=1)}(document,window.posthog||[]);
+    posthog.init(POSTHOG_API_KEY, {
+        // Send events through our own domain (see website/Caddyfile) so ad blockers don't drop them.
+        api_host: window.location.origin + '/phproxy',
+        ui_host: "https://us.posthog.com",
+        defaults: '2026-05-30',
+        person_profiles: 'identified_only',
+    })
+})();