add the initial draft of the design doc

2026-08-26 11:03:34 +00:00 · 2024-08-09 16:54:36 +10:00
parent 92e3b21589
commit 32448381ac
1 changed files with 157 additions and 0 deletions
@@ -0,0 +1,157 @@
 # Uncloud design
 Uncloud aims to provide a solution for deploying and running containerized applications and services across various
 platforms — from cloud VPSs to Raspberry Pis and powerful servers — or any combination of them.
 While existing prominent PaaS solutions like https://coolify.io/ and https://dokploy.com/ use a single machine or Docker
 Swarm as their underlying infrastructure, Uncloud takes a different approach. It aims to offer a multi-host experience
 similar to the serverless solutions provided by cloud providers but on users' own machines. Users should be able to add
 machines as compute resources without worrying about control plane high availability and cluster management. For
 example, a user could combine a cloud VPS with a spare Raspberry Pi to create a unified computing environment.
 Uncloud aims to replace the concept of a "cluster" with a "network" of machines. It's similar to Tailscale's design
 where users can add machines to a network to enable secure communication between them. The primary challenges in this
 design are:
 1. Building such a network
 2. Orchestrating containers across it
 3. Exposing services to the internet
 ## Network of machines
 While Tailscale is an obvious solution for establishing an overlay network, it has some drawbacks:
 1. Users must register an account and generate a key for each machine.
 2. Every container requires a Tailscale proxy to communicate with other machines and containers.
 The Tailscale subnet router feature could potentially be used to route traffic from the machine to the internal Docker
 network.
 As an alternative, we can configure a simple flat WireGuard mesh network during the machine setup process. For example:
 | CIDR            | Description                                              |
 |-----------------|----------------------------------------------------------|
 | `10.210.0.0/16` | The entire WireGuard mesh network                        |
 | `10.210.X.0/24` | `/24` subnet assigned to machine `X`                     |
 | `10.210.X.1/32` | Machine `X` address is the first address from the subnet |
 | `10.210.X.Y/32` | Container `Y` address running on machine `X`             |
 Testing has shown that inter-container and machine routing works well when `/24` is assigned to the Docker network
 bridge, while the WireGuard interface is configured with just the `/32` machine address.
 This setup allows containers to communicate with each other on the same machine or across machines without address
 translation. This design also enables the future implementation of [ACLs](https://tailscale.com/kb/1018/acls) and
 security groups to restrict traffic between machines and containers if needed.
 The peer discovery and NAT traversal techniques for the WireGuard mesh are heavily inspired by the
 Talos [KubeSpan](https://www.talos.dev/v1.7/talos-guides/network/kubespan/) design.
 ## Orchestration
 The main question that drives the design of the orchestration system is can we build a system that doesn't require a
 centralised control plane? The short answer is yes, we can. However, it's not easy as decentralised systems are
 inherently more complex than centralised ones.
 The existing solutions such as Kubernetes, Docker Swarm, and Nomad use a centralised control plane to provide a single
 point of entry for the user to interact with the cluster. I'm envisioning a system where all machines are equal, yet
 each one can represent the entire cluster and receive commands from the user. Where a user can be a CLI tool or a web
 interface that can run on any machine in the network. In case of network partitioning, the user should continue to be
 able to interact with each partition separately to manage services running on them. The system should be able to
 reconcile the state of the network when the partition is healed. Such an architecture favors Availability and Partition
 tolerance (AP) over Consistency and Availability (CA) in the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem).
 ### Shared state
 One of the ways to achieve this is to store the entire shared state on every machine and keep them all in sync. A couple
 of techniques can be used to assist with this:
 a [Conflict-Free Replicated Data Type](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) (CRDT) and a
 peer-to-peer [gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol).
 By composing the state using CRDT data structures allows any machine to modify its own copy of the state independently,
 concurrently, and without coordination with other machines. The CRDT automatically resolves conflicts that might arise
 from concurrent updates. Although the machines may have different states at any given time, they're guaranteed to
 eventually converge to the *same* state. The tradeoff is that the *same* state doesn't always mean the intended one from
 each user's point of view.
 There is a distributed key-value store implementation in Go ([ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt)) from
 the IPFS ecosystem that uses
 [Merkle CRDTs](https://research.protocol.ai/publications/merkle-crdts-merkle-dags-meet-crdts/psaras2020.pdf). It's
 independent of the permanent storage implementation and the transport layer for broadcasting and receiving updates from
 peers.
 The gossip protocol can be used to propagate state updates across the network. There is a great project from HashiCorp
 called [Serf](https://www.serf.io/) that provides a gossip protocol implementation in addition to cluster membership,
 failure detection, and basic orchestration that is decentralized, fault-tolerant and highly available. It's used in
 Nomad and Consul internally.
 [`experiment/serf_crdt.go`](../experiment/serf_crdt.go) is an experimental distributed key-value store
 using [BadgerDB](https://github.com/dgraph-io/badger) as the persistent storage for
 [ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt) and Serf as the gossip protocol that uses its custom user events
 and queries features. The implementation is very inefficient but the results are promising. A couple of machines are
 able to replicate changes to the store across the internet in a fraction of a second and converge their copies if they
 get out of sync. Although, the convergence is quite slow at the moment due to the inefficient broadcast and query
 implementation.
 There is a hypothesis that in practice there shouldn't be much churn and conflicts in the state of relatively small
 deployments. The updates should be propagated quickly enough thus the eventually consistent behaviour should be
 acceptable.
 ### Container scheduling
 There are two fundamental approaches to design an orchestration system: declarative and imperative. I believe that
 Uncloud should use a hybrid approach that favours the imperative over the declarative one where possible.
 The imperative approach allows the errors to be handled in a more predictable way. While the declarative approach allows
 to decouple components of the system by using a shared state and events for communication.
 For example, the user can run a command to start a container on a specific machine. The command can call the target
 machine directly to start the container, handle the errors accordingly, and return the result to the user.
 Alternatively, the command can update the local state declaring that a container should be started on a specific machine
 and wait until it's propagated to the target machine and reconciled. Arguably, the latter approach is more complex, less
 predictable, and has more edge cases to handle.
 However, asynchronously updating the configuration of DNS servers or reverse proxies in accordance to the state changes
 caused by started or stopped containers is likely a more reliable approach than doing it imperatively.
 Not sure if we really need this yet, but more complex scenarios like running and watching a replica set of containers
 might require a sort of coordination between machines. I believe this can be achieved by using consensus and leader
 election algorithms on demand.
 Maybe for now we should only aim for a simpler and more static container orchestrator where container scheduling can
 only be initiated by a user. Docker on each machine will ensure that the containers are running and restarted on
 failures, of course. But they won't be moved to other machines automatically.
 ## Service discovery
 Service discovery using DNS has its own drawbacks but it's likely the simplest way to implement it. Like in Docker and
 Docker Swarm, the Uncloud agent on every machine will expose an internal DNS server to the machine itself and the
 containers running on it. The DNS server will resolve machine, container, and service names to their respective IP
 addresses within the mesh network.
 DNS servers will watch running containers through the shared state and update their DNS records accordingly. The
 following table is an example list of available DNS names:
 | DNS name                                           | Resolves to                                                                |
 |----------------------------------------------------|----------------------------------------------------------------------------|
 | `<machine-name>.machine.internal`                  | Mesh IP of the machine. Machine names must be unique within the org        |
 | `<container-name>.<machine-name>.machine.internal` | Mesh IP of the container on the machine                                    |
 | `<service-name>.internal`                          | Mesh IPs of all containers for the service                                 |
 | `lb.<service-name>.internal`                       | Virtual IP / LB IP that balances traffic to all containers for the service |
 A proper thinking about the name design is needed. Most likely we need to namespace them by project or environment name
 to allow for multiple instances of the same service and avoid conflicts.
 Virtual IPs for services using IPVS, ip/nftables, or eBPF can be implemented later if we see fit.
 ## Ingress
 Services can be exposed to the internet by running a reverse proxy on a machine(s) that listens on the public IP
 address. A user should be able to control which machine(s) should run a reverse proxy by assigning an appropriate role
 to them.
 Traefik or Caddy can be used as a reverse proxy that can automatically discover services running on the network using
 the internal DNS server. They both support automatic TLS certificate generation and renewal using Let's Encrypt.
 In case of network partitioning, DNS servers should adjust their records in accordance with what containers are
 available in their partition. Consequently, the reverse proxy should adjust its configuration to route traffic to only
 the available containers by resolving the updated DNS records.