From 32448381acf132f93869fffb4b3cea945b5032d7 Mon Sep 17 00:00:00 2001 From: Pavel Sviderski Date: Fri, 9 Aug 2024 16:54:36 +1000 Subject: [PATCH] add the initial draft of the design doc --- docs/design.md | 157 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 157 insertions(+) create mode 100644 docs/design.md diff --git a/docs/design.md b/docs/design.md new file mode 100644 index 00000000..ea8825bd --- /dev/null +++ b/docs/design.md @@ -0,0 +1,157 @@ +# Uncloud design + +Uncloud aims to provide a solution for deploying and running containerized applications and services across various +platforms — from cloud VPSs to Raspberry Pis and powerful servers — or any combination of them. + +While existing prominent PaaS solutions like https://coolify.io/ and https://dokploy.com/ use a single machine or Docker +Swarm as their underlying infrastructure, Uncloud takes a different approach. It aims to offer a multi-host experience +similar to the serverless solutions provided by cloud providers but on users' own machines. Users should be able to add +machines as compute resources without worrying about control plane high availability and cluster management. For +example, a user could combine a cloud VPS with a spare Raspberry Pi to create a unified computing environment. + +Uncloud aims to replace the concept of a "cluster" with a "network" of machines. It's similar to Tailscale's design +where users can add machines to a network to enable secure communication between them. The primary challenges in this +design are: + +1. Building such a network +2. Orchestrating containers across it +3. Exposing services to the internet + +## Network of machines + +While Tailscale is an obvious solution for establishing an overlay network, it has some drawbacks: + +1. Users must register an account and generate a key for each machine. +2. Every container requires a Tailscale proxy to communicate with other machines and containers. + +The Tailscale subnet router feature could potentially be used to route traffic from the machine to the internal Docker +network. + +As an alternative, we can configure a simple flat WireGuard mesh network during the machine setup process. For example: + +| CIDR | Description | +|-----------------|----------------------------------------------------------| +| `10.210.0.0/16` | The entire WireGuard mesh network | +| `10.210.X.0/24` | `/24` subnet assigned to machine `X` | +| `10.210.X.1/32` | Machine `X` address is the first address from the subnet | +| `10.210.X.Y/32` | Container `Y` address running on machine `X` | + +Testing has shown that inter-container and machine routing works well when `/24` is assigned to the Docker network +bridge, while the WireGuard interface is configured with just the `/32` machine address. + +This setup allows containers to communicate with each other on the same machine or across machines without address +translation. This design also enables the future implementation of [ACLs](https://tailscale.com/kb/1018/acls) and +security groups to restrict traffic between machines and containers if needed. + +The peer discovery and NAT traversal techniques for the WireGuard mesh are heavily inspired by the +Talos [KubeSpan](https://www.talos.dev/v1.7/talos-guides/network/kubespan/) design. + +## Orchestration + +The main question that drives the design of the orchestration system is can we build a system that doesn't require a +centralised control plane? The short answer is yes, we can. However, it's not easy as decentralised systems are +inherently more complex than centralised ones. + +The existing solutions such as Kubernetes, Docker Swarm, and Nomad use a centralised control plane to provide a single +point of entry for the user to interact with the cluster. I'm envisioning a system where all machines are equal, yet +each one can represent the entire cluster and receive commands from the user. Where a user can be a CLI tool or a web +interface that can run on any machine in the network. In case of network partitioning, the user should continue to be +able to interact with each partition separately to manage services running on them. The system should be able to +reconcile the state of the network when the partition is healed. Such an architecture favors Availability and Partition +tolerance (AP) over Consistency and Availability (CA) in the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem). + +### Shared state + +One of the ways to achieve this is to store the entire shared state on every machine and keep them all in sync. A couple +of techniques can be used to assist with this: +a [Conflict-Free Replicated Data Type](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) (CRDT) and a +peer-to-peer [gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol). + +By composing the state using CRDT data structures allows any machine to modify its own copy of the state independently, +concurrently, and without coordination with other machines. The CRDT automatically resolves conflicts that might arise +from concurrent updates. Although the machines may have different states at any given time, they're guaranteed to +eventually converge to the *same* state. The tradeoff is that the *same* state doesn't always mean the intended one from +each user's point of view. + +There is a distributed key-value store implementation in Go ([ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt)) from +the IPFS ecosystem that uses +[Merkle CRDTs](https://research.protocol.ai/publications/merkle-crdts-merkle-dags-meet-crdts/psaras2020.pdf). It's +independent of the permanent storage implementation and the transport layer for broadcasting and receiving updates from +peers. + +The gossip protocol can be used to propagate state updates across the network. There is a great project from HashiCorp +called [Serf](https://www.serf.io/) that provides a gossip protocol implementation in addition to cluster membership, +failure detection, and basic orchestration that is decentralized, fault-tolerant and highly available. It's used in +Nomad and Consul internally. + +[`experiment/serf_crdt.go`](../experiment/serf_crdt.go) is an experimental distributed key-value store +using [BadgerDB](https://github.com/dgraph-io/badger) as the persistent storage for +[ipfs/go-ds-crdt](https://github.com/ipfs/go-ds-crdt) and Serf as the gossip protocol that uses its custom user events +and queries features. The implementation is very inefficient but the results are promising. A couple of machines are +able to replicate changes to the store across the internet in a fraction of a second and converge their copies if they +get out of sync. Although, the convergence is quite slow at the moment due to the inefficient broadcast and query +implementation. + +There is a hypothesis that in practice there shouldn't be much churn and conflicts in the state of relatively small +deployments. The updates should be propagated quickly enough thus the eventually consistent behaviour should be +acceptable. + +### Container scheduling + +There are two fundamental approaches to design an orchestration system: declarative and imperative. I believe that +Uncloud should use a hybrid approach that favours the imperative over the declarative one where possible. + +The imperative approach allows the errors to be handled in a more predictable way. While the declarative approach allows +to decouple components of the system by using a shared state and events for communication. + +For example, the user can run a command to start a container on a specific machine. The command can call the target +machine directly to start the container, handle the errors accordingly, and return the result to the user. +Alternatively, the command can update the local state declaring that a container should be started on a specific machine +and wait until it's propagated to the target machine and reconciled. Arguably, the latter approach is more complex, less +predictable, and has more edge cases to handle. + +However, asynchronously updating the configuration of DNS servers or reverse proxies in accordance to the state changes +caused by started or stopped containers is likely a more reliable approach than doing it imperatively. + +Not sure if we really need this yet, but more complex scenarios like running and watching a replica set of containers +might require a sort of coordination between machines. I believe this can be achieved by using consensus and leader +election algorithms on demand. + +Maybe for now we should only aim for a simpler and more static container orchestrator where container scheduling can +only be initiated by a user. Docker on each machine will ensure that the containers are running and restarted on +failures, of course. But they won't be moved to other machines automatically. + +## Service discovery + +Service discovery using DNS has its own drawbacks but it's likely the simplest way to implement it. Like in Docker and +Docker Swarm, the Uncloud agent on every machine will expose an internal DNS server to the machine itself and the +containers running on it. The DNS server will resolve machine, container, and service names to their respective IP +addresses within the mesh network. + +DNS servers will watch running containers through the shared state and update their DNS records accordingly. The +following table is an example list of available DNS names: + +| DNS name | Resolves to | +|----------------------------------------------------|----------------------------------------------------------------------------| +| `.machine.internal` | Mesh IP of the machine. Machine names must be unique within the org | +| `..machine.internal` | Mesh IP of the container on the machine | +| `.internal` | Mesh IPs of all containers for the service | +| `lb..internal` | Virtual IP / LB IP that balances traffic to all containers for the service | + +A proper thinking about the name design is needed. Most likely we need to namespace them by project or environment name +to allow for multiple instances of the same service and avoid conflicts. + +Virtual IPs for services using IPVS, ip/nftables, or eBPF can be implemented later if we see fit. + +## Ingress + +Services can be exposed to the internet by running a reverse proxy on a machine(s) that listens on the public IP +address. A user should be able to control which machine(s) should run a reverse proxy by assigning an appropriate role +to them. + +Traefik or Caddy can be used as a reverse proxy that can automatically discover services running on the network using +the internal DNS server. They both support automatic TLS certificate generation and renewal using Let's Encrypt. + +In case of network partitioning, DNS servers should adjust their records in accordance with what containers are +available in their partition. Consequently, the reverse proxy should adjust its configuration to route traffic to only +the available containers by resolving the updated DNS records.