- Uses Dual Stack IPv6 + IPv4 over Wireguard
- Sites are in a fully-connected wireguard mesh
- Sites are BGP neighbors
- Cross-site devices are connected to more than one site
- Routing to cross-site devices is managed with BGP
- Clients are never connected directly and should only communicate through sites: although direct connection is possible and can be used, inconsistensies may occur when automatically updating IP addresses of the peers, and manual reconfiguration of those clients would be necessary in such cases.
- If site is not a part of full mesh, routing to is is managed is BGP
- No NAT required with IPv6, very powerful access-management is possible on firewall-level
- To avoid connectivity issues, sites that do not have static public IP address, must use persistent keepalive when connected to other sites
#todo/infra/research
- how will routing work with two routes existing through different sites?
- BGP Multipath?
- latency-based routing?
| base | network | site | client | |
|---|---|---|---|---|
| subnet | 10/8 | 10.181/16 | 10.181.X/24 | 10.181.0.X/32 |
| value | 10 | 181 | 0-255 | 0-255 |
Global and Network IDs: RFC3879
Note: G and N are shorthands for global and network prefixes Note: Network ID is divergent from RFC and takes up 2 bytes
The network is subdivided into 3 namespaces:
- site: the geographical location enclosing clients and devices
- client: the unique client id in the network, an owner of the device, should be consistent across sites
- device: the physical device capable of receiving or sending traffic, has a 8-bit subnet for device-local networking (docker containers, VMs, ip-over-usb periphery, etc...)
A special case exists: not every device is in geographically immutable location, thus a global "default" site must exist for cross-site devices.
Base network hierarchy
| base | global id | version | network id | site_group | site id | client_group | client id | device-group | device id | device-local | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| bits | 8 | 40 | 8 | 8 | 4 | 12 | 4 | 16 | 4 | 16 | 8 |
| subnet | fd::/8 | fd75:0707:0036::/48 | /56 | /64 | /68 | /80 | /84 | /100 | /104 | /120 | /128 |
| value | fd | 7507070036 | 0-FF | 0-FF | 1-F | 0-FFF | 0-F | 0-FFFF | 0-F | 0xFFFF | 0-FF |
| width | 1 | 1 | 256 | 256 | 15 | 4096 | 16 | 65536 | 16 | 65536 | 256 |
Global site namespace, site_group=0
| base | global id | version | network id | site_group | client_group | client id | device-group | device id | device-local | |
|---|---|---|---|---|---|---|---|---|---|---|
| bits | 8 | 40 | 8 | 8 | 4 | 4 | 16 | 4 | 28 | 8 |
| subnet | fd::/8 | fd75:0707:0036::/48 | /56 | /64 | /68 | /72 | /88 | /92 | /120 | /128 |
| value | fd | 7507070036 | 0-FF | 0-FF | 0 | 1-F | 0-FFFF | 1-F | 0-0xFFFFFFF | 0-FF |
| width | 1 | 1 | 256 | 256 | 1 | 15 | 65536 | 15 | 268435456 | 256 |
IMPORTANT: as entity groups are part of the address, they should not be changed frequently. For now assumption is that address is never reused, and any change will just take a new address from the pool.
Additionally, for some entities, changing the group is not feasible at all: sites should never change their address, as we can't update all client configurations.
Following that quality, we can use sgroup+sid pair as site identifier.
client_id is unique across the given network