Last active
March 29, 2024 14:50
-
-
Save edharman/fe647e88bb4610a461c4803d2b3cbfab to your computer and use it in GitHub Desktop.
Running containers using systemd-nspawn
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Containers provide a quick way to configure and test new operating system versions, deployment scripts etc in minimal containers which | |
are merely files stored on your container host. | |
One aspect that some folk may find daunting is the networking aspect, to make containers useful the default networking which will allow | |
- Container->Internet connectivity means that the container will not be visible to your local network so say ssh'ing in from say your | |
desktop machine or launching Xsessions just won't work. | |
So before you start we need to configure a bridged network on the principle lan interface on your host and then that bridged interface | |
can be shared by your containers and will allow unfettered inbound and outbound connectivity to your other local hosts and devices. | |
Unfortunately setting this up can be confusing because each linux distro will have it's own preferred method for controlling the network | |
and even these will vary across Desktop and Server variants of the same distro... | |
My approach to this is simple - since I am proposing to use systemd to manage the containers, we can use systemd-networkd to configure | |
and control the network - | |
So let's assume you have a suitable host with a suitable version of Ubuntu server or desktop installed and in which case it is likely | |
that NetworkManager is controlling your network interfaces. | |
Open a terminal session on your host and install the following packages - | |
sudo apt install systemd systemd-container | |
sudo mkdir /etc/systemd/nspawn | |
Check your current IP configuration to ascertain the current interface name and IP configuration details - | |
If using DHCP that's all you need to know | |
If using a static IP then you will need | |
IP address and subnet | |
Gateway address | |
DNS address | |
Make a note of these for later.. | |
Next we'll create the configuration files to switch over from using your existing interface name to a new bridge name, nothing will | |
actually happen at this stage since NetworkManager is still running and managing your hosts interfaces.. | |
All systemd-networkd configuration files live in a root owned directory- | |
/etc/systemd/network | |
To configure a bridge we need to create 3 files, one defines the bridge name and interface type, the second associates (binds) the bridge name with a physical interface, and the third sets the bridge interface network parameters - | |
So 1st lets create the bridge - | |
sudo su | |
<your root pwd> | |
# nano /etc/systemd/network/hostbridge.netdev | |
[NetDev] | |
Name=br0 | |
Kind=bridge | |
Save and exit the file | |
Next lets create the bind - | |
# nano /etc/systemd/network/10-bind.network | |
[Match] | |
Name=enp1s0f0 <- substitute with your interface name e.g. eth0 | |
[Network] | |
Bridge=br0 | |
LinkLocalAddressing=no | |
IPv6AcceptRA=no | |
Save and exit the file | |
Finally lets create the interface configuration - | |
# nano /etc/systemd/network/hostbridge.network | |
[Match] | |
Name=br0 | |
[Network] | |
DHCP=ipv4 | |
Save and exit the file | |
Note: this assumes you are using DHCP on the host, if you are using a static address then substitute the final DHCP line with - | |
Address=192.168.0.130/24 | |
Gateway=192.168.0.254 | |
DNS=192.168.0.120 | |
substituting your specific IP addresses for each line from your earlier notes.... | |
Note: If you have multiple network devices on your host but one or more are not connected then by default systemd-networkd will wait upon boot until either all are online or timeout after 2 minutes. | |
To circumvent this you can tell the daemon to only wait until one interface has become ready before continuing the system startup - | |
sudo systemctl edit systemd-networkd-wait-online.service | |
navigate to the 3rd line from the top and add - | |
[Service] | |
ExecStart= | |
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --any | |
CTRL-X to save and exit | |
This creates an override file in /etc/systemd/system that will enable the boot to proceed as soon as one interface becomes ready, and on my rather old hardware once the machine has moved past POST it boots in under 3 secs. | |
Now we have the required configuration files we can now turn off NetworkManager and switch over to systemd-networkd - | |
sudo systemctl stop NetworkManager <- stop the service | |
sudo systemctl mask Network-Manager <- prevent the service starting on boot | |
sudo systemctl start systemd-networkd | |
sudo systemctl enable systemd-networkd <- enable it to start on next boot | |
If all worked ok, your network interfaces br0 and physical interface should both be up, and e.g. ping to a foreign address - google or whatever should work ok. | |
You can check the interfaces using ip a - | |
enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000 | |
link/ether 00:1b:78:59:2b:2e brd ff:ff:ff:ff:ff:ff | |
br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 | |
link/ether f6:50:82:b4:22:4d brd ff:ff:ff:ff:ff:ff | |
inet 192.168.0.240/24 metric 1024 brd 192.168.0.255 scope global dynamic br0 | |
valid_lft 73419sec preferred_lft 73419sec | |
inet6 fe80::f450:82ff:feb4:224d/64 scope link | |
valid_lft forever preferred_lft forever | |
...and using systend's network status tool networkctl - | |
IDX LINK TYPE OPERATIONAL SETUP | |
1 lo loopback carrier unmanaged | |
2 eno1 ether no-carrier configuring <onboard and unused nic | |
3 enp1s0f0 ether enslaved configured < port 0 of a dual port PCI nic that is connected | |
4 enp1s0f1 ether no-carrier configuring < disconnected 2nd port of PCI nic | |
5 br0 bridge routable configured < my virtual bridge | |
12 vb-rms-debi4QQB ether degraded unmanaged < an active container using the shared br0 - hence vb prefix | |
6 links listed. | |
If you have multiple network onterfaces on your host you may experience slow boot times because by default the systemd-wait-online service by default will wait for all available interfaces to come up, and it will wait and eventually timeout after 2 minutes. | |
To prevent this create an override file and allow startup to proceed if any interface becomes ready - | |
sudo systemctl edit systemd-networkd-wait-online.service | |
and place this block within the predefined gap below the template stanza - | |
[Service] | |
ExecStart= | |
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --any | |
save and exit - this will write the override to /etc | |
So now we have an environment that allows us to install a new container, attach it to the bridge and let it have connectivity to my local lan + devices and the internet. | |
To build a new container using a Debian or Ubuntu distro you can use a program called debootstrap so - | |
On the host machine - | |
sudo apt install debootstrap | |
The syntax of how to use it is pretty simple - | |
sudo debootstrap -components=main,universe <codename> <container name> <repository-url> | |
the components bit just means the core binaries and universe repositories | |
<codename> is the distro specific version e.g. bookworm, jammy etc. to see what releases your version of debootstrap supports - | |
ls /usr/share/debootstrap/scripts | |
By default systemd-nspawn looks for containers in - | |
/var/lib/machines | |
..so if you have a particular location you'd prefer to locate these then prepend that to the container name and create a soft link in /var/lib/machines pointing to the real machine location so that the systemd tools can find it- | |
sudo ln -s <real machine path/name> </var/lib/machines/machine-name> | |
So to create a container called rms-ubuntu-lts based upon Ubuntu Jammy stored in the default location - | |
sudo debootstrap --include=systemd-container,openssh-server --components=main,universe jammy /var/lib/machines/rms-ubuntu-LTS http://archive.ubuntu.com/ubuntu | |
The container needs some specific packages - dbus-broker and systemd-container to work so these are included as add-ons | |
to the base build | |
Depending on your internet connectivity and local disk speeds this may take 10 or more minutes to download and install, but you get a running commentary along the way.. | |
This does install the bare minimum to make a bootable system but things like man pages and even the man package itself are not included. | |
It makes this a pretty minimal but fast process and you can always add stuff later if you wish! | |
To give an idea as to size a Ubuntu Jammy build + RMS + one nights capture +gdm3 (the Desktop environment) occupies 18GB so still pretty | |
small. | |
Post the base system installation debootstrap images require some housekeeping - | |
By default the only account on the system is root, and it has no password set and also for whatever reason the container inherits the hostname of the parent machine - which can lead to some confusion!! | |
So next step is to launch a root shell in the container in single user mode - | |
To exit this shell when finished you hold down the Ctrl key and type 3 ]'s in quick sucession - so CTRl]]] | |
systemd-nspawn -D <path to container/container name> | |
Note: you are root so no sudo is necessary | |
Add a regular user - | |
adduser rms | |
newpassword prompt | |
repeat password | |
accept defaults | |
grant user sudo privs - note for Debian distros sudo itself is not installed and at present your container has no active network so cannot access any remote repositories. Workaround is to set a password for root using the passwd cmd and fix later on when the network | |
is configured and working.Also Debian have implemented tighter security on containers such that e.g. unprivileged users cannot use - | |
/usr/bin/ping. | |
I will document these later. | |
usermod -aG sudo rms | |
Change the hostname - best to set this to the container name again to avoid later confusion.... | |
echo "myhostname" >/etc/hostname | |
NOTE: It is way simpler to have the container hostname agree with the actual container directory name - that way you know what each container ...erm contains! | |
If your existing container directory doesn't match the contained o/s hostname then after exiting the container you can simply issue a | |
sudo mv <oldname> <newname> | |
This is possible because as far as the host is concerned a container is just a file so can be copied around, moved, cloned or whatever without breaking anything! | |
set systemd-networkd to start on-boot | |
systemctl enable systemd-networkd | |
exit the container -CTRl]]] | |
The regular way of managing and controlling contaiiners is via the cmdline tool machinectl and it by default will look for containers in | |
/var/lib/machines | |
..and it looks for per-container configuration files in - | |
/etc/systemd/nspawn with files named in the format <containername>.nspawn | |
typical contents of a .nspawn would be | |
[Network] | |
Bridge=br0 | |
- this tells systemd that this host will grab an auto generated interface (in practice named host0) which will act as a slave to the parents bridge interface br0 | |
So to launch and background the container - | |
machinectl start <hostname> | |
The machinectl cmd requires privileged authentication so it will prompt for your password and then return | |
a bare machinectl will list currently running containers - | |
(vRMS) user@rms-test:~/source/RMS$ machinectl | |
MACHINE CLASS SERVICE OS VERSION ADDRESSES | |
ubuntu-test container systemd-nspawn ubuntu 22.04 192.168.0.247… | |
If you configured DHCP networking for the container then it should acquire an IP address and register it's hostname in your DNS | |
so you could e.g. ssh into it by name - | |
ssh ubuntu-test | |
You can also login to the container using machinectl - | |
(vRMS) eharman@rms-test:~/source/RMS$ machinectl login ubuntu-test | |
==== AUTHENTICATING FOR org.freedesktop.machine1.login === | |
Authentication is required to log into a local container. | |
Authenticating as: ed (eharman) | |
Password: | |
==== AUTHENTICATION COMPLETE === | |
Connected to machine ubuntu-test. Press ^] three times within 1s to exit session. | |
Ubuntu 22.04 LTS ubuntu-test pts/2 | |
ubuntu-test login: | |
If you want to use a graphical desktop there are a few extra packages to install and use rdp to display it. | |
Note: running this in a container does consume some resources and significant network bandwidth - | |
login to the container | |
sudo apt install --no-install-recommends ubuntu-desktop | |
sudo apt install xrdp | |
start the login greeter screen - | |
sudo systemctl restart gdm3 | |
Then on your remote system launch the RDP client, enter the destination hostname or IP address and remember to enable the options to | |
use different credentials if required and then click Connect. | |
You should then see the default Ubuntu desktop login screen, login using your local credentials and launch GUI apps. | |
I'm unsure why by default no desktop icons are displayed - it may be because this is a minimal install and so many of the required apps are missing, however you can enable them by installing - | |
sudo apt install gnome-shell-extension-desktop-icons-ng | |
then install the app to set preferences | |
sudo apt install gnome-shell-extension-prefs | |
To enable Desktop icons open the desktop extensions app and enable Desktop Icons NG (DING) | |
I haven't quite sorted launching from a click but right mouse and open works albeit showlivestream consumes over 40% util on 4 cores.... | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment