If you would like to install extra packages on OpenWrt, but you have run out of space on your router’s internal flash memory, then this tutorial is for you.
The plan is to copy the OpenWrt’s root filesystem onto an external USB flash drive, and tell the router to switch to that when it boots up.
All you need is a standard USB flash drive, a USB capable router running OpenWrt, and about 30 mins.
The OpenWrt wiki pages for this subject are very good. If you’re interested in reading about all the different options, then take a look (for example, it’s possible to mount the USB drive on /opt and install extras here, but some packages expect to be installed in root and won’t behave nicely).
In case you’d like to look it up later, the type of extroot that I’m talking about here is external root (aka pivot root), not external overlay (aka pivot-overlay).
This assumes that you have already formatted your external USB drive with a journaling filesystem (like EXT3 or EXT4). If you haven’t, do this first (the partition managers in Ubuntu and Kubuntu do this very easily, do a google search for a tutorial if you’re unsure how to do this).
To use pivot root, you must be using a version of OpenWrt that is newer than v12 (so 12.09 Attitude Adjustment is fine, but Backfire is not).
First, connect to OpenWrt via ssh or telnet.
First, install the package that will swap your root from the router’s inbuilt flash memory to the external USB flash device:
opkg update
opkg install block-mount
Before you do this, make sure your router is secure (e.g. SSH login isn’t possible from WAN, wifi interfaces are password protected etc).
This is important because if something goes wrong when your router boots up then OpenWrt will be unable to boot to the USB flash drive, leaving you with whatever your present setup is until you can make it boot to the USB drive properly.
First, install fdisk, a tool to tell you info about attached devices:
opkg update
opkg install fdisk
Now, insert your USB drive and run the following command to get some details about it:
root@OpenWrt:~# fdisk -l
...
Disk /dev/sda: 4229 MB, 4229496832 bytes
255 heads, 63 sectors/track, 514 cylinders, total 8260736 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00933453
Device Boot Start End Blocks Id System
/dev/sda1 63 8257409 4128673+ 83 Linux
You see your USB drive listed under “Device”. Now make a mount point for the drive. You can choose whatever you like instead of sda1, just remember what you chose later!
mkdir /mnt/sda1
Mount the drive on the mount point you just created (replace /dev/sda2 with whatever you got from the fdisk command, and /mnt/sda1 with whatever you chose as your mount point):
mount /dev/sda2 /mnt/sda1
Now copy the root filesystem from the router’s inbuilt flash to the USB drive with these commands (if you chose a name other than /mnt/sda1 in the previous step, then replace /mnt/sda1 in line 3):
mkdir -p /tmp/cproot
mount --bind / /tmp/cproot
tar -C /tmp/cproot -cvf - . | tar -C /mnt/sda1 -xf -
umount /tmp/cproot
You now have a USB drive with a copy of your router’s filesystem on it. The next step is to make it automatically mount when you boot up, and use it as root.
Open /etc/config/fstab with your favourite text editor:
nano /etc/config/fstab
If nano is not installed, you can use vi (which is installed by default, but is horrible!) or you can just install nano with:
opkg update opkg install nano
Add the following to the file:
config mount
option target /
option device /dev/sda1
option fstype ext3
option options rw,sync
option enabled 1
option enabled_fsck 0
When you reboot, you should now be running OpenWrt from your USB stick.
Check your current mounts with this command:
mount
Output should be something similar to the following:
root@OpenWrt:~# mount
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime,size=63340k)
tmpfs on /dev type tmpfs (rw,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,noatime,mode=600)
/dev/sda1 on / type ext4 (rw,sync,relatime,user_xattr,barrier=1,data=ordered)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)
See how much space you’ve created for exciting packages with this command:
df -h
Which should show you something like this:
root@OpenWrt:~# df -h
Filesystem Size Used Available Use% Mounted on
rootfs 3.9G 157.1M 3.6G 4% /
/dev/root 2.0M 2.0M 0 100% /rom
tmpfs 61.9M 932.0K 60.9M 1% /tmp
tmpfs 512.0K 0 512.0K 0% /dev
/dev/sda1 3.9G 157.1M 3.6G 4% /
You’re all done. Have fun building your personalised OpenWrt cathedral!
Ever since I obtained the Netgear router, I replaced the OEM firmware with OpenWRT firmware, though it was with the stock image and using the internal storage, which is about 4MB.
I was happy to find that I was able to upgrade the firmware to the latest version, so I installed 18.06.4. I managed to configure it so that it would hand out IP addresses via DHCP, act as a DNS server and resolve the hostnames. I don’t really intend on the router doing much more, to be quite honest, just find that making it do too much doesn’t make sense and may make it more vulnerable.
But I did try to install several packages and ended up filling up the internal storage to 100%, which is not the first time I ran into that issue. So I looked into using the USB port on the router and moving the root to the external storage.
I looked into the documentation from OpenWRT on the Extroot Configuration and created a simple firmware version that contained the necessary tools to mount the USB thumb drive as root.
After downloading the image builder package from the Releases page at OpenWRT, I proceeded to check that my router was still supported by running the following command
make info | grep -A 2 -i 'wndr3700'
Be sure to run that command on the directory that is created when extracting the contents of the image builder archive. The output of that command should look something like this
wndr3700:
NETGEAR WNDR3700
Packages: kmod-usb-core kmod-usb-ohci kmod-usb2 kmod-usb-ledtrig-usbport kmod-leds-wndr3700-usb
If you are doing this for another router, replace wndr3700 with whatever your router’s model number is.
Now we can proceed to create the base firmware for the router by running the command below
make image PROFILE=wndr3700 PACKAGES="kmod-fs-ext4 kmod-usb-storage kmod-usb-ohci kmod-usb-uhci"
This will take a while to complete, specially when other components need to be downloaded to the system. If all ended successfully, then we should have several .bin files created under bin/targets/ar71xx/generic .
Since I was already running OpenWRT on the router, I used the bin file with sysupgrade in the name. Just upload the firmware through the web interface and reset the router to factory defaults.
Because the firmware I created for the router is the base version, it doesn’t have a web interface, so access is only available via ssh and there is no default password for root, also it is only accessible from the LAN side.
Once we have access to the router, I first proceeded to update the packages and install the block-mount package by running the following commands
opkg update
opkg install block-mount
Validate that the necessary kernel modules are loaded by running the command
lsmod | grep -Ei 'sd_mod|usb-storage|ext4'
The output should be something along the lines of
crc16 960 1 ext4
crypto_hash 8288 3 ext4,jbd2,crc32c_generic
ext4 326304 0
jbd2 45616 1 ext4
mbcache 2768 1 ext4
scsi_mod 86752 2 usb_storage,sd_mod
sd_mod 25424 0
I wanted to make sure that after I made the necessary changes, that they were in fact applied, so I checked the current status of storage by running the df -h command before and after making the changes, being sure to document all of the output.
Since I have a lot of headless devices running in my network, I always want to make sure that they’re able to boot up successfully without user interaction, so I always run through a reboot process after configuring a service or system.
At this point we can mount the USB thumb drive, be sure to format the storage device with ext4 partition type previously as we will not run through that process in the router due to lack of tools. Mount the storage device using the command below
mount /dev/sda1 /mnt
If there are multiple partitions made on the USB thumb drive, then be sure to replace /dev/sda1 accordingly. I’m using an 8GB thumb drive so I only created one partition. Now, 8GB is quite overkill, but I didn’t have any smaller thumb drives, not that they’re easily available anyways.
Once mounted, verify that it is in fact mounted, we need to copy over all of the current data in /overlay to the /mnt directory and we do this by running this command
tar -C /overlay/ -c . -f - | tar -C /mnt/ -xf -
Once this copy process is completed, run the sync command, then unmount the thumb drive with the command umount /dev/sda1 and run the command below to create the fstab for this thumb drive
block detect > /etc/config/fstab
The file that is created should look something like the one below
config 'global'
option anon_swap '0'
option anon_mount '0'
option auto_swap '1'
option auto_mount '1'
option delay_root '5'
option check_fs '0'
config 'mount'
option target '/mnt/sda1'
option uuid '01234567-89AB-CDEF-1234-567890ABCDEF'
option enabled '0'
We need to change it to reflect that this is the new /overlay by changing the target and enabled options to the ones below
config 'mount'
option target '/overlay'
option uuid '01234567-89AB-CDEF-1234-567890ABCDEF'
option enabled '1'
Save the file and enable the fstab configuration with the command below
/etc/init.d/fstab enable
At this point reboot the router and wait for it to come back up. There are a couple of steps more that we need to take, but at this point the router should come back up and be using the thumb drive as the primary storage, we can validate this using the df -h command.
We also need to check that uci sees the changes, so run the following command
uci show fstab
If the output comes back like the one below
fstab.@global[0]=global
fstab.@global[0].anon_swap='0'
fstab.@global[0].anon_mount='0'
fstab.@global[0].auto_swap='1'
fstab.@global[0].auto_mount='1'
fstab.@global[0].delay_root='5'
fstab.@global[0].check_fs='0'
fstab.@mount[0]=mount
fstab.@mount[0].target='/mnt/sda1'
fstab.@mount[0].uuid='01234567-89AB-CDEF-1234-567890ABCDEF'
fstab.@mount[0].enabled='0'
We will need to make some changes by running the two commands below
uci set fstab.@mount[0].target='/overlay'
uci set fstab.@mount[0].enabled='1'
uci commit fstab
service fstab boot
We can reboot the device once more and make sure that the external storage is still being mounted.
Rootfs on External Storage (extroot) As TurrisOS is a fork of OpenWrt, most of the principles of OpenWrt are applicable to TurrisOS. Extroot is one of them.
What is extroot? Almost every router has a small amount of persistent memory. Turris Omnia has 8 GB of storage but developers say that this storage was not designed to be heavily used for read/write. Sometimes it can be an issue, for example if it is needed to:
Have permanent /var directory instead of symbolic link to /tmp Install more than 8 GB of software but don't want to alter opkg target installation pathes Use software that requires a lot of read/write operations to the root filesystem but don't want to damage the internal memory. There are several workarounds which involve creating symbolic links from the router's internal filesystem to the one which is mounted from USB device. But there is a choice which might be more convenient and reliable in some cases: extroot.
In OpenWrt there is a possibility to replace the root filesystem which is located in the internal memory with the one from external device during the boot. Original instructions for OpenWrt to achieve that can be found here.
How to do extroot on Turris Omnia If you fail, there is a possibility to soft-brick your device, so please do the backups! If you failed and router doesn't boot, please follow these instructions.
USB device is needed, make sure it is working. If it is a hard drive, check that is has enough power from USB port to work and check its cable. At least one partition on the USB device must be empty. One of the partitions will be used for extroot and will be mounted as / (root). Others can be used for any custom mounts including swapping. Extroot only supports Ext4, so if you have a partition of other type, use mkfs.ext4 to re-format the partition. Please stop all non-core activities like torrent client, DLNA server, Samba server etc. Clone the internal filesystem to the external device. For example, if the device appeared as sda1:
mkdir -p /tmp/introot
mkdir -p /tmp/extroot
mount --bind / /tmp/introot
mount /dev/sda1 /tmp/extroot
tar -C /tmp/introot -cvf - . | tar -C /tmp/extroot -xf -
umount /tmp/introot
umount /tmp/extroot
In LuCI, navigate to System → Mount Points → Add. Check Enable this mount. Choose your external USB device from UUID combo box. Choose Use as root filesystem (/) from Mount point combo box. Hit SAVE & APPLY Now configuration should look similar to this:
# cat /etc/config/fstab
config global
option anon_swap '0'
option anon_mount '0'
option auto_swap '1'
option auto_mount '1'
option delay_root '5'
option check_fs '0'
config mount
option uuid '1f0e8e00-b854-49d2-97ee-4b57bdaccb1d'
option target '/'
option enabled '1'
Copy /etc/config/fstab to the external filesystem:
mount /dev/sda1 /tmp/extroot
cp /etc/config/fstab /tmp/extroot/etc/config/fstab
umount /tmp/extroot
Cross the fingers and reboot. After reboot, mounts should look like that:
# mount
/dev/mmcblk0p1 on /rom type btrfs (ro,noatime,ssd,space_cache,commit=X,subvolid=ABC,subvol=/@)
...
/dev/sda1 on / type ext4 (rw,relatime,data=ordered)
...
From now all your data would be stored on the external drive. You can find the internal root filesystem mounted to /rom. Notes I'd recommend to keep essential files like the contents of /etc/config in sync between extroot and internal memory. I cannot say for sure at which exact moment the internal memory is replaced with the external one. It has to be /etc/rc.d/S40fstab, so scripts with level less then 40 have to be loaded from internal memory while the others - from the external storage. Since /rom is obviously is read-only, to copy changed files from the extroot to internal memory, you'd have to remount it for read/write:
mount /rom -o remount,rw
Please be aware that schnapps most probably will stop to work with extroot or will work with errors. It uses Btrfs snapshot feature to work but Brtfs is not supported by extroot. If you can shed some light on that, please update this article. You can now make /var permanent (reboot required):
cp -r /tmp /var-tmp
rm /var
mv /var-tmp /var
- backup your installed packages
opkg list-installed | cut -f 1 -d ' ' > /root/installed_packages.txt - Flash normally, keeping settings.
- The router will boot with all settings, without the extra packages. Setup internet access if it is not working already.
- Install packages to access the usb drive
opkg install block-mount kmod-fs-ext4 kmod-usb-storage e2fsprogs kmod-usb-ohci kmod-usb-uhci fdisk - Format the usb drive and copy overlay
mount /dev/sda1 /mnt ; tar -C /overlay -cvf - . | tar -C /mnt -xf - ; umount /mnt - Fix fstab and reboot
- If all goes well it will boot with overlay of usb as root.
- Install extra packages and optionally add old overlay to fstab.