Last active
April 29, 2024 12:33
-
-
Save edharman/1fbb2fd84c5c9d5811a5c79bce6cbfd6 to your computer and use it in GitHub Desktop.
RMS image build
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Install QEMU support for the target image platform - | |
For binary support only - | |
apt-get install qemu-user-static | |
For full system emulation - where supported - | |
apt-get install qemu-system | |
Note: Qemu provides full support for some platforms e.g. RPi2/3 but not for RPi4/5 however binary support is sufficient for | |
our purposes. | |
Later we will need some Container tools to enable us to log into the image, install these - | |
sudo apt install systemd-container | |
By default release images are shrunk to minimise free space on the image and thus reduce the download size. | |
To give some usefull free space we can increase the partition size and then expand the root filesystem to add this extra space - | |
If you were burning to pysical media the raspi-config Expand filesystem performs the same task, however it expands the partition | |
to use all available free space - this is quite likely *not* what you want on your parent host... | |
A .img file is just a block based file container for the partition tables - | |
so first we need to add some space to the container | |
then expand the root partition | |
then grow the root filesystem to encompass the new space | |
First lets take a look at the latest RPi4 Bullseye RMS image and see how much is free- | |
our uncompressed image is in our home directory | |
test@rms-test:~$ ls | |
RMS_RPi4Bullseye_image_20231213.img | |
test@rms-test:~$ | |
First we set the image up on the next free loopdevice -the -f finds next free loopdev and the -P tells it to search the image | |
for partition tables and the --show displays the loop device it found- | |
test@rms-test:~$ sudo losetup -fP --show RMS_RPi4Bullseye_image_20231213.img | |
[sudo] password for test: | |
/dev/loop6 | |
So /dev/loop6p2 should be our root partition -/dev/loop6p1 is our boot partition which we don't need, lets create a staging dir to mount it on then we can inspect it - | |
mkdir temp | |
..and mount partition 2 to it - | |
sudo mount /dev/loop6/p2 temp | |
now check size and freespace - | |
df -h temp | |
Filesystem Size Used Avail Use% Mounted on | |
/dev/loop6p2 13G 12G 392M 97% /home/test/temp | |
As suspected the partition has been shrunk and there's only 392M free, so if we say want to tweak the image and run a short test | |
capture we need to add some space, so lets add 10GB - | |
First we need to unount it and remove it from the loopdev - | |
sudo umount temp | |
sudo losetup -D | |
Then append some space on the end - | |
test@rms-test:~$ sudo dd if=/dev/zero bs=10M count=1024 >>RMS_RPi4Bullseye_image_20231213.img | |
1024+0 records in | |
1024+0 records out | |
10737418240 bytes (11 GB, 10 GiB) copied, 29.8849 s, 359 MB/s # <-- its an ssd so pretty quick | |
Now just as a check, take a look at the image file - | |
test@rms-test:~$ fdisk -l RMS_RPi4Bullseye_image_20231213.img | |
Disk RMS_RPi4Bullseye_image_20231213.img: 22.75 GiB, 24422978048 bytes, 47701129 sectors | |
Units: sectors of 1 * 512 = 512 bytes | |
Sector size (logical/physical): 512 bytes / 512 bytes | |
I/O size (minimum/optimal): 512 bytes / 512 bytes | |
Disklabel type: dos | |
Disk identifier: 0xc5c7b7be | |
Device Boot Start End Sectors Size Id Type | |
RMS_RPi4Bullseye_image_20231213.img1 8192 532479 524288 256M c W95 FAT32 (LBA) | |
RMS_RPi4Bullseye_image_20231213.img2 532480 26729608 26197129 12.5G 83 Linux | |
So the partitions occupy less than 13GB but the image is now 22.75 GiB, lets expand the 2nd partition to occupy that space - | |
test@rms-test:~$ sudo growpart RMS_RPi4Bullseye_image_20231213.img 2 | |
CHANGED: partition=2 start=532480 old: size=26197129 end=26729609 new: size=47168616 end=47701096 | |
First present it the loopdev - | |
test@rms-test:~$ sudo losetup -fP --show RMS_RPi4Bullseye_image_20231213.img | |
/dev/loop6 | |
Now expand the filesystem into that new space - | |
We have to force a filesystem integrity check before resizing it - | |
test@rms-test:~$ sudo e2fsck -f /dev/loop6p2 | |
e2fsck 1.46.5 (30-Dec-2021) | |
Pass 1: Checking inodes, blocks, and sizes | |
Inode 1070 extent tree (at level 1) could be narrower. Optimize<y>? yes | |
Pass 1E: Optimizing extent trees | |
Pass 2: Checking directory structure | |
Pass 3: Checking directory connectivity | |
Pass 4: Checking reference counts | |
Pass 5: Checking group summary information | |
rootfs: ***** FILE SYSTEM WAS MODIFIED ***** | |
rootfs: 306308/812800 files (0.4% non-contiguous), 3032331/3274641 blocks | |
Finally resize the filesystem - | |
test@rms-test:~$ sudo resize2fs /dev/loop6p2 | |
resize2fs 1.46.5 (30-Dec-2021) | |
Resizing the filesystem on /dev/loop6p2 to 5896077 (4k) blocks. | |
The filesystem on /dev/loop6p2 is now 5896077 (4k) blocks long. | |
Then mount it and check free - | |
test@rms-test:~$ sudo mount /dev/loop6p2 temp | |
test@rms-test:~$ df -h temp | |
Filesystem Size Used Avail Use% Mounted on | |
/dev/loop6p2 23G 12G 9.9G 54% /home/test/temp | |
-we now have 9.9G free :-) | |
The root filesystem is now mounted on temp and looks like a regular root '/' filesystem - | |
test@rms-test:~$ ls temp | |
bin boot dev etc home lib lost+found media mnt opt proc root run sbin srv sys tmp usr var | |
NOTE: you msy be tempted to add/edit files directly prior to booting the container, this will likely end in tears since the security model will detect this and you may find that from within the container even root cannot manipulate them. | |
Now because the QEMU emulator libraries we loaded earlier do not support RPi4 or 5 hardware we can't natively boot the system | |
where ordinarily init owns the operating process tree, instead we can only boot into single user mode aka runlevel=1 | |
So to do this - | |
test@rms-test:~$ sudo systemd-nspawn -D temp | |
Spawning container temp on /home/test/temp. | |
Press ^] three times within 1s to kill container. | |
root@temp:~# uname -m | |
aarch64 | |
We have a shell session as user root and as you can see from the output of uname -m the system architecture is aarch64 and not that of the host - x86_64 | |
But remember this is a pre-built RMS image - lets logout and launch it again as user rms - | |
test@rms-test:~$ sudo systemd-nspawn -D temp -u rms | |
Spawning container temp on /home/test/temp. | |
Press ^] three times within 1s to kill container. | |
(vRMS) rms@temp:~/source/RMS $ | |
We are logged in as user RMS, we have executed user rms's .bashrc and our default location is | |
/home/rms/source/RMS | |
and we have activated the vRMS virtualenv | |
So we can pip install, add/remove packages and even run non-gui apps like StartCapture and everything will work. | |
Moreover if we quit the session and unmount the image, any changes we made are persisted to the image on-disk.. | |
Lets try -1st edit .config and assign a valid camera IP to the device capture string, we don't need to change anything else | |
- we'll end up with data captures in ~/RMS/ArchivedFiles/XX0001.... directories, save the changes. | |
from the cmd prompt lets capture 6 minutes of camera output - | |
python -m RMS.StartCapture -d 0.1 | |
Log output - | |
2024/04/27 13:52:35-INFO-StartCapture-line:767 - Program start | |
2024/04/27 13:52:35-INFO-StartCapture-line:768 - Station code: XX0001 | |
2024/04/27 13:52:35-INFO-StartCapture-line:782 - Program version: 20231213_130459, 46dfa1f662a7db2f82c38d4d3ddffa556ca3b74a | |
2024/04/27 13:52:35-INFO-StartCapture-line:810 - Freeing up disk space... | |
2024/04/27 13:52:35-INFO-DeleteOldObservations-line:241 - Need 5.12 GB for next night | |
2024/04/27 13:52:35-INFO-StartCapture-line:829 - Running for 0.1 hours... | |
2024/04/27 13:52:35-INFO-StartCapture-line:239 - Data directory: /home/rms/RMS_data/CapturedFiles/XX0001_20240427_135235_516310 | |
2024/04/27 13:52:35-INFO-DownloadPlatepar-line:25 - Checking for new platepar on the server... | |
2024/04/27 13:52:35-DEBUG-DownloadPlatepar-line:27 - Can't contact the server: RSA private key file not found. | |
2024/04/27 13:52:35-INFO-Reprocess-line:87 - No platepar file found! | |
2024/04/27 13:52:35-INFO-StartCapture-line:258 - Initializing frame buffers... | |
2024/04/27 13:52:35-INFO-StartCapture-line:282 - Initializing frame buffers done! | |
2024/04/27 13:52:35-INFO-QueuedPool-line:204 - Loaded 0 backed up results... | |
2024/04/27 13:52:35-INFO-QueuedPool-line:204 - Using 1 cores | |
2024/04/27 13:52:35-INFO-StartCapture-line:96 - Press Ctrl+C to stop capturing... | |
2024/04/27 13:52:36-INFO-BufferedCapture-line:148 - Camera IP ping successful! | |
2024/04/27 13:52:36-INFO-BufferedCapture-line:163 - Initializing the video device... | |
2024/04/27 13:52:36-INFO-BufferedCapture-line:164 - Device: rtsp://192.168.0.150:554/user=admin&password=&channel=1&stream=0.sdp | |
2024/04/27 13:52:38-INFO-BufferedCapture-line:219 - Video device opened! | |
2024/04/27 13:52:38-INFO-BufferedCapture-line:331 - Grabbing a new block of 256 frames... | |
2024/04/27 13:52:47-INFO-BufferedCapture-line:474 - New block of raw frames available for compression with starting time: 1714225958.731498 | |
2024/04/27 13:52:47-INFO-BufferedCapture-line:331 - Grabbing a new block of 256 frames... | |
2024/04/27 13:52:47-DEBUG-Compression-line:274 - Compressing frame block with start time at: 1714225958.731498 | |
2024/04/27 13:52:53-DEBUG-Compression-line:295 - Compression time: 6.167 s | |
2024/04/27 13:52:55-DEBUG-Compression-line:302 - Saving time: 1.932 s | |
2024/04/27 13:52:55-INFO-DetectionTools-line:61 - Loaded mask: mask.bmp | |
.... | |
.... | |
.... | |
2024/04/27 13:59:02-INFO-Reprocess-line:378 - Generating a timelapse... | |
2024/04/27 13:59:22-INFO-Reprocess-line:511 - Archiving detections to /home/rms/RMS_data/ArchivedFiles/XX0001_20240427_135235_516310 | |
2024/04/27 13:59:22-INFO-ArchiveDetections-line:171 - Generating thumbnails... | |
2024/04/27 13:59:23-INFO-ArchiveDetections-line:192 - Generating a stack of detections... | |
2024/04/27 13:59:29-INFO-ArchiveDetections-line:210 - Captured stack saved to: /home/rms/RMS_data/CapturedFiles/XX0001_20240427_135235_516310/XX0001_20240427_135235_516310_captured_stack.jpg | |
2024/04/27 13:59:29-INFO-ArchiveDetections-line:237 - Detected stack could not be saved! | |
2024/04/27 13:59:29-DEBUG-shutil-line:1039 - changing into '/home/rms/RMS_data/ArchivedFiles/XX0001_20240427_135235_516310' | |
2024/04/27 13:59:29-INFO-shutil-line:899 - Creating tar archive | |
2024/04/27 13:59:30-DEBUG-shutil-line:1067 - changing back to '/home/rms/source/RMS' | |
So everything looks good. | |
NOTE: whilst it is old hardware - Intel i3 circa 2013 the compression times look poor @~6s per frame | |
this is an artefact of the emulation not being efficient, the save times to an SSD whilst not stellar are | |
respectable @2secs so not terrible..! | |
the resultant ff_intervals shows numerous dropped frames - an artefact of the slwness of the emulation on what is a pretty old and lo-spec host. | |
Now one might think - what's the point? | |
I have a newer Intel i5 Gen-11 Nuc I use as a HTPC hanging off the back of my TV, I installed the two packages at the head | |
of this posting, scp'd the RPi image to it and ran a capture again after running RMS_Update so utilising the new more accurate | |
frame timing code - | |
Compression time: 3.091 s | |
Saving time: 0.679 s | |
The resultant ff_intervals image was flat - no dropped frames, Median 2.48s +/-0.000s etc | |
Press ] 3 times in a second and that kills the container, unmount the image, losetup -D to remove the loopback ...and you have a portable RMS image. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment