The docs on https://developer.apple.com/documentation/virtualization/running_intel_binaries_in_linux_vms_with_rosetta#4239539 are sparse and incomplete and don't tell you at all how to set this up. Lets trial and error our way through this.
In theory this can be enabled at the host side with
try? rosettaDirectoryShare.setCachingOptions(.defaultUnixSocket).defaultUnixSocket evaluates to /run/rosettad/rosetta.sock.
When this is enabled, a second file $mountPoint/rosettad will be available:
/run/rosetta/rosettad
usage: /run/rosetta/rosettad <command> [<args>]
valid commands:
cfg <input binary>
Constructs a control-flow graph for an x86-64 binary.
translate <input binary> <output binary>
Translates an x86-64 binary to an arm64 binary.
daemon [<cache path>]
Runs the aot translation daemon, using <cache path> as its cache directory.
If <path> is empty, the daemon defaults to caching in "$HOME/.cache/rosettad".
digest <input file>
Computes the sha256 digest of the given file.When running rosettad daemon it creates a socket in $HOME/.cache/rosettad/uds/rosetta.sock instead of /run/rosettad/rosetta.sock.
how /run/rosetta/rosetta connects to rosettad is completely unclear to me.
The quote
There are two modes of operation for AOT caching: The first is communication using a Unix Domain Socket where the Virtualization framework shares a file that represents the socket between the Rosetta daemon and Rosetta runtime through a symlink or bind-mount. The second is communication through an abstract socket where the framework defines a shared name rather than a shared file.
seems inaccurate. Virtualization.Framework isn't bind-mounting a Unix Domain Socket. It's mounting a binary called rosettad.
$ sudo -i
# nix run nixpkgs#hello --system x86_64-linux
Hello World!
# find ~/.cache/rosetta*
/root/.cache/rosetta
A folder /run/.cache/rosetta is also created. but empty.
Now also start the rosettad daemon:
# /run/rosetta/rosettad daemon &
[1] 90561
# find ~/.cache/rosetta*
/root/.cache/rosetta
/root/.cache/rosettad
/root/.cache/rosettad/uds
/root/.cache/rosettad/uds/rosetta.sock
the socket is created at /root/.cache/rosettad/uds/rosetta.sock.
But running an x86 binary doesn't really give anything
[root@utm:~/.cache]# nix run nixpkgs#hello --system x86_64-linux
Hello, world!
[root@utm:~/.cache]# find ~/.cache/rosetta*
/root/.cache/rosetta
/root/.cache/rosettad
/root/.cache/rosettad/uds
/root/.cache/rosettad/uds/rosetta.sock
Lets try to make the symlink manually?
# ln -s $HOME/.cache/rosettad/uds/rosetta.sock /run/rosettad/rosetta.sock
[root@utm:~/.cache]# nix run nixpkgs#hello --system x86_64-linux
Hello, world!
[root@utm:~/.cache]# find ~/.cache/rosetta*
/root/.cache/rosetta
/root/.cache/rosetta/58e6fd10b53795fd496dd588f0152abc.flu
/root/.cache/rosetta/0ced6549d448d1bd719c6202a33ca024.flu
/root/.cache/rosetta/a39f512ad80ab0e8db9151236992bfe7.flu
/root/.cache/rosettad
/root/.cache/rosettad/uds
/root/.cache/rosettad/uds/rosetta.sock
/root/.cache/rosettad/995d3bc2706188f35cfcbc4051022f5d012ac3e8833ba6ce7d7bae85bdb6d4f7.aotcache
/root/.cache/rosettad/77174e421083f351df2b230d9b86e49f8a1055e8e054ee0a8c679be42169b38f.aotcache
/root/.cache/rosettad/f03cb898d6687145fab8f67ba894bedb9c7e19805205cf79a5d06235ec5faea1.aotcache
Now it does work! Success!
[arian@utm:~]$ nix run nixpkgs#cowsay --system x86_64-linux -- hey
_____
< hey >
-----
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
[arian@utm:~]$ sudo find /root/.cache/rosetta /root/.cache/rosettad
/root/.cache/rosetta
/root/.cache/rosetta/58e6fd10b53795fd496dd588f0152abc.flu
/root/.cache/rosetta/0ced6549d448d1bd719c6202a33ca024.flu
/root/.cache/rosetta/a39f512ad80ab0e8db9151236992bfe7.flu
/root/.cache/rosettad
/root/.cache/rosettad/uds
/root/.cache/rosettad/uds/rosetta.sock
/root/.cache/rosettad/995d3bc2706188f35cfcbc4051022f5d012ac3e8833ba6ce7d7bae85bdb6d4f7.aotcache
/root/.cache/rosettad/77174e421083f351df2b230d9b86e49f8a1055e8e054ee0a8c679be42169b38f.aotcache
/root/.cache/rosettad/f03cb898d6687145fab8f67ba894bedb9c7e19805205cf79a5d06235ec5faea1.aotcache
No new entries. Permission issue?
Lets first create a dedicated systemd service
$ systemctl edit --full --force rosettad.service
$ systemctl cat rosettad.service
# /run/systemd/system/rosettad.service
[Unit]
RequiresMountFor=/run/rosetta
[Service]
RuntimeDirectory=rosettad
CacheDirectory=rosettad
ExecStart=/run/rosetta/rosettad daemon $CACHE_DIRECTORY
$ systemctl start $rosettad.service
[arian@utm:~]$ systemctl status rosettad.service
● rosettad.service
Loaded: loaded (/run/systemd/system/rosettad.service; static)
Active: active (running) since Mon 2024-07-29 10:06:47 UTC; 2min 46s ago
Main PID: 92302 (rosettad)
IP: 0B in, 0B out
IO: 0B read, 0B written
Tasks: 1 (limit: 38336)
Memory: 252.0K (peak: 560.0K)
CPU: 1ms
CGroup: /system.slice/rosettad.service
└─92302 /run/rosetta/rosettad daemon /var/cache/rosettad
Jul 29 10:06:47 utm systemd[1]: Started rosettad.service.
Lets create the symlink again (TODO: automate with systemd-tmpfiles):
[arian@utm:~]$ sudo ln -s /var/cache/rosettad/uds/rosetta.sock /run/rosettad/rosetta.sock
$ nix run nixpkgs#cowsay --system x86_64-linux -- hey
_____
< hey >
-----
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
[arian@utm:~]$ ls -la /var/cache/rosettad
total 0
drwxr-xr-x 1 root root 6 Jul 29 10:06 .
drwxr-xr-x 1 root root 30 Jul 29 10:06 ..
drwxr-xr-x 1 root root 24 Jul 29 10:06 uds
Hmm still nothing. Odd. However running as root does work
[arian@utm:~]$ sudo nix run nixpkgs#cowsay --system x86_64-linux -- hey
[sudo] password for arian:
_____
< hey >
-----
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
[arian@utm:~]$ ls /var/cache/rosettad/
267aefea02f89b2b41a595d481e113e2b71762be702a02693063cd7f471f50dc.aotcache a367ecc9903ab2a225610dcee6812392b205c691f46601c22b648f2d3612676f.aotcache
46f21842c4ad7b67f6c6e68897105eec1bbb67556b2e857fe5ec5a433bb0c067.aotcache e78061297a7674c5fdae9414340c7a8bf7e2d79683a08800344920ce06834453.aotcache
5029bfcb3d05842b2d2331875bcf1ba94121efbc7dd911c58d576380e2d2729f.aotcache ea88f3428c300a9d8fe37c955bb1a98e90b946df68c0406a2afc26e53322d0e4.aotcache
77174e421083f351df2b230d9b86e49f8a1055e8e054ee0a8c679be42169b38f.aotcache f03cb898d6687145fab8f67ba894bedb9c7e19805205cf79a5d06235ec5faea1.aotcache
979b27c31dd176688abb6889b4b97810372ccd06ac1e9819b782f9c4afe353e2.aotcache f3764e058289f03df19a117a55530dfab65797b237aa5e864c2fa6526df35731.aotcache
9b1c71fee94e08677559d06c28bcf7f3a79c23aaebe4031ef0cf06322cbc14b0.aotcache f3dac3a504283786eeba591b6e046adbece90b0e0bfa082c300022a333203328.aotcache
9da0216a2250267d29273dc1913f15be7096d085c5fdf64a1da8ce6f30ecf3d6.aotcache uds
Lets try to change the permssions of the socket to allow write access from groups and users:
$ sudo chmod ga+w /var/cache/rosettad/uds/rosetta.sock
$ sudo rm -rf /var/cache/rosettad/*.aotcache
And now it does seem to work:
[arian@utm:/var/cache/rosettad]$ nix run nixpkgs#cowsay --system x86_64-linux -- hey
_____
< hey >
-----
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
[arian@utm:~]$ ls /var/cache/rosettad
267aefea02f89b2b41a595d481e113e2b71762be702a02693063cd7f471f50dc.aotcache a367ecc9903ab2a225610dcee6812392b205c691f46601c22b648f2d3612676f.aotcache
46f21842c4ad7b67f6c6e68897105eec1bbb67556b2e857fe5ec5a433bb0c067.aotcache e78061297a7674c5fdae9414340c7a8bf7e2d79683a08800344920ce06834453.aotcache
5029bfcb3d05842b2d2331875bcf1ba94121efbc7dd911c58d576380e2d2729f.aotcache ea88f3428c300a9d8fe37c955bb1a98e90b946df68c0406a2afc26e53322d0e4.aotcache
77174e421083f351df2b230d9b86e49f8a1055e8e054ee0a8c679be42169b38f.aotcache f03cb898d6687145fab8f67ba894bedb9c7e19805205cf79a5d06235ec5faea1.aotcache
979b27c31dd176688abb6889b4b97810372ccd06ac1e9819b782f9c4afe353e2.aotcache f3764e058289f03df19a117a55530dfab65797b237aa5e864c2fa6526df35731.aotcache
9b1c71fee94e08677559d06c28bcf7f3a79c23aaebe4031ef0cf06322cbc14b0.aotcache f3dac3a504283786eeba591b6e046adbece90b0e0bfa082c300022a333203328.aotcache
9da0216a2250267d29273dc1913f15be7096d085c5fdf64a1da8ce6f30ecf3d6.aotcache uds
And the .flu files end up in the user's homedir:
[arian@utm:/var/cache/rosettad]$ ls ~/.cache/rosetta/
0ced6549d448d1bd719c6202a33ca024.flu 5a9d181ae5d235fdc1e88c9b67642699.flu b359582657097fec2c8ebe95b0e8ab2a.flu e31f93b3c4206974b5b42d2ec86b989e.flu
48326ec74f19e8a3b6b3526f273a7539.flu 65c827546f0ed538778d2ad7f0b92ee0.flu b4e820894f22c456f5a4429a8338f05c.flu
4c52fc315150b88ce141554ed600e93b.flu 961be7d532c2aa9f7ab5373968aa2784.flu b693a5a5de4355cce52d4e6131db360a.flu
5496122c174d3c94ec13e76228c4c8c6.flu a39f512ad80ab0e8db9151236992bfe7.flu b9f237d61a1a624ea345c41b2015000e.flu
The Apple documentation is inaccurate. It's up to the guest to set up the bind-mount/symlink to /run/rosettad/rosetta.sock for
AOT caching to work and to fix any permissions resulting from that.
The socket that is created does not have ga+w permissions which means nobody can connect to it except root.
We can't change permissions with a bind mount; so a bind mount can never be used. Instead a symlink should be set up
and then the original file's permissions need to be changed.
Explore how the Abstract Domain Socket stuff works. It sounds conceptually a lot simpler!