EDIT: check out fix-python instead, make Python run "as usual" on NixOS!
- You are using NixOS
- You start working on a Python project
- You manage the dependencies in a classic Python virtual environment using pip or poetry
You won't be able to import some of those libraries.
$ nix run nixpkgs#python3 -- -m venv .venv
$ . .venv/bin/activate
(.venv) $ pip install numpy
(.venv) $ python -c 'import numpy'
You'll get for instance:
libz.so.1: cannot open shared object file: No such file or directory
When we peak at the installation of numpy
, there are a bunch of compiled files (executables or libraries).
Some of those are not properly linked.
$ nix shell nixpkgs#file --command find .venv/lib/python3.9/site-packages/numpy -type f -executable -exec sh -c "file -i '{}' | grep -qE 'x-(.*); charset=binary'" \; -print -exec sh -c "ldd '{}' | grep 'not found'" \;
.venv/lib/python3.9/site-packages/numpy/fft/_pocketfft_internal.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/bit_generator.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_common.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_pcg64.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_mt19937.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_bounded_integers.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_philox.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/mtrand.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_sfc64.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_generator.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/linalg/_umath_linalg.cpython-39-x86_64-linux-gnu.so
libz.so.1 => not found
.venv/lib/python3.9/site-packages/numpy/linalg/lapack_lite.cpython-39-x86_64-linux-gnu.so
libz.so.1 => not found
.venv/lib/python3.9/site-packages/numpy/core/_rational_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_operand_flag_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_umath_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_multiarray_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_struct_ufunc_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_simd.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so
libz.so.1 => not found
In the example above, we can see that lapack_lite.cpython-39-x86_64-linux-gnu.so
, one of the binary files provided by the numpy
wheel, does not find the libz.so.1
library.
-
Install the dependencies, or enter a shell
$ nix shell nixpkgs#file nixpkgs#patchelf github:GuillaumeDesforges/nix-patchtools
-
Set the environment variable
libs
(used byautopatchelf
) to a list of paths to all the required libraries.$ export libs=$(nix eval --expr 'let pkgs = import (builtins.getFlake "nixpkgs") {}; in pkgs.lib.strings.makeLibraryPath (with pkgs; [ gcc.cc zlib glibc ])' --impure | sed 's/^"\(.*\)"$/\1/'):$(find $(pwd) -name '*.libs' | tr '\n' ':' | sed 's/:$//')
-
Save the following script to a file and run it
TMP="__tmp" mkdir -p $TMP cd $TMP find "$(realpath "../.venv")" -type f -executable -exec sh -c "file -i '{}' | grep -qE 'x-(.*); charset=binary'" \; -print \ | while read file do echo "file=$file" RPATH=$(patchelf --print-rpath "$file") echo "rpath=$RPATH" autopatchelf "$file" if [[ "$RPATH" =~ .*(^|:)[^/].* ]]; then echo "repatch rpath" patchelf --set-rpath "" "$file" autopatchelf "$file" fi done cd .. rm -R $TMP
After running the script, check the result:
$ . .venv/bin/activate
(.venv) $ python -c 'import numpy'
In the special case of CUDA for pytorch:
You need to add
$(readlink /run/opengl-driver)/lib
to$libs
.Also, since the lib is not required, autopatchelf will not add it, so the bash script does not suffice.
What you can do instead is set the RPATH of all executables to
$libs
: