OpenCV with TIM-VX is developped to run deep learning models on NPUs (on Khadas VIM3 etc.) using OpenCV with TIM-VX integrated. Code is under review at opencv/opencv#21036. Inference performance on the NPU on Khadas VIM3 can be found in OpenCV Zoo: https://github.com/opencv/opencv_zoo#models--benchmark-results.
We provide two ways compiling OpenCV with TIM-VX backend:
- (Recommanded) Compile OpenCV together with TIM-VX.
- Compile OpenCV with TIM-VX library installed previously.
Before compiling, make sure you have the following dependencies installed:
- git
- cmake (3.14 or above)
- (optional) python3 (3.6 or above), python3-numpy
Install packages on Ubuntu:
sudo apt install git cmake
# python3 is needed if you want to run examples from OpenCV Zoo
sudo apt install python3 python3-dev
pip3 install numpy
NOTE: If running on Khadas VIM3, you will need to replace galcore.ko
with the one extracted from aarch64_A311D_6.4.9.tgz
, which can be downloaded from https://github.com/VeriSilicon/TIM-VX/releases.
# replace everytime system is rebooted
sudo rmmod galcore
sudo insmod /path/to/aarch64_A311D_6.4.9/galcore.ko
# replace once for all
sudo find /usr/lib -name galcore.ko
mv /path/to/aarch64_A311D_6.4.9/galcore.ko <found_galcore_path_in_usr_lib>
sudo reboot # reboot to take effect
To compile OpenCV together with TIM-VX, all you need to do is adding the option -D WITH_TIMVX=ON
into your cmake command:
git clone https://github.com/fengyuentau/opencv.git
cd opencv
git checkout timvx_backend_support
mkdir build & cd build
# '-D BUILD_opencv_python3=ON' is needed for OpenCV's Python interface
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=./install \
-D WITH_TIMVX=ON ..
# NOTE: Make sure you see `TIMVX: YES` in the CMake output.
# Copy the value of `VIVANTE SDK path` for running on TIM-VX backend.
make -j8
make install
LD_LIBRARY_PATH
and VIVANTE_SDK_DIR
is needed to run OpenCV with TIM-VX backend:
export LD_LIBRARY_PATH=/path/to/<VIVANTE_SDK_path>/lib:$LD_LIBRARY_PATH
export VIVANTE_SDK_DIR=/path/to/<VIVANTE_SDK_path>
# On x86_64 Linux, a typical <VIVANTE_SDK_path> is /home/user/opencv/build/3rdparty/libtim-vx/TIM-VX-c1f6866b697000934d1d61ca40937248c262fa9d/prebuilt-sdk/x86_64_linux
# On Khadas VIM3, a typical <VIVANTE_SDK_path> is /home/user/opencv/build/3rdparty/libtim-vx/VIVANTE_SDK/vim3_aarch64
Our cmake scripts detect system architecture automatically to determine which SDK is used. For x86_64 Linux, SDK for x86_64 simulation is used; For Ubuntu on Khadas VIM3, the aarch64_A311D_6.4.8.tgz
SDK is used instead. Otherwise, building OpenCV with TIM-VX will be turned off.
git clone https://github.com/VeriSilicon/TIM-VX.git
cd TIM-VX # lets say ${TIM-VX} is the root of the cloned TIM-VX repositry
git checkout -b v1.1.37 v1.1.37
mkdir build & cd build
# build for x86_64 Linux
cmake -D BUILD_SHARED_LIBS=OFF ..
# build on Khadas VIM3
# Download aarch64_A311D_6.4.9.tgz from https://github.com/VeriSilicon/TIM-VX/releases
# You will need to replace 'drivers' with 'lib' in file ${TIM-VX}/cmake/local_sdk.cmake
cmake -D EXTERNAL_VIV_SDK=/path/to/aarch64_A311D_6.4.9 -D BUILD_SHARED_LIBS=OFF ..
make -j8
make install # installs to ./install by default
git clone https://github.com/fengyuentau/opencv.git
cd opencv
git checkout timvx_backend_support
mkdir build & cd build
# VIVANTE_SDK_DIR is /path/to/TIM-VX/prebuilt-sdk/x86_64_linux for x86_64
# or /path/to/aarch64_A311D_6.4.9 for Khadas VIM3.
# '-D BUILD_opencv_python3=ON' is needed for OpenCV's Python interface.
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=./install \
-D WITH_TIMVX=ON \
-D TIMVX_INSTALL_DIR=/path/to/TIM-VX/build/install \
-D VIVANTE_SDK_DIR=/path/to/VIVANTE-SDK
..
make -j8
make install
LD_LIBRARY_PATH
and VIVANTE_SDK_DIR
is needed as well to run OpenCV with TIM-VX backend:
export LD_LIBRARY_PATH=/path/to/<VIVANTE_SDK_path>/lib:$LD_LIBRARY_PATH
export VIVANTE_SDK_DIR=/path/to/<VIVANTE_SDK_path>
We have done performance tests on OpenCV Zoo: https://github.com/opencv/opencv_zoo#models--benchmark-results. You can reproduce the results as follows:
# Install git-lfs from https://git-lfs.github.com/
git clone https://github.com/opencv/opencv_zoo && cd opencv_zoo
git lfs install
git lfs pull
cd benchmark
# edit the config of the model you want to run, switch backend to TIM-VX and target to npu
vim config/face_detection_yunet.yaml
# in vim, press i to edit:
backend: "default" -> backend: "timvx"
target: "cpu" -> target: "npu"
# in vim, press esc and type :wq to save & exit
# run benchmarking
PYTHONPATH=.. python benchmark.py --cfg ./config/face_detection_yunet.yaml
Please ensure you have set LD_LIBRARY_PATH
and VIVANTE_SDK_DIR
properly as noted above.
Please make sure you have installed python3, python3-dev and numpy:
sudo apt install python3 python3-dev
pip3 install numpy
Remove cmake cache under build
:
# lets say ${opencv} is the root of the cloned OpenCV repositry
cd ${opencv}/build
rm CMakeCache.txt
Add the following options to your cmake command:
-D PYTHON3_INCLUDE_DIR=/path/to/python/include # A typical one is /usr/include/python3.8, where you can find Python.h
-D PYTHON3_EXECUTABLE=/path/to/python_binary # A typical one is /usr/bin/python3.8, where you can locate the binary
-D PYTHON3_LIBRARY=/path/to/libpython3.x.so # A typical one is /usr/lib/aarch64-linux-gnu/libpython3.8.so.1.0, where you can locate the shared library
Then run make -j 4
and make install
to re-build OpenCV. The built Python interface cv2.x.so
should be located under install/lib/python3.8/site-packages/cv2/python-3.8/
.
By default, you can locate the OpenCV's Python interface cv2.x.so
under ${opencv}/build/install/lib/python3.8/site-packages/cv2/python-3.8
once compilation is finished.
You may create a symbolic link to cv2.x.so
under python3's package directory. For example,
# '/usr/lib/python3/dist-packages/cv2.so': you will need to find your own location to replace in your system
ln -sf ${opencv}/build/install/lib/python3.8/site-packages/cv2/python-3.8/cv2.cpython-38-aarch64-linux-gnu.so /usr/lib/python3/dist-packages/cv2.so
OpenCV should be ready for Python:
python3 -c "import cv2; print(cv2.__version__)"
VIVANTE_SDK_DIR
is set for TIM-VX to find VIVANTE_SDK_DIR/include/CL/cl_viv_vx_ext.h
. A workaround to avoid setting VIVANTE_SDK_DIR
is put cl_viv_vx_ext.h
along with executable binary files.
LD_LIBRARY_PATH
is set for TIM-VX to find OpenVX and other dependencies. TIM-VX is built on top of OpenVX and other shared libraries. Integrating TIM-VX into OpenCV DNN also brings the dependencies for OpenCV DNN.