-
80-NL315-14 A
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
|
Snapdragon Neural Processing Engine SDK
Reference Guide
|
This tutorial describes the steps needed to create a UDO package and execute the Inception-V3 model using the package. The Softmax operation has been chosen in this tutorial to demonstrate the implementation of a UDO with SNPE.
The SNPE SDK provides the resources for this example under
Information on UDO in general is available at UDO Overview.
Information on running the Inception-V3 network without UDO is available at Inception-V3 Tutorial.
The following tutorial assumes that general SNPE setup has been followed to support SDK environment, TensorFlow environment, and desired platform dependencies. The steps listed in this tutorial use the Tensorflow model in the form of inception_v3_2016_08_28_frozen.pb. For details on acquiring the Inception-V3 model visit Tutorials Setup.
Here are the steps to develop and run a UDO
2.) Framework Model Conversion to a DLC
5.) Model Execution
Steps 1-4 are run offline on the x86 host and are necessary for execution in step 5. Step 5 provides information on execution using the SNPE command-line executable snpe-net-run. Optionally, the user can perform steps 1-4 automatically using the provided setup script.
Generating the SoftmaxUdo package requires the snpe-udo-package-generator tool and the provided UDO plugin: Softmax.json. The plugin is located under $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/config. More information about creating a UDO plugin can be found here.
Generate the SoftmaxUdo Package using the following:
export SNPE_UDO_ROOT=$SNPE_ROOT/share/SnpeUdo snpe-udo-package-generator -p $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/config/Softmax.json -o $SNPE_ROOT/models/inception_v3/
This command creates the Softmax based package at $SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage.
For more information on the snpe-udo-package-generator tool visit here.
Converting the Tensorflow Inception-V3 model to DLC requires the snpe-tensorflow-to-dlc tool. The snpe-tensorflow-to-dlc tool consumes the same Softmax.json used in package generation via the --udo command line option. In this step, <INCEPTION_V3_PATH> refers to the path to the inception_v3 pb file. For example, after running the setup_inceptionv3.py script <INCEPTION_V3_PATH> is $SNPE_ROOT/models/inception_v3/tensorflow.
Convert Inception-V3 with the following:
snpe-tensorflow-to-dlc --input_network <INCEPTION_V3_PATH>/inception_v3_2016_08_28_frozen.pb --input_dim 'input' 1,299,299,3 --out_node InceptionV3/Predictions/Reshape_1 --output_path $SNPE_ROOT/models/inception_v3/dlc/inception_v3_udo.dlc --allow_unconsumed_nodes --udo $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/config/Softmax.json
This will generate a DLC named inception_v3_udo.dlc containing the Softmax as UDO at $SNPE_ROOT/models/inception_v3/dlc.
The generated package creates the skeleton of the operation implementation, which must be filled by the user to create a functional UDO. Additionally, the skeleton for a user implemented validation function can be populated to validate information about the UDO passed from the SNPE runtime. The rest of the code scaffolding for compatibility with SNPE is provided by the snpe-udo-package-generator.
The UDO implementations and validation function for this tutorial are provided under $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/src.
Note: Implementing validation is user-specific and not a strict requirement for using a UDO in SNPE. Only CPU Validation is provided as example.
CPU Implementations (Android and x86)
The files in the package that need to be implemented for CPU are
The provided example implementations for these files are at the locations
Copy the provided implementations to the package:
cp -f $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/src/CPU/SoftmaxImplLibCpu.cpp $SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage/jni/src/CPU/ cp -f $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/src/reg/SoftmaxUdoPackageCpuImplValidationFunctions.cpp $SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage/jni/src/reg/
Optionally, the user can provide their own implementations in the package.
GPU Implementations
The file in the package that needs to be implemented for GPU is
The provided example implementation is at the location
Copy the provided implementation to the package:
cp -f $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/src/GPU/SoftmaxImplLibGpu.cpp $SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage/jni/src/GPU/
Optionally, the user can provide their own implementations in the package.
DSP Implementations
Similar to all other SNPE runtimes, a registration library and an implementation library are required to run inference on a network with UDO layers on SNPE DSP. The registration library will run on CPU, and specifies the DSP implementation library of the UDO.
Please note that only C files are supported for UDO on DSP runtime.
The file in the package that need to be implemented for DSP are
The provided example implementation is at the location
Copy the provided implementations to the package:
cp -f $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/src/DSP/SoftmaxFloatImpl/SoftmaxImplLibDsp.c $SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage/jni/src/DSP/ cp -f $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/src/DSP/SoftmaxFloatImpl/SoftmaxImplLibDsp.h $SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage/include/
Optionally, the user can provide their own implementations in the package.
x86 Host Compilation
Compiling on x86 host uses the make build system. Compile the CPU implementations with the following:
cd $SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage make cpu_x86
The exepcted artifacts after compiling for CPU on x86 host are
Android CPU Runtime Compilation
Compilation for the CPU runtime on Android uses Android NDK. The ANDROID_NDK_ROOT environment variable must be set to the directory containing ndk-build in order to compile the package.
export ANDROID_NDK_ROOT=<path_to_android_ndk>
It is suggested to add ANDROID_NDK_ROOT to the PATH environment variable to access ndk-build.
export PATH=$ANDROID_NDK_ROOT:$PATH
Target architecture must also be specified when compiling the package.
export UDO_APP_ABI=<target_architecture>
This tutorial uses arm64-v8a architectures - it is recommended but not required to use arm64-v8a as the target architecture for the remainder of the tutorial. If no target architecture is supplied both arm64-v8a and armeabi-v7a are targeted.
Once the ANDROID_NDK_ROOT is part of PATH, compile the package for Android CPU target:
cd $SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage make cpu_android PLATFORM=$UDO_APP_ABI
The expected artifacts after compiling for Android CPU are
Note: ndk-build attempts to build implementation libraries for both CPU and GPU runtime. This may result in compilation failures if OpenCL libraries cannot be found. This dependency can be worked around by removing GPU modules in SoftmaxUdoPackage/jni/Android.mk. Alternatively, GPU can be removed as a core type in Softmax.json during package generation in step 2.
Android GPU Runtime Compilation
Compilation for the Android GPU runtime uses the same Android NDK toolchain as the CPU. The remainder of the tutorial assumes ANDROID_NDK_ROOT has been set and added to PATH. Additionally, the CL_INCLUDE_PATH and CL_LIBRARY_PATH environment variables must be set to compile for Android GPU.
export CL_INCLUDE_PATH=<path_to_cl_include> export CL_LIBRARY_PATH=<path_to_libOpenCL.so>
Note: libOpenCL.so is platform dependent. It is the user's responsibility to obtain libOpenCL.so. libOpenCL.so and Open CL headers do not ship with the SNPE SDK.
With environment set, compile the package for Android GPU target:
cd $SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage make gpu PLATFORM=$UDO_APP_ABI
The expected artifacts after compiling for Android GPU are
Hexagon DSP Runtime Compilation
Compilation for the DSP runtime makes use of the make system. In order to build the DSP implementation libraries, Hexagon-SDK needs to be installed and set up. For details, follow the setup instructions on $HEXAGON_SDK_ROOT/docs/readme.html page, where HEXAGON_SDK_ROOT is the location of your Hexagon-SDK installation.
Note: This SNPE release supports building UDO DSP implementation libraries using Hexagon-SDK 3.5.1/3.5.2.
Make sure that HEXAGON_SDK_ROOT, HEXAGON_TOOLS_ROOT and SDK_SETUP_ENV=Done is set.
export HEXAGON_SDK_ROOT=<path to hexagon sdk installation> export HEXAGON_TOOLS_ROOT=$HEXAGON_SDK_ROOT/tools/HEXAGON_Tools/8.3.07 export SDK_SETUP_ENV=Done
With the environment set up, compile for DSP with the following:
cd $SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage make dsp PLATFORM=$UDO_ABI
The expected artifacts after compiling for Hexagon DSP are
Note: For DSP, PLATFORM will only determine the ABI of the registration library.
The SNPE SDK provides an option to automatically perform steps of dlc conversion, package generation, package implementation, and package compilation for UDO as outlined in steps 1-4 above. The option is an extension of the Inception V3 setup script. To enable Inception-V3 setup for UDO, run the script with the --udo or -u option.
usage: $SNPE_ROOT/models/inception_v3/scripts/setup_inceptionv3.py [-h] -a ASSETS_DIR [-d] [-r RUNTIME] [-u]
Prepares the inception_v3 assets for tutorial examples.
required arguments:
-a ASSETS_DIR, --assets_dir ASSETS_DIR
directory containing the inception_v3 assets
optional arguments:
-d, --download Download inception_v3 assets to inception_v3 example
directory
-r RUNTIME, --runtime RUNTIME
Choose a runtime to set up tutorial for. Choices: cpu,
gpu, dsp, aip, all. 'all' option is only supported
with --udo flag
-u, --udo Generate and compile a user-defined operation package
to be used with inception_v3. Softmax is simulated as
a UDO for this script.The --udo extension is compatible with options normally used by the setup_inceptionv3.py script. When the --udo option is enabled, the -r or --runtime option controls the runtime for the package implementation and compilation. Additionally, the --udo option supports use of an 'all' runtime option to create and compile the SoftmaxUdo Package for the CPU, GPU, and DSP/AIP runtimes. Selecting the 'aip' or 'dsp' runtime options additionally compiles x86 libraries in order to quantize the model. Selecting the 'cpu' runtime option compiles for both x86 and Android targets. Compilation for Android target will be skipped if ANDROID_NDK_ROOT is not set. If no runtime option is provided the package is compiled for the CPU runtime.
The command to use the setup script for UDO is:
python3 $SNPE_ROOT/models/inception_v3/scripts/setup_inceptionv3.py -a ~/tmpdir -d -u -r <runtime_of_choice>
For instance, to create and compile a package containing the libraries for all runtimes run:
python3 $SNPE_ROOT/models/inception_v3/scripts/setup_inceptionv3.py -a ~/tmpdir -d -u -r all
This will populate the artifacts in Step 4.
Note: Setup script compiles for arm64-v8a platform architecture. To compile for different target set PLATFORM when using make. See step 4 for more.
Execution using snpe-net-run
Executing Inception-V3 for UDO is largely the same as use of snpe-net-run without UDO.
The SNPE SDK provides Linux and Android binaries of snpe-net-run under
For UDO, snpe-net-run consumes the registration library through the --udo_package_path option. LD_LIBRARY_PATH must also be updated to include the runtime-specific artifacts generated from package compilation.
x86 Host Execution
To execute the network on x86 host, run:
cd $SNPE_ROOT/models/inception_v3 export LD_LIBRARY_PATH = $LD_LIBRARY_PATH:$SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage/libs/x86/ snpe-net-run --container dlc/inception_v3_udo.dlc --input_list data/cropped/raw_list.txt --udo_package_path SoftmaxUdoPackage/libs/x86/libUdoSoftmaxUdoPackageReg.so
Android Target Execution
The tutorial for execution on Android targets will use the arm64-v8a architecture. This portion of the tutorial is generic to all runtimes (CPU, GPU, DSP).
# architecture: arm64-v8a - compiler: clang - STL: libc++ export SNPE_TARGET_ARCH=aarch64-android-clang6.0 export SNPE_TARGET_STL=libc++_shared.so
Then, push SNPE binaries and libraries to the target device:
adb shell "mkdir -p /data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/bin"
adb shell "mkdir -p /data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/lib"
adb push $SNPE_ROOT/lib/$SNPE_TARGET_ARCH/$SNPE_TARGET_STL \
/data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/lib
adb push $SNPE_ROOT/lib/$SNPE_TARGET_ARCH/*.so \
/data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/lib
adb push $SNPE_ROOT/bin/$SNPE_TARGET_ARCH/snpe-net-run \
/data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/bin
Next, update environment variables on the target device to include the SNPE libraries and binaries:
adb shell export SNPE_TARGET_ARCH=aarch64-android-clang6.0 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/lib export PATH=$PATH:/data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/bin
Lastly, push the Inception-V3 UDO model and input data to the device:
cd $SNPE_ROOT/models/inception_v3 mkdir data/rawfiles && cp data/cropped/*.raw data/rawfiles/ adb shell "mkdir -p /data/local/tmp/inception_v3_udo" adb push data/rawfiles /data/local/tmp/inception_v3_udo/cropped adb push data/target_raw_list.txt /data/local/tmp/inception_v3_udo adb push dlc/inception_v3_udo.dlc /data/local/tmp/inception_v3_udo rm -rf data/rawfiles
Android CPU Execution
Once the model and data have been placed on the device, place the UDO libraries on the device:
cd $SNPE_ROOT/models/inception_v3 adb shell "mkdir -p /data/local/tmp/inception_v3_udo/cpu" adb push SoftmaxUdoPackage/libs/arm64-v8a/libUdoSoftmaxUdoPackageImplCpu.so /data/local/tmp/inception_v3_udo/cpu adb push SoftmaxUdoPackage/libs/arm64-v8a/libUdoSoftmaxUdoPackageReg.so /data/local/tmp/inception_v3_udo/cpu adb push SoftmaxUdoPackage/libs/arm64-v8a/libc++_shared.so /data/local/tmp/inception_v3_udo/cpu
Now set required environment variables and run snpe-net-run on device:
adb shell cd /data/local/tmp/inception_v3_udo/ export LD_LIBRARY_PATH=/data/local/tmp/inception_v3_udo/cpu/:$LD_LIBRARY_PATH snpe-net-run --container inception_v3_udo.dlc --input_list target_raw_list.txt --udo_package_path cpu/libUdoSoftmaxUdoPackageReg.so
Android GPU Execution
The procedure for execution using the GPU runtime is largely similar to execution procedure for CPU. First, use the following to place the UDO libraries on device:
cd $SNPE_ROOT/models/inception_v3 adb shell "mkdir -p /data/local/tmp/inception_v3_udo/gpu" adb push SoftmaxUdoPackage/libs/arm64-v8a/libUdoSoftmaxUdoPackageImplGpu.so /data/local/tmp/inception_v3_udo/gpu adb push SoftmaxUdoPackage/libs/arm64-v8a/libUdoSoftmaxUdoPackageReg.so /data/local/tmp/inception_v3_udo/gpu adb push SoftmaxUdoPackage/libs/arm64-v8a/libc++_shared.so /data/local/tmp/inception_v3_udo/gpu adb push SoftmaxUdoPackage/libs/arm64-v8a/libOpenCL.so /data/local/tmp/inception_v3_udo/gpu
Now set required environment variables and run snpe-net-run to device:
adb shell cd /data/local/tmp/inception_v3_udo/ export LD_LIBRARY_PATH=data/local/tmp/inception_v3_udo/gpu/:$LD_LIBRARY_PATH snpe-net-run --container inception_v3_udo.dlc --input_list target_raw_list.txt --udo_package_path gpu/libUdoSoftmaxUdoPackageReg.so --use_gpu
Hexagon DSP Execution
The procedure for execution on device for DSP is largely the same as CPU and GPU. However, the DSP runtime requires quantized network parameters. While DSP allows unquantized DLCs, it is generally recommended to quantize DLCs for improved performance. The tutorial will use a quantized DLC as an illustrative example. Quantizing the DLC requires the snpe-dlc-quantize tool.
To quantize the DLC for use on DSP:
cd $SNPE_ROOT/models/inception_v3/ snpe-dlc-quantize --input_dlc dlc/inception_v3_udo.dlc --input_list data/cropped/raw_list.txt --udo_package_path SoftmaxUdoPackage/libs/x86-64_linux_clang/libUdoSoftmaxUdoPackageReg.so --output_dlc dlc/inception_v3_udo_quantized.dlc
For more information on snpe-dlc-quantize visit quantization. For information on UDO-specific quantization visit Quantizing a DLC with UDO. For information on DSP/AIP runtime visit DSP Runtime or AIP Runtime.
Now push the quantized model to device:
adb push dlc/inception_v3_udo_quantized.dlc /data/local/tmp/inception_v3_udo
Note: Please refer to UDO DSP for Quantized DLC tutorial for executing on the DSP runtime using quantized dlc.
Before executing on the DSP, push the SNPE libraries for DSP to device:
adb shell "mkdir -p /data/local/tmp/snpeexample/dsp/lib"
adb push $SNPE_ROOT/lib/dsp/*.so \
/data/local/tmp/snpeexample/dsp/lib
Now push DSP-specific UDO libraries to device:
cd $SNPE_ROOT/models/inception_v3 adb shell "mkdir -p /data/local/tmp/inception_v3_udo/dsp" adb push SoftmaxUdoPackage/libs/dsp/*.so /data/local/tmp/inception_v3_udo/dsp adb push SoftmaxUdoPackage/libs/arm64-v8a/libUdoSoftmaxUdoPackageReg.so /data/local/tmp/inception_v3_udo/dsp # Pushes reg lib
Then set required environment variables and run snpe-net-run on device:
adb shell cd /data/local/tmp/inception_v3_udo/ export LD_LIBRARY_PATH=/data/local/tmp/inception_v3_udo/dsp/:$LD_LIBRARY_PATH export ADSP_LIBRARY_PATH="/data/local/tmp/inception_v3_udo/dsp/;/data/local/tmp/snpeexample/dsp/lib;/system/lib/rfsa/adsp;/system/vendor/lib/rfsa/adsp;/dsp" snpe-net-run --container inception_v3_udo_quantized.dlc --input_list target_raw_list.txt --udo_package_path dsp/libUdoSoftmaxUdoPackageReg.so --use_dsp
AIP Execution
Because UDOs are not supported on the HTA hardware, executing on the AIP runtime defaults to the DSP UDO implementations. HTA hardware runs exclusively on quantized models and therefore as with the DSP runtime, a quantized model will be used. The command to quantize the DLC for AIP is:
cd $SNPE_ROOT/models/inception_v3/ snpe-dlc-quantize --input_dlc dlc/inception_v3_udo.dlc --input_list data/cropped/raw_list.txt --udo_package_path SoftmaxUdoPackage/libs/x86-64_linux_clang/libUdoSoftmaxUdoPackageReg.so --output_dlc dlc/inception_v3_udo_quantized.dlc --enable_hta
Now push the quantized model to device:
adb push dlc/inception_v3_udo_quantized.dlc /data/local/tmp/inception_v3_udo
Before executing using the AIP runtime, push the SNPE libraries for DSP to device with these commands:
adb shell "mkdir -p /data/local/tmp/snpeexample/dsp/lib"
adb push $SNPE_ROOT/lib/dsp/*.so \
/data/local/tmp/snpeexample/dsp/lib
Now push DSP-specific UDO libraries to device:
cd $SNPE_ROOT/models/inception_v3 adb shell "mkdir -p /data/local/tmp/inception_v3_udo/dsp" adb push SoftmaxUdoPackage/libs/dsp/*.so /data/local/tmp/inception_v3_udo/dsp adb push SoftmaxUdoPackage/libs/arm64-v8a/libUdoSoftmaxUdoPackageReg.so /data/local/tmp/inception_v3_udo/dsp # Pushes reg lib
Then set required environment variables and run snpe-net-run on device:
adb shell cd /data/local/tmp/inception_v3_udo/ export LD_LIBRARY_PATH=/data/local/tmp/inception_v3_udo/dsp/:$LD_LIBRARY_PATH export ADSP_LIBRARY_PATH="/data/local/tmp/inception_v3_udo/dsp/;/data/local/tmp/snpeexample/dsp/lib;/system/lib/rfsa/adsp;/system/vendor/lib/rfsa/adsp;/dsp" snpe-net-run --container inception_v3_udo_quantized.dlc --input_list target_raw_list.txt --udo_package_path dsp/libUdoSoftmaxUdoPackageReg.so --use_aip
Integration with Android APK
This portion of the tutorial outlines how to integrate SNPE UDO libraries and Java API for package registration into an example application. Generally, for native shared libraries to be discoverable by the application they must be placed in the project under
<project>/app/src/main/jniLibs/<platform_abi>
Once the libraries are accessible by the application, the registration library can be registered using the provided Java API. This process will be replicated with the example Image Classifiers application. The following assumes that the rest of the example application setup has been followed. The tutorial will issue instructions for platforms with arm64-v8a ABI.
First, create the neccessary directories to contain the UDO libraries. The following steps will populate all runtime implementation libraries.
mkdir app/src/main/jniLibs/ cp -a $SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage/libs/arm64-v8a/ app/src/main/jniLibs/
If DSP is to be used as the runtime, copy the implementation library with the following:
cp $SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage/libs/dsp/*.so app/src/main/jniLibs/arm64-v8a/
If not already done, running setup_inceptionv3.sh will add the Inception-V3 model enabled with UDO to the project.
bash ./setup_inceptionv3.sh
Now the Java API can be registered. Edit the file $SNPE_ROOT/examples/android/image-classifiers/app/src/main /java/com/qualcomm/qti/snpe/imageclassifiers/tasks/LoadNetworkTask.java
To contain this line
@Override
protected NeuralNetwork doInBackground(File... params) {
NeuralNetwork network = null;
try {
SNPE.addOpPackage(mApplication,"libUdoSoftmaxUdoPackageReg.so"); // Add this line to register package
final SNPE.NeuralNetworkBuilder builder = new SNPE.NeuralNetworkBuilder(mApplication)
...
Now the APK can be built and exercised
./gradlew assembleDebug
80-NL315-14 A
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION