Snapdragon Neural Processing Engine SDK
Reference Guide
UDO DSP tutorial for Quantized DLC

Overview

This tutorial describes the steps needed to create a UDO package for DSP runtime and execute the Inception-V3 model using the package. The Softmax operation has been chosen in this tutorial to demonstrate the implementation of a UDO with SNPE.

The SNPE SDK provides the resources for this example under

  • $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax

Information on UDO in general is available at UDO Overview.

Information on running the Inception-V3 network without UDO is available at Inception-V3 Tutorial.

Information on running the Inception-V3 network for CPU and GPU runtime is available at Inception-V3 UDO Tutorial.

Prerequisites

The following tutorial assumes that general SNPE setup has been followed to support SDK environment, TensorFlow environment, and desired platform dependencies. The steps listed in this tutorial use the Tensorflow model in the form of inception_v3_2016_08_28_frozen.pb. For details on acquiring the Inception-V3 model visit Tutorials Setup.

Introduction

Here are the steps to develop and run a UDO

1.) Package Generation

2.) Framework Model Conversion to a DLC

3.) Package Implementation

4.) Package Compilation

5.) Model Execution

Step 1: Package Generation

Generating the SoftmaxUdo package requires the snpe-udo-package-generator tool and the provided UDO plugin: Softmax_Quant.json. The plugin is located under $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/config. More information about creating a UDO plugin can be found here.

Generate the SoftmaxUdo Package using the following: 

export SNPE_UDO_ROOT=$SNPE_ROOT/share/SnpeUdo
mkdir $SNPE_ROOT/models/inception_v3/udo_dsp
snpe-udo-package-generator -p $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/config/Softmax_Quant.json -o $SNPE_ROOT/models/inception_v3/udo_dsp/

This command creates the Softmax based package at $SNPE_ROOT/models/inception_v3/udo_dsp/SoftmaxUdoPackage.

For more information on the snpe-udo-package-generator tool visit here.

Step 2: Framework model Conversion to a DLC

Information for converting a model to a DLC is available at Inception V3 UDO Model Conversion. Here we need to use the Softmax.json provided at the location $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/config/Softmax.json

This will generate a DLC named inception_v3_udo.dlc containing the Softmax as UDO at $SNPE_ROOT/models/inception_v3/dlc.

Step 3: Package Implementations

The generated package creates the skeleton of the operation implementation, which must be filled by the user to create a functional UDO. Additionally, the skeleton for a user implemented validation function can be populated to validate information about the UDO passed from the SNPE runtime. The rest of the code scaffolding for compatibility with SNPE is provided by the snpe-udo-package-generator.

The UDO implementations and validation function for this tutorial are provided under $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/src.

DSP Implementations

A registration library and an implementation library are required to run inference on a network with UDO layers on SNPE DSP. The registration library will run on CPU, and specifies the DSP implementation library of the UDO.

Please note that only C files are supported for UDO on DSP runtime.

The file in the package that need to be implemented for DSP are

  • SoftmaxUdoPackage/jni/src/DSP/SoftmaxImplLibDsp.c
  • SoftmaxUdoPackage/include/SoftmaxImplLibDsp.h

The provided example implementation is at the location

  • $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/src/DSP/SoftmaxInt8Impl/SoftmaxImplLibDsp.c
  • $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/src/DSP/SoftmaxInt8Impl/SoftmaxImplLibDsp.h

Copy the provided implementations to the package: 

cp -f $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/src/DSP/SoftmaxInt8Impl/SoftmaxImplLibDsp.c $SNPE_ROOT/models/inception_v3/udo_dsp/SoftmaxUdoPackage/jni/src/DSP/
cp -f $SNPE_ROOT/examples/NativeCpp/UdoExample/Softmax/src/DSP/SoftmaxInt8Impl/SoftmaxImplLibDsp.h $SNPE_ROOT/models/inception_v3/udo_dsp/SoftmaxUdoPackage/include/

Optionally, the user can provide their own implementations in the package.

Step 4: Package Compilation

Hexagon DSP Runtime Compilation

Compilation for the DSP runtime makes use of the make system. In order to build the DSP implementation libraries, Hexagon-SDK needs to be installed and set up. For details, follow the setup instructions on $HEXAGON_SDK_ROOT/docs/readme.html page, where HEXAGON_SDK_ROOT is the location of your Hexagon-SDK installation.

Note: This SNPE release supports building UDO DSP implementation libraries using Hexagon-SDK 3.5.1/3.5.2.

Make sure that HEXAGON_SDK_ROOT, HEXAGON_TOOLS_ROOT and SDK_SETUP_ENV=Done is set. 

export HEXAGON_SDK_ROOT=<path to hexagon sdk installation>
export HEXAGON_TOOLS_ROOT=$HEXAGON_SDK_ROOT/tools/HEXAGON_Tools/8.3.07
export SDK_SETUP_ENV=Done

The ANDROID_NDK_ROOT environment variable must be set to the directory containing ndk-build in order to compile the package. 

export ANDROID_NDK_ROOT=<path_to_android_ndk>

It is suggested to add ANDROID_NDK_ROOT to the PATH environment variable to access ndk-build. 

export PATH=$ANDROID_NDK_ROOT:$PATH

Target architecture must also be specified when compiling the package. 

export UDO_APP_ABI=<target_architecture>

This tutorial uses arm64-v8a architectures - it is recommended but not required to use arm64-v8a as the target architecture for the remainder of the tutorial. If no target architecture is supplied both arm64-v8a and armeabi-v7a are targeted.

With the environment set up, compile for DSP with the following: 

cd $SNPE_ROOT/models/inception_v3/udo_dsp/SoftmaxUdoPackage
make dsp PLATFORM=$UDO_APP_ABI

The expected artifacts after compiling for Hexagon DSP are

  • SoftmaxUdoPackage/libs/dsp/libUdoSoftmaxUdoPackageImplDsp.so
  • SoftmaxUdoPackage/libs/$UDO_APP_ABI/libUdoSoftmaxUdoPackageReg.so

Note: For DSP, PLATFORM will only determine the ABI of the registration library.

Model Execution

Execution using snpe-net-run

Executing Inception-V3 for UDO is largely the same as use of snpe-net-run without UDO.

The SNPE SDK provides Linux and Android binaries of snpe-net-run under

  • $SNPE_ROOT/bin/x86_64-linux-clang
  • $SNPE_ROOT/bin/arm-android-clang6.0
  • $SNPE_ROOT/bin/aarch64-android-clang6.0
  • $SNPE_ROOT/bin/aarch64-linux-gcc4.9
  • $SNPE_ROOT/bin/arm-linux-gcc4.9sf
  • $SNPE_ROOT/bin/aarch64-oe-linux-gcc6.4
  • $SNPE_ROOT/bin/arm-oe-linux-gcc6.4hf

For UDO, snpe-net-run consumes the registration library through the --udo_package_path option. LD_LIBRARY_PATH must also be updated to include the runtime-specific artifacts generated from package compilation.

Android Target Execution

The tutorial for execution on Android targets will use the arm64-v8a architecture.

# architecture: arm64-v8a - compiler: clang - STL: libc++
export SNPE_TARGET_ARCH=aarch64-android-clang6.0
export SNPE_TARGET_STL=libc++_shared.so

Then, push SNPE binaries and libraries to the target device: 

adb shell "mkdir -p /data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/bin"
adb shell "mkdir -p /data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/lib"

adb push $SNPE_ROOT/lib/$SNPE_TARGET_ARCH/$SNPE_TARGET_STL \
      /data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/lib
adb push $SNPE_ROOT/lib/$SNPE_TARGET_ARCH/*.so \
      /data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/lib
adb push $SNPE_ROOT/bin/$SNPE_TARGET_ARCH/snpe-net-run \
      /data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/bin

Next, update environment variables on the target device to include the SNPE libraries and binaries: 

adb shell
export SNPE_TARGET_ARCH=aarch64-android-clang6.0
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/lib
export PATH=$PATH:/data/local/tmp/snpeexample/$SNPE_TARGET_ARCH/bin

Lastly, push the Inception-V3 UDO model and input data to the device: 

cd $SNPE_ROOT/models/inception_v3
mkdir data/rawfiles && cp data/cropped/*.raw data/rawfiles/
adb shell "mkdir -p /data/local/tmp/inception_v3_udo"
adb push data/rawfiles /data/local/tmp/inception_v3_udo/cropped
adb push data/target_raw_list.txt /data/local/tmp/inception_v3_udo
adb push dlc/inception_v3_udo.dlc /data/local/tmp/inception_v3_udo
rm -rf data/rawfiles

Hexagon DSP Execution

The procedure for execution on device for DSP is largely the same as CPU and GPU. However, the DSP runtime requires quantized network parameters. While DSP allows unquantized DLCs, it is generally recommended to quantize DLCs for improved performance. The tutorial will use a quantized DLC as an illustrative example. Quantizing the DLC requires the snpe-dlc-quantize tool.

Note: In the below command one should use input dlc generated at Model DLC Conversion. Also provide the path of the registration lib generated after compiling x86 Host under the argument "udo_package_path".More information about compiling x86 can be found here.

To quantize the DLC for use on DSP: 

cd $SNPE_ROOT/models/inception_v3/
snpe-dlc-quantize --input_dlc dlc/inception_v3_udo.dlc --input_list data/cropped/raw_list.txt --udo_package_path SoftmaxUdoPackage/libs/x86-64_linux_clang/libUdoSoftmaxUdoPackageReg.so --output_dlc dlc/inception_v3_udo_quantized.dlc

For more information on snpe-dlc-quantize visit quantization. For information on UDO-specific quantization visit Quantizing a DLC with UDO. For information on DSP/AIP runtime visit DSP Runtime or AIP Runtime.

Now push the quantized model to device: 

adb push dlc/inception_v3_udo_quantized.dlc /data/local/tmp/inception_v3_udo

Before executing on the DSP, push the SNPE libraries for DSP to device: 

adb shell "mkdir -p /data/local/tmp/snpeexample/dsp/lib"
adb push $SNPE_ROOT/lib/dsp/*.so \
      /data/local/tmp/snpeexample/dsp/lib

Now push DSP-specific UDO libraries to device: 

cd $SNPE_ROOT/models/inception_v3/udo_dsp
adb shell "mkdir -p /data/local/tmp/inception_v3_udo/dsp"
adb push SoftmaxUdoPackage/libs/dsp/*.so /data/local/tmp/inception_v3_udo/dsp
adb push SoftmaxUdoPackage/libs/arm64-v8a/libUdoSoftmaxUdoPackageReg.so /data/local/tmp/inception_v3_udo/dsp # Pushes reg lib

Then set required environment variables and run snpe-net-run on device: 

adb shell
cd /data/local/tmp/inception_v3_udo/
export LD_LIBRARY_PATH=/data/local/tmp/inception_v3_udo/dsp/:$LD_LIBRARY_PATH
export ADSP_LIBRARY_PATH="/data/local/tmp/inception_v3_udo/dsp/;/data/local/tmp/snpeexample/dsp/lib;/system/lib/rfsa/adsp;/system/vendor/lib/rfsa/adsp;/dsp"
snpe-net-run --container inception_v3_udo_quantized.dlc --input_list target_raw_list.txt --udo_package_path dsp/libUdoSoftmaxUdoPackageReg.so --use_dsp

AIP Execution

Because UDOs are not supported on the HTA hardware, executing on the AIP runtime defaults to the DSP UDO implementations. HTA hardware runs exclusively on quantized models and therefore as with the DSP runtime, a quantized model will be used.

Note: In the below command one should use input dlc generated at Model DLC Conversion. Also provide the path of the registration lib generated after compiling x86 Host under the argument "udo_package_path".More information about compiling x86 can be found here.

The command to quantize the DLC for AIP is: 

cd $SNPE_ROOT/models/inception_v3/
snpe-dlc-quantize --input_dlc dlc/inception_v3_udo.dlc --input_list data/cropped/raw_list.txt --udo_package_path SoftmaxUdoPackage/libs/x86-64_linux_clang/libUdoSoftmaxUdoPackageReg.so --output_dlc dlc/inception_v3_udo_quantized.dlc --enable_hta

Now push the quantized model to device: 

adb push dlc/inception_v3_udo_quantized.dlc /data/local/tmp/inception_v3_udo

Before executing using the AIP runtime, push the SNPE libraries for DSP to device with these commands: 

adb shell "mkdir -p /data/local/tmp/snpeexample/dsp/lib"
adb push $SNPE_ROOT/lib/dsp/*.so \
      /data/local/tmp/snpeexample/dsp/lib

Now push DSP-specific UDO libraries to device: 

cd $SNPE_ROOT/models/inception_v3/udo_dsp
adb shell "mkdir -p /data/local/tmp/inception_v3_udo/dsp"
adb push SoftmaxUdoPackage/libs/dsp/*.so /data/local/tmp/inception_v3_udo/dsp
adb push SoftmaxUdoPackage/libs/arm64-v8a/libUdoSoftmaxUdoPackageReg.so /data/local/tmp/inception_v3_udo/dsp # Pushes reg lib

Then set required environment variables and run snpe-net-run on device: 

adb shell
cd /data/local/tmp/inception_v3_udo/
export LD_LIBRARY_PATH=/data/local/tmp/inception_v3_udo/dsp/:$LD_LIBRARY_PATH
export ADSP_LIBRARY_PATH="/data/local/tmp/inception_v3_udo/dsp/;/data/local/tmp/snpeexample/dsp/lib;/system/lib/rfsa/adsp;/system/vendor/lib/rfsa/adsp;/dsp"
snpe-net-run --container inception_v3_udo_quantized.dlc --input_list target_raw_list.txt --udo_package_path dsp/libUdoSoftmaxUdoPackageReg.so --use_aip

Integration with Android APK

This portion of the tutorial outlines how to integrate SNPE UDO libraries and Java API for package registration into an example application. Generally, for native shared libraries to be discoverable by the application they must be placed in the project under 

<project>/app/src/main/jniLibs/<platform_abi>

Once the libraries are accessible by the application, the registration library can be registered using the provided Java API. This process will be replicated with the example Image Classifiers application. The following assumes that the rest of the example application setup has been followed. The tutorial will issue instructions for platforms with arm64-v8a ABI.

First, create the neccessary directories to contain the UDO libraries. The following steps will populate all runtime implementation libraries. 

mkdir app/src/main/jniLibs/
cp -a $SNPE_ROOT/models/inception_v3/SoftmaxUdoPackage/libs/arm64-v8a/ app/src/main/jniLibs/

If DSP is to be used as the runtime, copy the implementation library with the following: 

cp $SNPE_ROOT/models/inception_v3/udo_dsp/SoftmaxUdoPackage/libs/dsp/*.so app/src/main/jniLibs/arm64-v8a/

If not already done, running setup_inceptionv3.sh will add the Inception-V3 model enabled with UDO to the project. 

bash ./setup_inceptionv3.sh

Now the Java API can be registered. Edit the file $SNPE_ROOT/examples/android/image-classifiers/app/src/main /java/com/qualcomm/qti/snpe/imageclassifiers/tasks/LoadNetworkTask.java

To contain this line 

@Override
    protected NeuralNetwork doInBackground(File... params) {
        NeuralNetwork network = null;
        try {
            SNPE.addOpPackage(mApplication,"libUdoSoftmaxUdoPackageReg.so"); // Add this line to register package
            final SNPE.NeuralNetworkBuilder builder = new SNPE.NeuralNetworkBuilder(mApplication)
        ...

Now the APK can be built and exercised 

./gradlew assembleDebug