Defining a UDO

A User-Defined Operation (UDO) allows users to integrate their custom operations with SNPE to enable execution on any supported hardware accelerator. The UDO mechanism accepts a specification of a custom operation (defined below), and processes that information to process a model containing that custom operation. This section explains how such a UDO can be specified. See Overview of UDO for more details about UDO and Preparing a model with UDO for details on how to convert a model that contains a UDO into a SNPE DLC for supported frameworks.

The UDO Configuration Specification

As described in the Overview of UDO section, a user can express the attributes of their custom operation with a configuration specification file. This UDO configuration (henceforth known as UDO config) is a description of the operation created using the Javascript Object Notation (JSON) syntax and formatting. The configuration file syntax defines fields that can be specified in key-value pairs, and arranged in a tree-like structure in accordance with JSON rules. The fields are pre-determined and will ultimately be parsed into the required information that constitutes a UDO. The information provided should be generic and independent of a particular model, meaning model-specific parameters or names need not be part of the configuration. The information will be used to identify the op within a framework model, and then ultimately serialized into the DLC model. This implies that any changes in the config would require re-generation of the DLC model to ensure the correct information is serialized.

UDO Config Field Description

The details of the aforementioned UDO config file can be described below.

{
    "UdoPackage_0":
    {
        "Operators": [
            {
                "type": "",
                "inputs":[
                    {"name":"", "data_type": "FLOAT_32", "static": true,
                        "tensor_layout": "NHWC"},
                ],
                "outputs":[
                    {"name":"","data_type": "FLOAT_32"}
                ],
                "scalar_params": [
                    {"name":"scalar_param_1", "data_type": "INT_32"}
                ],
                "tensor_params": [
                    {"name":"tensor_param_1", "data_type": "FLOAT_32", "tensor_layout": "NHWC"},
                ],
                "core_types": ["CPU", "GPU", "DSP"]
            }
        ],
        "UDO_PACKAGE_NAME": "MyCustomUdoPackage",
    }
}

The above description is simply a generic configuration file to aid the definition of the fields that the user can fill. Required fields are provided with a specific value, while optional fields are denoted with empty strings. Note that an optional field only implies that there is either a default value if it is not provided, or that an empty string will be used. The full details of each available field is described hierarchically below:

• UdoPackage: Every UDO package can be described as "UdoPackage_i" where i indicates the order in which the packages will be generated. The user is also free to use empty strings but the dictionary structure is necessary.¹
• Operators: This is a child node of a particular UdoPackage indicating the number of operators present. ⁶
  • type: defines the type of the operation.
  • inputs: a list of input tensors to the operation. Each input is a dictionary object. ²
    • name: An optional field that describes the name of the input tensor. Since the name of the input tensor is variable, the user is not required to provide this.
    • data_type: A string field that describes the data-type of this input tensor. Only one data-type can be defined per tensor. The supported data-types are:⁷
      • FLOAT_16
      • FLOAT_32
      • FIXED_4
      • FIXED_8
      • FIXED_16
      • UINT_8
      • UINT_16
      • UINT_32
      • STRING
    • static: A boolean field that is required if the input data is static i.e data is provided in the model. This field needs to be set if the input tensor will contain data, otherwise the input will be treated dynamically, and the data will not be serialized.
    • tensor_layout: A string field that describes the canonical dimension format of the input tensor. The supported values are: ⁵
      • NCHW
      • NHWC
  • outputs: A list of output tensors to the operation.²
  • scalar_params: A list of scalar-valued attributes.¹
    • name: A required field that describes the name of the scalar parameter.
    • data_type: A required field that describes the data-type supported by this scalar parameter.
  • tensor_params: A list of tensor-valued attributes.^{2 3}
  • core_types: The intended IP cores for this particular operation. The supported core_types:
    • CPU
    • GPU
    • DSP
• UDO_PACKAGE_NAME: The name of the UDO Package, which can be any valid string.¹

Creating a UDO config

The user should aim to fill out the fields described in the config above to adequately describe a UDO. In some cases, the information required in this config could be easily obtained from framework documentation about the operation. However, there may be subtle caveats, therefore the user is encouraged to ensure that inputs, outputs and params are properly categorized and described. A potential caveat is that inputs can be mis-categorized as parameters and vice versa, if the config is written only according to documentation. In this scenario, a useful tip is to visualize the model using an open source tool such as Netron (found here: https://github.com/lutzroeder/netron) to assist with crafting the UDO config correctly.

Once an adequately descriptive config has been created, it can be used as an argument to the framework converters as described in Preparing a model with UDO.

Notes:

1. More than one udo package can be defined in a single config file. Users should note that the package name specified here must match the package name used in creating the corresponding package.
2. Each input, output and tensor parameter is categorized as the same kind of tensor object, meaning that all the fields are shared. The names of inputs and outputs are not required since the config is a generic description of an operation.
3. In the case of the parameters, the name field is always required.
4. The tensor layout is a convention to indicate the arrangement of data within the tensor. Therefore a tensor layout of NHWC means that the data is organized in (batch x height x width x channel), where channel is the fastest changing dimension. Note this is the default arrangement for SNPE, and that may have implications on a model containing a UDO if other tensor layouts are selected. Notably, if a tensor layout of NCHW is selected, then the data and/or tensor parameters may need to be reshaped to the SNPE default to maintain dimensional understanding. Should the user encounter this scenario, they may notice the introduction of intermediary permute layers prior to the UDO layer which will ultimately feed the tensors in question. These caveats should be visible as either converter warnings, debug messages or through outputs of the visualization tools described in Tools. For more details on tensor layouts, the user can consult the section:Input Image Formatting of the documentation.
5. DSP runtime can handle multiple operations in one UDO package, however the provided skeleton code is tailored to define only one operation in one package. For CPU and GPU coretypes, there can be an arbitrary number of operators defined per UDO package.
6. The data-type of a tensor determines both how the data contained in the tensor will be stored in the DLC, and the type of memory handed over by SNPE during runtime execution. While tensors get stored within the DLC in the exact data-type specified by the UDO definition, there may be runtime restrictions on the type of memory users can expect to receive depending on the chosen core-type. Users should visit the following section: Compiling a UDO package for more details.