The initial PR for NNAPI EP

[guoyu-wang]

Description: The initial PR for NNAPI EP
Supported Operators,
Add/Sub/Mul/Div/Relu/Transpose/Reshape/BatchNormalization/GlobalAveragePool/GlobalMaxPool/ AveragePool/MaxPool/Conv/Cast/Softmax/Identity/Abs/Exp/Floor/Log/Sigmoid/Neg/Sin/Sqrt/Tanh
Gemm/MatMul (A*B’+C only)

Motivation and Context
The current NNAPI EP is using external DNNLibrary, which does not support many scenarios, such as dynamic input shape, require Android API level 27 (requires libneuralnetworks.so which is not available on Android API 26-), has to manually convert NCHW->NHWC and back at input/output, difficult to maintain in case bug fixes or new features are needed, extra binary size increase (~800k for x86-64), …
The current NNAPI EP is still working using –use_dnnlibrary option, as we will eventually remove it.

There are 3 major components in the implementation

1. ModelBuilder, this will generate a compiled NNAPI model by converting the given ONNX model, it will also process initializers, register inputs/outputs, and call OpBuilder to convert operators.
2. OpBuilder(s), this interface will convert operators to NNAPI operation, as well as checking if the operators are supported, processing initializers and add intermediate steps if necessary
3. Model, this will be the final component for execution after the 1. ModelBuilder generates the compiled NNAPI model, will set the input/output buffers to the NNAPI model for execution

Not included in this PR

1. Android CI changes to validate the new NNAPI EP
   Will send a separated PR for this
2. For the NNAPI model builder, switch the ONNX input from model to graph since we do not need the model after the NNAPI model is generated. As using graph may enable better graph partitioning, and easier properties checking (such as initializers and inputs). Currently we are using similar logic as TRT EP.
3. NHWC support, as for now, we only support NCHW which is the data format for ONNX, however, NNAPI works more efficiently using NHWC
4. Certain unit test cases for the operators, such as Conv/BatchNormalization with initializers, (the current tests do not use
5. Dynamic output shape support, this can be supported on Android API level 29
6. Documentation, will add when most the issues are addressed and when we are ready to remove the use_dnnlibrary option

7. We currently do not handle concurrent calls to the same model object, need add ability to lock the model until the predict is run.
   

8. Check the input/initializer of each individual operator while calling GetCapability()
   

[skottmckay]

ASSERT_TRUE(status.IsOK());

Would be good to check where each node was partitioned to to validate things were assigned to the NNAPI EP as expected. #Pending

---

Refers to: onnxruntime/test/providers/nnapi/nnapi_basic_test.cc:99 in a138cf2. [](commit_id = a138cf2, deletion_comment = False)

[guoyu-wang]

ASSERT_TRUE(status.IsOK());

Would need to think about how to verify this

---

In reply to: 647959128 [](ancestors = 647959128)

---

Refers to: onnxruntime/test/providers/nnapi/nnapi_basic_test.cc:99 in a138cf2. [](commit_id = a138cf2, deletion_comment = False)