[QNN EP] Add session option to disable fallback to default CPU EP

[adrianlizarraga]

Description

Adds the session config option disable_cpu_ep_fallback to allow the user to prevent the CPU EP from handling
nodes not supported by other execution providers.

// Graph nodes that are not supported by the execution providers (EPs) explicitly added to the session are
// assigned (i.e., "fallback") to the CPU EP by default.
//
// This option allows the user to disable the fallback of unsupported graph nodes to the CPU EP.
// If this option is set to "1", session creation will fail if the execution providers other than the CPU EP cannot
// fully support all of the nodes in the graph.
//
// It is invalid to set this option and explicitly add the CPU EP to the session. In this case, session creation
// will also fail with an error.
//
// Option values:
// - "0": CPU EP fallback is not disabled. [DEFAULT]
// - "1": CPU EP fallback is disabled.
static const char* const kOrtSessionOptionsDisableCPUEPFallback = "session.disable_cpu_ep_fallback";

Example use

#include "core/session/onnxruntime_cxx_api.h"
#include "core/session/onnxruntime_session_options_config_keys.h"

int main(int argc, char** argv) {
    Ort::SessionOptions so;
    so.AddConfigEntry(kOrtSessionOptionsDisableCPUEPFallback, "1");  // Disable fallback to the CPU EP.

    onnxruntime::ProviderOptions options;
#if defined(_WIN32)
    options["backend_path"] = "QnnCpu.dll";
#else
    options["backend_path"] = "libQnnCpu.so";
#endif

    so.AppendExecutionProvider("QNN", options);

    const ORTCHAR_T* ort_model_path = ORT_MODEL_FOLDER "qnn_ep_partial_support.onnx";
    Ort::Session session(*ort_env, ort_model_path, so);  // Throws exception if nodes fallback to CPU
    // ...

Motivation and Context

Makes it easier for application developers to ensure that the entire model runs on specific EPs. This is critical for Qualcomm/scenarios. If the compute cannot be offloaded to the NPU, running on CPU is not acceptable. (could be the difference between 90 second inference and 6 seconds inference)

[pranavsharma]

This almost looks like a violation of the contract promised by the ORT framework of falling back to the less preferred EPs. It should be the framework that decides whether to fall back or not (based on user config) and fail the session creation accordingly. GetCapability should only report what nodes it can run. Looks like we could have a session level framework config, something like 'disable_ep_fallback' which is false by default. The partitioner can then choose to consult other EPs in the preferred list.

[adrianlizarraga]

This almost looks like a violation of the contract promised by the ORT framework of falling back to the less preferred EPs. It should be the framework that decides whether to fall back or not (based on user config) and fail the session creation accordingly. GetCapability should only report what nodes it can run. Looks like we could have a session level framework config, something like 'disable_ep_fallback' which is false by default. The partitioner can then choose to consult other EPs in the preferred list.

Thanks Pranav. I'll look into an alternative solution with a session config option, perhaps using a check similar to what CUDA is doing here:

onnxruntime/onnxruntime/core/session/inference_session.cc

Line 1536 in ea7b2de

} else if (!AreAllNodesInMainGraphAssignedToOneEp(graph, onnxruntime::kCudaExecutionProvider)) {

[jywu-msft]

This almost looks like a violation of the contract promised by the ORT framework of falling back to the less preferred EPs. It should be the framework that decides whether to fall back or not (based on user config) and fail the session creation accordingly. GetCapability should only report what nodes it can run. Looks like we could have a session level framework config, something like 'disable_ep_fallback' which is false by default. The partitioner can then choose to consult other EPs in the preferred list.

yes, seems like there should be a framework level option to disable default CPU fallback.
the current contract is we defer partition/priority decisions to the user. it is reasonable for them to request we disable that behavior.
This is critical for qualcomm/scnearios. if the compute cannot be offloaded to the NPU, running on CPU is not acceptable. (could be the difference between 90 second inference and 6 seconds inference)

[pranavsharma]

Looks good. Some minor changes.

[HectorSVC]