You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
WebKit has a few objections to MLContextOptions.deviceType:
MLContextOptions.deviceType is currently unimplementable currently via CoreML
CoreML is the only public framework on macOS for running computations on the NPU / ANE. The reason as pointed out #623 (comment) and possibly elsewhere, CoreML allows you to specify MLComputeUnitsCPUOnly, MLComputeUnitsCPUAndGPU, MLComputeUnitsAll, and MLComputeUnitsCPUAndNeuralEngine, but not MLComputeUnitsNeuralEngine only for instance. Even assuming we could request CoreML to add support for only the ANE, that support would be limited to a future version of macOS and therefore could only be supported on currently unreleased versions of macOS.
Also note, MLComputeUnitsCPUAndNeuralEngine, is only available on macOS 13.0 and later and I'm not sure if some browser vendors want to support earlier versions of macOS.
MLContextOptions.deviceType, as currently specified, would lead to additional fragmentation, works on some devices, fails on others
It would be preferable to fallback instead of failing, otherwise a website which specifies NPU and runs on a variety of NPUs, but maybe not the NPU of a certain device, the website would not run if the author does not handle fallback. This seems unnecessary, when the browser could fallback to the GPU or CPU themselves.
The browser has better insight into workloads than the website author
It was pointed out in #322 that a website author may want their models to not run on the GPU so they don't impact WebGPU rendering performance. While a good idea in theory, the browser implementation and native frameworks have much greater insight into the graphics chip utilization and the most appropriate resource to run a model on without impacting target frameworks.
The answer to this question, where to run the model, can not be easily decided by the website as the website does not know the underlying hardware it is running on due to privacy reasons and possibly other reasons. Additionally, while browser vendor can trivially compute the amount of time a GPUCommandBuffer takes to submit from WebGPU, such timing is not available to the website. For example, if all the GPUCommandBuffers in the WebGPU workload complete in <3ms on a Mac Studio, and the website calls requestAnimationFrame every ~16ms, then restricting the WebNN model from running on the GPU seems unnecessary. However on a less powerful device, like an iPhone 12 mini, if the WebGPU workload is falling behind the requested update rate, the browser implementation may move the model processing to the device least in use. To the website, a browser implementation may present both devices as having the same capabilities for privacy reasons, so there is no way for the website to make the correct decision which device to run on.
Often with graphics intensive WebGPU applications, we find the GPU is still the least utilized processor as the CPU overhead can be high. Long running shaders or complex compute kernels are two exceptions. In any case this is device dependent and not something a website author can easily predict.
To summarize it seems MLDeviceType is useful for testing in browser implementations, but seems counter-productive for framework authors and general websites / wasm applications.
The text was updated successfully, but these errors were encountered:
WebKit has a few objections to MLContextOptions.deviceType:
MLContextOptions.deviceType is currently unimplementable currently via CoreML
CoreML is the only public framework on macOS for running computations on the NPU / ANE. The reason as pointed out #623 (comment) and possibly elsewhere, CoreML allows you to specify MLComputeUnitsCPUOnly, MLComputeUnitsCPUAndGPU, MLComputeUnitsAll, and MLComputeUnitsCPUAndNeuralEngine, but not MLComputeUnitsNeuralEngine only for instance. Even assuming we could request CoreML to add support for only the ANE, that support would be limited to a future version of macOS and therefore could only be supported on currently unreleased versions of macOS.
Also note, MLComputeUnitsCPUAndNeuralEngine, is only available on macOS 13.0 and later and I'm not sure if some browser vendors want to support earlier versions of macOS.
MLContextOptions.deviceType, as currently specified, would lead to additional fragmentation, works on some devices, fails on others
Under Section 4 we see the wording:
It would be preferable to fallback instead of failing, otherwise a website which specifies NPU and runs on a variety of NPUs, but maybe not the NPU of a certain device, the website would not run if the author does not handle fallback. This seems unnecessary, when the browser could fallback to the GPU or CPU themselves.
The browser has better insight into workloads than the website author
It was pointed out in #322 that a website author may want their models to not run on the GPU so they don't impact WebGPU rendering performance. While a good idea in theory, the browser implementation and native frameworks have much greater insight into the graphics chip utilization and the most appropriate resource to run a model on without impacting target frameworks.
The answer to this question, where to run the model, can not be easily decided by the website as the website does not know the underlying hardware it is running on due to privacy reasons and possibly other reasons. Additionally, while browser vendor can trivially compute the amount of time a GPUCommandBuffer takes to submit from WebGPU, such timing is not available to the website. For example, if all the GPUCommandBuffers in the WebGPU workload complete in <3ms on a Mac Studio, and the website calls requestAnimationFrame every ~16ms, then restricting the WebNN model from running on the GPU seems unnecessary. However on a less powerful device, like an iPhone 12 mini, if the WebGPU workload is falling behind the requested update rate, the browser implementation may move the model processing to the device least in use. To the website, a browser implementation may present both devices as having the same capabilities for privacy reasons, so there is no way for the website to make the correct decision which device to run on.
Often with graphics intensive WebGPU applications, we find the GPU is still the least utilized processor as the CPU overhead can be high. Long running shaders or complex compute kernels are two exceptions. In any case this is device dependent and not something a website author can easily predict.
To summarize it seems MLDeviceType is useful for testing in browser implementations, but seems counter-productive for framework authors and general websites / wasm applications.
The text was updated successfully, but these errors were encountered: