Issues using IoBindings #66
Assignees
Labels
Comments
decahedron1
added
documentation
|
Sorry for the delay. The proper usage of use anyhow;
use ndarray::{ArrayD, CowArray};
use ort::tensor::OrtOwnedTensor;
use ort::{AllocationDevice, AllocatorType, Environment, ExecutionProvider, GraphOptimizationLevel, IoBinding, MemType, MemoryInfo, SessionBuilder, Value};
fn main() -> anyhow::Result<()> {
tracing_subscriber::fmt::init();
let environment = Environment::builder()
.with_name("test")
.with_execution_providers([ExecutionProvider::CUDA(Default::default())])
.build()?
.into_arc();
let session = SessionBuilder::new(&environment)?
.with_optimization_level(GraphOptimizationLevel::Level1)?
.with_intra_threads(1)?
.with_model_from_file("examples/net.onnx")?;
let input_tensor = tch::Tensor::arange(16 * 2, (tch::Kind::Float, tch::Device::cuda_if_available())).view([16, 2]);
// First option: ndarray
let input_array: ArrayD<f32> = input_tensor.as_ref().try_into()?;
let input_cow_array = CowArray::from(&input_array);
let output_array: OrtOwnedTensor<f32, _> = session.run(vec![Value::from_array(session.allocator(), &input_cow_array)?])?[0].try_extract()?;
println!("{:?}", output_array);
// Second option: IO Bindings
- let mut io_bindings = IoBinding::new(&session)?;
+ let mut io_bindings = session.bind()?;
let value = Value::from_array(session.allocator(), &input_cow_array)?;
- let _ = io_bindings.bind_input("some_input", value)?;
+ let _ = io_bindings.bind_input("some_input", &value)?;
let output_mem_info = MemoryInfo::new(AllocationDevice::CPU, 0, AllocatorType::Device, MemType::Default)?;
let _ = io_bindings.bind_output("some_output", output_mem_info)?;
+
+ session.run_with_binding(&io_bindings)?;
let outputs = io_bindings.outputs()?;
for (output_name, output_value) in outputs {
let output_array: OrtOwnedTensor<f32, _> = output_value.try_extract()?;
println!("{output_name}: {output_array:?}");
}
Ok(())
}
I'm not sure if it does.
I'll look into this 🙂 |
|
Thank you very much! Regarding the use anyhow;
use ndarray::{ArrayD, CowArray};
use ort::tensor::OrtOwnedTensor;
use ort::{AllocationDevice, AllocatorType, Environment, ExecutionProvider, GraphOptimizationLevel, IoBinding, MemType, MemoryInfo, SessionBuilder, Value};
fn main() -> anyhow::Result<()> {
tracing_subscriber::fmt::init();
let environment = Environment::builder()
.with_name("test")
.with_execution_providers([ExecutionProvider::CUDA(Default::default())])
.build()?
.into_arc();
let session = SessionBuilder::new(&environment)?
.with_optimization_level(GraphOptimizationLevel::Level1)?
.with_intra_threads(1)?
.with_model_from_file("examples/net.onnx")?;
let input_tensor = tch::Tensor::arange(16 * 2, (tch::Kind::Float, tch::Device::cuda_if_available())).view([16, 2]);
// First option: ndarray
let input_array: ArrayD<f32> = input_tensor.as_ref().try_into()?;
let input_cow_array = CowArray::from(&input_array);
let output_array: OrtOwnedTensor<f32, _> = session.run(vec![Value::from_array(session.allocator(), &input_cow_array)?])?[0].try_extract()?;
println!("{:?}", output_array);
// Second option: IO Bindings
let mut io_bindings = session.bind()?;
let value = Value::from_array(session.allocator(), &input_cow_array)?;
let _ = io_bindings.bind_input("some_input", &value)?;
let output_mem_info = MemoryInfo::new(AllocationDevice::CPU, 0, AllocatorType::Device, MemType::Default)?;
let _ = io_bindings.bind_output("some_output", output_mem_info)?;
session.run_with_binding(&io_bindings)?;
let outputs = io_bindings.outputs()?;
for (output_name, output_value) in outputs {
let output_array: OrtOwnedTensor<f32, _> = output_value.try_extract()?;
println!("{output_name}: {output_array:?}");
}
Ok(())
}Does not compile: Updating |
decahedron1
added a commit
that referenced
this issue
Jul 23, 2023
|
What should I put in inputname? Edit: Find the Input Name in the model. let onnx = OnnxRuntime::new(r"best.onnx", 416, 0.5, 0.5, 0).unwrap();
let array: ndarray::ArrayBase<ndarray::OwnedRepr<f32>, ndarray::Dim<ndarray::IxDynImpl>> =
Array::zeros((1, 3, 416, 416)).into_dyn();
// let result = onnx.inference(array).unwrap();
let mut io_bindings = onnx.session.bind().unwrap();
let array = CowArray::from(array);
let value = Value::from_array(onnx.session.allocator(), &array).unwrap();
let _ = io_bindings.bind_input("some_input", value).unwrap(); |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello,
I am trying to work with the
IoBindingsthat were recently added, and I am facing a few issues. I could not find documentations or examples in the crate illustrating how this would work -- I am attempting to reproduce a minimal example in Python usingonnxruntime: the gist can be found here. I am attaching the tiny onnx model file (net.zip) to this issue, but it can be created again by running the notebook linked above.Here are the current issues I am facing:
IoBindingsis not public:ort/src/io_binding.rs
Line 19 in cb21def
(crate)visibility modifier, but am I missing something on how theIoBindingshould be created?Dropimplementation for theIoBindingsdoes not include bound input and output. I believebind_inputneeds to take ownership of its value in the current implementation:Failed to get tensor type and shape: the ort_value must contain a constructed tensor or sparse tensorThe small Rust binary I am using for testing is included below for reference. This would require adding the tch dependency to access libtorch, please let me know if you have any issues doing so:
I have also tried extracting the values from the output memory info as follows:
but the values from the tensor are incorrect, so I guess the memory is not read from the right location.
Finally, eventually I think it would be great to be able to run something similar to the Python interface: where we can create 2 torch tensors (input and placeholder output), register pointers to these tensors in the io bindings, and calling
session.run()would populate the tensor output. This would probably require being allowed to pass raw pointers to the io-bindings, maybe a "dangerous" module could be created to allow such usecase.Thank you
The text was updated successfully, but these errors were encountered: