Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]Error log is misleading when IPU device is inaccessible #790

Open
tianboh opened this issue Jan 6, 2022 · 0 comments
Open

[BUG]Error log is misleading when IPU device is inaccessible #790

tianboh opened this issue Jan 6, 2022 · 0 comments
Assignees

Comments

@tianboh
Copy link
Collaborator

tianboh commented Jan 6, 2022

Describe the bug
As title.

Screenshots

2022-01-06T06:45:13.321528Z popart:popart 904.904 I: Start to create Executable...
2022-01-06T06:45:13.321620Z popart:popart 904.904 I: Computation is not initialized. init it first
2022-01-06T06:45:13.321629Z popart:popart 904.904 I: PIPELINE not used for this run, Set the global virtual group to ipu 0
2022-01-06T06:45:18.424014Z popart:popart 904.904 E: Must pass a valid deviceInfo to InferenceSession::createFromOnnxModel

[0] _odla_computation::init(bool)
[1] odla_CreateExecutable
[2] tensorflow::halo::MlirIpuExecutable::Init()
[3] tensorflow::halo::HaloCompilationCache::CompileImpl(std::unique_ptr<tensorflow::halo::HaloCompilerInput, std::default_delete<tensorflow::halo::HaloCompilerInput> >, tensorflow::NameAttrList const&, std::map<int, tensorflow::Tensor, std::less<int>, std::allocator<std::pair<int const, tensorflow::Tensor> > > const&, std::set<int, std::less<int>, std::allocator<int> > const&, std::map<int, tensorflow::halo::OptionalTensor, std::less<int>, std::allocator<std::pair<int const, tensorflow::halo::OptionalTensor> > > const&, tensorflow::OpKernelContext*, tensorflow::halo::Executable**)
[4] tensorflow::halo::HaloCompilationCache::Compile(std::unique_ptr<tensorflow::halo::HaloCompilerInput, std::default_delete<tensorflow::halo::HaloCompilerInput> >, tensorflow::NameAttrList const&, std::map<int, tensorflow::Tensor, std::less<int>, std::allocator<std::pair<int const, tensorflow::Tensor> > > const&, std::set<int, std::less<int>, std::allocator<int> > const&, std::map<int, tensorflow::halo::OptionalTensor, std::less<int>, std::allocator<std::pair<int const, tensorflow::halo::OptionalTensor> > > const&, tensorflow::OpKernelContext*, tensorflow::halo::Executable**)
[5] tensorflow::halo::HaloLaunchOp::CompileToLocalExecutable(tensorflow::OpKernelContext*, tensorflow::NameAttrList const&, std::map<int, tensorflow::halo::OptionalTensor, std::less<int>, std::allocator<std::pair<int const, tensorflow::halo::OptionalTensor> > >*, tensorflow::halo::Executable**)
[6] tensorflow::halo::HaloLaunchOp::CompileAndRun(tensorflow::OpKernelContext*, tensorflow::halo::DoneHelper*, bool*)
[7] tensorflow::halo::HaloLaunchOp::ComputeAsync(tensorflow::OpKernelContext*, std::function<void ()>)



2022-01-06T06:45:18.424154Z popart:popart 904.904 E: Session::createFromOnnxModel failed:Must pass a valid deviceInfo to InferenceSession::createFromOnnxModel
2022-01-06T06:45:18.494088Z popart:popart 904.904 E: Failed to init computation when compiling.
2022-01-06 14:45:18.637901: I tensorflow/compiler/bridge/kernels/halo_launch.cc:524] Done Local compile:Internal: Create executable failed
2022-01-06 14:45:18.639146: W tensorflow/compiler/bridge/kernels/halo_launch.cc:436] cluster: cluster_0_halo_launch fallbacks to TF as Create executable failed
2022-01-06 14:45:18.639214: I tensorflow/compiler/bridge/kernels/halo_launch.cc:440] cluster: cluster_0_halo_launch fallback to TF.
2022-01-06 14:45:18.640200: W tensorflow/compiler/bridge/kernels/halo_launch.cc:378] HaloLaunch execute function failed: Handle: 140125997346536 not found.
Encount error(s) when warmup: Handle: 140125997346536 not found.
	 [[{{node cluster_0_halo_launch}}]]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants