-
Notifications
You must be signed in to change notification settings - Fork 505
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Manually init absl log to avoid log spam #6890
Conversation
@lsy323 I consider this as a regression, since in 2.0 we fixed this issue once. I am debating whether we should backport this to 2.3. What do you think? |
@@ -2206,6 +2207,7 @@ void InitXlaModuleBindings(py::module m) { | |||
!xtensor->CurrentDataHandle()->HasValue(); | |||
}); | |||
m.def("_init_xla_lazy_backend", []() { | |||
absl::InitializeLog(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmmm, I'm not confident about this change. IIRC this function errors out if it's called twice, and it is for sure called in the GPU plugin: https://github.com/openxla/xla/blob/df104c0963bf1597358dc759da3fa3d2639de03c/xla/pjrt/c/pjrt_c_api_gpu.cc#L28
If you're considering a backport, a more conservative change would be to add the initialization for TPU only sine we're sure libtpu
does not init logging:
xla/torch_xla/csrc/runtime/pjrt_registry.cc
Line 111 in 0b9513c
} else if (device_type == "TPU") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sg, will update
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's not backport this since it has risks.
01f48fa
to
3919c07
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What tool did you use to format the bazel build file and should I be using it?
I need to find out haha, I was just using the vscode with our dev container and the vscode decided to format the bazel file. |
This should get rid of
on every init.
torch_xla/csrc/BUILD
was messed by the formater, my only change was to add"@com_google_absl//absl/log:initialize",