Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wait for profile peek task to stop before exit #51409

Closed

Conversation

IanButterworth
Copy link
Member

Trying to help with spurious reports of #50873

Copy link
Member

@timholy timholy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Closes #51003

@IanButterworth IanButterworth added the backport 1.10 Change should be backported to the 1.10 release label Sep 20, 2023
@IanButterworth
Copy link
Member Author

Before this precompiling DifferentialEquations I got

┌ ArrayInterface → ArrayInterfaceBandedMatricesExt
│  [pid 16932] waiting for IO to finish:
│   TYPE[FD/PID]       @UV_HANDLE_T->DATA
│   timer              @0x600002e75100->0x111324580
└  
┌ ExponentialUtilities
│  [pid 16870] waiting for IO to finish:
│   TYPE[FD/PID]       @UV_HANDLE_T->DATA
│   timer              @0x60000787af00->0x17c0dacb0
└  
┌ ChainRulesCore
│  [pid 16721] waiting for IO to finish:
│   TYPE[FD/PID]       @UV_HANDLE_T->DATA
│   timer              @0x6000003efb00->0x10dff95d0
└  

With this

┌ ArrayInterface → ArrayInterfaceBandedMatricesExt
│  [pid 63460] waiting for IO to finish:
│   TYPE[FD/PID]       @UV_HANDLE_T->DATA
│   timer              @0x600003efba00->0x110d5e560
└  

base/Base.jl Outdated Show resolved Hide resolved
@vtjnash
Copy link
Member

vtjnash commented Sep 20, 2023

I don't get it. Where do you see a Timer or sleep call here?

@timholy
Copy link
Member

timholy commented Sep 21, 2023

You know more about this than I do, but just in case: in Base.__init__ we spawn profile_printing_listener which waits on PROFILE_PRINT_COND. I don't fully follow the Base.uv_unref(cond.handle) operations, but presumably the notification doesn't always arrive before Julia tries to exit. On slack it was found that these warnings get printed disproportionately for fast-to-precompile "packages," which fits many extensions.

@vtjnash
Copy link
Member

vtjnash commented Sep 26, 2023

We can observe this does not seem to be the fix by generating an empty package:

(@v1.11) pkg> generate Hello

(@v1.11) pkg> dev ./Hello

julia> while true; Base.compilecache(Base.identify_package(Main, "Hello")); end
[ Info: Precompiling Hello [b78c8773-8b09-44b1-ab9b-12867abe6258]
("/home/vtjnash/.julia/compiled/v1.11/Hello/PkeV8_ikVpV.ji", "/home/vtjnash/.julia/compiled/v1.11/Hello/PkeV8_ikVpV.so")

@PallHaraldsson
Copy link
Contributor

PallHaraldsson commented Sep 28, 2023

I don't like this for many reasons.

Such as it may not work. I found exit_on_sigint, and Julia exit is not always called to exit, rather the C function, and also that C function is exported, and if called should it also do this logic?

First a question. When your main strict ends, is exit() or similar called? I couldn't locate, and think you would want it called in almost all situations when you exit.

But I note C and C++ have many ways to exit a program, I would like to know how many Julia has, all the ways; e.g. similar to _Exit() and std::quick_exit()?

https://en.cppreference.com/w/cpp/utility/program/quick_exit

https://stackoverflow.com/questions/9758495/what-is-the-difference-between-stdquick-exit-and-stdabort-and-why-was-stdq

[..] There's a simple and very tempting workaround for this problem: call _exit() instead. Kaboom, program ended, the operating system brooms up the shrapnel. But clearly without any cleanup at all, very messy sometimes with artifacts like a half-written file or an incomplete dbase transaction.

std::quick_exit() offers the alternative. Similar to _exit() but with still the option to execute some code, whatever was registered with at_quick_exit.

I guess you need and want to have similar exit to C/C++'s default, I thought quick_exit would be faster/fastest, it seems not. I would like a very fast or immediate exit, and I fear that Tasks will take a long time to finish, or will not, deadlock. I suppose you still may not want to just abort by default but I would at least want to have that option.

I like the ideal of https://en.wikipedia.org/wiki/Crash-only_software

But since most software doesn't live up to this software (yet) then I guess Julia programs may need to allow Tasks to finish by default. It would be great if Julia itself would adopt that ideal however. It seems this is being implemented for Julia itself since Julia doesn't live up to it.

@IanButterworth
Copy link
Member Author

Sorry, I should've closed this. It doesn't seem to be the issue. The issue appears to be Pkg.precompile not getting the ordering of deps quite right, and the internal precompile process of packages hitting the pid locks when they shouldn't

@IanButterworth IanButterworth deleted the ib/wait_profile_peek branch September 28, 2023 20:43
@KristofferC KristofferC removed the backport 1.10 Change should be backported to the 1.10 release label Oct 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants