Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use cloudpickle for serializing NegLogParameterPriors #1467

Merged
merged 7 commits into from
Sep 26, 2024

Conversation

dweindl
Copy link
Member

@dweindl dweindl commented Sep 16, 2024

Cloudpickle is able to handle more complex objects than pickle.

See #1465
Closes #1465

Cloudpickle is able to handle more complex objects than pickle.

See ICB-DCM#1465
Closes ICB-DCM#1465

If the new process are forked, we could skip the pickling, but at this point, I don't think that's necessary.
@codecov-commenter
Copy link

codecov-commenter commented Sep 16, 2024

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.83%. Comparing base (ab61c4a) to head (48e4743).

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1467      +/-   ##
===========================================
- Coverage    82.87%   82.83%   -0.05%     
===========================================
  Files          163      163              
  Lines        13760    13765       +5     
===========================================
- Hits         11404    11402       -2     
- Misses        2356     2363       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@dweindl dweindl marked this pull request as ready for review September 16, 2024 19:00
Copy link
Collaborator

@PaulJonasJost PaulJonasJost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to perhaps test with specifically a model where it previously failed (e.g. Schwen_PONE2014 from #1465)?

@dweindl
Copy link
Member Author

dweindl commented Sep 17, 2024

Does it make sense to perhaps test with specifically a model where it previously failed (e.g. Schwen_PONE2014 from #1465)?

Generally, yes. I already have that lying around. But it will be another slow test due to model import, that's why I left it out so far.

@PaulJonasJost
Copy link
Collaborator

Does it make sense to perhaps test with specifically a model where it previously failed (e.g. Schwen_PONE2014 from #1465)?

Generally, yes. I already have that lying around. But it will be another slow test due to model import, that's why I left it out so far.

As we always have the problem to balance between "elaborate" test vs time. Would it make sense to have a set of tests that only run in the (I think weekly) master test?

@dweindl
Copy link
Member Author

dweindl commented Sep 23, 2024

As we always have the problem to balance between "elaborate" test vs time. Would it make sense to have a set of tests that only run in the (I think weekly) master test?

In principle, yes, but this should not become an excuse for not writing more efficient tests.

@dweindl
Copy link
Member Author

dweindl commented Sep 23, 2024

Does it make sense to perhaps test with specifically a model where it previously failed (e.g. Schwen_PONE2014 from #1465)?

Added a simpler test.

@dweindl dweindl changed the title SacessOptimizer: Use cloudpickle for passing data to workers Use cloudpickle for serializing NegLogParameterPriors Sep 24, 2024
@dweindl dweindl self-assigned this Sep 25, 2024
Copy link
Collaborator

@PaulJonasJost PaulJonasJost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks :) If cloudpickle is more powerful then pickle, should we switch to cloudpickle everywhere?

@dweindl
Copy link
Member Author

dweindl commented Sep 26, 2024

If cloudpickle is more powerful then pickle, should we switch to cloudpickle everywhere?

It seems pickle got the job done in most cases so far. I'd only do it where necessary.

@dweindl dweindl merged commit 61c912d into ICB-DCM:develop Sep 26, 2024
18 checks passed
@dweindl dweindl deleted the fix_1465_sacess_cloudpickle branch September 26, 2024 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Using SacessOptimizer on objectives with parameter priors leads to pickle error
3 participants