Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add memory and core estimate function #308

Merged
merged 24 commits into from
Nov 21, 2024

Conversation

FedeMPouzols
Copy link
Collaborator

@FedeMPouzols FedeMPouzols commented Nov 20, 2024

Resolves #263.

This adds the function estimate_conversion_memory_and_cores() to convert_msv2_to_processing_set.
A simple example is added to the ps_vis tutorial (tried to keep this as short as possible to prevent loss of focus on the MSv4).
The estimates seem reliable enough, with two caveats:

  • MSv2s with large pointing tables relative to main table => will tend to underestimate
  • There is currently a ~5-6% overestimation to cover for that plus other possible buffer allocations in to_zarr()

I'd suggest we follow-up on this with an issue to add memory estimates of pointing and sys_cal. That would make it possible to have a more precise model and reduce the overestimation.


Note well:

This contribution is made under the current ALMA software agreements. 
(c) European Southern Observatory, 2024 
Copyright by ESO (in the framework of the ALMA collaboration)

@FedeMPouzols FedeMPouzols linked an issue Nov 20, 2024 that may be closed by this pull request
5 tasks
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@Jan-Willem Jan-Willem merged commit 10fabac into main Nov 21, 2024
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Memory and Core estimate function
3 participants