Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MDR Operator #3826

Merged
merged 1 commit into from
Dec 12, 2023
Merged

MDR Operator #3826

merged 1 commit into from
Dec 12, 2023

Conversation

pnorbert
Copy link
Contributor

Refactor operator using MGARD-x MDR feature.
Useless in its current form and does not have testing yet but it can refactor float/double arrays.
Reconstruction is full due to lack of support for reader side operator arguments, but MDR itself supports accuracy.

@pnorbert
Copy link
Contributor Author

pnorbert commented Dec 6, 2023

Well, I will need to consolidate all the commits into one or few before merge

… new MDR extension.

 - Only works with BP5 for now since BP5 deserializer had to be modified for reading with accuracy
 - Use mgard cmake flag MGARD_ENABLE_MDR to set ADIOS2_HAVE_MGARD_MDR flag and build refactor operator conditionally
 - Added Accuracy struct to adios types (error, norm, and rel|abs flag)
 - Set/GetAccuracy() functions to Variable, VariableBase and Operator classes.
 - BP5 deserializer now prepares the operator with accuracy before calling Decompress().
 - bpls new option --error "double,double|inf,rel|abs" added.
 - Add tests for MDR operator and Accuracy default behavior (for all operators)
 - Add rounding error constants adios2::ops::mdr::DOUBLE_ROUNDING_ERROR_LIMIT and FLOAT_ROUNDING_ERROR_LIMIT,
      beacuse GetAccuracy() is limited by rounding errors in MDR
Copy link
Member

@eisenhauer eisenhauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really, my only question would be whether or not we want to embed this particular definition of Accuracy as an ADIOS type (including fixing it in external bindings, etc.). I'm assuming that other lossy compressors might have other ways to express accuracy as well (or at least their own nomenclatures). Should we consider leaving the accuracy expression in string form instead? Prefix these types with MGD or include them in specific namespaces so that if/when we have future lossy compressors we're inclusive of their definitions of accuracy?

@pnorbert
Copy link
Contributor Author

pnorbert commented Dec 11, 2023

Really, my only question would be whether or not we want to embed this particular definition of Accuracy as an ADIOS type (including fixing it in external bindings, etc.).

Excellent question. I was aiming to create a definition that should encompass all compression/refactoring, instead of a simple error value. However, there is the danger that we don't get this right.

BTW, MDR does not seem to work with norms other than inf (Linf norm) and 0.0 (L2 norm) but this may change, and only with absolute error but we aim to provide both relative and absolute errors on user request. So it's not clear how a user requesting a1 accuracy should deal with the a2 accuracy returned by the operator after the read.

So, I did not want a simple double value, but also not a some hierarchy of classes for accuracy when defining the interface and came up with this triplet as a probably good enough first try. A string instead would be the safest but ugliest first try in my opinion, wanted something better than that.

@eisenhauer
Copy link
Member

Sounds reasonable. We could always add other overloaded get/put operations with string or attribute if we got in a bind with this version...

@eisenhauer eisenhauer merged commit 931941e into ornladios:master Dec 12, 2023
34 of 35 checks passed
@vicentebolea
Copy link
Collaborator

This PR added some memory leaks that were caught by our newly working (albeit not completely) asan build as seen here: https://app.circleci.com/pipelines/github/ornladios/ADIOS2/7912/workflows/9dd388ff-4437-4450-80aa-1c4e5cfeda6a/jobs/58636

@pnorbert
Copy link
Contributor Author

pnorbert commented Dec 12, 2023 via email

@pnorbert pnorbert deleted the mdr branch December 12, 2023 13:33
pnorbert added a commit to pnorbert/ADIOS2 that referenced this pull request Dec 12, 2023
* master:
  Have HDF5 write raise error if operator(s) requested (ornladios#3951)
  fix for ASAN issue related to JoinedDimArray handling in BP5 deserializer (ornladios#3963)
  New operator MDR, for refactoring floating point arrays using MGARD's new MDR extension. (ornladios#3826)
  restricted http transport from windows builds.
  XMLConfigTest: Add RemoveIO test
  adios2::core::ADIOS: Initialize new IO objects with config file
  removed unsused variable
  Update readme for heat transfer example with new location and build instructions
  Ignore tests with defects for now
  Adapt libfabric dataplane of SST to Cray CXI provider (ornladios#3672)
  ci: fix path to lsan suppressions, fix broken gh status post
  Use adios2_mode_readRandomAccess in matlab open to make it work for BP5 (ornladios#3956)
  Add Global Array Capabilities and Limitations
  Add Section for Anatomy of an ADIOS Program
  Enable Shell-Check for gh-actions scripts
  Enable Shell-Check for circle CI scripts
  Enable Shell-Check for tau contract scripts
  Enable Shell-Check for scorpio contract scripts
  Enable Shell-Check for lammps contract scripts
  Delete VTK code in examples
  Fix MATLAB bindings for MacOS (ornladios#3950)
  Set the compiler for the Kokkos DataMan example to what is used to build Kokkos
  Fix the HIP architecture CMAKE variable (ornladios#3931)
  perfstubs 2023-11-27 (845d0702) (ornladios#3944)
  Revert "Only rank 0 should print the initialization message in perfstub"
  Formatting
  Formatting
  Revision
  Added buffered data receive in the client side.
  A socket version of HTTP connector. Proxy server host is hardwired to "localhost" and port to 9999 Remote bpls: bpls -E bp4 -T "Library=HTTP" /remote_path/myVector_cpp.bp -d bpInts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants