MDR Operator #3826
MDR Operator #3826
Conversation
|
Well, I will need to consolidate all the commits into one or few before merge |
… new MDR extension.
- Only works with BP5 for now since BP5 deserializer had to be modified for reading with accuracy
- Use mgard cmake flag MGARD_ENABLE_MDR to set ADIOS2_HAVE_MGARD_MDR flag and build refactor operator conditionally
- Added Accuracy struct to adios types (error, norm, and rel|abs flag)
- Set/GetAccuracy() functions to Variable, VariableBase and Operator classes.
- BP5 deserializer now prepares the operator with accuracy before calling Decompress().
- bpls new option --error "double,double|inf,rel|abs" added.
- Add tests for MDR operator and Accuracy default behavior (for all operators)
- Add rounding error constants adios2::ops::mdr::DOUBLE_ROUNDING_ERROR_LIMIT and FLOAT_ROUNDING_ERROR_LIMIT,
beacuse GetAccuracy() is limited by rounding errors in MDR
There was a problem hiding this comment.
Really, my only question would be whether or not we want to embed this particular definition of Accuracy as an ADIOS type (including fixing it in external bindings, etc.). I'm assuming that other lossy compressors might have other ways to express accuracy as well (or at least their own nomenclatures). Should we consider leaving the accuracy expression in string form instead? Prefix these types with MGD or include them in specific namespaces so that if/when we have future lossy compressors we're inclusive of their definitions of accuracy?
Excellent question. I was aiming to create a definition that should encompass all compression/refactoring, instead of a simple error value. However, there is the danger that we don't get this right. BTW, MDR does not seem to work with norms other than inf (Linf norm) and 0.0 (L2 norm) but this may change, and only with absolute error but we aim to provide both relative and absolute errors on user request. So it's not clear how a user requesting So, I did not want a simple double value, but also not a some hierarchy of classes for accuracy when defining the interface and came up with this triplet as a probably good enough first try. A string instead would be the safest but ugliest first try in my opinion, wanted something better than that. |
|
Sounds reasonable. We could always add other overloaded get/put operations with string or attribute if we got in a bind with this version... |
|
This PR added some memory leaks that were caught by our newly working (albeit not completely) asan build as seen here: https://app.circleci.com/pipelines/github/ornladios/ADIOS2/7912/workflows/9dd388ff-4437-4450-80aa-1c4e5cfeda6a/jobs/58636 |
|
I think these leaks are in other branches as well and doesn't seem to be
related to this code, even though the new test fails
…On Mon, Dec 11, 2023, 10:17 PM Vicente Bolea ***@***.***> wrote:
This PR added some memory leaks that were caught by our newly working
(albeit not completely) asan build as seen here:
https://app.circleci.com/pipelines/github/ornladios/ADIOS2/7912/workflows/9dd388ff-4437-4450-80aa-1c4e5cfeda6a/jobs/58636
—
Reply to this email directly, view it on GitHub
<#3826 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAYYYLOYNKO3YSCVPIDAU4TYI7EF5AVCNFSM6AAAAAA5H4JE32VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJRGIZTMMJXGU>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
* master: Have HDF5 write raise error if operator(s) requested (ornladios#3951) fix for ASAN issue related to JoinedDimArray handling in BP5 deserializer (ornladios#3963) New operator MDR, for refactoring floating point arrays using MGARD's new MDR extension. (ornladios#3826) restricted http transport from windows builds. XMLConfigTest: Add RemoveIO test adios2::core::ADIOS: Initialize new IO objects with config file removed unsused variable Update readme for heat transfer example with new location and build instructions Ignore tests with defects for now Adapt libfabric dataplane of SST to Cray CXI provider (ornladios#3672) ci: fix path to lsan suppressions, fix broken gh status post Use adios2_mode_readRandomAccess in matlab open to make it work for BP5 (ornladios#3956) Add Global Array Capabilities and Limitations Add Section for Anatomy of an ADIOS Program Enable Shell-Check for gh-actions scripts Enable Shell-Check for circle CI scripts Enable Shell-Check for tau contract scripts Enable Shell-Check for scorpio contract scripts Enable Shell-Check for lammps contract scripts Delete VTK code in examples Fix MATLAB bindings for MacOS (ornladios#3950) Set the compiler for the Kokkos DataMan example to what is used to build Kokkos Fix the HIP architecture CMAKE variable (ornladios#3931) perfstubs 2023-11-27 (845d0702) (ornladios#3944) Revert "Only rank 0 should print the initialization message in perfstub" Formatting Formatting Revision Added buffered data receive in the client side. A socket version of HTTP connector. Proxy server host is hardwired to "localhost" and port to 9999 Remote bpls: bpls -E bp4 -T "Library=HTTP" /remote_path/myVector_cpp.bp -d bpInts
Refactor operator using MGARD-x MDR feature.
Useless in its current form and does not have testing yet but it can refactor float/double arrays.
Reconstruction is full due to lack of support for reader side operator arguments, but MDR itself supports accuracy.