Fix: Static Builds Against C-Blosc2 #3715

ax3l · 2023-07-26T00:35:00Z

This uses the new C-Blosc2 Blosc2Config.cmake packages, so we can do purely static builds.

depends on upstream CMake: Install Blosc2Config.cmake Blosc/c-blosc2#537
- part of the next release, C-Blosc2 v2.10.1

ax3l · 2023-08-07T17:08:56Z

C-Blosc2 v2.10.1 contains the fix! 🎉
https://github.com/Blosc/c-blosc2/releases/tag/v2.10.1

vicentebolea

@ax3l many thanks for this contributions. If you do not have cycles to follow up my request let me know and I can take of it.

cmake/DetectOptions.cmake

source/adios2/CMakeLists.txt

- Give priority to Blosc2Config.cmake file rather than FindBlosc2.cmake. - Prefer shared Blosc2 library. Co-Authored-By: Axel Huebl <axel.huebl@plasma.ninja>

vicentebolea · 2023-08-10T22:30:47Z

@scottwittenburg I am adding a blosc2 dependency to the spack builds. Note that Blosc2 does not have a Spack pkg and the reason to install this in our Spack images is that they are our standard images that we test most of the configurations. I am not too strong about it, we can move them to the nonspack build. Let me know.

On another note I changed the tag naming while building the builds so that we can avoid having to retag and we can build a leaf image without rebuilding the base image (if no changes in base).

scottwittenburg · 2023-08-10T22:49:38Z

@vicentebolea

Regarding building a non-spack blosc in the spack image: does blosc share any dependencies with adios2 which might be picked up from the system when building and possibly conflict later? Maybe it's nothing to worry about, I'm just thinking out loud.

Regarding your change to remove the explicit tagging step in favor of just using the right tag when building: sounds good to me.

Thanks!

vicentebolea · 2023-08-10T22:51:13Z

@vicentebolea

Regarding building a non-spack blosc in the spack image: does blosc share any dependencies with adios2 which might be picked up from the system when building and possibly conflict later? Maybe it's nothing to worry about, I'm just thinking out loud.

Good question. I do not see this possibly happening we are building a very minimal blosc2

scottwittenburg · 2023-08-10T23:11:08Z

Good question. I do not see this possibly happening we are building a very minimal blosc2

I see by default it turns on zlib, zstd, and lz4, and also by default it uses it's own internal "thirdparty" versions of those.

c-blosc2 internal dependency versions:

lz4-1.9.4
zlib-1.2.13
zstd-1.5.5

adios2 spack image contains:

lz4@1.9.4
zlib@1.2.13
zstd@1.5.2

So there's a pretty close match between what we have in the spack image, and what c-blosc2 needs, so if problems arise from doing it this way we could try to get the c-blosc2 build to use our spack-installed dependencies. Or we could also just move c-blosc2 to the non-spack build.

By the way, our image also contains c-blosc@1.21.1, but I guess that's something different. Maybe there's a c-blosc2 spack package in the works?

Actually, there is a c-blosc2 package in spack, and it has the version mentioned above. At that's what it looks like from this merged PR: spack/spack#39297

scottwittenburg · 2023-08-10T23:16:37Z

However, it does seem that the adios2 spack package only knows about c-blosc, nothing about c-blosc2, so maybe that's why we can't use the spack package.

vicentebolea · 2023-08-10T23:22:28Z

However, it does seem that the adios2 spack package only knows about c-blosc, nothing about c-blosc2, so maybe that's why we can't use the spack package.

@scottwittenburg thanks for pointing out this, that is indeed the Spack package for blosc2, I did made a search using spack search to [bB]losc2 without luck but indeed the name is c-blosc2. Will have to update our adios2 pkg to use c-blocs2 rather than c-blocs.

scottwittenburg · 2023-08-10T23:27:31Z

Will have to update our adios2 pkg to use c-blocs2 rather than c-blocs.

Yeah. But I think it makes sense to go ahead with the current approach here for now, as that will take some time to get merged and appear in the E4S image we're using as a base.

vicentebolea · 2023-08-10T23:28:23Z

Yeah. But I think it makes sense to go ahead with the current approach here for now, as that will take some time to get merged and appear in the E4S image we're using as a base.

I am adding c-blocs2 to our spec.yml in the meanwhile.

scripts/ci/images/specs.yaml

vicentebolea · 2023-08-11T01:45:37Z

@scottwittenburg we need to install this manually for the time being since the blosc2 in the E4S spack is older than what we req. I am building it with the adios2-ci-serial environment activated

scottwittenburg · 2023-08-11T15:53:45Z

I see the cmakecache.txt now reflect that blosc2 is enabled and found in your manual install directory. So this looks good to me, once you remove the tmp from the image names.

vicentebolea · 2023-08-11T16:09:16Z

I see the cmakecache.txt now reflect that blosc2 is enabled and found in your manual install directory. So this looks good to me, once you remove the tmp from the image names.

Done

ax3l · 2023-08-16T20:49:48Z

cmake/DetectOptions.cmake

+  if (Blosc2_VERSION VERSION_GREATER_EQUAL 2.10.1)
+    set(adios2_blosc2_tgt Blosc2::blosc2_$<IF:$<BOOL:${ADIOS2_Blosc2_PREFER_SHARED}>,shared,static>)


@vicentebolea there is a bug here:
ADIOS2_Blosc2_PREFER_SHARED should not assume there is always a shared c-blosc2 library available.

C-Blosc2 builds allow to control static and shared builds independently, thus this will cause issues with ADIOS2_Blosc2_PREFER_SHARED=ON (default) for purely static builds (prefer shared != require shared).

Thanks for pointing this out.

ADIOS2_Blosc2_PREFER_SHARED should not assume there is always a shared c-blosc2 library available.

We do not assume this, a couple of lines above we decide this. However, the prefer is not respected since if the user force ADIOS2_Blosc2_PREFER_SHARED=ON it will fail with an static lib

Yes, that's what I mean, thx! Do you want to patch this or track it for later patching?

A possible solution at #3767

ornladios/ADIOS2#3715 (comment)

* Update blosc -> blosc2, ADIOS2 -> v2.9.0-rc1 ADIOS2 should be replace by v2.9.0 finally * Fix Blosc2 CMake params * Guard against CMAKE_OSX_ARCHITECTURES * For now, deactivate the pthread patch in ADIOS2 * Test small patch for blosc2 CMake * Add -fPIC flag * MacOS compatibility * -DADIOS2_USE_Blosc2=ON * Build atl and ffs manually to avoid CMake import error * Revert "Build atl and ffs manually to avoid CMake import error" This reverts commit c5469a3. * RC1 -> full release * Don't build ADIOS1 * -DADIOS2_USE_MHS=OFF * Use -DADIOS2_INSTALL_GENERATE_CONFIG=OFF workaround See ornladios/ADIOS2#3348 (comment) * Do not disable find calls * Revert "Do not disable find calls" This reverts commit 7065b89. * Try setting ADIOS2_PATH * Keep ADIOS1 + c-blosc1 * Undo `build.yml` Changes * ADIOS2: no c-blosc1 pthread issue * HDF5: 1.14.1-2 * macOS CI: Now has 3 cores * C-Blosc2 Updates - CMake options - Windows - build stamp * Windows Updates * HDF5 Patches * C-Blosc 2.9.3 + CMake Patch * ARM: H5Detect * Windows: Avoid CXX20 in C-Blosc2 * macOS arm64 HDF5: H5detect cross-compile * HDF5 arm64: cross-compiling H5detect * [Patch] ADIOS2 c-blosc2 windows.h std::min * [Patch] C-Blosc2: External Zlib * [Hack] Windows: Ignore Ext. ZLIB for c-blosc2 Just build again and close eyes. * ppc64le: Skip C-Blosc & ADIOS1 Due to time limits on Travis-CI. * Unix: Keep HDF5 1.12.2 for Now - issues with cross-compile cannot quickly be solved for macOS arm64/aarch64 - disable HDF5 support in ADIOS2 for now * [Patch] C-Blosc2 PUBLIC Linkage Zlib * Use Blosc2Config.cmake Package * Cleanup and `cat` ADIOS2 Config File * [Patch] [Hack] CMake --trace-expand * Cleanup: cat adios2-config.cmake * [Travis] PPC64le: `travis_wait 30` Takes longer now w/o output to the terminal than the default wait time of 10min. https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received * [ADIOS2] CMake --trace-expand * Undo Trace Expand Hacks * [Travis] PPC64le: `travis_wait 45` * ADIOS2: w/ SST * Blosc2: CMake Config PR Merged * macOS (x86): 10.15 -> 11.0 The macOS 10.15 build images on GH Actions are gone, so we remove this EOL platform for good now. * Windows: Skip Building Blosc1 Co-authored-by: Franz Pöschel <franz.poeschel@gmail.com> * Python: 3.8+ Skip older Python releases, 3.7 is EOL since last month. * [Patch] ADIOS2: Remove FindBlosc2.cmake * C-Blosc 2.10.1 * Win: ADIOS 2.9.1 * ADIOS2_Blosc2_PREFER_SHARED=OFF ornladios/ADIOS2#3715 (comment) * cibuildwheel 2.12.1 -> 2.14.1 For `pp310-manylinux_aarch64` --------- Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

ax3l · 2023-08-18T16:02:09Z

@vicentebolea @scottwittenburg just FYI, for transitive, external dependencies in C-Blosc2 you might want to use v2.10.2+
#3714 (comment)

ornladios#3715

ax3l requested a review from vicentebolea as a code owner July 26, 2023 00:35

ax3l force-pushed the fix-blosc2-findpackage branch from a1aa202 to 6f3ae6f Compare July 26, 2023 00:35

vicentebolea requested changes Aug 7, 2023

View reviewed changes

cmake/DetectOptions.cmake Outdated Show resolved Hide resolved

source/adios2/CMakeLists.txt Outdated Show resolved Hide resolved

vicentebolea force-pushed the fix-blosc2-findpackage branch 2 times, most recently from 4481696 to 592efa2 Compare August 10, 2023 21:47

vicentebolea requested review from anagainaru and caitlinross as code owners August 10, 2023 21:47

vicentebolea changed the base branch from release_29 to master August 10, 2023 21:47

vicentebolea added the backport:release label Aug 10, 2023

vicentebolea added this to the v2.9.2 milestone Aug 10, 2023

vicentebolea force-pushed the fix-blosc2-findpackage branch 2 times, most recently from beb9966 to de9df0a Compare August 10, 2023 22:15

cmake: Add Blosc2 2.10.1 compatibility.

8333d5e

- Give priority to Blosc2Config.cmake file rather than FindBlosc2.cmake. - Prefer shared Blosc2 library. Co-Authored-By: Axel Huebl <axel.huebl@plasma.ninja>

vicentebolea force-pushed the fix-blosc2-findpackage branch from de9df0a to eba7417 Compare August 10, 2023 22:27

vicentebolea mentioned this pull request Aug 10, 2023

adios2: correct c-blosc dependency spack/spack#39385

Merged

vicentebolea force-pushed the fix-blosc2-findpackage branch 2 times, most recently from 7fd8481 to b965912 Compare August 10, 2023 23:55

vicentebolea reviewed Aug 10, 2023

View reviewed changes

scripts/ci/images/specs.yaml Outdated Show resolved Hide resolved

vicentebolea force-pushed the fix-blosc2-findpackage branch from b965912 to eba7417 Compare August 11, 2023 01:41

CI: added blosc2 dependency to Spack build

59fd7dc

CI: use full image tag

47082ef

vicentebolea force-pushed the fix-blosc2-findpackage branch from ae4d0c7 to 47082ef Compare August 11, 2023 16:08

vicentebolea enabled auto-merge August 11, 2023 16:08

vicentebolea disabled auto-merge August 11, 2023 16:09

scottwittenburg approved these changes Aug 11, 2023

View reviewed changes

vicentebolea enabled auto-merge August 11, 2023 16:15

vicentebolea approved these changes Aug 11, 2023

View reviewed changes

vicentebolea self-assigned this Aug 11, 2023

vicentebolea merged commit 276383a into ornladios:master Aug 11, 2023
25 checks passed

vicentebolea mentioned this pull request Aug 14, 2023

Add blosc to our CI github actions images #3747

Closed

ax3l deleted the fix-blosc2-findpackage branch August 16, 2023 20:47

ax3l commented Aug 16, 2023

View reviewed changes

ax3l added a commit to franzpoeschel/openPMD-api that referenced this pull request Aug 16, 2023

ADIOS2_Blosc2_PREFER_SHARED=OFF

086c69f

ornladios/ADIOS2#3715 (comment)

ax3l restored the fix-blosc2-findpackage branch August 16, 2023 21:03

ax3l added a commit to franzpoeschel/openPMD-api that referenced this pull request Aug 16, 2023

ADIOS2_Blosc2_PREFER_SHARED=OFF

2c97be6

ornladios/ADIOS2#3715 (comment)

vicentebolea mentioned this pull request Aug 16, 2023

Change ADIOS2_Blosc2_PREFER_SHARED behavior #3766

Closed

ax3l added a commit to ax3l/ADIOS2 that referenced this pull request Aug 20, 2023

Backport C-Blosc2 Static Build

540ee9b

ornladios#3715

This was referenced Aug 20, 2023

cmake: correct ADIOS2_Blosc2_PREFER_SHARED behavior #3767

Merged

Blosc2 USE ON: Fix Module Fallback #3774

Merged

ax3l deleted the fix-blosc2-findpackage branch August 20, 2023 05:09

vicentebolea mentioned this pull request Aug 24, 2023

Static build on Summit issues #3128

Open

This was referenced Oct 19, 2023

Backports from master #3860

Closed

Backport from master #3861

Merged

ax3l mentioned this pull request Nov 10, 2023

2.9.2 w/ C-Blosc2 v2.11.1 Static #3903

Open

vicentebolea removed the backport:release label Feb 21, 2024

BrewTestBot mentioned this pull request Apr 4, 2024

adios2 2.10.0 Homebrew/homebrew-core#167957

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix: Static Builds Against C-Blosc2 #3715

Fix: Static Builds Against C-Blosc2 #3715

ax3l commented Jul 26, 2023 •

edited

Loading

ax3l commented Aug 7, 2023

vicentebolea left a comment

vicentebolea commented Aug 10, 2023

scottwittenburg commented Aug 10, 2023

vicentebolea commented Aug 10, 2023

scottwittenburg commented Aug 10, 2023

scottwittenburg commented Aug 10, 2023

vicentebolea commented Aug 10, 2023

scottwittenburg commented Aug 10, 2023

vicentebolea commented Aug 10, 2023

vicentebolea commented Aug 11, 2023

scottwittenburg commented Aug 11, 2023

vicentebolea commented Aug 11, 2023

ax3l Aug 16, 2023 •

edited

Loading

vicentebolea Aug 16, 2023

ax3l Aug 18, 2023

vicentebolea Aug 18, 2023

ax3l commented Aug 18, 2023

		if (Blosc2_VERSION VERSION_GREATER_EQUAL 2.10.1)
		set(adios2_blosc2_tgt Blosc2::blosc2_$<IF:$<BOOL:${ADIOS2_Blosc2_PREFER_SHARED}>,shared,static>)

Fix: Static Builds Against C-Blosc2 #3715

Fix: Static Builds Against C-Blosc2 #3715

Conversation

ax3l commented Jul 26, 2023 • edited Loading

ax3l commented Aug 7, 2023

vicentebolea left a comment

Choose a reason for hiding this comment

vicentebolea commented Aug 10, 2023

scottwittenburg commented Aug 10, 2023

vicentebolea commented Aug 10, 2023

scottwittenburg commented Aug 10, 2023

scottwittenburg commented Aug 10, 2023

vicentebolea commented Aug 10, 2023

scottwittenburg commented Aug 10, 2023

vicentebolea commented Aug 10, 2023

vicentebolea commented Aug 11, 2023

scottwittenburg commented Aug 11, 2023

vicentebolea commented Aug 11, 2023

ax3l Aug 16, 2023 • edited Loading

Choose a reason for hiding this comment

vicentebolea Aug 16, 2023

Choose a reason for hiding this comment

ax3l Aug 18, 2023

Choose a reason for hiding this comment

vicentebolea Aug 18, 2023

Choose a reason for hiding this comment

ax3l commented Aug 18, 2023

ax3l commented Jul 26, 2023 •

edited

Loading

ax3l Aug 16, 2023 •

edited

Loading