Add dev.eessi.io support including project subdirectories #885

Neves-P · 2025-01-24T09:19:14Z

This PR replaces open WIP PR #804 (to merge from a more up to date and tested branch) and adds support for building into the dev.eessi.io repository.

Documentation for how to install into dev.eessi.io is available in https://www.eessi.io/docs/adding_software/adding_development_software/, with maintainer documentation to come in EESSI/dev.eessi.io-example#12

With this PR, software builds are installed into /cvmfs/dev.eessi.io/2023.06/$EESSI_DEV_PROJECT/. $EESSI_DEV_PROJECT is determined automatically during the build process and comes from the name of the GitHub repository that the bot is listening according to the format: dev.eessi.io-EESSI_DEV_PROJECT. See: https://github.com/EESSI/dev.eessi.io-example

Existing builds to dev.eessi.io were tested with the code that is about to be merged as verified in EESSI/dev.eessi.io-example#14

…tware-layer into feature/dev.eessi.io

…minimal_eessi_env

…ILD_INSTALLPATH override

This reverts commit a37ae0b.

This reverts commit 39d67eb.

This reverts commit 28c6b7e.

This reverts commit 3fc95e1.

This reverts commit 9f0b2df.

This reverts commit b3399f2.

This reverts commit 72ba311.

This reverts commit 35bd8a1.

This reverts commit a4d61d6.

This reverts commit 1ecd93b.

Neves-P · 2025-02-14T18:33:24Z

Talked with @bedroge about this and decided to try just adding a simple if statement to EESSI extend that would account for builds targeting the development repository, changes in: ee2bcb1

I'd like to clarify to things:

How to test this without potentially breaking working code. I suppose that the change to the EESSI-extend module needs to actually be ingested first so that the new module file is used?
Does this actually work, or is there a better or improved way of achieving this.

@ocaisa , would you have any suggestions regarding this?

PS: I'm marking this a draft to prevent accidental merges as this could break production code. This PR also got extremely long with many commits reverting small changes and applying them again. I'm happy to open a new one from a clean branch to prevent polluting the repo's commit history with many trial and error commits.

ocaisa · 2025-02-17T13:32:39Z

Talked with @bedroge about this and decided to try just adding a simple if statement to EESSI extend that would account for builds targeting the development repository, changes in: ee2bcb1

In general, I think this is the way to go. I would like to target the dev repos just the same as we do project extensions. There will need to be a step that generates the path with the subdirectory, but that could all be done with a single environment variable (if the variable EESSI_DEV_PROJECT is set, we are targeting /cvmfs/dev.eessi.io and should use the value of the variable to decide the subdirectory). That would mean you set the variable to the project and there is only a very small amount of special treatment to determine EESSI_PROJECT_INSTALL

I'd like to clarify to things:

How to test this without potentially breaking working code. I suppose that the change to the EESSI-extend module needs to actually be ingested first so that the new module file is used?

EESSI-extend is just an easyconfig that creates a module file, you can install it locally.

Does this actually work, or is there a better or improved way of achieving this.

@ocaisa , would you have any suggestions regarding this?

Rather than complicate our lives with

/cvmfs/dev.eessi.io/versions/2023.06/example/software/linux/aarch64/neoverse_n1

perhaps we should be using

/cvmfs/dev.eessi.io/example/versions/2023.06/software/linux/aarch64/neoverse_n1

then you have direct mapping /cvmfs/software.eessi.io -> /cvmfs/dev.eessi.io/example

Neves-P · 2025-02-17T15:26:05Z

Thanks @ocaisa , indeed that makes it a lot simpler.

In fact, I think we don't need to even touch EESSI-extend (or just tweak the check to see if the project directory exists), because we should be able to just pass /cvmfs/dev.eessi.io/example as EESSI_PROJECT_INSTALL in one go from theEESSI-install.sh. I'll give this a try now on the example repo.

ocaisa · 2025-02-17T16:42:11Z

EESSI-extend-easybuild.eb

@@ -106,7 +106,6 @@ elseif (os.getenv("EESSI_SITE_INSTALL") ~= nil) then
  easybuild_installpath = os.getenv("EESSI_SITE_SOFTWARE_PATH")
 else
  -- Deal with user and project installs
-  project_install = os.getenv("EESSI_PROJECT_INSTALL")


This doesn't look right...

Changed in e012bd4. (Still shows up in the review for some reason, maybe because I pushed while a review was open?)

Still here...

EESSI-extend-easybuild.eb

EESSI-install-software.sh

Co-authored-by: ocaisa <alan.ocais@cecam.org>

…-P/software-layer into feature/dev.eessi.io-merge

ocaisa · 2025-02-18T12:27:03Z

create_tarball.sh

+# install to $eessi_version/$eessi_dev_project if defined, otherwise install
+# to $eessi_version
+install_prefix_dir=${eessi_version}${eessi_dev_project:+/$eessi_dev_project}
+echo "Setting Install prefix directory to: ${install_prefix_dir}"
+
+if [ -z ${install_prefix_dir} ]; then
+    echo "\${install_prefix_dir} is empty?!" >&2
+    exit 2
+fi


What is this supposed to match? It doesn't look like it could match the new location /cvmfs/dev.eessi.io/${eessi_dev_project}/versions/...

From the comment, shouldn't it be $eessi_dev_project/versions/$eessi_version

What about keeping the code as was, but updating line 23 such that

cvmfs_repo=${EESSI_CVMFS_REPO}${eessi_dev_project:+/$eessi_dev_project}

ocaisa · 2025-02-18T12:37:16Z

init/minimal_eessi_env

+# If $EESSI_DEV_PROJECT is defined, then we are building for dev.eessi.io
+# and add a subdirectory to $EESSI_PREFIX
+if [ -z ${EESSI_DEV_PROJECT} ]; then
+    export EESSI_PREFIX=$EESSI_CVMFS_REPO/versions/$EESSI_VERSION
+else
+    echo "\$EESSI_DEV_PROJECT: ${EESSI_DEV_PROJECT}; building for dev.eessi.io"
+    export EESSI_PREFIX=$EESSI_CVMFS_REPO_OVERRIDE/versions/$EESSI_VERSION/$EESSI_DEV_PROJECT
+    # Set EESSI_PREFIX to the dev project path for EESSI-extend
+    export EESSI_CVMFS_INSTALL=$EESSI_PREFIX
+fi


Ditto here, you could just tweak the definition of EESSI_CVMFS_REPO_OVERRIDE

Neves-P and others added 30 commits November 1, 2024 11:26

Add EESSI_DEV_PROJET to prefix

8a6cd13

Merge branch 'EESSI:2023.06-software.eessi.io' into feature/dev.eessi.io

ee6aea9

Print EESSI_CVMFS_REPO to debug

0e9402b

Merge branch 'feature/dev.eessi.io' of https://github.com/Neves-P/sof…

8e628be

…tware-layer into feature/dev.eessi.io

Check if running in subshell is the cause of some variables being reset

888b874

Revert to run minimal_eessi_env in subshell and add log line

1af43f1

Fix condition to build with subdir in prefix in dev.eessi.io

25e8a5f

Add some logs to see where EESSI_CVMFS_REPO is being ovewritten

3e5e2af

Add more log output for EESSI_DEV_PROJECT variable

722ca22

export EESSI_DEV_PROJECT to compat layer env

1c3f20a

Expose cvmfs and version override in eessi_defaults

7c9dee8

Also expose EESSI_CVMFS_REPO in defaults

d5d1522

Try exporting EESSI_CVMFS_INSTALL

fb45d7e

Overwrite EASYBUILD_INSTALLPATH and use EESSI_CVMFS_REPO_OVERRIDE in …

a1f11fe

…minimal_eessi_env

Load $EASYBUILD_INSTALLPATH modules in dev.eessi.io

dd9fccd

Use EESSI_DEV_PROJECT for check instead of EESSI_CVMFS_REPO in EASYBU…

821455b

…ILD_INSTALLPATH override

Fix running install_scripts.sh only if not in dev.eessi.io

a19c79c

Remove debug printing to log

afaa94f

Improved logging when $EESSI_DEV_PROJECT is set

2c3e74e

Fix tarball creations in dev.eessi.io

3d9424d

Actually set install_prefix_dir

028f935

Log EESSI_DEV_PROJECT and export it

4f06864

Fix test checking for ${install_prefix_dir}

9eca5e4

Look for installed software in the right directory

0e68f18

Fix error reporting of input in create_tarball.sh

82d36c7

Fix installpath to include cpu target

3248438

Merge branch 'EESSI:2023.06-software.eessi.io' into feature/dev.eessi.io

d162ef3

Track EASYBUILD_EXPERIMENTAL set

4db1083

Fix typo in path

720d6e9

Use $TOPDIR to load_eessi_extend_module.sh

e126f78

Neves-P added 16 commits February 2, 2025 20:38

Revert "Set EESSI_PROJECT_INSTALL in minimal_eessi_env"

228fe3b

This reverts commit a37ae0b.

Revert "Try EESSI_PROJECT_INSTALL without minimal_eessi_env override"

72395ea

This reverts commit 39d67eb.

Revert "Use EESSI_SITE_INSTALL instead"

b5b9916

This reverts commit 28c6b7e.

Revert "Restore right part of path"

00fc025

This reverts commit 3fc95e1.

Revert "Remove more redundant parts of path"

cb41a85

This reverts commit 9f0b2df.

Revert "Shorten EESSI_PROJECT_INSTALL path"

ab7245a

This reverts commit b3399f2.

Revert "Unset EESSI_CVMFS_INSTALL on dev.eessi.io builds"

d9bed4f

This reverts commit 72ba311.

Revert "Fix syntax error"

2dd7650

This reverts commit 35bd8a1.

Revert "Add some debugging infop"

5bb1c56

This reverts commit a4d61d6.

Revert "Set EESSI_DEV_PROJECT via EESSI-extend's EESSI_PROJECT_INSTALL"

dc95dbd

This reverts commit 1ecd93b.

Try again EESSI_PROJECT_INSTALL; temp override EESSI_CVMFS_REPO

e5517b6

Also unset $EESSI_CVMFS_INSTALL

6c9e295

Fix typo in unset

b71e2f9

Need to export EESSI_PROJECT_INSTALL

9211451

Again, use previous apprach

48c61ef

Handle dev.eessi.io via EESSI_extend-easybuild.eb

ee2bcb1

Neves-P requested a review from ocaisa February 14, 2025 18:26

Neves-P marked this pull request as draft February 14, 2025 18:33

Try installing to dev.eessi.io/project/version

e012bd4

Unset EESSI_CVMFS_INSTALL

367e30b

ocaisa requested changes Feb 17, 2025

View reviewed changes

Neves-P commented Feb 18, 2025

View reviewed changes

EESSI-extend-easybuild.eb Show resolved Hide resolved

ocaisa reviewed Feb 18, 2025

View reviewed changes

EESSI-install-software.sh Show resolved Hide resolved

Neves-P and others added 3 commits February 18, 2025 10:34

Create installdir if it doesn't exist

e17d4f3

Co-authored-by: ocaisa <alan.ocais@cecam.org>

Correct place for eessi_dev_project in software_dir in create_tarball.sh

2b96be5

Merge branch 'feature/dev.eessi.io-merge' of https://github.com/Neves…

2e0bc48

…-P/software-layer into feature/dev.eessi.io-merge

ocaisa requested changes Feb 18, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add dev.eessi.io support including project subdirectories #885

Add dev.eessi.io support including project subdirectories #885

Neves-P commented Jan 24, 2025

Neves-P commented Feb 14, 2025

ocaisa commented Feb 17, 2025

Neves-P commented Feb 17, 2025

ocaisa Feb 17, 2025

Neves-P Feb 18, 2025 •

edited

Loading

ocaisa Feb 18, 2025

ocaisa Feb 18, 2025

ocaisa Feb 18, 2025

ocaisa Feb 18, 2025

Add dev.eessi.io support including project subdirectories #885

Are you sure you want to change the base?

Add dev.eessi.io support including project subdirectories #885

Conversation

Neves-P commented Jan 24, 2025

Neves-P commented Feb 14, 2025

ocaisa commented Feb 17, 2025

Neves-P commented Feb 17, 2025

ocaisa Feb 17, 2025

Choose a reason for hiding this comment

Neves-P Feb 18, 2025 • edited Loading

Choose a reason for hiding this comment

ocaisa Feb 18, 2025

Choose a reason for hiding this comment

ocaisa Feb 18, 2025

Choose a reason for hiding this comment

ocaisa Feb 18, 2025

Choose a reason for hiding this comment

ocaisa Feb 18, 2025

Choose a reason for hiding this comment

Neves-P Feb 18, 2025 •

edited

Loading