environment and repo issues w/ eCLM

[mvhulten]

When I follow the workflow, src/TSMP2 is pulled in. From there one must ./build_tsmp2.sh --eclm to build just eCLM. This fails of course when I do not load modules, which I will do on the next try.

I think we should try standardise on Stages/2024 (almost DONE in TSMP2 repo; TODO here).

But trying to check consistency between repo, HEAD and environment, I see a detached HEAD and am unsure what is going on and what I should do:

vanhulten1@jrlogin09:.../wfe-test/src/TSMP2$ git branch -v
* (HEAD detached at d61a0d4) d61a0d4 change submodule remote url of COSMO
  master                     15ff6eb build_tsmp2: correct check for alternative `parflow` (#41)
vanhulten1@jrlogin09:.../wfe-test/src/TSMP2$ git show -v
commit d61a0d49d418d9eb3040b293ad389435060d0ca8 (HEAD)
Author: Stefan Poll <s.poll@fz-juelich.de>
Date:   Fri Sep 13 10:44:36 2024 +0200

    change submodule remote url of COSMO

diff --git a/.gitmodules b/.gitmodules
index e6b1861..d31661e 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -28,5 +28,5 @@
        branch = tsmp-patches-v0.1
 [submodule "models/cosmo5.01_fresh"]
        path = models/cosmo5.01_fresh
-       url = https://icg4geo.icg.kfa-juelich.de/ModelSystems/tsmp_src/cosmo5.01_fresh.git
+       url = https://gitlab.jsc.fz-juelich.de/sdlts/tsmp_components_mirrors/cosmo5.01_fresh.git
        branch = tsmp-oasis
vanhulten1@jrlogin09:.../wfe-test/src/TSMP2$ git pull
You are not currently on a branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.

    git pull <remote> <branch>

vanhulten1@jrlogin09:.../wfe-test/src/TSMP2$ git remote -v
origin  https://github.com/HPSCTerrSys/TSMP2.git (fetch)
origin  https://github.com/HPSCTerrSys/TSMP2.git (push)
vanhulten1@jrlogin09:.../wfe-test/src/TSMP2$ git pull origin master
From https://github.com/HPSCTerrSys/TSMP2
 * branch            master     -> FETCH_HEAD
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint: 
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint: 
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.

[mvhulten]

Commit d61a0d49d418d9eb3040b293ad389435060d0ca0 is not part of the TSMP2 repo.

Commit d61a0d49d418d9eb3040b293ad389435060d0ca8 is (HPSCTerrSys/TSMP2@d61a0d4) part of the TSMP2 repo.

Where does this commit come from?

[jjokella]

Hi @mvhulten the detached HEAD is something that has been bugging me as well. If someone has a nice way of dealing with this, this would help.

What I do: Check the commit hash, find the commit hash in the history of the main branch and then checkout a branch at this commit.

[jjokella]

Ok, I guess you just now did the same thing. Try this link: HPSCTerrSys/TSMP2@d61a0d49d4 Somehow too much detail in the hash has made the link go bad.

OK, the commit does not belong to any branch.

This happens, when the commit has been changed / rebased / squashed after it was saved in the submodule (for example if the submodule was specified during development of a PR and the PR was afterwards squashed).

So the task would be to specify the corresponding commit in the history of TSMP2 that best fits (e.g. the squashed PR commit).

[mvhulten]

Right, thanks for the clarification @jjokella!

I messed up including a really non-existing hash (used to test!). Fixed comment above.

In strict Git lingo, it is part of the repo but not of any branch (because of explanation you gave).

[jjokella]

In strict Git lingo, it is part of the repo but not of any branch (because of explanation you gave).

Fixed in my comment above. Thanks.

[mvhulten]

@s-poll, in .gitmodules the TSMP2 module is at branch dev_frontend. Why is this?

I'll presume it is not important and I it should be master that soon has the changes from stages/2024, but please confirm if this is ok.

[s-poll]

Indeed the TSMP2 submodule should be updated after stages/2024 to the master. The WFE will take automatically the environment, which is used in build_tsmp2.sh (see find command here).

Detached head are occurring as the submodules are at a specific commit.

However, I am wondering why your eCLM build fails. There should be no need to load environments as it should be taken by default.

[mvhulten]

No, that line doesn't do that. It puts instead all file names in a variable. Later in sim_run.sh the source internal command is called and the first file in the variable is sourced (and other strings in the array are passed as arguments—not used in those source files (and I have never seen that anyway)).

Funny, the actual behaviour you may not notice. But let's make this work in theory, not just in practice.

Loading the environment in both TSMP2 and the WFE must be handled in a more controlled way.

[s-poll]

Actually, there should be only one file as a result otherwise the build is not clean. We could add a warning/abort message when there is more than one file listened.

Please be aware that you should clean-up (remove the bin & bld directory) when using TSMP2 build with a new environment as mentioned in PR24 of TSMP2.

[mvhulten]

First, a short correction about detached heads.

Detached heads are commits that are not associated with any branch. In principle it is okay if a submodule are not on a branch, but it is… quirky and maybe sometimes more difficult to make sense of.

[mvhulten]

I think I do see what you intend with displaying an abort message when there are multiple environment files. I suppose this hangs on the idea that TSMP2 should provide no more than one environment file, because then it is univocal in what environment is to be used.

However, this is the WFE point of view, not per se that of TSMP2. Practically, it can be great, at least for TSMP2, to provide several environment variables (but others may comment on that), and this is fine, because it is the task only for the WFE to make a simulation unambiguously reproducible.

I propose to accept that TSMP2 may provide multiple environment files and that the WFE picks one; that is really a task of the WFE. WDYT?

[kvrigor]

I see a detached HEAD and am unsure what is going on and what I should do

Unfortunately we can't do anything about DETACHED heads since git submodule work this way. It clones a "sub-repo" and then pins it to a specific version by doing a git checkout $COMMIT_HASH. Note that running git checkout $COMMIT_HASH always ends up with a DETACHED head.

In the context of submodules, I interpret a DETACHED head as "tampering with this sub-repo is not encouraged". I believe this aligns with the reproducibility principle of the WFE. If you want to ensure that build_tsmp2.sh builds the correct version most up-to-date version of components, you can run these:

# Make sure all models/* are updated
git submodule update --init --remote
git submodule update --force

# Check if sub-repo versions are correct, which means
# no diffs under models/* 
git status

I propose to accept that TSMP2 may provide multiple environment files and that the WFE picks one; that is really a task of the WFE. WDYT?

We're planning to do this on the Stages/2025 PR, which would support multiple compiler toolchains and hence the possibility of multiple environment files.

[s-poll]

IMO the provision of several environment files in TSMP2 is desired and was already discussed in this TSMP2 PR. However, the WFE should only take the one, which was used for building the model system. Thus, this number should always be always one. The find-command for the environment is done in the directory of the binaries.

For the git submodules I have nothing to add to Pauls statement, except that I would not call it "correct version of components" but rather most up-to-date version.

[mvhulten]

You're right. I was just confusing things in my last comment.

Now (with Stages/2024) eCLM cloning and compilation shows no problems anymore for me.