singularity def issue with switching to easybuild user to build eb package

[shahzebsiddiqui]

Getting weird issue where singularity is creating the user easybuild but it is not switching to easybuild user when reaching eb <easyconfig>.eb --robot line.

As you can see i ran id and tail /etc/passwd and seems also issue with way log output is streamed but easybuild is uid 1000

id
+ id
tail /etc/passwd | grep easybuild
+ grep easybuild
+ tail /etc/passwd
# use EasyBuild to install specified software
eb binutils-2.28.eb --robot
+ eb binutils-2.28.eb --robot
easybuild:x:1000:1000::/home/easybuild:/bin/bash
ERROR: You seem to be running EasyBuild with root privileges which is not wise, so let's end this here.
== temporary log file in case of crash /scratch/tmp/eb-CxVXMk/easybuild-lUdmUu.log
FATAL:   failed to execute %post proc: exit status 1
FATAL:   While performing build: while running engine: while running /usr/local/libexec/singularity/bin/starter: exit status 255

This seemed to happen all of a sudden. I tried testing this in version 3.4.x and 3.5.x of singularity and got same issue. Right now i am using eb version 4.0.x

Singularity.binutils-2.28.txt

build (1).log

[shahzebsiddiqui]

switching to easybuild user is not working

# use EasyBuild to install specified software
eb Anaconda3-5.3.0.eb --robot
+ eb Anaconda3-5.3.0.eb --robot
/bin/eb: line 77: which: command not found
ERROR: You seem to be running EasyBuild with root privileges which is not wise, so let's end this here.
== temporary log file in case of crash /scratch/tmp/eb-6A5vDL/easybuild-4_ubs8.log
FATAL:   failed to execute %post proc: exit status 1
FATAL:   While performing build: while running engine: while running /usr/local/libexec/singularity/bin/starter: exit status 255
FATAL:   While performing build: build image size <= 0

The workaround is to change the following code line

easybuild-framework/easybuild/tools/containers/singularity.py

Line 98 in bf663f7

su - easybuild

to

su - easybuild << 'EOF'

And replace the exit statement

easybuild-framework/easybuild/tools/containers/singularity.py

Line 126 in bf663f7

exit

to

EOF

I wont bother making the change now @boegel as other things will need to be replaced too.

[vanzod]

I stumbled into the same issue.
The solution @shahzebsiddiqui provided works and it is a viable option IMHO.

[boegel]

I'm looking into fixing this, but when I'm switching to su - easybuild << EOF, I'm running into a weird error when updating the Lmod cache, where $LMOD_DIR is defined, but only when checking via env, not when trying to use it. I'm seeing this:

env | grep LMOD_DIR
LMOD_DIR=/usr/share/lmod/lmod/libexec

echo "LMOD_DIR: "
LMOD_DIR:

mkdir -p /app/lmodcache
/update_lmod_system_cache_files -d /app/lmodcache -t /app/lmodcache/timestamp /app/modules/all
-bash: line 27: /update_lmod_system_cache_files: No such file or directory

[shahzebsiddiqui]

I never really understood why we have to update lmod cache inside a container, this was something you added @boegel. If we didn't add this line, were you able to access modules inside the container.

Clearly this filepath /update_lmod_system_cache_files is not valid. I have a base container https://github.com/shahzebsiddiqui/eb-singularity/blob/master/Singularity.Easybuild-1.0 that i used to build containers, this container is published in Sylabs to use as a base image. I kinda stopped putting any effort in this since there was no clear direction what were the next steps.