add support for --extended-dry-run/-x (REVIEW) #1388

boegel · 2015-09-11T20:25:16Z

TODO:

Example output:

$ eb make-3.82-GCC-4.8.2.eb -x
== temporary log file in case of crash /tmp/eb-ksVC07/easybuild-MmAugb.log

== processing EasyBuild easyconfig /home/example/eb/easybuild-easyconfigs/easybuild/easyconfigs/m/make/make-3.82-GCC-4.8.2.eb

Important note: the actual build & install procedure that will be performed may diverge
(slightly) from what is outlined below, due to conditions in the easyblock which are
incorrectly handled in a dry run.
Any errors that may occur are ignored and reported as warnings, on a per-step basis.
Please be aware of this, and only use the information below for quick debugging purposes.

*** DRY RUN using 'ConfigureMake' easyblock (easybuild.easyblocks.generic.configuremake @ /home/example/eb/easybuild-easyblocks/easybuild/easyblocks/generic/configuremake.pyc) ***

== building and installing make/3.82-GCC-4.8.2...
fetching files... [DRY RUN]

[fetch_step method]
Available download URLs for sources/patches:
  * http://ftpmirror.gnu.org/make/$source

List of sources:
  * make-3.82.tar.bz2 will be downloaded to /home/example/eb/sources/m/make/make-3.82.tar.bz2

List of patches:
(none)

creating build dir, resetting environment... [DRY RUN]

[check_readiness_step method]

[make_builddir method]

[reset_changes method]

[handle_iterate_opts method]

unpacking... [DRY RUN]

[checksum_step method]
* expected checksum for make-3.82.tar.bz2: (none)

[extract_step method]
  running command "tar xjf /home/example/eb/sources/m/make/make-3.82.tar.bz2"
  (in /home/example/eb/build/make/3.82/GCC-4.8.2)

patching... [DRY RUN]

[patch_step method]

preparing... [DRY RUN]

[prepare_step method]
Defining build environment, based on toolchain (options) and specified dependencies...

Loading toolchain module...

module load GCC/4.8.2

Loading modules for dependencies...


Full list of loaded modules:
  1) GCC/4.8.2

Defining build environment...

  export CC="gcc"
  export CFLAGS="-O2"
  export CXX="g++"
  export CXXFLAGS="-O2"
  export F77="gfortran"
  export F90="gfortran"
  export F90FLAGS="-O2"
  export FFLAGS="-O2"
  export FLIBS="-lgfortran"
  export LDFLAGS="-L/home/example/eb/software/GCC/4.8.2/lib"
  export LIBS="-lm -lpthread"
  export OPTFLAGS="-O2"
  export PRECFLAGS=""

configuring... [DRY RUN]

[configure_step method]
  running command " ./configure --prefix=/home/example/eb/software/make/3.82-GCC-4.8.2 "
  (in /home/example/eb/build/make/3.82/GCC-4.8.2/make-3.82)

building... [DRY RUN]

[build_step method]
  running command " make -j 4 "
  (in /home/example/eb/build/make/3.82/GCC-4.8.2/make-3.82)

testing... [DRY RUN]

[test_step method]

installing... [DRY RUN]

[stage_install_step method]

[make_installdir method]

[install_step method]
  running command " make install "
  (in /home/example/eb/build/make/3.82/GCC-4.8.2/make-3.82)

taking care of extensions... [DRY RUN]

[extensions_step method]

postprocessing... [DRY RUN]

[post_install_step method]

sanity checking... [DRY RUN]

[sanity_check_step method]
Sanity check paths - file ['files']
  * bin/make
Sanity check paths - (non-empty) directory ['dirs']
  (none)
Sanity check commands
  (none)

cleaning up... [DRY RUN]

[cleanup_step method]

creating module... [DRY RUN]

[make_module_step method]
Generating module file /home/example/eb/modules/all/make/3.82-GCC-4.8.2, with contents:

    #%Module
    proc ModulesHelp { } {
        puts stderr { make-3.82: GNU version of make utility - Homepage: http://www.gnu.org/software/make/make.html
        }
    }

    module-whatis {Description: make-3.82: GNU version of make utility - Homepage: http://www.gnu.org/software/make/make.html}

    set root /home/example/eb/software/make/3.82-GCC-4.8.2

    conflict make

    if { ![ is-loaded GCC/4.8.2 ] } {
        module load GCC/4.8.2
    }

    setenv  EBROOTMAKE      "$root"
    setenv  EBVERSIONMAKE       "3.82"
    setenv  EBDEVELMAKE     "$root/easybuild/make-3.82-GCC-4.8.2-easybuild-devel"

    # Built with EasyBuild version 2.4.0


permissions... [DRY RUN]

[permissions_step method]

packaging... [DRY RUN]

[package_step method]

== COMPLETED: Installation ended successfully

Important note: the actual build & install procedure that will be performed may diverge
(slightly) from what is outlined above, due to conditions in the easyblock which are
incorrectly handled in a dry run.
Any errors that may occur are ignored and reported as warnings, on a per-step basis.
Please be aware of this, and only use the information above for quick debugging purposes.

(no ignored errors during dry run)

== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-ksVC07/easybuild-MmAugb.log* have been removed.
== Temporary directory /tmp/eb-ksVC07 has been removed.

hpcugentbot · 2015-09-11T20:33:29Z

EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2032/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

pforai · 2015-09-11T20:41:40Z

woot! great visibility improvement. totally dig this. 👍

hpcugentbot · 2015-09-11T20:59:47Z

EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2033/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

hpcugentbot · 2015-09-11T22:01:33Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2034/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

hpcugentbot · 2015-09-11T22:44:54Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2035/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

hpcugentbot · 2015-09-11T23:26:54Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2036/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

hpcugentbot · 2015-09-12T00:10:20Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2037/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

hpcugentbot · 2015-09-12T00:52:52Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2038/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

hpcugentbot · 2015-09-12T01:36:12Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2039/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

hpcugentbot · 2015-09-12T02:20:57Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2040/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

verdurin · 2015-09-12T07:39:18Z

Looks very useful.

Occurs to me that showing where files would be fetched from would be a good addition.

boegel · 2015-09-12T08:50:44Z

@verdurin: you mean download URL and path where it'll be downloaded to?

verdurin · 2015-09-12T09:00:08Z

@boegel - exactly, yes.

pescobar · 2015-09-12T09:57:25Z

this looks really useful.

I would add to verdurin comment that if the source tarball is found locally it would be nice to show the full path to the source file.

boegel · 2015-09-12T10:52:54Z

I think I'll go with something like:

[fetch_step method]
List of required files:
* make-3.82.tar.bz2 found at /home/kehoste/easybuild/sources/make-3.82.tar.bz2

vs

[fetch_step method]
List of required files:
* make-3.82.tar.bz2 downloaded from http://ftpmirror.gnu.org/make/make-3.82.tar.bz2 to /home/kehoste/easybuild/sources/make-3.82.tar.bz2

verdurin · 2015-09-13T06:44:17Z

Yes, that's what's needed here.

hpcugentbot · 2015-10-28T07:26:02Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2243/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

hpcugentbot · 2015-10-28T17:06:49Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2249/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

@rjeschmi

…wing suggesting by @rjeschmi

…an unknown build option *should* fail

…upport for specifying a default fallback value

… cleanups

boegel · 2015-10-28T23:08:32Z

I gave this another full review myself, and added some final minor tweaks.

Since the documentation is also update (see easybuilders/easybuild#159), I feel this is ready to merge as soon as Jenkins gives the green light.

Thanks to everyone who contributed to this in one way or another, in my view this is going to be a killer feature.

hpcugentbot · 2015-10-28T23:36:20Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2257/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

add support for --extended-dry-run/-x (REVIEW)

boegel added 4 commits September 11, 2015 19:08

move set_tmpdir from tools/config.py to tools/options.py

035b1c7

defined --extended-dry-run configure/build option

f93d7dd

add support for specifying default value for (missing) build option

d82c0b7

implement support for extended dry run

8038ac3

dry run for patch_step

7f68850

boegel added 3 commits September 11, 2015 23:19

fix infinite recursion in extract_step

0362ebb

don't use dry run version of extract_step + tweak extract_file function

7dec3ad

dry run for patch_perl_script_autoflush

9b41c6b

boegel added 4 commits September 12, 2015 00:06

add apply_regex_substitutions function

b21c3ea

dry run for apply_regex_substitutions

f363dd8

catch errors and continue under --extended-dry-run

71f5f57

catch all exceptions under --extended-dry-run

ffc690e

boegel mentioned this pull request Sep 11, 2015

use apply_regex_substitutions in WRF easyblock easybuilders/easybuild-easyblocks#685

Merged

boegel added this to the v2.4.0 milestone Sep 13, 2015

boegel mentioned this pull request Oct 27, 2015

Add support for 'whatis' easyconfig parameter #1271

Merged

move printing list of loaded modules to toolchain.prepare

cec8082

add self.dry_run_msg method in EasyBlock class

2256baa

avoid double quotes when ignoring errors

07f1d71

boegel mentioned this pull request Oct 28, 2015

add support for using regular expressions in 'patches' #1400

Open

rename 'forced' named argument of run_cmd to 'force_in_dry_run' follo…

4586c08

…wing suggesting by @rjeschmi

boegel mentioned this pull request Oct 28, 2015

use apply_regex_substitutions in easyblocks, where appropriate easybuilders/easybuild-easyblocks#725

Open

26 tasks

boegel added 11 commits October 28, 2015 21:55

Merge branch 'develop' into extended_dry_run

4c079b2

fix typo

ee95ee1

don't provide default fallback value in build_option method, quering …

6578d04

…an unknown build option *should* fail

ensure clear error message when quering undefined build option, add s…

1bd4f64

…upport for specifying a default fallback value

avoid inline return in filetools.py functions

efed3d7

remove unused named argument 'silent' in ModulesTool.load() method

6b798c8

provide default fallback value for defining Toolchain.dry_run + minor…

5c7bd24

… cleanups

make sure extended_dry_run build option is always defined

87da9c6

ensure clear error message for failing test in test__list_toolchains

aac7b2e

fix .load() statements in toolchain.py, no named argument 'silent'

59444af

fix broken test

81199ce

boegel added a commit that referenced this pull request Oct 28, 2015

Merge pull request #1388 from boegel/extended_dry_run

1a14ebf

add support for --extended-dry-run/-x (REVIEW)

boegel merged commit 1a14ebf into easybuilders:develop Oct 28, 2015

boegel deleted the extended_dry_run branch October 28, 2015 23:38

boegel mentioned this pull request Jan 23, 2016

add support for --dump-toolchain-env #1091

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add support for --extended-dry-run/-x (REVIEW) #1388

add support for --extended-dry-run/-x (REVIEW) #1388

boegel commented Sep 11, 2015

hpcugentbot commented Sep 11, 2015

pforai commented Sep 11, 2015

hpcugentbot commented Sep 11, 2015

hpcugentbot commented Sep 11, 2015

hpcugentbot commented Sep 11, 2015

hpcugentbot commented Sep 11, 2015

hpcugentbot commented Sep 12, 2015

hpcugentbot commented Sep 12, 2015

hpcugentbot commented Sep 12, 2015

hpcugentbot commented Sep 12, 2015

verdurin commented Sep 12, 2015

boegel commented Sep 12, 2015

verdurin commented Sep 12, 2015

pescobar commented Sep 12, 2015

boegel commented Sep 12, 2015

verdurin commented Sep 13, 2015

hpcugentbot commented Oct 28, 2015

hpcugentbot commented Oct 28, 2015

boegel commented Oct 28, 2015

hpcugentbot commented Oct 28, 2015

add support for --extended-dry-run/-x (REVIEW) #1388

add support for --extended-dry-run/-x (REVIEW) #1388

Conversation

boegel commented Sep 11, 2015

hpcugentbot commented Sep 11, 2015

pforai commented Sep 11, 2015

hpcugentbot commented Sep 11, 2015

hpcugentbot commented Sep 11, 2015

hpcugentbot commented Sep 11, 2015

hpcugentbot commented Sep 11, 2015

hpcugentbot commented Sep 12, 2015

hpcugentbot commented Sep 12, 2015

hpcugentbot commented Sep 12, 2015

hpcugentbot commented Sep 12, 2015

verdurin commented Sep 12, 2015

boegel commented Sep 12, 2015

verdurin commented Sep 12, 2015

pescobar commented Sep 12, 2015

boegel commented Sep 12, 2015

verdurin commented Sep 13, 2015

hpcugentbot commented Oct 28, 2015

hpcugentbot commented Oct 28, 2015

boegel commented Oct 28, 2015

hpcugentbot commented Oct 28, 2015