add support for --include-* (REVIEW) #1301

boegel · 2015-06-18T14:16:33Z

Add support to easily include additional custom easyblocks, module naming schemes and toolchains/toolchain components, without having to fiddle with $PYTHONPATH or Python packaging magic (__init__.py), etc.

TODO list:

--include-easyblocks
--include-module-naming-schemes
--include-toolchains
unit tests
- include_* functions
- --include-* options combined with --list-toolchains, --list-easyblocks, --avail-module-naming-schemes
updated documentation (see document --include (REVIEW) easybuild#129)

note: requires easybuilders/easybuild-easyblocks#617 in order to work correctly

hpcugentbot · 2015-06-18T14:16:59Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1785/
EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1785/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

boegel · 2015-06-18T14:18:56Z

Currently only the --include-easyblocks part works:

$ eb --list-easyblocks | grep foo

$ eb --include-easyblocks=$PWD/foo.py,$PWD/generic/foobar.py --list-easyblocks | grep foo
|-- EB_foo
|-- foobar

If also enhanced the output of --list-easyblocks=detailed:

$ eb --include-easyblocks=$PWD/foo.py,$PWD/generic/foobar.py --list-easyblocks=detailed | grep foo
|-- EB_foo (easybuild.easyblocks.foo @ /var/folders/8s/_frgh9sj6m744mxt5w5lyztr0000gn/T/eb-yFeTNf/included-easyblocks/easybuild/easyblocks/foo.py)
|-- foobar (easybuild.easyblocks.generic.foobar @ /var/folders/8s/_frgh9sj6m744mxt5w5lyztr0000gn/T/eb-yFeTNf/included-easyblocks/easybuild/easyblocks/generic/foobar.py)

wpoely86 · 2015-06-18T14:22:21Z

easybuild/main.py

@@ -163,9 +163,6 @@ def main(testing_data=(None, None, None)):
        new_umask = int(options.umask, 8)
        old_umask = os.umask(new_umask)

-    # set temporary directory to use
-    eb_tmpdir = set_tmpdir(options.tmpdir)


why this change?

I need a temporary directory to symlink the included easyblocks into the correct namespace.

When --include-easyblocks is combined with --list-easyblocks, this point is never reached (--list-easyblocks is handled via tools/options.py, and so we don't return back to main.py after parsing options).

I now set the temporary directory as a part of postprocessing the parsed options, so earlier than I was here. Should be fine (can't hurt to do it earlier).

boegel · 2015-06-18T14:22:46Z

When combined with easybuilders/easybuild-easyblocks#617, this also reflects easyblocks being overridden:

$ eb --include-easyblocks=$PWD/foo.py,$PWD/gcc.py,$PWD/generic/\*.py --list-easyblocks=detailed | grep -C 2 GCC
|   |-- EB_g2clib (easybuild.easyblocks.g2clib @ /Users/kehoste/work/easybuild-easyblocks/easybuild/easyblocks/g/g2clib.py)
|   |-- EB_g2lib (easybuild.easyblocks.g2lib @ /Users/kehoste/work/easybuild-easyblocks/easybuild/easyblocks/g/g2lib.py)
|   |-- EB_GCC (easybuild.easyblocks.gcc @ /var/folders/8s/_frgh9sj6m744mxt5w5lyztr0000gn/T/eb-5Zqzj1/included-easyblocks/easybuild/easyblocks/gcc.py)
|   |-- EB_GHC (easybuild.easyblocks.ghc @ /Users/kehoste/work/easybuild-easyblocks/easybuild/easyblocks/g/ghc.py)
|   |-- EB_Go (easybuild.easyblocks.go @ /Users/kehoste/work/easybuild-easyblocks/easybuild/easyblocks/g/go.py)


$ eb --list-easyblocks=detailed | grep -C 2 GCC
|   |-- EB_g2clib (easybuild.easyblocks.g2clib @ /Users/kehoste/work/easybuild-easyblocks/easybuild/easyblocks/g/g2clib.py)
|   |-- EB_g2lib (easybuild.easyblocks.g2lib @ /Users/kehoste/work/easybuild-easyblocks/easybuild/easyblocks/g/g2lib.py)
|   |-- EB_GCC (easybuild.easyblocks.gcc @ /Users/kehoste/work/easybuild-easyblocks/easybuild/easyblocks/g/gcc.py)
|   |-- EB_GHC (easybuild.easyblocks.ghc @ /Users/kehoste/work/easybuild-easyblocks/easybuild/easyblocks/g/ghc.py)
|   |-- EB_Go (easybuild.easyblocks.go @ /Users/kehoste/work/easybuild-easyblocks/easybuild/easyblocks/g/go.py)

fgeorgatos · 2015-06-18T14:34:26Z

easybuild/tools/include.py

+__path__ = extend_path(__path__, __name__)
+"""
+
+


any double space to trim above, all is intentional?

we separate constants from functions (and top-level functions between each other) with a double newline

fgeorgatos · 2015-06-18T14:41:49Z

first visual review ok. very interesting PR, I am sure it will get quickly popular.

I hope the lack of --include-module-naming-schemes, --include-toolchains will not delay merging it in before 2.2.0 (if so, just turn the latter two into experimental facilities).

boegel · 2015-06-18T14:46:55Z

@fgeorgatos: the idea is to get some early feedback, finishing the other two options shouldn't be that hard, with what's in place already (it pretty analogous)

…lude-toolchains + refactor

boegel · 2015-06-20T05:54:14Z

implementation finished, first stab at dedicated unit tests for include_* functions included

should also add tests for --include-* options combined with --list-toolchains, --list-easyblocks, --avail-module-naming-schemes

hpcugentbot · 2015-06-20T05:59:59Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1788/
EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1788/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

hpcugentbot · 2015-06-22T10:02:29Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1789/
EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1789/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

JensTimmerman · 2015-06-25T15:26:41Z

easybuild/tools/include.py

+        handle = open(init_path, 'w')
+        handle.write(PKG_INIT_BODY)
+        handle.close()
+    except (IOError, OSError) as err:


this is fine,
but it's sometimes easier to have
with open(newfile, 'w') as handle:

yeah, I keep forgetting we're no longer stuck to Python 2.4 ;)

…* options

hpcugentbot · 2015-06-27T19:32:13Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1800/
EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1800/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

boegel · 2015-06-29T10:00:43Z

@JensTimmerman: please rereview

JensTimmerman · 2015-06-29T15:10:11Z

ok

hpcugentbot · 2015-07-03T14:43:07Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1838/
EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1838/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

hpcugentbot · 2015-07-03T15:32:16Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1842/
EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1842/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

hpcugentbot · 2015-07-03T16:52:56Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1847/
EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1847/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

hpcugentbot · 2015-07-06T09:37:13Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1848/
EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1848/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

boegel · 2015-07-08T07:01:22Z

OK, I'm calling this, merging to be included in EB v2.2.0.

All TODO's are ticked off, extensive unit tests are in place, and a documentation update is ready to go as well (see easybuilders/easybuild#129).

Thanks for the feedback on this @fgeorgatos, @JensTimmerman, @wpoely86 and @rjeschmi!

add support for --include-* (REVIEW)

add support for --include-*

2362b20

wpoely86 reviewed Jun 18, 2015
View reviewed changes

fgeorgatos reviewed Jun 18, 2015
View reviewed changes

boegel added 3 commits June 19, 2015 13:48

guard __import__ statement in import_available_modules for ImportErrors

2516973

complete implementation for --include-module-naming-schemes and --inc…

6ea369a

…lude-toolchains + refactor

add unit tests for --include-*

9654529

don't redefine options.tmpdir, fix broken tests

ba66131

only set up include symlinks if something to include has been specified

b2bd260

boegel mentioned this pull request Jun 22, 2015

tweak easybuild.easyblocks package declaration to better support custom easyblocks repo easybuilders/easybuild-easyblocks#617

Merged

boegel modified the milestone: v2.2 Jun 24, 2015

JensTimmerman reviewed Jun 25, 2015
View reviewed changes

boegel added 2 commits June 26, 2015 15:49

reorganize test easyblocks to mimic easybuild-easyblocks setup

9d9bd3d

fix remarks + enhance and extend unit tests, to also cover --include-…

2616f46

…* options

boegel changed the title ~~add support for --include-* (WIP)~~ add support for --include-* (REVIEW) Jun 29, 2015

verify whether import of included modules works as expected

d73f2c6

fix broken test

9439f96

boegel added 2 commits July 3, 2015 17:35

Merge branch 'develop' into include

429e01c

make sure custom foo easyblock is 'unloaded' again

8b02f1b

fix comment

6f0c94a

boegel added a commit that referenced this pull request Jul 8, 2015

Merge pull request #1301 from boegel/include

49c0f3c

add support for --include-* (REVIEW)

boegel merged commit 49c0f3c into easybuilders:develop Jul 8, 2015

boegel deleted the include branch July 8, 2015 07:01

boegel mentioned this pull request Jul 8, 2015

document --include (REVIEW) easybuilders/easybuild#129

Merged

boegel mentioned this pull request Jul 15, 2015

add support for specyfing location of easyblocks via EasyBuild configuration #1010

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add support for --include-* (REVIEW) #1301

add support for --include-* (REVIEW) #1301

boegel commented Jun 18, 2015

hpcugentbot commented Jun 18, 2015

boegel commented Jun 18, 2015

wpoely86 Jun 18, 2015

boegel Jun 18, 2015

boegel commented Jun 18, 2015

fgeorgatos Jun 18, 2015

boegel Jun 18, 2015

fgeorgatos commented Jun 18, 2015

boegel commented Jun 18, 2015

boegel commented Jun 20, 2015

hpcugentbot commented Jun 20, 2015

hpcugentbot commented Jun 22, 2015

JensTimmerman Jun 25, 2015

boegel Jun 25, 2015

hpcugentbot commented Jun 27, 2015

boegel commented Jun 29, 2015

JensTimmerman commented Jun 29, 2015

hpcugentbot commented Jul 3, 2015

hpcugentbot commented Jul 3, 2015

hpcugentbot commented Jul 3, 2015

hpcugentbot commented Jul 6, 2015

boegel commented Jul 8, 2015

		__path__ = extend_path(__path__, __name__)
		"""

add support for --include-* (REVIEW) #1301

add support for --include-* (REVIEW) #1301

Conversation

boegel commented Jun 18, 2015

hpcugentbot commented Jun 18, 2015

boegel commented Jun 18, 2015

wpoely86 Jun 18, 2015

Choose a reason for hiding this comment

boegel Jun 18, 2015

Choose a reason for hiding this comment

boegel commented Jun 18, 2015

fgeorgatos Jun 18, 2015

Choose a reason for hiding this comment

boegel Jun 18, 2015

Choose a reason for hiding this comment

fgeorgatos commented Jun 18, 2015

boegel commented Jun 18, 2015

boegel commented Jun 20, 2015

hpcugentbot commented Jun 20, 2015

hpcugentbot commented Jun 22, 2015

JensTimmerman Jun 25, 2015

Choose a reason for hiding this comment

boegel Jun 25, 2015

Choose a reason for hiding this comment

hpcugentbot commented Jun 27, 2015

boegel commented Jun 29, 2015

JensTimmerman commented Jun 29, 2015

hpcugentbot commented Jul 3, 2015

hpcugentbot commented Jul 3, 2015

hpcugentbot commented Jul 3, 2015

hpcugentbot commented Jul 6, 2015

boegel commented Jul 8, 2015