{devel}[foss/2016b] TensorFlow v1.0.1, SWIG v3.0.10, wheel v0.30.0a0, ... #4412

gppezzi · 2017-03-29T14:52:46Z

(created using eb --new-pr)

wpoely86 · 2017-03-29T14:58:08Z

easybuild/easyconfigs/s/SWIG/SWIG-3.0.10-foss-2016b-Python-3.5.2.eb

+
+name = 'SWIG'
+version = '3.0.10'
+pyver = '3.5.2'


you can drop this, EB provides it.

wpoely86 · 2017-03-29T14:59:01Z

easybuild/easyconfigs/s/SWIG/SWIG-3.0.10-foss-2016b-Python-3.5.2.eb

+    ('PCRE', '8.38'),
+]
+
+# fix for TensorFlow


please explain a bit more? Why does Tensorflow require an rpath in SWIG?

This was an issue on Cray and using older version of Bazel, I've just removed the fix and it seems to work now.

wpoely86 · 2017-03-29T14:59:57Z

easybuild/easyconfigs/t/TensorFlow/TensorFlow-1.0.0-foss-2016b-Python-3.5.2.eb

+
+sanity_check_paths={
+    'files': ['%(builddir)s/tensorflow-%(version)s-cp35-cp35m-linux_x86_64.whl'],
+    'dirs': ['%(installdir)s/lib/python%(pyshortver)s/site-packages'],


you can drop installdir here

wpoely86 · 2017-03-29T15:00:19Z

easybuild/easyconfigs/t/TensorFlow/TensorFlow-1.0.0-foss-2016b-Python-3.5.2.eb

+]
+
+sanity_check_paths={
+    'files': ['%(builddir)s/tensorflow-%(version)s-cp35-cp35m-linux_x86_64.whl'],


why check something in the builddir? it gets deleted at the end?

No idea, I'll blame one of the co-authors 😄
Fixed.

wpoely86 · 2017-03-29T15:00:29Z

easybuild/easyconfigs/t/TensorFlow/TensorFlow-1.0.0-foss-2016b-Python-3.5.2.eb

+}
+
+moduleclass = 'devel'
+


no trailing empty lines

wpoely86 · 2017-03-29T15:02:20Z

easybuild/easyconfigs/t/TensorFlow/tensorflow-v1.0.0-Python-3.5.2-foss.patch

+
+sed -i s\@'$EBROOTGCCCORE'@"$EBROOTGCCCORE"@g third_party/gpus/crosstool/CROSSTOOL.tpl
+sed -i s\@'$EBROOTBINUTILS'@"$EBROOTBINUTILS"@g third_party/gpus/crosstool/CROSSTOOL.tpl
+sed -i s\@'$EBROOTGCCCORE'@"$EBROOTGCCCORE"@g third_party/gpus/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc.tpl


we should put things like this in an easyblock

+1, but thanks for sharing @gppezzi, this is going to be a major help!

I'm not against, but I don't have time to work on this :)

wpoely86 · 2017-03-29T15:02:33Z

easybuild/easyconfigs/t/TensorFlow/tensorflow-v1.0.0-Python-3.5.2-foss.patch

@@ -0,0 +1,103 @@
+diff --git a/configure-cscs.sh b/configure-cscs.sh


claim authorship and explain what you patch does

gppezzi · 2017-03-29T16:04:13Z

easybuild/easyconfigs/t/TensorFlow/TensorFlow-1.0.0-foss-2016b-Python-3.5.2.eb

+# Patch to remove hardcoded paths for GCC and Binutils
+patches = [('tensorflow-v%(version)s-Python-%(pyver)s-foss.patch')]
+
+with_configure = False


does this have any effect on a CmdCp easyblock?

well, CmdCp derives from MakeCp, so it's taken into account, but the default value is already False:

$ eb -a -e CmdCp | grep with_configure with_configure* Run configure script before building [default: False]

gppezzi · 2017-03-29T16:25:13Z

Test report by @gppezzi
SUCCESS
Build succeeded for 5 out of 5 (4 easyconfigs in this PR)
leone2 - Linux RHEL 6.7, Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, Python 2.7.12
See https://gist.github.com/1eefb0944a837f2672ec3fa5be9a4b10 for a full test report.

boegel · 2017-03-29T20:19:32Z

Test report by @boegel
SUCCESS
Build succeeded for 33 out of 33 (4 easyconfigs in this PR)
node2520.golett.os - Linux centos linux 7.3.1611, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Python 2.7.5
See https://gist.github.com/75dbaa98a94f78994c167df3574033fe for a full test report.

boegel · 2017-03-29T20:19:50Z

Test report by @boegel
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in this PR)
node2156.delcatty.os - Linux centos linux 7.3.1611, Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz, Python 2.7.5
See https://gist.github.com/3e20876b4d8d14d680e5821b37ab3d2d for a full test report.

RvDijk · 2017-04-19T11:26:41Z

What is the status on this PR?
Also, is it possible that the protobuf-python module also needs to be loaded in order for this to work?

gppezzi · 2017-04-19T12:02:10Z

Status on this PR: the reviewers suggested to write a custom easyblock for fixing the hardcoded absolute paths (instead of this hack).

I'm not planning to work on this anytime soon, so PRs against my branch are welcome.

@RvDijk did you have problems because of missing protobuf?

I've removed it from the dependency list because Bazel was ignoring it and building a new one inside its own workspace (see fix for protobuf here).

RvDijk · 2017-04-19T12:14:28Z

@gppezzi leaving out the protobuf-python module gives me the following when importing the tensorflow module in Python (after having succesfully installed this tensorflow eb):

>>> import tensorflow as tf
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/software/software/tensorflow/1.0.1-foss-2016a-Python-3.5.2/lib/python3.5/site-packages/tensorflow/__init__.py", line 24, in <module>
from tensorflow.python import *
File "/software/software/tensorflow/1.0.1-foss-2016a-Python-3.5.2/lib/python3.5/site-packages/tensorflow/python/__init__.py", line 75, in <module>
from tensorflow.core.framework.graph_pb2 import *
File "/software/software/tensorflow/1.0.1-foss-2016a-Python-3.5.2/lib/python3.5/site-packages/tensorflow/core/framework/graph_pb2.py", line 6, in <module>
from google.protobuf import descriptor as _descriptor
ImportError: No module named 'google'

And in this case including protobuf-python (3.2.0) fixed it. Could be a local issue if this does not occur anywhere else. Does it work for you without it?

gppezzi · 2017-04-20T08:38:04Z

@RvDijk thanks for reporting this, I'm including protobuf-python as dependency.

Maybe we should write a comment mentioning that this protobuf-python version is not the one used for building TF but it is needed at runtime?

Unless someone wants to find a way to re-use the contents of the bazel workspace 😄

RvDijk · 2017-04-20T09:45:16Z

@gppezzi it would be nice to mention it, otherwise it might confuse others.
I wonder if figuring out the bazel workspace for re-use is worth the time... 😃

boegel · 2017-07-12T21:02:41Z

This may also be useful: https://github.com/truatpasteurdotfr/docker-centos6-tensorflow/blob/master/runme.sh

boegel · 2017-08-12T07:56:33Z

More useful information for building Tensorflow from source with Intel compilers: https://software.intel.com/en-us/articles/tensorflow-optimizations-on-modern-intel-architecture

boegel · 2017-09-15T16:07:35Z

More: https://gist.github.com/taylorpaul/3e4e405ffad1be8355bf742fa24b41f0

boegel · 2017-09-15T18:33:20Z

yet another one: https://github.com/mkandes/naked-singularity/tree/master/definition-files/us/ucsd/sdsc/comet/tensorflow

boegel · 2017-11-07T13:48:58Z

Finally made some progress here... First step: an easyblock for Bazel, see easybuilders/easybuild-easyblocks#1286, plus accompanying easyconfig file for latest Bazel version, see #5311.

bartoldeman · 2017-11-08T16:11:41Z

notes:
https://wiki.calculquebec.ca/w/Tensorflow/en
see
https://wiki.calculquebec.ca/w/Tensorflow_v1.2.x

boegel · 2018-02-20T12:53:22Z

@gppezzi We now have a custom easyblock for TensorFlow, see easybuilders/easybuild-easyblocks#1287, and easyconfigs for building TensorFlow from source, both CPU-only (see #5819) and with GPU support (see #5769).

So this can be closed?

gppezzi · 2018-02-20T15:45:18Z

Sure.

TensorFlow v1.0.0, dependencies and patch

5050b3b

gppezzi changed the title ~~{devel}[foss/2016b] TensorFlow v1.0.0, SWIG v3.0.10, wheel v0.30.0a0, ...~~ WIP {devel}[foss/2016b] TensorFlow v1.0.0, SWIG v3.0.10, wheel v0.30.0a0, ... Mar 29, 2017

wpoely86 reviewed Mar 29, 2017

View reviewed changes

gppezzi added 2 commits March 29, 2017 17:56

TF version bump to 1.0.1

1d5104d

fixed remarks

77630c8

gppezzi commented Mar 29, 2017

View reviewed changes

gppezzi changed the title ~~WIP {devel}[foss/2016b] TensorFlow v1.0.0, SWIG v3.0.10, wheel v0.30.0a0, ...~~ {devel}[foss/2016b] TensorFlow v1.0.1, SWIG v3.0.10, wheel v0.30.0a0, ... Mar 29, 2017

gppezzi mentioned this pull request Mar 29, 2017

Tensorflow easyconfig #4101

Closed

boegel added this to the 3.2.0 milestone Mar 29, 2017

adding protobuf-python as dependency

bddd2cc

boegel modified the milestones: 3.2.0, 3.3.0 May 2, 2017

boegel mentioned this pull request May 31, 2017

Tensorflow lack of optimisation? #4644

Closed

boegel modified the milestones: 3.3.0, 3.4.0 Jun 25, 2017

boegel mentioned this pull request Jul 12, 2017

{devel}[foss/2016b] Tensorflow v1.2.1, protobuf-python v3.3.0, protobuf v3.3.0 #4871

Merged

boegel modified the milestones: 3.5.0, 3.4.0 Sep 6, 2017

boegel modified the milestones: 3.5.0, next release Dec 6, 2017

boegel modified the milestones: 3.5.1, 3.6.0 Jan 11, 2018

gppezzi closed this Feb 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{devel}[foss/2016b] TensorFlow v1.0.1, SWIG v3.0.10, wheel v0.30.0a0, ... #4412

{devel}[foss/2016b] TensorFlow v1.0.1, SWIG v3.0.10, wheel v0.30.0a0, ... #4412

gppezzi commented Mar 29, 2017

wpoely86 Mar 29, 2017

gppezzi Mar 29, 2017

wpoely86 Mar 29, 2017

gppezzi Mar 29, 2017

wpoely86 Mar 29, 2017

gppezzi Mar 29, 2017

wpoely86 Mar 29, 2017

gppezzi Mar 29, 2017

wpoely86 Mar 29, 2017

wpoely86 Mar 29, 2017

boegel Mar 29, 2017

gppezzi Mar 30, 2017

wpoely86 Mar 29, 2017

gppezzi Mar 29, 2017

gppezzi Mar 29, 2017

boegel Mar 29, 2017

gppezzi commented Mar 29, 2017

boegel commented Mar 29, 2017

boegel commented Mar 29, 2017

RvDijk commented Apr 19, 2017

gppezzi commented Apr 19, 2017

RvDijk commented Apr 19, 2017

gppezzi commented Apr 20, 2017

RvDijk commented Apr 20, 2017

boegel commented Jul 12, 2017

boegel commented Aug 12, 2017

boegel commented Sep 15, 2017

boegel commented Sep 15, 2017

boegel commented Nov 7, 2017

bartoldeman commented Nov 8, 2017

boegel commented Feb 20, 2018

gppezzi commented Feb 20, 2018

		@@ -0,0 +1,103 @@
		diff --git a/configure-cscs.sh b/configure-cscs.sh

		}

		moduleclass = 'devel'

{devel}[foss/2016b] TensorFlow v1.0.1, SWIG v3.0.10, wheel v0.30.0a0, ... #4412

{devel}[foss/2016b] TensorFlow v1.0.1, SWIG v3.0.10, wheel v0.30.0a0, ... #4412

Conversation

gppezzi commented Mar 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gppezzi commented Mar 29, 2017

boegel commented Mar 29, 2017

boegel commented Mar 29, 2017

RvDijk commented Apr 19, 2017

gppezzi commented Apr 19, 2017

RvDijk commented Apr 19, 2017

gppezzi commented Apr 20, 2017

RvDijk commented Apr 20, 2017

boegel commented Jul 12, 2017

boegel commented Aug 12, 2017

boegel commented Sep 15, 2017

boegel commented Sep 15, 2017

boegel commented Nov 7, 2017

bartoldeman commented Nov 8, 2017

boegel commented Feb 20, 2018

gppezzi commented Feb 20, 2018