Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes to support building R-keras and dependencies on POWER9 #9544

Closed

Conversation

@edmondac edmondac changed the title Changes to support building R-keras and dependencies on POWER9 WIP: Changes to support building R-keras and dependencies on POWER9 Dec 18, 2019
@edmondac edmondac changed the title WIP: Changes to support building R-keras and dependencies on POWER9 Changes to support building R-keras and dependencies on POWER9 Dec 18, 2019
@edmondac
Copy link
Contributor Author

Patch for R base 3.6.2 from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=946836

@edmondac
Copy link
Contributor Author

@edmondac edmondac changed the title Changes to support building R-keras and dependencies on POWER9 WIP: Changes to support building R-keras and dependencies on POWER9 Dec 19, 2019
@edmondac
Copy link
Contributor Author

I took the config.guess from the easybuild ConfigureMake sources

@boegelbot
Copy link
Collaborator

Travis test report: 2/2 runs failed - see https://travis-ci.org/easybuilders/easybuild-easyconfigs/builds/627723995

Only showing partial log for 1st failed test suite run 18994.1;
full log at https://travis-ci.org/easybuilders/easybuild-easyconfigs/jobs/627723996

...
ERROR: test__parse_easyconfig_Mesa-19.1.7-GCCcore-8.3.0.eb (test.easyconfigs.easyconfigs.EasyConfigTest)
Test for parsing of easyconfig Mesa-19.1.7-GCCcore-8.3.0.eb
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 1, in innertest
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 745, in template_easyconfig_test
    ecs = process_easyconfig(spec)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1791, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: "Failed to process easyconfig /home/travis/build/easybuilders/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesa/Mesa-19.1.7-GCCcore-8.3.0.eb: No software-specific easyblock 'EB_Mesa' found for Mesa"

======================================================================
ERROR: test__parse_easyconfig_Mesa-19.2.1-GCCcore-8.3.0.eb (test.easyconfigs.easyconfigs.EasyConfigTest)
Test for parsing of easyconfig Mesa-19.2.1-GCCcore-8.3.0.eb
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 1, in innertest
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 745, in template_easyconfig_test
    ecs = process_easyconfig(spec)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1791, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: "Failed to process easyconfig /home/travis/build/easybuilders/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesa/Mesa-19.2.1-GCCcore-8.3.0.eb: No software-specific easyblock 'EB_Mesa' found for Mesa"

======================================================================
ERROR: test_dep_versions_per_toolchain_generation (test.easyconfigs.easyconfigs.EasyConfigTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 413, in test_dep_versions_per_toolchain_generation
    self.process_all_easyconfigs()
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 119, in process_all_easyconfigs
    EasyConfigTest.ordered_specs = resolve_dependencies(EasyConfigTest.parsed_easyconfigs, modules_tool(), retain_all_deps=True)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/tools/robot.py", line 436, in resolve_dependencies
    processed_ecs = process_easyconfig(path, validate=not retain_all_deps, hidden=hidden)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1791, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: "Failed to process easyconfig /home/travis/build/easybuilders/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesa/Mesa-19.1.7-GCCcore-8.3.0.eb: No software-specific easyblock 'EB_Mesa' found for Mesa"

======================================================================
ERROR: test_sanity_check_paths (test.easyconfigs.easyconfigs.EasyConfigTest)
Make sure specified sanity check paths adher to the requirements.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 476, in test_sanity_check_paths
    self.process_all_easyconfigs()
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 119, in process_all_easyconfigs
    EasyConfigTest.ordered_specs = resolve_dependencies(EasyConfigTest.parsed_easyconfigs, modules_tool(), retain_all_deps=True)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/tools/robot.py", line 436, in resolve_dependencies
    processed_ecs = process_easyconfig(path, validate=not retain_all_deps, hidden=hidden)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1791, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: "Failed to process easyconfig /home/travis/build/easybuilders/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesa/Mesa-19.1.7-GCCcore-8.3.0.eb: No software-specific easyblock 'EB_Mesa' found for Mesa"

======================================================================
FAIL: test_changed_files_pull_request (test.easyconfigs.easyconfigs.EasyConfigTest)
Specific checks only done for the (easyconfig) files that were changed in a pull request.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 713, in test_changed_files_pull_request
    self.assertTrue(False, error_msg)
AssertionError: Failed to find parsed easyconfig for Mesa-19.1.7-GCCcore-8.3.0.eb (and could not isolate it in easyconfigs archive either)

----------------------------------------------------------------------
Ran 8851 tests in 280.340s

FAILED (failures=1, errors=4)
ERROR: Not all tests were successful.

*bleep, bloop, I'm just a bot (boegelbot v20180813.01)*Please talk to my owner @boegel if you notice you me acting stupid),or submit a pull request to https://github.com/boegel/boegelbot fix the problem.

@edmondac
Copy link
Contributor Author

R builds on POWER9 now, with the exception of extensions OpenCL and gpuR - which require the OpenCL headers and library. On Intel they ship in CUDA, but the CUDA ppc64le .run file doesn't include OpenCL. I'm talking to IBM about where we should get OpenCL on ppc64le...

@branfosj
Copy link
Member

@zao
Copy link
Contributor

zao commented Dec 22, 2019

@edmondac By "Intel" do you mean actual Intel as in not even on AMD processors, or just regular old boring x86-64?

It's hard to tell at a glance which one you is meant when skimming through the quite rapid feed of EasyBuild mails.

@branfosj
Copy link
Member

branfosj commented Dec 31, 2019

@edmondac #9593 adds pocl 1.4. I've done a test with the CUDA version and it allows the gpuR and OpenCL extensions to install.

The log shows the following at the end of installing gpuR, which is certainly encouraging:

** testing if installed package can be loaded from final location
Number of platforms: 1
- platform: The pocl project: OpenCL 1.2 pocl 1.4, RelWithDebInfo, LLVM 9.0.1, RELOC, SLEEF, CUDA, POCL_DEBUG
  - context device index: 0
    - pthread-POWER9, altivec supported
  - context device index: 1
    - Tesla V100-SXM2-16GB
checked all devices
completed initialization

@edmondac edmondac closed this Jan 6, 2020
@edmondac edmondac reopened this Jan 6, 2020
@edmondac
Copy link
Contributor Author

edmondac commented Jan 6, 2020

@edmondac By "Intel" do you mean actual Intel as in not even on AMD processors, or just regular old boring x86-64?

Sorry, I mean x86-64

@boegelbot
Copy link
Collaborator

Travis test report: 2/2 runs failed - see https://travis-ci.org/easybuilders/easybuild-easyconfigs/builds/633212341

Only showing partial log for 1st failed test suite run 19162.1;
full log at https://travis-ci.org/easybuilders/easybuild-easyconfigs/jobs/633212342

...
ERROR: test__parse_easyconfig_Mesa-19.1.7-GCCcore-8.3.0.eb (test.easyconfigs.easyconfigs.EasyConfigTest)
Test for parsing of easyconfig Mesa-19.1.7-GCCcore-8.3.0.eb
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 1, in innertest
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 746, in template_easyconfig_test
    ecs = process_easyconfig(spec)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1791, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: "Failed to process easyconfig /home/travis/build/easybuilders/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesa/Mesa-19.1.7-GCCcore-8.3.0.eb: No software-specific easyblock 'EB_Mesa' found for Mesa"

======================================================================
ERROR: test__parse_easyconfig_Mesa-19.2.1-GCCcore-8.3.0.eb (test.easyconfigs.easyconfigs.EasyConfigTest)
Test for parsing of easyconfig Mesa-19.2.1-GCCcore-8.3.0.eb
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 1, in innertest
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 746, in template_easyconfig_test
    ecs = process_easyconfig(spec)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1791, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: "Failed to process easyconfig /home/travis/build/easybuilders/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesa/Mesa-19.2.1-GCCcore-8.3.0.eb: No software-specific easyblock 'EB_Mesa' found for Mesa"

======================================================================
ERROR: test_dep_versions_per_toolchain_generation (test.easyconfigs.easyconfigs.EasyConfigTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 413, in test_dep_versions_per_toolchain_generation
    self.process_all_easyconfigs()
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 119, in process_all_easyconfigs
    EasyConfigTest.ordered_specs = resolve_dependencies(EasyConfigTest.parsed_easyconfigs, modules_tool(), retain_all_deps=True)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/tools/robot.py", line 436, in resolve_dependencies
    processed_ecs = process_easyconfig(path, validate=not retain_all_deps, hidden=hidden)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1791, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: "Failed to process easyconfig /home/travis/build/easybuilders/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesa/Mesa-19.1.7-GCCcore-8.3.0.eb: No software-specific easyblock 'EB_Mesa' found for Mesa"

======================================================================
ERROR: test_sanity_check_paths (test.easyconfigs.easyconfigs.EasyConfigTest)
Make sure specified sanity check paths adher to the requirements.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 476, in test_sanity_check_paths
    self.process_all_easyconfigs()
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 119, in process_all_easyconfigs
    EasyConfigTest.ordered_specs = resolve_dependencies(EasyConfigTest.parsed_easyconfigs, modules_tool(), retain_all_deps=True)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/tools/robot.py", line 436, in resolve_dependencies
    processed_ecs = process_easyconfig(path, validate=not retain_all_deps, hidden=hidden)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1791, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: "Failed to process easyconfig /home/travis/build/easybuilders/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesa/Mesa-19.1.7-GCCcore-8.3.0.eb: No software-specific easyblock 'EB_Mesa' found for Mesa"

======================================================================
FAIL: test_changed_files_pull_request (test.easyconfigs.easyconfigs.EasyConfigTest)
Specific checks only done for the (easyconfig) files that were changed in a pull request.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 714, in test_changed_files_pull_request
    self.assertTrue(False, error_msg)
AssertionError: Failed to find parsed easyconfig for Mesa-19.1.7-GCCcore-8.3.0.eb (and could not isolate it in easyconfigs archive either)

----------------------------------------------------------------------
Ran 8931 tests in 283.040s

FAILED (failures=1, errors=4)
ERROR: Not all tests were successful.

*bleep, bloop, I'm just a bot (boegelbot v20180813.01)*Please talk to my owner @boegel if you notice you me acting stupid),or submit a pull request to https://github.com/boegel/boegelbot fix the problem.

@easybuilders easybuilders deleted a comment from boegelbot Jan 11, 2020
@easybuilders easybuilders deleted a comment from boegelbot Jan 11, 2020
@easybuilders easybuilders deleted a comment from boegelbot Jan 11, 2020
@easybuilders easybuilders deleted a comment from boegelbot Jan 11, 2020
@boegelbot
Copy link
Collaborator

Travis test report: 2/2 runs failed - see https://travis-ci.org/easybuilders/easybuild-easyconfigs/builds/640812406

Only showing partial log for 1st failed test suite run 19572.1;
full log at https://travis-ci.org/easybuilders/easybuild-easyconfigs/jobs/640812407

...
ERROR: test__parse_easyconfig_Mesa-19.1.7-GCCcore-8.3.0.eb (test.easyconfigs.easyconfigs.EasyConfigTest)
Test for parsing of easyconfig Mesa-19.1.7-GCCcore-8.3.0.eb
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 1, in innertest
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 759, in template_easyconfig_test
    ecs = process_easyconfig(spec)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1793, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: "Failed to process easyconfig /home/travis/build/easybuilders/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesa/Mesa-19.1.7-GCCcore-8.3.0.eb: No software-specific easyblock 'EB_Mesa' found for Mesa"

======================================================================
ERROR: test__parse_easyconfig_Mesa-19.2.1-GCCcore-8.3.0.eb (test.easyconfigs.easyconfigs.EasyConfigTest)
Test for parsing of easyconfig Mesa-19.2.1-GCCcore-8.3.0.eb
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 1, in innertest
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 759, in template_easyconfig_test
    ecs = process_easyconfig(spec)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1793, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: "Failed to process easyconfig /home/travis/build/easybuilders/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesa/Mesa-19.2.1-GCCcore-8.3.0.eb: No software-specific easyblock 'EB_Mesa' found for Mesa"

======================================================================
ERROR: test_dep_versions_per_toolchain_generation (test.easyconfigs.easyconfigs.EasyConfigTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 413, in test_dep_versions_per_toolchain_generation
    self.process_all_easyconfigs()
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 119, in process_all_easyconfigs
    EasyConfigTest.ordered_specs = resolve_dependencies(EasyConfigTest.parsed_easyconfigs, modules_tool(), retain_all_deps=True)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/tools/robot.py", line 436, in resolve_dependencies
    processed_ecs = process_easyconfig(path, validate=not retain_all_deps, hidden=hidden)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1793, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: "Failed to process easyconfig /home/travis/build/easybuilders/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesa/Mesa-19.1.7-GCCcore-8.3.0.eb: No software-specific easyblock 'EB_Mesa' found for Mesa"

======================================================================
ERROR: test_sanity_check_paths (test.easyconfigs.easyconfigs.EasyConfigTest)
Make sure specified sanity check paths adher to the requirements.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 476, in test_sanity_check_paths
    self.process_all_easyconfigs()
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 119, in process_all_easyconfigs
    EasyConfigTest.ordered_specs = resolve_dependencies(EasyConfigTest.parsed_easyconfigs, modules_tool(), retain_all_deps=True)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/tools/robot.py", line 436, in resolve_dependencies
    processed_ecs = process_easyconfig(path, validate=not retain_all_deps, hidden=hidden)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1793, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: "Failed to process easyconfig /home/travis/build/easybuilders/easybuild-easyconfigs/easybuild/easyconfigs/m/Mesa/Mesa-19.1.7-GCCcore-8.3.0.eb: No software-specific easyblock 'EB_Mesa' found for Mesa"

======================================================================
FAIL: test_changed_files_pull_request (test.easyconfigs.easyconfigs.EasyConfigTest)
Specific checks only done for the (easyconfig) files that were changed in a pull request.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/easybuilders/easybuild-easyconfigs/test/easyconfigs/easyconfigs.py", line 727, in test_changed_files_pull_request
    self.assertTrue(False, error_msg)
AssertionError: Failed to find parsed easyconfig for Mesa-19.1.7-GCCcore-8.3.0.eb (and could not isolate it in easyconfigs archive either)

----------------------------------------------------------------------
Ran 9017 tests in 292.089s

FAILED (failures=1, errors=4)
ERROR: Not all tests were successful.

*bleep, bloop, I'm just a bot (boegelbot v20180813.01)*Please talk to my owner @boegel if you notice you me acting stupid),or submit a pull request to https://github.com/boegel/boegelbot fix the problem.

@Flamefire
Copy link
Contributor

As this seemingly went stale I extracted the fix for the long double issue for R 3.6.2 into #9830 which also reduces the amount of changes

@edmondac
Copy link
Contributor Author

From @Flamefire on #9538

Seems it is not so reasonable to remove liquidSVM. It was "recently" updated on CRAN and it is failing due to this on nvcc in GCCcore/8.3.0/include/c++/8.3.0/type_traits:

#if !defined(__STRICT_ANSI__) && defined(_GLIBCXX_USE_FLOAT128)
   template<>
      struct __is_floating_point_helper<__float128>
     : public true_type { };
#endif

Some more links on that: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1717257, or even migueldiascosta@480a4df#diff-9d734c9069d6eb20c2678afc2959918aR65-R66

@edmondac
Copy link
Contributor Author

As this seemingly went stale I extracted the fix for the long double issue for R 3.6.2 into #9830 which also reduces the amount of changes

It's not stale - there are just several things that need to be merged first:

easybuilders/easybuild-easyblocks#1892 then #9764

AND

#9694

@edmondac edmondac changed the title WIP: Changes to support building R-keras and dependencies on POWER9 Changes to support building R-keras and dependencies on POWER9 Feb 13, 2020
@Flamefire
Copy link
Contributor

It's not stale - there are just several things that need to be merged first:

Ah ok, they weren't mentioned initially (I guess?) and there was WIP. Anyway I'd suggest to continue with #9830 as the patch has better description and the unrelated changes are already removed.

@edmondac
Copy link
Contributor Author

OK - so all that is left on here are the changes to R for ppc64le.

easybuilders/easybuild-easyblocks#1949 and #9830 will cover that - so I'll close this PR.

@edmondac edmondac closed this Mar 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants