Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't bind MPI Ranks to process to solve OpenMP timeouts (#2628) #2638

Merged

Conversation

bartlettroscoe
Copy link
Member

@bartlettroscoe bartlettroscoe commented Apr 25, 2018

CC: @trilinos/nox, @fryeguy52

Description

This solved the timeouts for the NOX test suite (#2628) for me on ceerws1113.

How Has This Been Tested?

Before this change, on ceerws1113, when I ran:

$ ./checkin-test-atdm.sh gnu-opt-openmp \
   --enable-all-packages=off --no-enable-fwd-packages \
  --enable-packages=NOX --local-do-all

it produced the ctest output:

93% tests passed, 7 tests failed out of 105

Label Time Summary:
NOX    = 7002.16 sec (105 tests)

Total Test time (real) = 1050.16 sec

The following tests FAILED:
         71 - NOX_LOCA_TcubedTP_MPI_2 (Timeout)
         72 - NOX_LOCA_TcubedTP_stratimikos_MPI_2 (Timeout)
         77 - NOX_LOCA_BrusselatorHopf_MPI_2 (Timeout)
         80 - NOX_LOCA_BrussXYZT_Sequential_MPI_2 (Timeout)
         81 - NOX_LOCA_BrussXYZT_SequentialIPS_MPI_2 (Timeout)
         82 - NOX_LOCA_BrussXYZT_SequentialOPS_MPI_2 (Timeout)
         84 - NOX_LOCA_BrussXYZT_BlockDiagonal_MPI_2 (Timeout)

After this change, it produces the test output:

100% tests passed, 0 tests failed out of 105

Label Time Summary:
NOX    =  59.11 sec (105 tests)

Total Test time (real) =   9.28 sec

Checklist

  • My commit messages mention the appropriate GitHub issue numbers.
  • All new and existing tests passed.
  • No new compiler warnings were introduced.

This solved the timeouts for the NOX test suite for me on ceerws1113.
@bartlettroscoe bartlettroscoe added the client: ATDM Any issue primarily impacting the ATDM project label Apr 25, 2018
@bartlettroscoe bartlettroscoe requested a review from etphipp April 25, 2018 16:42
Copy link
Contributor

@etphipp etphipp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to do this for the other architectures instead of just rhel6?

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

  • Build Num: 563
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.9.3
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 2638
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 2628-address-timeouts
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA 118bbba
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 5c408a6

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

  • Build Num: 280
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.8.4
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 2638
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 2628-address-timeouts
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA 118bbba
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 5c408a6

Using Repos:

Repo: TRILINOS (bartlettroscoe/Trilinos)
  • Branch: 2628-address-timeouts
  • SHA: 118bbba
  • Mode: TEST_REPO

Pull Request Author: bartlettroscoe

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

  • Build Num: 563
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.9.3
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 2638
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 2628-address-timeouts
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA 118bbba
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 5c408a6

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

  • Build Num: 280
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.8.4
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 2638
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 2628-address-timeouts
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA 118bbba
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 5c408a6


CDash Test Results for PR# 2638.

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ etphipp ]!

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Pull Request MUST BE MERGED MANUALLY BY Project Team - Master Automerge is disabled (in .cfg file)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
client: ATDM Any issue primarily impacting the ATDM project
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants