Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: Add MPIEvaluator tutorial for multi-node HPC systems, including DelftBlue #308

Merged
merged 2 commits into from
Nov 16, 2023

Conversation

EwoutH
Copy link
Collaborator

@EwoutH EwoutH commented Nov 3, 2023

This PR introduces a two-part tutorial for the MPIEvaluator introduced in #299.

The first part (d26f54e) focuses on the general setup and usage of MPIEvaluator in multi-node HPC systems:

  • Utilizing the new MPIEvaluator for HPC systems.
  • Instructions on setting up the necessary environment, including the installation of OpenMPI and mpi4py.
  • Detailed step-by-step guidance for defining, testing, and executing a model on an MPI cluster.
  • Demonstrations of saving and loading results using pickle.

The second part (9e04566) extends the tutorial with specific instructions for the DelftBlue supercomputer, which uses the SLURM scheduler:

  • Detailed guide to crafting a SLURM script specifically for the DelftBlue supercomputer.
  • Insights into DelftBlue's capabilities and hardware specifications.
  • Comprehensive steps for environment setup, user authentication, file transfers, job scheduling, and retrieval of results on DelftBlue.

Together, the tutorial provides an end-to-end guide for leveraging the MPIEvaluator on HPC systems.

@EwoutH EwoutH added the docs label Nov 3, 2023
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@EwoutH EwoutH requested a review from quaquel November 3, 2023 13:37
@EwoutH EwoutH force-pushed the MPI-tutorial branch 2 times, most recently from 5364a18 to b426fb0 Compare November 3, 2023 13:59
@EwoutH
Copy link
Collaborator Author

EwoutH commented Nov 3, 2023

I have no idea why the ReadtheDocs build is failing.

nbsphinx.NotebookError: NoSuchKernel in indepth_tutorial/mpi-evaluator.ipynb:
No such kernel named python3

I compared the notebook metadata to general-introduction.ipynb and directed-search.ipynb, and it looks practically identical. directed-search.ipynb:

 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },

mpi-evaluator.ipynb

 "metadata": {
  "kernelspec": {
   "name": "python3",
   "language": "python",
   "display_name": "Python 3 (ipykernel)"
  },

@EwoutH EwoutH force-pushed the MPI-tutorial branch 3 times, most recently from e3bb224 to 7ba1ddc Compare November 3, 2023 14:45
@EwoutH EwoutH removed the request for review from quaquel November 3, 2023 14:49
@EwoutH
Copy link
Collaborator Author

EwoutH commented Nov 3, 2023

One of those cases you spend more than an hour to find that one exact line code is a sea of possible configuration options. But it now works.

Ready for review, merge (by fast-forward) after #299.

@coveralls
Copy link

Coverage Status

coverage: 80.585%. remained the same
when pulling 0616fca on MPI-tutorial
into 65e0fc0 on master.

- Introduced a detailed tutorial on how to use the new MPIEvaluator from EMAworkbench.
- Covered environment setup, including OpenMPI and mpi4py installations.
- Provided step-by-step instructions for defining, testing, and running a model on an MPI cluster.
- Illustrated how to save and load results using pickle.

This is the first part of the tutorial, the second part will follow in the next commit and cover how to run on HPC systems with a SLURM scheduler, like DelftBlue.
Added a detailed guide on how to run the model on the DelftBlue supercomputer, which utilizes the SLURM scheduler:
  - Steps for creating a SLURM script tailored for DelftBlue.
  - Overview of the DelftBlue supercomputer's capabilities.
  - Instructions on environment setup, logging in, file transfer, job scheduling, and result retrieval for DelftBlue.
@EwoutH
Copy link
Collaborator Author

EwoutH commented Nov 15, 2023

@quaquel Thanks for reviewing, I implemented the feedback and cleaned up the commits. If you approve we can fast-forward merge.

Edit: CI doesn't run on docs changes, and currently there is a required check which will prevent merging. Personally I don't think we need the required checks, we won't merge (blindly) when CI fails anyway.

@EwoutH
Copy link
Collaborator Author

EwoutH commented Nov 16, 2023

Required statuses must pass before merging

Can you remove the required check? Otherwise I cannot merge.

@EwoutH EwoutH merged commit 1831bd8 into master Nov 16, 2023
@EwoutH EwoutH deleted the MPI-tutorial branch November 16, 2023 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants