Documentation Example

[gtdang]

Documentation Example

OOD Jupyter Notebook Python with virtualenv and GPU

Tags: GPU, Jupyter Notebook, OOD, Python, virtualenv

Set up for an OOD Jupyter Notebook with python virtual environment /w tensorflow installation and GPU set-up.

Several students from Deep Learning for Scientists and Engineers (APMA-2070 / ENGN 2912) came in for help on 3/10/23 for help with this set-up.

1. GPU setup on Oscar

   1. GPU setup is a one-time set-up and should be available to all of your future sessions to load.
   2. In the OSCAR Shell Access enter the following in order:

   module load python/3.7.4  # alternatively, 3.9.0 or 3.11.0
   module load cuda/11.3.1
   module load cudnn/8.2.0
   module load openssl/3.0.0

   b. Modified from the documentation at the following page: https://docs.ccv.brown.edu/oscar/gpu-computing/ampere-architecture-gpus

2. Create a python virtual environment using virtualenv and install tensorflow

   cd ~
   virtualenv -p python3 <myenv>
   source <myenv>/bin/activate 
   pip install tensorflow

   • Install any other packages you may need with pip while your environment is activated.
   • You can start from this point if you need to create new environments after the first time you set up the GPU

3. Install jupyter notebook and create the jupyter kernel

   pip install notebook
   pip install ipykernel
   python -m ipykernel install --user --name=<myenv> where <myenv>

   • https://docs.ccv.brown.edu/oscar/connecting-to-oscar/open-ondemand/using-python-or-conda-environments-in-the-jupyter-app

4. Back at the OOD dashboard start a request for a Jupyter Notebook for Python Environments GUI and fill in the following fields.

   1. Python Module on Oscar: Select the python version you loaded in step 1 (python/3.7.4 for this example)

   2. Python Virtual Environment: Use the full path to <myenv> set up in step 2. It should look something like /users/<USERNAME>/<myenv>

   3. Modules: List all the modules loaded in step 1 separated by a space.

      python/3.7.4 cuda/11.3.1 cudnn/8.2.0 openssl/3.0.0

   4. Partition: gpu

Links

1. https://docs.ccv.brown.edu/oscar/gpu-computing/ampere-architecture-gpus
2. https://docs.ccv.brown.edu/oscar/connecting-to-oscar/open-ondemand/using-python-or-conda-environments-in-the-jupyter-app
3. https://docs.ccv.brown.edu/oscar/software/python-instal

[gtdang]

@singhsaluja @prasadbandarkar @rainxfan @phall-brown @paulstey
I wrote up some notes last Friday on the common issue students from (APMA-2070 / ENGN 2912) were coming in for help (Could use a look-over). @mirestrepo suggested a good place to store these notes could be the GitHub discussions here. We could share this with others, or have it as a resource for users/staff to search for common issues. Do you think this format would be helpful?

[gtdang]

I think it's the challenge of piecing the correct order of operations for a particular use case. There is information in the documentation pages but you have to know which are applicable and what order to do them in. These things can be different on a case-by-case basis and are not clear to inexperienced users. So it might be useful to have a resource with common cases that we see.

I tried to link all the relevant pages in the documentation with each step above. We did need to adapt the information from the pages to their specific use cases though. Like Tensorflow instead of PyTorch, etc.

[phall-brown]

@paulstey Our current docs include instructions for using both virtualenv and conda, though that's in the context of working with python packages on Oscar. Our documentation for using the Jupyter app via OOD also cover both. I wonder if there is an issue with organization (rather than completeness/accuracy) in our documentation that leads people to overlook the instructions.

Regardless, I think it would be very useful to have example cases available in our docs, and the instructions @gtdang has put together for this use case is an excellent place to start.

[phall-brown]

@gtdang Totally agree with your last comment (didn't see it until after I typed mine). It's easy for users to get lost in the docs if they don't know exactly what they are looking for. Examples will go a long way to helping that.

[gtdang]

@phall-brown Do you think this would be a suitable location to house these examples? We can create an "Examples" discussion category and link to it from the Oscar documentation.

[phall-brown]

@gtdang Good question. Within the documentation I could see it being useful to have a separate "Examples" section, broken down by topic, with links to relevant examples (within gitbooks? as gists?). I could also see utility in having links to relevant examples within the existing docs themselves (e.g., at the bottom of each page)? If not for the management headache it would create, I would suggest having examples in both places.

One thing to be careful with: as with all documentation, any examples we put up will eventually go stale as the system or software changes over time. It might be useful to include some indication of when each example was last checked, so users don't waste too much time and effort trying to reproduce examples that are severely out of date and likely no longer working.

[paulstey]

This is great, @gtdang!! Thank you so much!! Just out of curiosity, is there any reason we wouldn't want to use the latest Python on Oscar? Our latest version is 3.11, isn't it? Does the above only work with Python 3.7 or 3.9?

Again, thank you so much for writing this up, George!!

[gtdang]

Oh I'm not sure. The documentation page has uses 3.7.4 in the example and we used 3.9 with one student on Friday.

[paulstey]

Ah, okay, perfect! Yes, the Python 3.11 module is one of the newer ones, so I'm guessing it's just not widely used or represented in the docs yet.

Thanks again, George!!

[gtdang]

Ok! I'll add it to the notes.

[digicosmos86]

I wonder if it might be a good idea to let faculty know that they can create conda environments (or whatever env manager) themselves, and the oscar team could help them make the environments accessible to their students? All the students need to do is to launch the notebook environment with the environments already created. That would really simplify the process especially for beginners and ensures that everyone taking the same course is using the same software environment. Might be a huge advantage for using Oscar. I tried this when I was doing DsCov and it saved me lots of time and sweat. Thanks! Paul

…

On Mon, Mar 13, 2023 at 1:58 PM Paul Stey ***@***.***> wrote: @phall-brown <https://github.com/phall-brown>: I might be wrong, but don't our docs still suggest using Conda for Python environments? I would definitely prefer pointing people towards virtualenv rather than conda envs. And as you note, I also don't think our docs lay this out as an example use case, which I think is super beneficial! — Reply to this email directly, view it on GitHub <#38 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEJNTY2HWOZ7C6PK7DU3F63W35N5XANCNFSM6AAAAAAVZL4LMQ> . You are receiving this because you are subscribed to this thread.Message ID: <brown-ccv/oscar-documentation/repo-discussions/38/comments/5294685@ github.com>

-- <http://ccv.brown.edu/> Paul Xu, PhD <https://github.com/digicosmos86> Data Scientist Center for Computation and Visualization Brown University | OIT Pronouns: he/him/his

[phall-brown]

@digicosmos86 Services like this should probably be documented on the Classroom Services page of the CCV website. Totally agree that this kind of information should be advertised to faculty. Not sure how best to do that, beyond the website itself, though. Maybe worth having an informational session on "CCV Services for Classroom Instruction" as part of the Bootcamp? Or as part of the Monday afternoon workshop series? Any thoughts, @paulstey ?

[gtdang]

Hi at the documentation meeting this week we discussed the format and use of these discussion pages for future examples and questions. We decided that they should be formatted with an initial question and then add a reply with the answer. That way we can mark the reply as an answer. See #40 for an example of this thread reformatted into the Q&A format.

Other things discussed:

• All topics will be contained in the Q&A category. Other categories will be removed.
• The repo's labels will be updated to enabling tagging of topics.
• We will pin guidelines at the top of the discussion page.

I will keep this page as a legacy of this discussion.