SoftwareTools.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CMake\n",
    "\n",
    "* cmake can take \"defines\" on the command line that can be used to set paths in the makefiles\n",
    "    * Example:\n",
    "~~~~~~~~\n",
    "cmake -DCMAKE_EXE_LINKER_FLAGS_INIT=\"-L/Users/stephenh/projects/py-ncepbufr/src\" ..\n",
    "~~~~~~~~\n",
    "\n",
    "* Other controls  \n",
    "    https://cmake.org/cmake/help/v3.0/manual/cmake-variables.7.html\n",
    "    \n",
    "    CMAKE_ARGC  \n",
    "    CMAKE_ARGV0  \n",
    "    CMAKE_BINARY_DIR  \n",
    "    CMAKE_COMMAND  \n",
    "    ...  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Git Flow\n",
    "\n",
    "## When the developer doesn't have push permission to the github repository\n",
    "\n",
    "### Developer in their local repo\n",
    "~~~~~~~~\n",
    "git flow init\n",
    "\n",
    "git flow feature start <name>\n",
    "\n",
    "make edits\n",
    "\n",
    "git flow feature publish <name>\n",
    "~~~~~~~~\n",
    "\n",
    "### Developer on GitHub\n",
    "\n",
    "~~~~~~~~\n",
    "initiate a pull request from branch featrue/<name> to branch develop\n",
    "~~~~~~~~\n",
    "\n",
    "### Owner on GitHub\n",
    "\n",
    "~~~~~~~~\n",
    "once feature/<name> is approved, process the pull request\n",
    "    merge feature/<name> into develop\n",
    "    delete feature/<name>\n",
    "~~~~~~~~\n",
    "\n",
    "### Developer in their local repo\n",
    "\n",
    "~~~~~~~~\n",
    "git branch -D feature/<name>\n",
    "\n",
    "git remote update -p  # -p prunes the remote feature/<name> branch from the local repo\n",
    "\n",
    "git checkout master\n",
    "git pull origin master\n",
    "\n",
    "git checkout develop\n",
    "git pull origin develop\n",
    "~~~~~~~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ecbuild\n",
    "\n",
    "* To enable features, such as OpenMP, you use the ecbuild_add_option macro\n",
    "    * The user sets an ENABLE\\_ environment var (via ecbuild_add_option)\n",
    "    * And the macro sets HAVE\\_ accordingly\n",
    "    \n",
    "    * Example: OpenMP\n",
    "        * User places the following in a CMakeLists.txt file\n",
    "\n",
    "~~~~~~~~\n",
    "ecbuild_add_option( FEATURE OMP\n",
    "                    DEFAULT ON\n",
    "                    DESCRIPTION \"Use OpenMP\" )\n",
    "~~~~~~~~\n",
    "\n",
    "* This macro will set the HAVE_OMP cmake variable which is then used downstream by other cmake processes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# How to resolve mege conflict with a git lfs file\n",
    "\n",
    "```\n",
    "cd ioda-bundle/iodaconv\n",
    "\n",
    "git checkout develop\n",
    "git pull\n",
    "\n",
    "git checkout feature/mycode\n",
    "git pull\n",
    "\n",
    "git merge develop\n",
    "\n",
    "# for each of the lfs files with conflicts\n",
    "git checkout --theirs path_to_file    # choose develop version\n",
    "git checkout --ours path_to_file      # choose feature/mycode version\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build and test fv3-bundle on S4, 1/18/22\n",
    "\n",
    "* When building my own jedi-stack using intel/impi, the formula for loading modules is a bit tricky. Here is what I did to get this accomplished.\n",
    "\n",
    "```\n",
    "git lfs install # I may have not needed this, but it did seem to help\n",
    "\n",
    "module use /data/users/sherbener/projects/jedi-stack/modulefiles/apps\n",
    "\n",
    "module load license_intal\n",
    "module load git/2.30.0\n",
    "module load miniconda/3.8-s4\n",
    "module load jedi/intel-impi\n",
    "module load odc/ecmwf-1.4.4\n",
    "module load jedi-gcc/8.3\n",
    "\n",
    "cd /data/users/sherbener/projects\n",
    "git clone https://github.com/jcsda-internal/fv3-bundle\n",
    "cd fv3-bundle\n",
    "git clone https://github.com/jcsda-internal/jedi-cmake jedicmake\n",
    "mkdir build\n",
    "cd build\n",
    "ecbuild --toolchain=../jedicmake/cmake/Toolchains/jcsda-S4-Intel.cmake ..\n",
    "make -j4\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build jedi-stack with debug settings on libraries, 1/31/22\n",
    "\n",
    "* MPICH\n",
    "    * Autoconfig\n",
    "        * `--enable-g=debug`\n",
    "        * `--disable-fast`\n",
    "        * `--enable-debug-info`\n",
    "\n",
    "* hdf5\n",
    "    * Autoconfig\n",
    "        * `--enable-build-mode=debug`\n",
    "        * `--enable-optimization=debug`\n",
    "\n",
    "* netcdf\n",
    "    * Autoconfig\n",
    "    * export CFLAGS, CXXFLAGS, FFLAGS, FCFLAGS\n",
    "        * \"-g -O0\"\n",
    "\n",
    "* pnetcdf\n",
    "    * Autocoonfig\n",
    "        * `--enable-debug`\n",
    "\n",
    "* eccodes\n",
    "    * cmake\n",
    "    * export CFLAGS, etc.\n",
    "\n",
    "* pio\n",
    "    * cmake\n",
    "    * export CFLAGS, etc.\n",
    "\n",
    "* There are a number of packages that have `--build=production` or `--build=release` settings which can be changed to `--build=Debug`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# fv3-bundle issue with building eckit, fckit, atlas, 2/23/22\n",
    "\n",
    "* Tried different combinations of running ecbuild using the `BUNDLE_SKIP_<REPO>` controls which controls whether you build the repo in your bundle (BUNDLE_SKIP_ = OFF) or use the installation in modules (BUNDLE_SKIP_ = ON, which is the default).\n",
    "\n",
    "| eckit | fckit | atlas | ecbuild status |\n",
    "|-------|-------|-------|----------------|\n",
    "| bundle | bundle | bundle | Fail |\n",
    "| bundle | bundle | module | Pass |\n",
    "| bundle | module | bundle | Fail |\n",
    "| bundle | module | moduel | Pass |\n",
    "| module | bundle | bundle | Pass |\n",
    "| module | bundle | module | Pass |\n",
    "| module | module | bundle | Pass |\n",
    "| module | module | moduel | Pass |\n",
    "\n",
    "* It appears that building eckit and atlas together in the bundle does not work. All other combinations do work.\n",
    "\n",
    "* Used `--trace-expand` to see what's going on with a failing combination\n",
    "    * Found that `ENABLE_OMP=OFF` when the atlas `atlas-import.cmake` file is being constructed\n",
    "    * This causes the `atlas_OMP, atlas_OMP_Fortran, atlas_OMP_CXX` features to be recorded in the `atlas-import.cmake` file as unavailable.\n",
    "    * Downstream oops is looking for these features, via a check script in eckit, that doesn't find them and throws the error\n",
    "\n",
    "* Found that adding `-DENABLE_OMP=ON` to the ecbuild command line fixes the problem since `ENABLE_OMP=ON` when the `atlas-import.cmake` file is being constructed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to debug python scripts with underlying C++ libraries, 5/12/22\n",
    "\n",
    "* The idea is to run python in one window, then run the debugger in another window (ie another process) and attach the debugger to the python process.\n",
    "    * You need to run the pyhton process where you can stop and start the script\n",
    "        * Either run python interactively or in pdb (python debugger) mode\n",
    "\n",
    "* The sequence goes like:\n",
    "\n",
    "```\n",
    "# in one window run:\n",
    "python3 -m pdb my_script.py\n",
    "\n",
    "# in another window, find the process id of the python job\n",
    "# that was started above\n",
    "ps aux | less\n",
    "\n",
    "# Then attach the debugger to that process and \"release\" that\n",
    "# process by telling the debugger to continue\n",
    "lldb\n",
    "(lldb) attach -pid py_proc_id\n",
    "(lldb) continue\n",
    "\n",
    "# in the debugger set a breakpoint in the C++ code\n",
    "(lldb) br s -f mysource.cpp -l myline\n",
    "\n",
    "# back in the python window tell the process to continue\n",
    "(pdb) c\n",
    "\n",
    "# After continuing the python process, you should see the lldb window\n",
    "# stop on the breakpoint and you can start debugging.\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EWOK flow, 6/1/22\n",
    "\n",
    "## How to test my feature branch\n",
    "\n",
    "* load ewok environment\n",
    "\n",
    "* clone fv3-bundle and build\n",
    "    * Include my feature branch\n",
    "\n",
    "* Change JEDI_BIN environment variable to point to my bundle build/bin directory\n",
    "    * Don't change JEDI_SRC variable\n",
    "\n",
    "* After bundle is built, in a new terminal\n",
    "\n",
    "```\n",
    "# set up modules, environment for ewok\n",
    "source /work/noaa/da/jedipara/ewok/bin/ewok\n",
    "\n",
    "# point to the applications using my feature branch\n",
    "JEDI_BIN=\"path-to-my-bundle/build/bin\"\n",
    "\n",
    "# create an R2D2 config file (if not done yet)\n",
    "# check the instruction in the ewok/README.md documentation\n",
    "#\n",
    "# location is: $HOME/.r2d2/config.yaml\n",
    "\n",
    "# create experiment\n",
    "# take note of experiment id at end of messaging from\n",
    "# the create_experiment.py script\n",
    "create_experiment.py $JEDI_SRC/ewok/experiments/<exp-id>.yaml\n",
    "\n",
    "# Error messages\n",
    "#\n",
    "# Look in:\n",
    "#   For most tasks: /work/noaa/da/herbener/ewok/tmp/ecflow/exp-id\n",
    "#   For hofx task: /work/noaa/da/herbener/ewok/tmp/ewok/datetime-id\n",
    "\n",
    "# note if env vars change you need to restart the server\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# spack-stack on EC2 instance, 6/10/22\n",
    "\n",
    "* Place a copy of the EWOK PEM file on your system and restrict access\n",
    "    * `~/.aws/ewok-tutorial-20220330.pem`\n",
    "    * `chmod 400 ~/.aws/ewok-tutorial-20220330.pem`\n",
    "\n",
    "* ssh to the instance\n",
    "    * `ssh -i /Users/steveherbener/.aws/ewok-tutorial-20220330.pem ec2-user@ec2-44-201-179-82.compute-1.amazonaws.com`\n",
    "\n",
    "* On the instance:\n",
    "\n",
    "```\n",
    "# set up modules and environment\n",
    "scl enable gcc-toolset-11 bash\n",
    "module use /home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/modulefiles/Core\n",
    "module load stack-gcc/11.2.1 stack-openmpi/4.1.3 stack-python/3.9.7\n",
    "module load jedi-fv3-env/main jedi-ewok-env/main\n",
    "\n",
    "# clone fv3-bundle and fix CMakeLists.txt files (until the\n",
    "# CMakeLists.txt files get updated)\n",
    "#\n",
    "# In top-level CMakeLists.txt replace the\n",
    "# include git_functions.cmake with:\n",
    "\n",
    "# Use external jedi-cmake or build in bundle\n",
    "if(DEFINED ENV{jedi_cmake_ROOT})\n",
    "  include( $ENV{jedi_cmake_ROOT}/share/jedicmake/Functions/git_functions.cmake )\n",
    "else()\n",
    "  ecbuild_bundle( PROJECT jedicmake GIT \"https://github.com/jcsda-internal/jedi-cmake.git\" BRANCH develop UPDATE )\n",
    "  include( jedicmake/cmake/Functions/git_functions.cmake )\n",
    "endif()\n",
    "\n",
    "# The first run of ecbuild will fail, then in\n",
    "# fv3-jedi/test/CMakeLists.txt replace the\n",
    "# include git-functions.cmake with:\n",
    "\n",
    "if(DEFINED ENV{jedi_cmake_ROOT})\n",
    "  include( $ENV{jedi_cmake_ROOT}/share/jedicmake/Functions/git_functions.cmake )\n",
    "else()\n",
    "  include( ${CMAKE_SOURCE_DIR}/jedicmake/cmake/Functions/git_functions.cmake )\n",
    "endif()\n",
    "\n",
    "# move the the sandpit area, clone fv3-bundle and proceed\n",
    "\n",
    "cd sandpit/steve\n",
    "git clone https://github.com/jcsda-intenal/fv3-bundle\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Git Submodules, 6/24/22\n",
    "\n",
    "* .gitmodule file\n",
    "    * branch specs are for navigation only\n",
    "\n",
    "* The hash on the commit that is used for the submodule is stored in the git metadata\n",
    "    * Can view this using github OR by running `git log`\n",
    "* The point of the feature branches in the parent repo is to have a branch with the correct submodule commit hash\n",
    "    * The submodule commit hash can be changed to test different versions of the submodule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DDT/MAP Arm Forge, 12/13/22\n",
    "\n",
    "* Dom's DDT tutorial\n",
    "    * Orion\n",
    "    * DDT memory debugger\n",
    "    * `ddt --connect` # prefix for connecting to remote GUI client on Mac\n",
    "    * DDT version on Orion is 22.0.2\n",
    "        * MacBook has 22.0.4 which is good (the major and minor version numbers need to match)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intel ifx, ifort update, 2/1/23\n",
    "\n",
    "* ifx is done and ready for beta testing\n",
    "    * backward compatibility with ifort\n",
    "    * Optimizations, machine code generation however are different\n",
    "        * Due to differnt backends: LLVM vs Intel\n",
    "\n",
    "* ifort is not going away soon (will probably last a couple years still)\n",
    "\n",
    "* ifort and ifx both produce ABI compatible binaries\n",
    "    * This should allow us to use ifort as a debug tool\n",
    "        * Say we have switched to icx, ifx and ifx throws a mysterious compiler error. We can switch to icx, ifort and see if we get better diagnostics. Or we can switch to icx, ifort for this particular file as a workaround (if ifort completes)\n",
    "\n",
    "* LLVM (icx, icpx and ifx) will not be available for Mac/Intel hardware\n",
    "    * We should be able to use icx, ifort in this case (see above)\n",
    "\n",
    "* New development will only go into ifx\n",
    "    * This include support for offloading code execution (ie, GPU)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# jedipara access, 6/14/24\n",
    "\n",
    "* Need on S4 and Discover\n",
    "\n",
    "* S4 Done\n",
    "  * `sudo -iu jedipara`\n",
    " \n",
    "* Discover\n",
    "  * `sudo -iu jedipara`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# WorkDay entering time, 7/19/24\n",
    "\n",
    "* Navigation for entering time\n",
    "  * Actions -> Enter Time by Type\n",
    "    * Create new row\n",
    "    * Click on first column\n",
    "    * Select Projects -> select account\n",
    "    * If there is no account nubmer to select\n",
    "      * select \"Account key not assigned\"\n",
    "      * Click second column\n",
    "      * Click down to \"All UCAR Division Keys\"\n",
    "      * Scroll to \"58\" and select\n",
    "      * Scroll to specific account and select"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# spack-stack, 10/21/24\n",
    "\n",
    "* LD_DEBUG is useful for tracking down shared library issues at runtime\n",
    "  * LD_DEBUG=libs -> show the libraries that are being selected for loading\n",
    "\n",
    "* Orion, intel, spack-stack-1.8.0 env\n",
    "  * tar command is messed up\n",
    "  * libz is in the spack-stack installation and in /usr/lib64\n",
    "  * libcrypt is in the intel compiler tools installation and in /usr/lib64\n",
    "  * tar needs to load the /usr/lib64 versions in order to work properly\n",
    "  * LD_DEBUG_libs tracing shows\n",
    "    * In the good environment (ie, right after login and before sourcing the spack-stack-1.8.0 env) the loading goes as intended and tar works\n",
    "    * In the spack-stack-1.8.0 environment, the loading goes astray when libcrypt is loaded (ie, the intel compiler version gets loaded instead of the /usr/lib64 version)\n",
    "  * In the spack-stack-1.8.0 environment, the following works for tar\n",
    "  * `LD_LIBRARY_PATH=\"\" tar <args>`\n",
    "  * This issue is being tracked in: https://github.com/JCSDA/spack-stack/issues/1355"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# spack-stack, 10/22/24\n",
    "\n",
    "* 1.9.0 release is targeted for end of this year\n",
    "  * LLVM based Intel compilers: icx, icpx, ifx\n",
    "\n",
    "* Dom will start the spack develop sync effort soon\n",
    "  * Need to get the openmpi fix into the auth spack repo soon\n",
    "\n",
    "* Carmeron, Mac Support, NOAA-EPIC\n",
    "  * They are also questioning the effort it takes to maintain the native Mac builds\n",
    "  * Looking at alternatives too"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# David H's instructions for gsi-addon env on S4, spack-stack-1.6.0, 10/28/24\n",
    "\n",
    "```\n",
    ". setup.sh\n",
    "spack stack create env --site <site name> --template unified-dev --name unified-env\n",
    "cd envs/unified-env\n",
    "spack env activate .\n",
    "spack concretize 2>&1 | tee log.concretize && spack install --verbose --fail-fast 2>&1 | tee log.install\n",
    "spack module lmod refresh\n",
    "spack stack setup-meta-modules\n",
    "spack env deactivate\n",
    "spack stack create env --site <site name> --template gsi-addon-dev --name gsi-addon --upstream $(pwd)/install\n",
    "cd ../gsi-addon\n",
    "vim spack.yaml # manually add \"compilers: [  '%gnu', '%intel' ]\"\n",
    "spack env activate .\n",
    "spack concretize 2>&1 | tee log.concretize && spack install --verbose --fail-fast 2>&1 | tee log.install\n",
    "spack module lmod refresh --upstream-modules\n",
    "spack stack setup-meta-modules\n",
    "```\n",
    "* Need to make sure the compiler specs are updated with spack-stack-1.6.0 ue-intel-2021.10.0 specs\n",
    "  * site/compilers.yaml\n",
    "  * site/packages.yaml\n",
    "  * Find all other places where \"intel\" exists and check those"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# spack-stack, 11/5/24\n",
    "\n",
    "* Tar issue\n",
    "  * Crux of issue is that libraries within the oneAPI installation are intended to the shared, but marked as static\n",
    "    * When one of these libraries gets loaded, the dynamic loader stops at that point since it thinks the library is static, and this prevents that library's dependencies from being loaded leading to undefined reference faults\n",
    "  * A workaround is to run patchelf on the mis-marked Intel libraries and change the mark to \"shared\" so that the dynamic loading process proceeds as intended\n",
    "  * One issue with the workaround is that we don't have permission to run patchelf on these libraries so we have to convince the IT folks to do it (and on Orion they are unwilling)\n",
    "  * Another more painful workaround is to install oneAPI in our spack-stack area, and then we can run patchelf to repair the libraries. Then recreate the intel build from this installation of oneAPI\n",
    "  * This spack-stack issue has details: https://github.com/JCSDA/spack-stack/issues/1355"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code review, 11/7/24\n",
    "\n",
    "* First meeting with interface team, used Liam's atlas PR as an example\n",
    "\n",
    "* Context: can we do the entire DA flow with single precision?\n",
    "\n",
    "* PR is to add sparse matrix multiply which can be offloaded to GPU\n",
    "  * sparse matrix multiply is paramterized\n",
    "  * `y = alpha * A * x + beta * y`\n",
    "  * alpha == 1, beta == 0 -> matrix multiply (A * x)\n",
    "  * alpha == 0, beta == 1 -> returns y\n",
    "  * Not sure if there are mixes inbetween that blend the alpha and beta sides\n",
    "  * Changes are in the linear algebra backend\n",
    "    * Routines to have either CPU or GPU do the computations\n",
    "    * `ExecutionSpace::Host` -> CPU\n",
    "    * `ExecutionSpace::Device` -> GPU\n",
    "  * \"HIC\" is an Atlas wrapper that handles CUDA, NVIDIA API signatures for different architectures (eg, AMD is different than Intel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# spack-stack, 12/9/24\n",
    "\n",
    "* LLVM compilers are getting more robust\n",
    "  * LLVM 19.1.4\n",
    "  * clang, clang++, flang-new (note not \"flang\")\n",
    "  * Dom is looking into this since there is concern about the future of Intel LLVM based compilers\n",
    "    * Intel going through leadership transitions\n",
    "    * Intel is using their AST frontend to translate source code to the LLVM intermediate form.\n",
    "      * Note any compiler based on LLVM needs to do this\n",
    "\n",
    "* Need to continue, for now, with Intel LLVM\n",
    "  * Lots of investment in AST compiler flags, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code review, 12/5/24\n",
    "\n",
    "* MPI gdb - for multi process debug\n",
    "\n",
    "* Liam has success with the SSH remote mode of vscode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AWS AMI, Steve G, 12/6/24\n",
    "\n",
    "* In the AWS CLI command example, \"key-name\" is the name of the key-pair on AWS\n",
    "  * Can view the key-pair entries in the AWS console\n",
    "\n",
    "* Still working on the Ubuntu image\n",
    "\n",
    "* Existing AMI for Ubuntu with Intel compiler\n",
    "\n",
    "* An AWS Snapshot is a backup mechinism, can take a snapshot of your instance at any time, and an AMI is simply a published snapshot.\n",
    "  * Can view snapshots in the AWS console\n",
    "  * On the AWS CLI example, replace \"ami-...\" with \"snap-...\" to build an instance from a snapshot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AWS AMI, 1/7/24\n",
    "\n",
    "* Need to declare \"profile\" in the AWS config files\n",
    "  * See the instructions in the spack-stack repo: configs/tier2/aws/ec2/README.md\n",
    "\n",
    "* After the instance is built\n",
    "  * `ssh -i <path>/steveh_us_east.old.pem ubuntu@<public_ip_address>`\n",
    "  * `gh auth login`\n",
    "    * use mobile app to go into github and complete the authentication\n",
    "    * This avoids having to put github credentials on the instance, but is not as convenient.\n",
    "\n",
    "* Try to get through skylab experiment\n",
    "\n",
    "* Still having trouble with Intel environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AWS AMI, 1/9/24\n",
    "\n",
    "* GCC env is missing the `spack stack setup-meta-modules` step\n",
    "  * Need to run as root to fix this: can use sudo\n",
    "\n",
    "* Once we get GCC and INTEL work, strip my data and credential info from the ami and take a snapshot\n",
    "\n",
    "* "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}