code/04-saxpy-workflow/NB04-saxpy-workflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pathfinder Workflow with SAXPY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lesson Objectives\n",
    "\n",
    "Upon completing this notebook you should be able to understand and apply the following concepts:\n",
    "\n",
    "- Understand how the Lucata Pathfinder workflow supports x86 execution, simulation, and execution in multiple configurations on the hardware.\n",
    "- Use basic Slurm commands to submit a job on the Pathfinder hardware.\n",
    "- Understand the debugging flow for the Pathfinder platform\n",
    "    - Use OpenCilk and the memoryweb library to test code on x86 systems\n",
    "    - Simulate your Lucata Cilk code with `emusim` and profile with `emusim_profile`\n",
    "    - Run a single node hardware job on the Pathfinder with Slurm\n",
    "    - Run multi-node and multi-chassis jobs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Environment Setup\n",
    "\n",
    "We rerun our environment setup for the notebook as with our previous notebooks. However, notice that we have added a new line with `X86FLAGS`, which we will use to compile our code on an x86 platform with OpenCilk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lucata Cilk compilation flags are -I/tools/emu/pathfinder-sw/22.09-beta/include/memoryweb/ -L/tools/emu/pathfinder-sw/22.09-beta/lib -lmemoryweb\n",
      "\n",
      "x86 flags are -I/tools/lucata/pathfinder-sw/22.09-beta/include/cilk/ -I/tools/emu/pathfinder-sw/22.09-beta/x86/include/emu_c_utils /tools/emu/pathfinder-sw/22.09-beta/x86/lib/libemu_c_utils.a\n"
     ]
    }
   ],
   "source": [
    "%load_ext slurm_magic\n",
    "import os\n",
    "#Used to display the code file directly within the notebook\n",
    "from IPython.display import Code\n",
    "\n",
    "#Set the path to the latest toolset \n",
    "LUCATA_BASE=\"/tools/emu/pathfinder-sw/22.09-beta\" \n",
    "\n",
    "#Get the path to where all code samples are\n",
    "os.environ[\"USER_NOTEBOOK_CODE\"]=os.path.dirname(os.getcwd())\n",
    "os.environ[\"X86FLAGS\"] = \"-I/tools/lucata/pathfinder-sw/22.09-beta/include/cilk/ -I/tools/emu/pathfinder-sw/22.09-beta/x86/include/emu_c_utils /tools/emu/pathfinder-sw/22.09-beta/x86/lib/libemu_c_utils.a\"\n",
    "os.environ[\"PATH\"]=os.pathsep.join([os.path.join(LUCATA_BASE,\"bin\"),os.environ[\"PATH\"]])\n",
    "os.environ[\"FLAGS\"]=\"-I\"+LUCATA_BASE+\"/include/memoryweb/\"+\" -L\"+LUCATA_BASE+\"/lib -lmemoryweb\"\n",
    "os.environ[\"LUCATA_BASE\"]=\"/tools/lucata/pathfinder-sw/22.09-beta/\"\n",
    "\n",
    "!printf \"Lucata Cilk compilation flags are $FLAGS\\n\\n\"  \n",
    "!printf \"x86 flags are $X86FLAGS\\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook goes along with the [Lucata workflow slides](https://github.com/gt-crnch-rg/lucata-pathfinder-tutorial/blob/8c5bfb3c1bf3dd5116887026917f8dcc59c47bb5/slides/lucata_tutorial/02_Lucata_Pathfinder_Tutorial_Workflow.pdf), so please follow along with the slides for a supplemental resource. \n",
    "\n",
    "## Lucata Pathfinder Workflow\n",
    "\n",
    "This figure shows the workflow for using the Pathfinder ecosystem and hardware. Since the Pathfinder is programmed using a variant of the Cilk programming language, code written for this platform can be run on x86 systems using the Lucata toolchain, some GCC versions (5-7), or an appropriate Clang branch like [OpenCilk](https://www.opencilk.org/).\n",
    "\n",
    "![Lucata Workflow](../resources/figs/lucata_pathfinder_workflow.png)\n",
    "\n",
    "This notebook takes one of our previous Saxpy examples and uses it as part of a workflow that shows how to run code on the x86 system, simulator, and finally on the hardware. Just to revisit, we are using the basic SAXPY \"1D allocation\" kernel from Notebook 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### X86 Execution\n",
    "As a first step, we need to update the code slightly to allow it to be compiled for x86 platforms using \"memoryweb_x86.h\". This compatibility header tells the compiler to compile for an x86 variant of Cilk rather than the Lucata version. The differences are that some Lucata-specific commands like `cilk_spawn_at` don't exist in most standard x86 Cilk APIs.\n",
    "\n",
    "Also, note the inclusion of the \"emu_c_utils\" header, which provides additional helper functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>pre { line-height: 125%; }\n",
       "td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }\n",
       "span.linenos { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }\n",
       "td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }\n",
       "span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }\n",
       ".output_html .hll { background-color: #ffffcc }\n",
       ".output_html { background: #f8f8f8; }\n",
       ".output_html .c { color: #3D7B7B; font-style: italic } /* Comment */\n",
       ".output_html .err { border: 1px solid #FF0000 } /* Error */\n",
       ".output_html .k { color: #008000; font-weight: bold } /* Keyword */\n",
       ".output_html .o { color: #666666 } /* Operator */\n",
       ".output_html .ch { color: #3D7B7B; font-style: italic } /* Comment.Hashbang */\n",
       ".output_html .cm { color: #3D7B7B; font-style: italic } /* Comment.Multiline */\n",
       ".output_html .cp { color: #9C6500 } /* Comment.Preproc */\n",
       ".output_html .cpf { color: #3D7B7B; font-style: italic } /* Comment.PreprocFile */\n",
       ".output_html .c1 { color: #3D7B7B; font-style: italic } /* Comment.Single */\n",
       ".output_html .cs { color: #3D7B7B; font-style: italic } /* Comment.Special */\n",
       ".output_html .gd { color: #A00000 } /* Generic.Deleted */\n",
       ".output_html .ge { font-style: italic } /* Generic.Emph */\n",
       ".output_html .gr { color: #E40000 } /* Generic.Error */\n",
       ".output_html .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       ".output_html .gi { color: #008400 } /* Generic.Inserted */\n",
       ".output_html .go { color: #717171 } /* Generic.Output */\n",
       ".output_html .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       ".output_html .gs { font-weight: bold } /* Generic.Strong */\n",
       ".output_html .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       ".output_html .gt { color: #0044DD } /* Generic.Traceback */\n",
       ".output_html .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       ".output_html .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       ".output_html .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       ".output_html .kp { color: #008000 } /* Keyword.Pseudo */\n",
       ".output_html .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       ".output_html .kt { color: #B00040 } /* Keyword.Type */\n",
       ".output_html .m { color: #666666 } /* Literal.Number */\n",
       ".output_html .s { color: #BA2121 } /* Literal.String */\n",
       ".output_html .na { color: #687822 } /* Name.Attribute */\n",
       ".output_html .nb { color: #008000 } /* Name.Builtin */\n",
       ".output_html .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       ".output_html .no { color: #880000 } /* Name.Constant */\n",
       ".output_html .nd { color: #AA22FF } /* Name.Decorator */\n",
       ".output_html .ni { color: #717171; font-weight: bold } /* Name.Entity */\n",
       ".output_html .ne { color: #CB3F38; font-weight: bold } /* Name.Exception */\n",
       ".output_html .nf { color: #0000FF } /* Name.Function */\n",
       ".output_html .nl { color: #767600 } /* Name.Label */\n",
       ".output_html .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       ".output_html .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       ".output_html .nv { color: #19177C } /* Name.Variable */\n",
       ".output_html .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       ".output_html .w { color: #bbbbbb } /* Text.Whitespace */\n",
       ".output_html .mb { color: #666666 } /* Literal.Number.Bin */\n",
       ".output_html .mf { color: #666666 } /* Literal.Number.Float */\n",
       ".output_html .mh { color: #666666 } /* Literal.Number.Hex */\n",
       ".output_html .mi { color: #666666 } /* Literal.Number.Integer */\n",
       ".output_html .mo { color: #666666 } /* Literal.Number.Oct */\n",
       ".output_html .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       ".output_html .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       ".output_html .sc { color: #BA2121 } /* Literal.String.Char */\n",
       ".output_html .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       ".output_html .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       ".output_html .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       ".output_html .se { color: #AA5D1F; font-weight: bold } /* Literal.String.Escape */\n",
       ".output_html .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       ".output_html .si { color: #A45A77; font-weight: bold } /* Literal.String.Interpol */\n",
       ".output_html .sx { color: #008000 } /* Literal.String.Other */\n",
       ".output_html .sr { color: #A45A77 } /* Literal.String.Regex */\n",
       ".output_html .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       ".output_html .ss { color: #19177C } /* Literal.String.Symbol */\n",
       ".output_html .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       ".output_html .fm { color: #0000FF } /* Name.Function.Magic */\n",
       ".output_html .vc { color: #19177C } /* Name.Variable.Class */\n",
       ".output_html .vg { color: #19177C } /* Name.Variable.Global */\n",
       ".output_html .vi { color: #19177C } /* Name.Variable.Instance */\n",
       ".output_html .vm { color: #19177C } /* Name.Variable.Magic */\n",
       ".output_html .il { color: #666666 } /* Literal.Number.Integer.Long */</style><div class=\"highlight\"><pre><span></span><span class=\"c1\">//saxpy-1d-workflow.c</span>\n",
       "<span class=\"cp\">#include</span><span class=\"w\"> </span><span class=\"cpf\">&lt;stdio.h&gt;</span><span class=\"cp\"></span>\n",
       "<span class=\"cp\">#include</span><span class=\"w\"> </span><span class=\"cpf\">&lt;stdlib.h&gt;</span><span class=\"cp\"></span>\n",
       "<span class=\"cp\">#include</span><span class=\"w\"> </span><span class=\"cpf\">&lt;cilk.h&gt;</span><span class=\"cp\"></span>\n",
       "\n",
       "<span class=\"c1\">//If x86 is specified use the x86 compatibility headers with Cilk; otherwise use the Lucata toolchain</span>\n",
       "<span class=\"cp\">#ifdef X86</span>\n",
       "<span class=\"w\">  </span><span class=\"cp\">#include</span><span class=\"w\"> </span><span class=\"cpf\">&quot;memoryweb_x86.h&quot;</span><span class=\"cp\"></span>\n",
       "<span class=\"w\">  </span><span class=\"cp\">#include</span><span class=\"w\"> </span><span class=\"cpf\">&lt;emu_c_utils.h&gt;</span><span class=\"cp\"></span>\n",
       "<span class=\"cp\">#else</span>\n",
       "<span class=\"w\">  </span><span class=\"cp\">#include</span><span class=\"w\"> </span><span class=\"cpf\">&quot;memoryweb.h&quot;</span><span class=\"cp\"></span>\n",
       "<span class=\"w\">  </span><span class=\"cp\">#include</span><span class=\"w\"> </span><span class=\"cpf\">&lt;emu_c_utils/emu_c_utils.h&gt;</span><span class=\"cp\"></span>\n",
       "<span class=\"cp\">#endif</span>\n",
       "\n",
       "<span class=\"kt\">void</span><span class=\"w\"> </span><span class=\"nf\">saxpy</span><span class=\"p\">(</span><span class=\"kt\">long</span><span class=\"w\"> </span><span class=\"n\">n</span><span class=\"p\">,</span><span class=\"w\"> </span><span class=\"kt\">long</span><span class=\"w\"> </span><span class=\"n\">a</span><span class=\"p\">,</span><span class=\"w\"> </span><span class=\"kt\">long</span><span class=\"w\"> </span><span class=\"o\">*</span><span class=\"n\">x</span><span class=\"p\">,</span><span class=\"w\"> </span><span class=\"kt\">long</span><span class=\"w\"> </span><span class=\"o\">*</span><span class=\"n\">y</span><span class=\"p\">)</span><span class=\"w\"></span>\n",
       "<span class=\"p\">{</span><span class=\"w\"></span>\n",
       "<span class=\"w\">  </span><span class=\"k\">for</span><span class=\"w\"> </span><span class=\"p\">(</span><span class=\"kt\">long</span><span class=\"w\"> </span><span class=\"n\">i</span><span class=\"w\"> </span><span class=\"o\">=</span><span class=\"w\"> </span><span class=\"mi\">0</span><span class=\"p\">;</span><span class=\"w\"> </span><span class=\"n\">i</span><span class=\"w\"> </span><span class=\"o\">&lt;</span><span class=\"w\"> </span><span class=\"n\">n</span><span class=\"p\">;</span><span class=\"w\"> </span><span class=\"n\">i</span><span class=\"o\">++</span><span class=\"p\">)</span><span class=\"w\"></span>\n",
       "<span class=\"w\">    </span><span class=\"n\">y</span><span class=\"p\">[</span><span class=\"n\">i</span><span class=\"p\">]</span><span class=\"w\"> </span><span class=\"o\">+=</span><span class=\"w\"> </span><span class=\"n\">a</span><span class=\"w\"> </span><span class=\"o\">*</span><span class=\"w\"> </span><span class=\"n\">x</span><span class=\"p\">[</span><span class=\"n\">i</span><span class=\"p\">];</span><span class=\"w\"></span>\n",
       "<span class=\"p\">}</span><span class=\"w\"></span>\n",
       "\n",
       "<span class=\"kt\">int</span><span class=\"w\"> </span><span class=\"nf\">main</span><span class=\"p\">(</span><span class=\"kt\">int</span><span class=\"w\"> </span><span class=\"n\">argc</span><span class=\"p\">,</span><span class=\"w\"> </span><span class=\"kt\">char</span><span class=\"w\"> </span><span class=\"o\">**</span><span class=\"n\">argv</span><span class=\"p\">)</span><span class=\"w\"></span>\n",
       "<span class=\"p\">{</span><span class=\"w\"></span>\n",
       "<span class=\"w\">  </span><span class=\"k\">if</span><span class=\"p\">(</span><span class=\"n\">argc</span><span class=\"w\"> </span><span class=\"o\">&lt;</span><span class=\"w\"> </span><span class=\"mi\">4</span><span class=\"p\">){</span><span class=\"w\">      </span>\n",
       "<span class=\"w\">      </span><span class=\"n\">printf</span><span class=\"p\">(</span><span class=\"s\">&quot;Pass at least 3 arguments!</span><span class=\"se\">\\n</span><span class=\"s\">&quot;</span><span class=\"p\">);</span><span class=\"w\"></span>\n",
       "<span class=\"w\">      </span><span class=\"n\">printf</span><span class=\"p\">(</span><span class=\"s\">&quot;saxpy-1d-workflow &lt;num_threads&gt; &lt;array_size&gt; &lt;constant&gt;</span><span class=\"se\">\\n</span><span class=\"s\">&quot;</span><span class=\"p\">);</span><span class=\"w\"></span>\n",
       "<span class=\"w\">      </span><span class=\"n\">exit</span><span class=\"p\">(</span><span class=\"mi\">1</span><span class=\"p\">);</span><span class=\"w\"></span>\n",
       "<span class=\"w\">  </span><span class=\"p\">}</span><span class=\"w\">  </span>\n",
       "<span class=\"w\">    </span>\n",
       "<span class=\"w\">  </span><span class=\"kt\">long</span><span class=\"w\"> </span><span class=\"n\">nth</span><span class=\"w\"> </span><span class=\"o\">=</span><span class=\"w\"> </span><span class=\"n\">atol</span><span class=\"p\">(</span><span class=\"n\">argv</span><span class=\"p\">[</span><span class=\"mi\">1</span><span class=\"p\">]);</span><span class=\"w\"> </span><span class=\"c1\">// number threads</span>\n",
       "<span class=\"w\">  </span><span class=\"kt\">long</span><span class=\"w\"> </span><span class=\"n\">size</span><span class=\"w\"> </span><span class=\"o\">=</span><span class=\"w\"> </span><span class=\"n\">atol</span><span class=\"p\">(</span><span class=\"n\">argv</span><span class=\"p\">[</span><span class=\"mi\">2</span><span class=\"p\">]);</span><span class=\"w\"> </span><span class=\"c1\">// array size</span>\n",
       "<span class=\"w\">  </span><span class=\"kt\">long</span><span class=\"w\"> </span><span class=\"n\">aval</span><span class=\"w\"> </span><span class=\"o\">=</span><span class=\"w\"> </span><span class=\"n\">atol</span><span class=\"p\">(</span><span class=\"n\">argv</span><span class=\"p\">[</span><span class=\"mi\">3</span><span class=\"p\">]);</span><span class=\"w\"> </span><span class=\"c1\">// constant</span>\n",
       "<span class=\"w\">  </span>\n",
       "<span class=\"w\">  </span><span class=\"kt\">long</span><span class=\"w\"> </span><span class=\"o\">*</span><span class=\"n\">x</span><span class=\"w\"> </span><span class=\"o\">=</span><span class=\"w\"> </span><span class=\"n\">mw_malloc1dlong</span><span class=\"p\">(</span><span class=\"n\">size</span><span class=\"p\">);</span><span class=\"w\"></span>\n",
       "<span class=\"w\">  </span><span class=\"kt\">long</span><span class=\"w\"> </span><span class=\"o\">*</span><span class=\"n\">y</span><span class=\"w\"> </span><span class=\"o\">=</span><span class=\"w\"> </span><span class=\"n\">mw_malloc1dlong</span><span class=\"p\">(</span><span class=\"n\">size</span><span class=\"p\">);</span><span class=\"w\"></span>\n",
       "\n",
       "<span class=\"w\">  </span><span class=\"k\">for</span><span class=\"w\"> </span><span class=\"p\">(</span><span class=\"kt\">long</span><span class=\"w\"> </span><span class=\"n\">i</span><span class=\"w\"> </span><span class=\"o\">=</span><span class=\"w\"> </span><span class=\"mi\">0</span><span class=\"p\">;</span><span class=\"w\"> </span><span class=\"n\">i</span><span class=\"w\"> </span><span class=\"o\">&lt;</span><span class=\"w\"> </span><span class=\"n\">size</span><span class=\"p\">;</span><span class=\"w\"> </span><span class=\"n\">i</span><span class=\"o\">++</span><span class=\"p\">)</span><span class=\"w\"> </span><span class=\"p\">{</span><span class=\"w\"></span>\n",
       "<span class=\"w\">    </span><span class=\"n\">x</span><span class=\"p\">[</span><span class=\"n\">i</span><span class=\"p\">]</span><span class=\"w\"> </span><span class=\"o\">=</span><span class=\"w\"> </span><span class=\"n\">i</span><span class=\"p\">;</span><span class=\"w\"> </span><span class=\"n\">y</span><span class=\"p\">[</span><span class=\"n\">i</span><span class=\"p\">]</span><span class=\"w\"> </span><span class=\"o\">=</span><span class=\"w\"> </span><span class=\"mi\">0</span><span class=\"p\">;</span><span class=\"w\"></span>\n",
       "<span class=\"w\">  </span><span class=\"p\">}</span><span class=\"w\"></span>\n",
       "\n",
       "<span class=\"w\">  </span><span class=\"kt\">long</span><span class=\"w\"> </span><span class=\"n\">grain</span><span class=\"w\"> </span><span class=\"o\">=</span><span class=\"w\"> </span><span class=\"n\">size</span><span class=\"w\"> </span><span class=\"o\">/</span><span class=\"w\"> </span><span class=\"n\">nth</span><span class=\"p\">;</span><span class=\"w\"> </span><span class=\"c1\">// elts per thread</span>\n",
       "\n",
       "<span class=\"w\">  </span><span class=\"c1\">//Timing flag to simulation only; ignored on x86 and HW</span>\n",
       "<span class=\"w\">  </span><span class=\"n\">starttiming</span><span class=\"p\">();</span><span class=\"w\"></span>\n",
       "<span class=\"w\">  </span><span class=\"k\">for</span><span class=\"w\"> </span><span class=\"p\">(</span><span class=\"kt\">long</span><span class=\"w\"> </span><span class=\"n\">i</span><span class=\"w\"> </span><span class=\"o\">=</span><span class=\"w\"> </span><span class=\"mi\">0</span><span class=\"p\">,</span><span class=\"w\"> </span><span class=\"n\">j</span><span class=\"w\"> </span><span class=\"o\">=</span><span class=\"w\"> </span><span class=\"mi\">0</span><span class=\"p\">;</span><span class=\"w\"> </span><span class=\"n\">i</span><span class=\"w\"> </span><span class=\"o\">&lt;</span><span class=\"w\"> </span><span class=\"n\">nth</span><span class=\"p\">;</span><span class=\"w\"> </span><span class=\"n\">i</span><span class=\"o\">++</span><span class=\"p\">,</span><span class=\"w\"> </span><span class=\"n\">j</span><span class=\"w\"> </span><span class=\"o\">+=</span><span class=\"w\"> </span><span class=\"n\">grain</span><span class=\"p\">)</span><span class=\"w\"></span>\n",
       "<span class=\"w\">    </span><span class=\"n\">cilk_spawn</span><span class=\"w\"> </span><span class=\"n\">saxpy</span><span class=\"p\">(</span><span class=\"n\">grain</span><span class=\"p\">,</span><span class=\"w\"> </span><span class=\"n\">aval</span><span class=\"p\">,</span><span class=\"w\"> </span><span class=\"o\">&amp;</span><span class=\"n\">x</span><span class=\"p\">[</span><span class=\"n\">j</span><span class=\"p\">],</span><span class=\"w\"> </span><span class=\"o\">&amp;</span><span class=\"n\">y</span><span class=\"p\">[</span><span class=\"n\">j</span><span class=\"p\">]);</span><span class=\"w\"></span>\n",
       "<span class=\"w\">  </span><span class=\"n\">cilk_sync</span><span class=\"p\">;</span><span class=\"w\"></span>\n",
       "<span class=\"w\">    </span>\n",
       "<span class=\"w\">  </span><span class=\"n\">printf</span><span class=\"p\">(</span><span class=\"s\">&quot;SAXPY complete!</span><span class=\"se\">\\n</span><span class=\"s\">&quot;</span><span class=\"p\">);</span><span class=\"w\"></span>\n",
       "<span class=\"p\">}</span><span class=\"w\"></span>\n",
       "</pre></div>\n"
      ],
      "text/latex": [
       "\\begin{Verbatim}[commandchars=\\\\\\{\\}]\n",
       "\\PY{c+c1}{//saxpy\\PYZhy{}1d\\PYZhy{}workflow.c}\n",
       "\\PY{c+cp}{\\PYZsh{}}\\PY{c+cp}{include}\\PY{+w}{ }\\PY{c+cpf}{\\PYZlt{}stdio.h\\PYZgt{}}\n",
       "\\PY{c+cp}{\\PYZsh{}}\\PY{c+cp}{include}\\PY{+w}{ }\\PY{c+cpf}{\\PYZlt{}stdlib.h\\PYZgt{}}\n",
       "\\PY{c+cp}{\\PYZsh{}}\\PY{c+cp}{include}\\PY{+w}{ }\\PY{c+cpf}{\\PYZlt{}cilk.h\\PYZgt{}}\n",
       "\n",
       "\\PY{c+c1}{//If x86 is specified use the x86 compatibility headers with Cilk; otherwise use the Lucata toolchain}\n",
       "\\PY{c+cp}{\\PYZsh{}}\\PY{c+cp}{ifdef X86}\n",
       "\\PY{+w}{  }\\PY{c+cp}{\\PYZsh{}}\\PY{c+cp}{include}\\PY{+w}{ }\\PY{c+cpf}{\\PYZdq{}memoryweb\\PYZus{}x86.h\\PYZdq{}}\n",
       "\\PY{+w}{  }\\PY{c+cp}{\\PYZsh{}}\\PY{c+cp}{include}\\PY{+w}{ }\\PY{c+cpf}{\\PYZlt{}emu\\PYZus{}c\\PYZus{}utils.h\\PYZgt{}}\n",
       "\\PY{c+cp}{\\PYZsh{}}\\PY{c+cp}{else}\n",
       "\\PY{+w}{  }\\PY{c+cp}{\\PYZsh{}}\\PY{c+cp}{include}\\PY{+w}{ }\\PY{c+cpf}{\\PYZdq{}memoryweb.h\\PYZdq{}}\n",
       "\\PY{+w}{  }\\PY{c+cp}{\\PYZsh{}}\\PY{c+cp}{include}\\PY{+w}{ }\\PY{c+cpf}{\\PYZlt{}emu\\PYZus{}c\\PYZus{}utils/emu\\PYZus{}c\\PYZus{}utils.h\\PYZgt{}}\n",
       "\\PY{c+cp}{\\PYZsh{}}\\PY{c+cp}{endif}\n",
       "\n",
       "\\PY{k+kt}{void}\\PY{+w}{ }\\PY{n+nf}{saxpy}\\PY{p}{(}\\PY{k+kt}{long}\\PY{+w}{ }\\PY{n}{n}\\PY{p}{,}\\PY{+w}{ }\\PY{k+kt}{long}\\PY{+w}{ }\\PY{n}{a}\\PY{p}{,}\\PY{+w}{ }\\PY{k+kt}{long}\\PY{+w}{ }\\PY{o}{*}\\PY{n}{x}\\PY{p}{,}\\PY{+w}{ }\\PY{k+kt}{long}\\PY{+w}{ }\\PY{o}{*}\\PY{n}{y}\\PY{p}{)}\n",
       "\\PY{p}{\\PYZob{}}\n",
       "\\PY{+w}{  }\\PY{k}{for}\\PY{+w}{ }\\PY{p}{(}\\PY{k+kt}{long}\\PY{+w}{ }\\PY{n}{i}\\PY{+w}{ }\\PY{o}{=}\\PY{+w}{ }\\PY{l+m+mi}{0}\\PY{p}{;}\\PY{+w}{ }\\PY{n}{i}\\PY{+w}{ }\\PY{o}{\\PYZlt{}}\\PY{+w}{ }\\PY{n}{n}\\PY{p}{;}\\PY{+w}{ }\\PY{n}{i}\\PY{o}{+}\\PY{o}{+}\\PY{p}{)}\n",
       "\\PY{+w}{    }\\PY{n}{y}\\PY{p}{[}\\PY{n}{i}\\PY{p}{]}\\PY{+w}{ }\\PY{o}{+}\\PY{o}{=}\\PY{+w}{ }\\PY{n}{a}\\PY{+w}{ }\\PY{o}{*}\\PY{+w}{ }\\PY{n}{x}\\PY{p}{[}\\PY{n}{i}\\PY{p}{]}\\PY{p}{;}\n",
       "\\PY{p}{\\PYZcb{}}\n",
       "\n",
       "\\PY{k+kt}{int}\\PY{+w}{ }\\PY{n+nf}{main}\\PY{p}{(}\\PY{k+kt}{int}\\PY{+w}{ }\\PY{n}{argc}\\PY{p}{,}\\PY{+w}{ }\\PY{k+kt}{char}\\PY{+w}{ }\\PY{o}{*}\\PY{o}{*}\\PY{n}{argv}\\PY{p}{)}\n",
       "\\PY{p}{\\PYZob{}}\n",
       "\\PY{+w}{  }\\PY{k}{if}\\PY{p}{(}\\PY{n}{argc}\\PY{+w}{ }\\PY{o}{\\PYZlt{}}\\PY{+w}{ }\\PY{l+m+mi}{4}\\PY{p}{)}\\PY{p}{\\PYZob{}}\\PY{+w}{      }\n",
       "\\PY{+w}{      }\\PY{n}{printf}\\PY{p}{(}\\PY{l+s}{\\PYZdq{}}\\PY{l+s}{Pass at least 3 arguments!}\\PY{l+s+se}{\\PYZbs{}n}\\PY{l+s}{\\PYZdq{}}\\PY{p}{)}\\PY{p}{;}\n",
       "\\PY{+w}{      }\\PY{n}{printf}\\PY{p}{(}\\PY{l+s}{\\PYZdq{}}\\PY{l+s}{saxpy\\PYZhy{}1d\\PYZhy{}workflow \\PYZlt{}num\\PYZus{}threads\\PYZgt{} \\PYZlt{}array\\PYZus{}size\\PYZgt{} \\PYZlt{}constant\\PYZgt{}}\\PY{l+s+se}{\\PYZbs{}n}\\PY{l+s}{\\PYZdq{}}\\PY{p}{)}\\PY{p}{;}\n",
       "\\PY{+w}{      }\\PY{n}{exit}\\PY{p}{(}\\PY{l+m+mi}{1}\\PY{p}{)}\\PY{p}{;}\n",
       "\\PY{+w}{  }\\PY{p}{\\PYZcb{}}\\PY{+w}{  }\n",
       "\\PY{+w}{    }\n",
       "\\PY{+w}{  }\\PY{k+kt}{long}\\PY{+w}{ }\\PY{n}{nth}\\PY{+w}{ }\\PY{o}{=}\\PY{+w}{ }\\PY{n}{atol}\\PY{p}{(}\\PY{n}{argv}\\PY{p}{[}\\PY{l+m+mi}{1}\\PY{p}{]}\\PY{p}{)}\\PY{p}{;}\\PY{+w}{ }\\PY{c+c1}{// number threads}\n",
       "\\PY{+w}{  }\\PY{k+kt}{long}\\PY{+w}{ }\\PY{n}{size}\\PY{+w}{ }\\PY{o}{=}\\PY{+w}{ }\\PY{n}{atol}\\PY{p}{(}\\PY{n}{argv}\\PY{p}{[}\\PY{l+m+mi}{2}\\PY{p}{]}\\PY{p}{)}\\PY{p}{;}\\PY{+w}{ }\\PY{c+c1}{// array size}\n",
       "\\PY{+w}{  }\\PY{k+kt}{long}\\PY{+w}{ }\\PY{n}{aval}\\PY{+w}{ }\\PY{o}{=}\\PY{+w}{ }\\PY{n}{atol}\\PY{p}{(}\\PY{n}{argv}\\PY{p}{[}\\PY{l+m+mi}{3}\\PY{p}{]}\\PY{p}{)}\\PY{p}{;}\\PY{+w}{ }\\PY{c+c1}{// constant}\n",
       "\\PY{+w}{  }\n",
       "\\PY{+w}{  }\\PY{k+kt}{long}\\PY{+w}{ }\\PY{o}{*}\\PY{n}{x}\\PY{+w}{ }\\PY{o}{=}\\PY{+w}{ }\\PY{n}{mw\\PYZus{}malloc1dlong}\\PY{p}{(}\\PY{n}{size}\\PY{p}{)}\\PY{p}{;}\n",
       "\\PY{+w}{  }\\PY{k+kt}{long}\\PY{+w}{ }\\PY{o}{*}\\PY{n}{y}\\PY{+w}{ }\\PY{o}{=}\\PY{+w}{ }\\PY{n}{mw\\PYZus{}malloc1dlong}\\PY{p}{(}\\PY{n}{size}\\PY{p}{)}\\PY{p}{;}\n",
       "\n",
       "\\PY{+w}{  }\\PY{k}{for}\\PY{+w}{ }\\PY{p}{(}\\PY{k+kt}{long}\\PY{+w}{ }\\PY{n}{i}\\PY{+w}{ }\\PY{o}{=}\\PY{+w}{ }\\PY{l+m+mi}{0}\\PY{p}{;}\\PY{+w}{ }\\PY{n}{i}\\PY{+w}{ }\\PY{o}{\\PYZlt{}}\\PY{+w}{ }\\PY{n}{size}\\PY{p}{;}\\PY{+w}{ }\\PY{n}{i}\\PY{o}{+}\\PY{o}{+}\\PY{p}{)}\\PY{+w}{ }\\PY{p}{\\PYZob{}}\n",
       "\\PY{+w}{    }\\PY{n}{x}\\PY{p}{[}\\PY{n}{i}\\PY{p}{]}\\PY{+w}{ }\\PY{o}{=}\\PY{+w}{ }\\PY{n}{i}\\PY{p}{;}\\PY{+w}{ }\\PY{n}{y}\\PY{p}{[}\\PY{n}{i}\\PY{p}{]}\\PY{+w}{ }\\PY{o}{=}\\PY{+w}{ }\\PY{l+m+mi}{0}\\PY{p}{;}\n",
       "\\PY{+w}{  }\\PY{p}{\\PYZcb{}}\n",
       "\n",
       "\\PY{+w}{  }\\PY{k+kt}{long}\\PY{+w}{ }\\PY{n}{grain}\\PY{+w}{ }\\PY{o}{=}\\PY{+w}{ }\\PY{n}{size}\\PY{+w}{ }\\PY{o}{/}\\PY{+w}{ }\\PY{n}{nth}\\PY{p}{;}\\PY{+w}{ }\\PY{c+c1}{// elts per thread}\n",
       "\n",
       "\\PY{+w}{  }\\PY{c+c1}{//Timing flag to simulation only; ignored on x86 and HW}\n",
       "\\PY{+w}{  }\\PY{n}{starttiming}\\PY{p}{(}\\PY{p}{)}\\PY{p}{;}\n",
       "\\PY{+w}{  }\\PY{k}{for}\\PY{+w}{ }\\PY{p}{(}\\PY{k+kt}{long}\\PY{+w}{ }\\PY{n}{i}\\PY{+w}{ }\\PY{o}{=}\\PY{+w}{ }\\PY{l+m+mi}{0}\\PY{p}{,}\\PY{+w}{ }\\PY{n}{j}\\PY{+w}{ }\\PY{o}{=}\\PY{+w}{ }\\PY{l+m+mi}{0}\\PY{p}{;}\\PY{+w}{ }\\PY{n}{i}\\PY{+w}{ }\\PY{o}{\\PYZlt{}}\\PY{+w}{ }\\PY{n}{nth}\\PY{p}{;}\\PY{+w}{ }\\PY{n}{i}\\PY{o}{+}\\PY{o}{+}\\PY{p}{,}\\PY{+w}{ }\\PY{n}{j}\\PY{+w}{ }\\PY{o}{+}\\PY{o}{=}\\PY{+w}{ }\\PY{n}{grain}\\PY{p}{)}\n",
       "\\PY{+w}{    }\\PY{n}{cilk\\PYZus{}spawn}\\PY{+w}{ }\\PY{n}{saxpy}\\PY{p}{(}\\PY{n}{grain}\\PY{p}{,}\\PY{+w}{ }\\PY{n}{aval}\\PY{p}{,}\\PY{+w}{ }\\PY{o}{\\PYZam{}}\\PY{n}{x}\\PY{p}{[}\\PY{n}{j}\\PY{p}{]}\\PY{p}{,}\\PY{+w}{ }\\PY{o}{\\PYZam{}}\\PY{n}{y}\\PY{p}{[}\\PY{n}{j}\\PY{p}{]}\\PY{p}{)}\\PY{p}{;}\n",
       "\\PY{+w}{  }\\PY{n}{cilk\\PYZus{}sync}\\PY{p}{;}\n",
       "\\PY{+w}{    }\n",
       "\\PY{+w}{  }\\PY{n}{printf}\\PY{p}{(}\\PY{l+s}{\\PYZdq{}}\\PY{l+s}{SAXPY complete!}\\PY{l+s+se}{\\PYZbs{}n}\\PY{l+s}{\\PYZdq{}}\\PY{p}{)}\\PY{p}{;}\n",
       "\\PY{p}{\\PYZcb{}}\n",
       "\\end{Verbatim}\n"
      ],
      "text/plain": [
       "//saxpy-1d-workflow.c\n",
       "#include <stdio.h>\n",
       "#include <stdlib.h>\n",
       "#include <cilk.h>\n",
       "\n",
       "//If x86 is specified use the x86 compatibility headers with Cilk; otherwise use the Lucata toolchain\n",
       "#ifdef X86\n",
       "  #include \"memoryweb_x86.h\"\n",
       "  #include <emu_c_utils.h>\n",
       "#else\n",
       "  #include \"memoryweb.h\"\n",
       "  #include <emu_c_utils/emu_c_utils.h>\n",
       "#endif\n",
       "\n",
       "void saxpy(long n, long a, long *x, long *y)\n",
       "{\n",
       "  for (long i = 0; i < n; i++)\n",
       "    y[i] += a * x[i];\n",
       "}\n",
       "\n",
       "int main(int argc, char **argv)\n",
       "{\n",
       "  if(argc < 4){      \n",
       "      printf(\"Pass at least 3 arguments!\\n\");\n",
       "      printf(\"saxpy-1d-workflow <num_threads> <array_size> <constant>\\n\");\n",
       "      exit(1);\n",
       "  }  \n",
       "    \n",
       "  long nth = atol(argv[1]); // number threads\n",
       "  long size = atol(argv[2]); // array size\n",
       "  long aval = atol(argv[3]); // constant\n",
       "  \n",
       "  long *x = mw_malloc1dlong(size);\n",
       "  long *y = mw_malloc1dlong(size);\n",
       "\n",
       "  for (long i = 0; i < size; i++) {\n",
       "    x[i] = i; y[i] = 0;\n",
       "  }\n",
       "\n",
       "  long grain = size / nth; // elts per thread\n",
       "\n",
       "  //Timing flag to simulation only; ignored on x86 and HW\n",
       "  starttiming();\n",
       "  for (long i = 0, j = 0; i < nth; i++, j += grain)\n",
       "    cilk_spawn saxpy(grain, aval, &x[j], &y[j]);\n",
       "  cilk_sync;\n",
       "    \n",
       "  printf(\"SAXPY complete!\\n\");\n",
       "}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Code('saxpy-1d-workflow.c')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then compile this code for execution on an x86 system as follows. Note that we are using the [OpenCilk version of Clang](https://www.opencilk.org/), which is not Lucata-specific! Note that we have added both `${X86FLAGS}` and `-DX86` to our compilation string so that `memoryweb_x86.h` is used instead of `memoryweb.h`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Makefile\n",
      "NB04-saxpy-workflow.html\n",
      "NB04-saxpy-workflow.ipynb\n",
      "saxpy-1d-workflow.c\n",
      "saxpy-1d-workflow.cdc\n",
      "saxpy-1d-workflow.mwx\n",
      "saxpy-1d-workflow-x86\n",
      "sbatch-hello-world.sh\n",
      "sbatch-saxpy-mn.sh\n",
      "sbatch-saxpy-sn.sh\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "+ /tools/lucata/opencilk-2.0.1/bin/clang -I/tools/lucata/pathfinder-sw/22.09-beta/include/cilk/ -I/tools/emu/pathfinder-sw/22.09-beta/x86/include/emu_c_utils /tools/emu/pathfinder-sw/22.09-beta/x86/lib/libemu_c_utils.a -DX86 saxpy-1d-workflow.c -o saxpy-1d-workflow-x86\n",
      "+ set +x\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "set -x\n",
    "/tools/lucata/opencilk-2.0.1/bin/clang ${X86FLAGS} -DX86 saxpy-1d-workflow.c -o saxpy-1d-workflow-x86\n",
    "set +x\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing out x86 SAXPY\n",
    "\n",
    "After compiling with OpenCilk, we run the Saxpy example with 8 threads, an array of size 2^20 and a constant value of 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SAXPY complete!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "real\t0m0.022s\n",
      "user\t0m0.013s\n",
      "sys\t0m0.006s\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "time CILK_NWORKERS=1 ./saxpy-1d-workflow-x86 8 1048576 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing SAXPY with additional x86 threads\n",
    "\n",
    "Rerun the example with more workers - the timing should go down slightly. Remember you may be running this notebook with a limited number of threads (by default usually 2), so adding more workers may not substantially speed this example up unless you run the notebook with more cores!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SAXPY complete!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "real\t0m0.020s\n",
      "user\t0m0.011s\n",
      "sys\t0m0.008s\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "time CILK_NWORKERS=2 ./saxpy-1d-workflow-x86 8 1048576 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Simulator Execution\n",
    "\n",
    "Once we have tested our program with x86 Cilk execution we can proceed to test with the Lucata simulator, `emusim.x`. This simulator is a single-threaded simulator that operates on a detailed SystemC model of the Pathfinder system. As such, it is somewhat slow and should normally be used in \"untimed\" mode to verify functionality. Since the Pathfinder hardware does not currently include a debugger or runtime profiler, the simulator should also be used to debug issues and check basic performance profiling.\n",
    "\n",
    "![Lucata Workflow](../resources/figs/lucata_pathfinder_workflow_2_emusim.png)\n",
    "\n",
    "The best way to limit the runtime of the simulator is to run in \"untimed\" mode, meaning that no relative clocks are used to estimate performance of the application. You can do this by either commenting out any `starttiming()` calls in your code or using the `--ignore_starttiming` flag when simulating your code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/net/tools/emu/pathfinder-sw/22.09-beta/bin/clang: /lib64/libtinfo.so.5: no version information available (required by /net/tools/emu/pathfinder-sw/22.09-beta/bin/clang)\n",
      "/net/tools/emu/pathfinder-sw/22.09-beta/bin/clang-6.0: /lib64/libtinfo.so.5: no version information available (required by /net/tools/emu/pathfinder-sw/22.09-beta/bin/clang-6.0)\n",
      "/net/tools/emu/pathfinder-sw/22.09-beta/bin/opt: /lib64/libtinfo.so.5: no version information available (required by /net/tools/emu/pathfinder-sw/22.09-beta/bin/opt)\n",
      "/net/tools/emu/pathfinder-sw/22.09-beta/bin/opt: /lib64/libtinfo.so.5: no version information available (required by /net/tools/emu/pathfinder-sw/22.09-beta/bin/opt)\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "emu-cc -o saxpy-1d-workflow.mwx $FLAGS saxpy-1d-workflow.c -lemu_c_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start untimed simulation with local date and time= Tue Sep 27 12:23:11 2022\n",
      "\n",
      "SAXPY complete!\n",
      "End untimed simulation with local date and time= Tue Sep 27 12:23:11 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "        SystemC 2.3.3-Accellera --- Sep  7 2022 09:15:59\n",
      "        Copyright (c) 1996-2018 by all Contributors,\n",
      "        ALL RIGHTS RESERVED\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "emusim.x --ignore_starttiming -m 24 --total_nodes 1 saxpy-1d-workflow.mwx 8 128 5.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Simulating with more nodes\n",
    "\n",
    "We can also simulate a larger system size as we have seen in previous notebook examples. Here we test with four simulated Pathfinder nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start untimed simulation with local date and time= Tue Sep 27 12:23:11 2022\n",
      "\n",
      "SAXPY complete!\n",
      "End untimed simulation with local date and time= Tue Sep 27 12:23:11 2022\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "        SystemC 2.3.3-Accellera --- Sep  7 2022 09:15:59\n",
      "        Copyright (c) 1996-2018 by all Contributors,\n",
      "        ALL RIGHTS RESERVED\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "emusim.x --ignore_starttiming -m 24 --total_nodes 4 saxpy-1d-workflow.mwx 8 128 5.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Hardware Execution\n",
    "\n",
    "Once our code works with the simulator, and we feel we have optimized it enough it is time to start running on the hardware. You should first try to run your code in single-node fashion on the Pathfinder and then scale up to multi-node execution. Note that the `emu_handler_and_loader` command is meant for single-node execution while `emu_multinode_exec` is meant for multinode execution. All multi-node jobs must be run from node 0 in the Pathfinder system.\n",
    "\n",
    "![Lucata Workflow](../resources/figs/lucata_pathfinder_workflow_3_emuhw.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using Slurm for hardware execution\n",
    "\n",
    "The Rogues Gallery recently has migrated to using Slurm for execution of code on the Pathfinder system. For more information on how to use Slurm and commands like sbatch, please see our [ReadTheDocs page for Slurm](https://gt-crnch-rg.readthedocs.io/en/main/general/using-slurm.html). \n",
    "\n",
    "Here we will cover basic execution of a single node and single-chassis job. For a single node job, we use the quality of service flag `-q single-node` and submit to the Pathfinder cluster `-M pathfinder`. The `--wrap \"emu_handler_and_loader 0 0 -- saxpy-1d-workflow.mwx 8 128 5.0\"` command just runs the standard `emu_handler_and_loader` as part of the Slurm job submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submitted batch job 134217841 on cluster pathfinder\n",
      "\n",
      "Showing Slurm output files:\n",
      "slurm-134217841.out\n",
      "\n",
      "Output from the run:\n",
      "SAXPY complete!\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "#Clean up any older Slurm output files\n",
    "rm -f *.out\n",
    "\n",
    "#Run the Saxpy command with Slurmemu_handler_and_loader saxpy-1d-workflow.mwx 8 128 5.0\n",
    "sbatch -M pathfinder -q single-node --wrap \"emu_handler_and_loader 0 0 -- saxpy-1d-workflow.mwx 8 128 5.0\"\n",
    " \n",
    "#If the job runs successfully, the output file should print out \"SAXPY complete!\"\n",
    "sleep 5  \n",
    "#List the Slurm output files.\n",
    "printf \"\\nShowing Slurm output files:\\n\"\n",
    "ls *.out  \n",
    "#Show the content of the latest output file\n",
    "printf \"\\nOutput from the run:\\n\"\n",
    "less slurm-*.out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using sbatch scripts\n",
    "\n",
    "While we have run the previous two commands from the command-line, it is more effective to create an \"batch file\" that we can reuse. This file is called `sbatch-saxpy-sn.sh` in the current directory. To run this script you would execute the following: `sbatch sbatch-saxpy-sn.sh`.\n",
    "\n",
    "```bash\n",
    "#!/bin/bash\n",
    "#==== Specify the time for the reservation (HH:MM:SS)\n",
    "#SBATCH --time=0:15:00\n",
    "#==== Create a useful name for output files\n",
    "#SBATCH --job-name=saxpy-1d-workflow\n",
    "#==== Specify the cluster to use\n",
    "#SBATCH -M pathfinder\n",
    "#==== Specify the QoS parameter. Valid options are single-node, single-chassis, multi-chassis\n",
    "#SBATCH -q single-node\n",
    "\n",
    "# Run the command on a single node and report the time it took\n",
    "time emu_handler_and_loader 0 0 -- saxpy-1d-workflow.mwx 8 2097152 5.0\n",
    "\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submitted batch job 134217842 on cluster pathfinder\n",
      "\n",
      "Output from the run:\n",
      "SAXPY complete!\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "#Clean up any older Slurm output files\n",
    "rm -f *.out\n",
    "\n",
    "#Run the same command but using a batch file instead of --wrap\n",
    "sbatch sbatch-saxpy-sn.sh\n",
    " \n",
    "#If the job runs successfully, the output file should print out \"SAXPY complete!\"\n",
    "sleep 10 \n",
    "#Show the content of the latest output file\n",
    "printf \"\\nOutput from the run:\\n\"\n",
    "less slurm-*.out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More information on Slurm queues on CRNCH RG\n",
    "\n",
    "We can also run commands like `sinfo` and `squeue` to see the entire state of the RG cluster. Here we run `sinfo --federation` because the Pathfinder is set up with a separate Slurm cluster instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>CLUSTER:</th>\n",
       "      <th>pathfinder</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>PARTITION</th>\n",
       "      <th>AVAIL</th>\n",
       "      <th>TIMELIMIT</th>\n",
       "      <th>NODES</th>\n",
       "      <td>STATE</td>\n",
       "      <td>NODELIST</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">rg-pathfinder*</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">up</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">180-00:00:</th>\n",
       "      <th>1</th>\n",
       "      <td>alloc</td>\n",
       "      <td>c2n1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>idle</td>\n",
       "      <td>c0n[0-7],c1n[0-7],c2n[0,2-7],c3n[0-7]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      CLUSTER:  \\\n",
       "PARTITION      AVAIL TIMELIMIT  NODES    STATE   \n",
       "rg-pathfinder* up    180-00:00: 1        alloc   \n",
       "                                31        idle   \n",
       "\n",
       "                                                                  pathfinder  \n",
       "PARTITION      AVAIL TIMELIMIT  NODES                               NODELIST  \n",
       "rg-pathfinder* up    180-00:00: 1                                       c2n1  \n",
       "                                31     c0n[0-7],c1n[0-7],c2n[0,2-7],c3n[0-7]  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Show the real-time status of the Pathfinder nodes.\n",
    "%sinfo --federation -M pathfinder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Debugging and Optimization Strategy for the Pathfinder\n",
    "\n",
    "If you've made it this far, you likely have some idea of the key steps for compiling and running programs with the Pathfinder. You might notice that there are several steps to compiling and running for the Pathfinder - this is because it is a novel parallel architecture that is constantly evolving!  \n",
    "\n",
    "However, in general there are three phases for running large applications at scale on the Pathfinder: 1) x86 testing for functional correctness and traditional Cilk-based timing information, 2) Lucata emusim testing for functional correctness, 3) Single-node and multi-node testing for timing results. The following steps break down this workflow into more detail and can be used as a reference for your development with the Pathfinder platform.  \n",
    "\n",
    "\n",
    "#### x86 Testing\n",
    "\n",
    "1) Compile on x86 (using memoryweb_x86.h)  \n",
    "2) Run on x86 with a single thread (CILK_NWORKERS = 1)  \n",
    "3) Run on x86 multi-threaded  (CILK_NWORKERS > 1)  \n",
    "\n",
    "#### Lucata simulation\n",
    "4) Compile for Lucata Cilk  \n",
    "5) Simulate  with emusim in untimed mode (--ignore_starttiming)  \n",
    "6) Simulate  on emusim with randomly initialized memory (--initialize_memory)  \n",
    "7) Simulate with emusim in timed mode. (starttiming() in code)  \n",
    "\n",
    "#### Pathfinder execution\n",
    "8) Execute on single-node HW  \n",
    "9) Execute on multi-node HW  \n",
    "10) Increase input size gradually. Always use the smallest input set that will finish/recreate the problem in a reasonable amount of time!  \n",
    "\n",
    "#### Pathfinder optimization\n",
    "11) Use the new hardware performance counters to help find areas for optimization.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises  \n",
    "\n",
    "To further your understanding of this topic we encourage you to try the following:  \n",
    "1) Restart the notebook and change the memory size and numbers of nodes that are simulated. How do the statistics change?  \n",
    "2) Investigate the other output files like the `.mps` file and understand how they are different for different applications.  \n",
    "More details on visualizing these files are in the next Notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Multi-node execution (Optional Exercise)\n",
    "\n",
    "Multi-node jobs run on a different chassis (c0 or c1 for our system), and they run in parallel across all the nodes (8 per chassis) and all the compute engines within a node (currently 24 per node. To run a multi-chassis job, we use the `-q single-chassis` flag which specifies that our code should run on multiple nodes within a chassis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "#Clean up any older Slurm output files\n",
    "rm -f *.out\n",
    "\n",
    "#Note that this may take a bit of time to complete. Uncomment if you'd like to test multi-node execution \n",
    "#sbatch sbatch-saxpy-mn.sh\n",
    "#less slurm-*.out"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}