21.10 Release merge (#14)

* Move the CoveredMask reset flag out of the if statement.

* Add user-defined chemistry tolerances.

* Add a Triple flame in Exec/Cases/TripleFlame

* While making new level from coarse, get 1 ghost cell right so that
AmrNewTime data have one ghost cell properly set.

* Add a simple 2D CH4/Air premixed bunsen case. Tested up to 5 levels.

* Add a 3Dversion of the premixed bunsen flame.

* Need to reset the covered cell mask after restart in case regrid is not called right after.

* Add option to restart an efield simulation from a non-efield one.

* When restarting from non-efield chk, initialize electro-neutral nE
field.

* Fix PlotFile to plot I_R(nE) with efield and write/read in phiV/nE in
chk.

* Machinery to restart from non-efield chk.

* Update Precond operator to implement the second approximation of Stilda.

* Make use of the absolute tolerance in MLGMRES.

* Add fillpatch functions for the non-linear state components.

* Enable Schur complement approximation 2.

* Define and ParmParse m_ef_schur_approx.

* Floor nE along with species if required.

* Fix typos in PeleLMBC.cpp

* Update PeleLMeX header for new fillpatch functions.

* Replace mechanism.h by mechanism.H

* Setup gravity.

* Use gravity in velocity forces.

* Replace .copy by .ParallelCopy.

* Add a lifting hot bubble case in RegTests.

* Default the pprocConvOrder.py to CoVo inputs.

* Enable CoVo in all directions.

* Default input.2d_CoVo to diagonal direction.

* Update pprocConvOrder.py with version checking the convergence order.

* Init convergence testing CI.

* Fix convergence CI (#6)

* Fix deps in convergence CI.

* Update pprocConvOrder.py.

* Fix Pele Physics (#7)

* Update make system to catch up with PP.

* Update sources for latest PP.

* Fix parsing of constant transport parameters.

* Switch AMReX-Hydro (#8)

* Add AMReX-Hydro to deps and GMake.

* Remove LMX Godunov folder from listed sources.

* Fix AMReX-Hydro makefile again and need MOL.

* Swtich to HydroUtils to predict velocity and use
create_umac_grown_constrained.

* Remove stranded Godunov include.

* Add m_advection_type. Only Godunov in LMeX.

* Switch to HydroUtils::ComputeFluxesOnBoxFromState for advection fluxes.

* Remove Godunov from LMeX sources.

* Rewritte the advective flux divergence to comply with AMReX-Hydro way.

* Update CI.

* Feature balance (#9)

* Add kin. energy derive function.

* Add kin.energy derive and ParmParse temporals input.

* Start setting-up run-time diagnostics.

* Update MLNorm0 to not account for fine-covered cells.

* Setup mass balance in runtime diagnostics.

* Add temporal keys to FlameSheet regtests

* Git ignore

* Add gnuplot script for mass balance.

* Fix runtime selection of reactor. (#10)

* Missing AMReX-Hydro home in convergence testing.

* Update Make.PeleLMeX

* Update reactor in Sources.

* Remove one call too many to deallocate of transport.

* Update FlameSheet GMake

* Add CVODE input keys.

* Udpate GMake in Periodic and HotBubble

* Fix typo on ReactorNull default:

* Make LMeX GPU-compatible (#11)

* Remove device from lineaChmeForcing.

* Can't init capture in host_device functions.

* Fix call to host function on device lambdas.

* Change OMP.

* Should be the same, but make sure.

* Don't init transport for incompressible flows.

* Missing Gpu Managed. Will be updated later.

* Minor clean up in PeriodicCases.

* Fix parm in FLameSheet too.

* Restore FlameSheet 3D.

* Remove unused.

* Add Sundials memory helper.

* Remove auto-TPL. Recompile all the source each time. Need fix.

* Restore make TPL in GH workflow for now.

* Fix velocity ghost cells for Nodal projection. Function (#12)

should only overwrite Inflow BCs.

* Implement closed chamber algorithm. (#13)

* Add auto-detection of closed chamber and unable overwrite.
Add PPquery of linear solves tolerances.

* Add GammmaInv kernel.

* Move MFSum in Utils and initialize uncovered volume computation.

* MAC projection function handles closed chamber corrections.

* Nodal projection functions handle -/+ of Sbar in RHS.

* Remove TODO comment.

* Pass dp0dt in diffusion forcing.

* Pass dp0dt in Advection forcing.

* Add pOld <-> pNew in advance function.

* Add adjustPandDivU in Eos file.

* Add accessor to divU levels vector

* Add declarations.

* Add ambient pressure to checkpoint file header.

* Uses pNew to get dPdt.

* Fix BL_PROF in UMAC

* Add an enclosed flame test to test closed chamber.

* Add a CI testing closed chamber.

.github/workflows/convergenceTesting.yml

@@ -0,0 +1,56 @@
+name: ConvergenceTesting
+
+on: 
+   workflow_dispatch:
+   schedule:
+      - cron: '0 2 * * *'
+
+
+jobs:
+  #Run the convergence test on CoVo
+  COVO_Conv:
+    name: Velocity-Advection
+    runs-on: ubuntu-18.04
+    env: 
+      {CXXFLAGS: "-Werror -Wshadow -Woverloaded-virtual -Wunreachable-code"}
+    steps:
+    - name: Cancel previous runs
+      uses: styfle/cancel-workflow-action@0.6.0
+      with:
+        access_token: ${{github.token}}
+    - uses: actions/checkout@v2
+    - name: System Dependencies
+      run: |
+         .github/workflows/dependencies/dependencies.sh
+         sudo apt-get install -y python3-setuptools
+         python3 -m pip install --user numpy
+         python3 -m pip install --user argparse
+         python3 -m pip install --user matplotlib
+    - name: Repo Dependencies
+      run: Utils/CloneDeps.sh
+    - name: Build AMReX Tools
+      env:
+         AMREX_HOME: ${GITHUB_WORKSPACE}/build/amrex
+      working-directory: ./build/amrex/Tools/Plotfile
+      run: |
+        make
+    - name: Build PeleLM
+      env:
+         AMREX_HOME: ${GITHUB_WORKSPACE}/build/amrex
+         PELE_PHYSICS_HOME: ${GITHUB_WORKSPACE}/build/PelePhysics
+         AMREX_HYDRO_HOME: ${GITHUB_WORKSPACE}/build/AMReX-Hydro
+         PELELMEX_HOME: ${GITHUB_WORKSPACE}
+      working-directory: ./Exec/RegTests/PeriodicCases
+      run: |
+        make -j 2 TPL COMP=gnu
+        make -j 2 COMP=gnu
+    - name: Run
+      working-directory: ./Exec/RegTests/PeriodicCases
+      run: |
+        cp ${GITHUB_WORKSPACE}/build/amrex/Tools/Plotfile/fcompare.gnu.ex .
+        sed -i "34s/.*/    resolution = [32,64,128,256,512]/" multiRuns.py
+        sed -i "s/mpiexec -n 1/mpiexec -n 4 --oversubscribe/g" multiRuns.py 
+        ./multiRuns.py --test_name GH_CI_CoVo --input_file input.2d_CoVo
+        sed -i "49s/.*/    resolution = [32,64,128,256,512]/" pprocConvOrder.py
+        ./pprocConvOrder.py ./fcompare.gnu.ex --test_name GH_CI_CoVo
+        cat Convergence_GH_CI_CoVo.dat

.github/workflows/linux.yml

@@ -3,9 +3,9 @@ name: PeleLMeX_linux
 on: 
    workflow_dispatch:
    push:
-     branches: [ main ]
+     branches: [ main, development ]
    pull_request:
-     branches: [ main ]
+     branches: [ main, development ]
 
 jobs:
   #Build the 2D FlameSheet RegTest with GNU7.5 and no MPI support
@@ -29,6 +29,7 @@ jobs:
          AMREX_HOME: ${GITHUB_WORKSPACE}/build/amrex
          PELE_PHYSICS_HOME: ${GITHUB_WORKSPACE}/build/PelePhysics
          PELELM_HOME: ${GITHUB_WORKSPACE}
+         AMREX_HYDRO_HOME: ${GITHUB_WORKSPACE}/build/AMReX-Hydro
       run: |
         cd Exec/RegTests/FlameSheet/
         make TPL COMP=gnu USE_MPI=FALSE
@@ -55,6 +56,7 @@ jobs:
          AMREX_HOME: ${GITHUB_WORKSPACE}/build/amrex
          PELE_PHYSICS_HOME: ${GITHUB_WORKSPACE}/build/PelePhysics
          PELELM_HOME: ${GITHUB_WORKSPACE}
+         AMREX_HYDRO_HOME: ${GITHUB_WORKSPACE}/build/AMReX-Hydro
       run: |
         cd Exec/RegTests/FlameSheet/
         make TPL COMP=gnu
@@ -81,6 +83,7 @@ jobs:
          AMREX_HOME: ${GITHUB_WORKSPACE}/build/amrex
          PELE_PHYSICS_HOME: ${GITHUB_WORKSPACE}/build/PelePhysics
          PELELM_HOME: ${GITHUB_WORKSPACE}
+         AMREX_HYDRO_HOME: ${GITHUB_WORKSPACE}/build/AMReX-Hydro
       run: |
         cd Exec/RegTests/FlameSheet/
         make TPL COMP=gnu USE_OMP=TRUE
@@ -107,10 +110,43 @@ jobs:
          AMREX_HOME: ${GITHUB_WORKSPACE}/build/amrex
          PELE_PHYSICS_HOME: ${GITHUB_WORKSPACE}/build/PelePhysics
          PELELM_HOME: ${GITHUB_WORKSPACE}
+         AMREX_HYDRO_HOME: ${GITHUB_WORKSPACE}/build/AMReX-Hydro
       working-directory: ./Exec/RegTests/PeriodicCases/
       run: |
+        make TPL COMP=gnu USE_MPI=FALSE
         make -j 2 COMP=gnu USE_MPI=FALSE
     - name: Run
       working-directory: ./Exec/RegTests/PeriodicCases/
       run: |
         ./PeleLMeX2d.gnu.ex input.2d_CoVo amr.max_step=2 amr.plot_int=-1 amr.check_int=-1
+
+  # Build and Run the EnclosedFlame RegTest with GNU7.5 and MPI support
+  EF2D_MPIRun:
+    name: GNU@7.5 MPI Run [EF2D]
+    runs-on: ubuntu-latest
+    env: 
+      {CXXFLAGS: "-Werror -Wshadow -Woverloaded-virtual -Wunreachable-code"}
+    steps:
+    - name: Cancel previous runs
+      uses: styfle/cancel-workflow-action@0.6.0
+      with:
+        access_token: ${{github.token}}
+    - uses: actions/checkout@v2
+    - name: System Dependencies
+      run: .github/workflows/dependencies/dependencies.sh
+    - name: Repo Dependencies
+      run: Utils/CloneDeps.sh
+    - name: Build
+      env:
+         AMREX_HOME: ${GITHUB_WORKSPACE}/build/amrex
+         PELE_PHYSICS_HOME: ${GITHUB_WORKSPACE}/build/PelePhysics
+         PELELM_HOME: ${GITHUB_WORKSPACE}
+         AMREX_HYDRO_HOME: ${GITHUB_WORKSPACE}/build/AMReX-Hydro
+      working-directory: ./Exec/RegTests/EnclosedFlame/
+      run: |
+        make TPL COMP=gnu USE_MPI=TRUE
+        make -j 2 COMP=gnu USE_MPI=TRUE
+    - name: Run
+      working-directory: ./Exec/RegTests/EnclosedFlame/
+      run: |
+        ./PeleLMeX2d.gnu.MPI.ex input.2d-regt amr.max_step=2 amr.plot_int=-1 amr.check_int=-1

.gitignore

@@ -0,0 +1,15 @@
+# Compilation/Executable
+*.ex
+tmp_build*
+
+
+# Run output
+*.p3d
+plt*
+*plt*old*
+Backtrace*
+chk*
+grdlog
+
+# OSX files
+.DS_Store

Exec/RegTests/EnclosedFlame/GNUmakefile

@@ -0,0 +1,29 @@
+TOP = ../../../..
+AMREX_HOME         ?= ${TOP}/amrex
+PELELMEX_HOME      ?= ${TOP}/PeleLMeX
+PELE_PHYSICS_HOME  ?= ${TOP}/PelePhysics
+AMREX_HYDRO_HOME   ?= ${TOP}/AMReX-Hydro
+
+
+# AMReX
+DIM             = 2
+DEBUG           = FALSE
+PRECISION       = DOUBLE
+VERBOSE         = FALSE
+TINY_PROFILE    = FALSE
+
+# Compilation
+COMP            = gnu
+USE_MPI         = TRUE
+USE_OMP         = FALSE
+USE_CUDA        = FALSE
+USE_HIP         = FALSE
+
+# PeleLMeX
+
+# PelePhysics
+Chemistry_Model = drm19
+Eos_Model		= Fuego
+Transport_Model = Simple
+
+include $(PELELMEX_HOME)/Utils/Make.PeleLMeX