diff --git a/Docs/source/manual/Tutorials.rst b/Docs/source/manual/Tutorials.rst index 2f58fe52..8b4d5bb6 100644 --- a/Docs/source/manual/Tutorials.rst +++ b/Docs/source/manual/Tutorials.rst @@ -11,6 +11,7 @@ This sections includes several self-contained tutorials. .. toctree:: :maxdepth: 1 + Tutorials_HotBubble.rst Tutorials_FlameSheet.rst Tutorials_FlowPastCyl.rst Tutorials_BFSFlame.rst diff --git a/Docs/source/manual/Tutorials_BFSFlame.rst b/Docs/source/manual/Tutorials_BFSFlame.rst index ff32a335..0c1a997c 100644 --- a/Docs/source/manual/Tutorials_BFSFlame.rst +++ b/Docs/source/manual/Tutorials_BFSFlame.rst @@ -34,20 +34,25 @@ Follow the steps listed below to get to this point: cd PeleLMeX/Exec/RegTests/EB_BackwardStepFlame Note that the makefile system is set up such that default paths are automatically set to the -submodules obtained with the recursive *git clone*, however the user can set its own dependencies +submodules obtained with the recursive *git clone*, however the user can set their own dependencies in the `GNUmakefile` by updating the top-most lines as follows: :: PELELMEX_HOME = AMREX_HOME = AMREX_HYDRO_HOME = PELE_PHYSICS_HOME = + SUNDIALS_HOME = -or directly through shell environement variables (using *bash* for instance): :: +or directly through shell environment variables (using *bash* for instance): :: export PELELMEX_HOME= export AMREX_HOME= export AMREX_HYDRO_HOME= export PELE_PHYSICS_HOME= + export SUNDIALS_HOME= + +Note that using the first option will overwrite any +environment variables you might have previously defined when using this `GNUmakefile`. You're good to go ! @@ -262,11 +267,13 @@ list of available mechanisms and more information regarding the EOS, chemistry a Eos_Model := Fuego Transport_Model := Simple -Finally, `PeleLMeX` utilizes the chemical kinetic ODE integrator `CVODE `_. This Third Party Librabry (TPL) is not shipped with the `PeleLMeX` distribution but can be readily installed through the makefile system of `PeleLMeX`. Note that compiling Sundials is necessary even if the simualtion does not involve reactions. To do so, type in the following command: :: +Finally, `PeleLMeX` utilizes the chemical kinetic ODE integrator `CVODE `_. This +Third Party Librabry (TPL) is shipped as a submodule of the `PeleLMeX` distribution and can be readily installed through the makefile system +of `PeleLMeX`. To do so, type in the following command: :: make -j4 TPL -Note that the installation of `CVODE` requires CMake 3.17.1 or higher. +Note that the installation of `CVODE` requires CMake 3.23.1 or higher. You are now ready to build your first `PeleLMeX` executable!! Type in: :: diff --git a/Docs/source/manual/Tutorials_FlameSheet.rst b/Docs/source/manual/Tutorials_FlameSheet.rst index 5420c35a..1da862df 100644 --- a/Docs/source/manual/Tutorials_FlameSheet.rst +++ b/Docs/source/manual/Tutorials_FlameSheet.rst @@ -9,15 +9,14 @@ Premixed flame sheet with harmonic perturbations Introduction ------------ - `PeleLMeX` primary objective is to enable simulation of reactive flows on platforms ranging from small personal computer to Exascale supercomputer. This short tutorial describes the case of a 2D laminar methane/hydrogen/air premixed flame, perturbed using harmonic fluctuations on the initial conditions. The goal of this tutorial is to demonstrate `PeleLMeX` basic controls when dealing with reactive simulations. -This document provides step by step instruction reviewing how to set-up the domain and boundary conditions, -and to construct an initial solution. +This document provides step by step instructions reviewing how to set-up the domain and boundary conditions, +and how to construct an initial solution. .. _sec:TUTO_FS::PrepStep: @@ -25,7 +24,9 @@ Setting-up your environment --------------------------- Getting a functioning environment in which to compile and run `PeleLMeX` is the first step of this tutorial. -Follow the steps listed below to get to this point: +Please review the requirements listed on the `PeleLMeX README `_ to ensure +you have a suitable compiler suite to build `PeleLMeX`. +Follow the steps listed below to get the source code and its dependent libraries: #. The first step consist in getting `PeleLMeX` and its dependencies. To do so, use a recursive *git clone*: :: @@ -36,20 +37,25 @@ Follow the steps listed below to get to this point: cd PeleLMeX/Exec/RegTests/FlameSheet Note that the makefile system is set up such that default paths are automatically set to the -submodules obtained with the recursive *git clone*, however the user can set its own dependencies +submodules obtained with the recursive *git clone*, however the user can set their own dependencies in the `GNUmakefile` by updating the top-most lines as follows: :: PELELMEX_HOME = AMREX_HOME = AMREX_HYDRO_HOME = PELE_PHYSICS_HOME = + SUNDIALS_HOME = -or directly through shell environement variables (using *bash* for instance): :: +or directly through shell environment variables (using *bash* for instance): :: export PELELMEX_HOME= export AMREX_HOME= export AMREX_HYDRO_HOME= export PELE_PHYSICS_HOME= + export SUNDIALS_HOME= + +Note that using the first option will overwrite any +environment variables you might have previously defined when using this `GNUmakefile`. You're good to go ! @@ -57,7 +63,7 @@ Case setup ---------- A `PeleLMeX` case folder generally contains a minimal set of files to enable compilation, -provide user-defined function defining initial and boundary conditions, input file(s) and +provide user-defined functions defining initial and boundary conditions, input file(s) and any additional files necessary for the simulation (solution of a Cantera 1D flame for instance). The following three files in particular are necessary: :: @@ -105,7 +111,7 @@ used here. All of the above information is provided in the first two blocks of t The base grid is decomposed into a 32x64 cells array, and initially 2 levels of refinement are used. When running serial, a single box is used on the base level as the ``amr.max_grid_size`` exceeds the number of cells in each direction. When running parallel, the base grid will be chopped into smaller -boxes in the limit that no box smaller than the ``amr.blocking_factor`` can be created (16:math:`^3` here). +boxes in the limit that no box smaller than the ``amr.blocking_factor`` can be created (16 :math:`^2` here). The refinement ratio between each level is set to 2 and `PeleLMeX` currently does not support refinement ratio of 4. Regrid operation will be performed every 5 steps. ``amr.n_error_buf`` specifies, @@ -152,8 +158,8 @@ periodicity of the initial solution. .. note:: The ``P_mean`` parameters, providing the initial thermodynamic pressure, is always needed in the ProbParm struct. -Looking now into ``pelelm_prob.cpp``, we can see how the user can overwrite the default values of the parameters -contained in `ProbParm` using AMReX's ParmParse: :: +Looking now into ``pelelm_prob.cpp``, we can see how the developer can provide access to the `ProbParm` parameters +to overwrite the default values using AMReX's ParmParse: :: void PeleLM::readProbParm() { @@ -201,7 +207,7 @@ Finally, ``pelelm_prob.H`` defines the two functions effectively filling the ini * ``pele::physics::PMF::PmfData::DataContainer const * pmf_data`` : the Cantera solution data struct -The reader is encouraged to look into the body of the `pelelm_initdata` function for more details, a skelettal +The reader is encouraged to look into the body of the `pelelm_initdata` function for more details, a skeletal version of the function reads: * Compute the coordinate of the cell center using the cell indices and the `geomdata`. @@ -227,7 +233,7 @@ A last function, ``zero_visc``, is included in ``pelelm_prob.H`` but is not used Numerical parameters ^^^^^^^^^^^^^^^^^^^^ -The ``PeleLM CONTROL`` block contains a few of the `PeleLMeX` algorithmic parameters. Many more +The ``PeleLMeX CONTROL`` block contains a few of the `PeleLMeX` algorithmic parameters. Many more unspecified parameters are relying on their default values which can be found in :doc:`LMeXControls`. Of particular interest are the ``peleLM.sdc_iterMax`` parameter controlling the number of SDC iterations (see :doc:`Model` for more details on SDC in `PeleLMeX`) and the @@ -260,7 +266,10 @@ The next few lines specify AMReX compilation options and compiler selection: :: USE_HIP = FALSE USE_SYCL = FALSE -It allows users to specify the number of spatial dimensions (2D), trigger debug compilation and other AMReX options. The compiler (``gnu``) and the parallelism paradigm (in the present case only MPI is used) are then selected. Note that on OSX platform, one should update the compiler to ``llvm``. +It allows users to specify the number of spatial dimensions (2D), trigger debug compilation and other AMReX options. +The compiler (``gnu``) and the parallelism paradigm (in the present case only MPI is used) are then selected. If MPI is not available on your +platform, please set ``USE_MPI = FALSE``. +Note that on OSX platform, one should update the compiler to ``llvm``. In `PeleLMeX`, the chemistry model (set of species, their thermodynamic and transport properties as well as the description of their of chemical interactions) is specified at compile time. Chemistry models available in `PelePhysics` can used in `PeleLMeX` by specifying the name of the folder in `PelePhysics/Support/Mechanisms/Models` containing the relevant files, for example: :: @@ -275,11 +284,13 @@ list of available mechanisms and more information regarding the EOS, chemistry a Note that the ``Chemistry_Model`` must be similar to the one used to generate the Cantera solution. -Finally, `PeleLMeX` utilizes the chemical kinetic ODE integrator `CVODE `_. This Third Party Librabry (TPL) is not shipped with the `PeleLMeX` distribution but can be readily installed through the makefile system of `PeleLMeX`. Note that compiling Sundials is necessary even if the simualtion do not involve reactions. To do so, type in the following command: :: +Finally, `PeleLMeX` utilizes the chemical kinetic ODE integrator `CVODE `_. This +Third Party Librabry (TPL) is shipped as a submodule of the `PeleLMeX` distribution and can be readily installed through the makefile system +of `PeleLMeX`. To do so, type in the following command: :: make -j4 TPL -Note that the installation of `CVODE` requires CMake 3.17.1 or higher. +Note that the installation of `CVODE` requires CMake 3.23.1 or higher. You are now ready to build your first `PeleLMeX` executable !! Type in: :: @@ -352,7 +363,7 @@ the initialization process and the solution will rapidly relax to adapt to the ` Let's now play with the problem parameters to see how the initial solution changes. For instance, decrease the amplitude of the perturbation, change the ``standoff`` parameter or deactivate the -initial projection by adding ``peleLM.do_init_proj=0`` to the ``PeleLM CONTROL`` block. Examples +initial projection by adding ``peleLM.do_init_proj=0`` to the ``PeleLMeX CONTROL`` block. Examples of the initial solution varying these parameters are displayed in :numref:`FS_InitTweaks`. .. figure:: images/tutorials/FS_InitSolTweaks.png diff --git a/Docs/source/manual/Tutorials_FlowPastCyl.rst b/Docs/source/manual/Tutorials_FlowPastCyl.rst index 05d86c21..8ea714a2 100644 --- a/Docs/source/manual/Tutorials_FlowPastCyl.rst +++ b/Docs/source/manual/Tutorials_FlowPastCyl.rst @@ -35,26 +35,26 @@ Follow the steps listed below to get to this point: cd PeleLMeX/Exec/RegTests/EB_FlowPastCylinder -#. Finally, setup the environment variables providing paths to `PeleLMeX` and its dependencies. This can done in - one of two ways: - - #. Directly into the `GNUmakefile` by updating the top-most lines as follows: :: +Note that the makefile system is set up such that default paths are automatically set to the +submodules obtained with the recursive *git clone*, however the user can set their own dependencies +in the `GNUmakefile` by updating the top-most lines as follows: :: PELELMEX_HOME = - AMREX_HOME =${PELELMEX_HOME}/Submodules/amrex - AMREX_HYDRO_HOME =${PELELMEX_HOME}/Submodules/AMReX-Hydro - PELE_PHYSICS_HOME =${PELELMEX_HOME}/Submodules/PelePhysics - + AMREX_HOME = + AMREX_HYDRO_HOME = + PELE_PHYSICS_HOME = + SUNDIALS_HOME = - #. Exporting shell environement variables (using *bash* for instance): :: +or directly through shell environment variables (using *bash* for instance): :: export PELELMEX_HOME= - export AMREX_HOME=${PELELMEX_HOME}/Submodules/amrex - export AMREX_HYDRO_HOME=${PELELMEX_HOME}/Submodules/AMReX-Hydro - export PELE_PHYSICS_HOME=${PELELMEX_HOME}/Submodules/PelePhysics + export AMREX_HOME= + export AMREX_HYDRO_HOME= + export PELE_PHYSICS_HOME= + export SUNDIALS_HOME= - Both options require to provide the path to where you cloned `PeleLMeX`. Note that using the first option will overwrite any - environement variables you might have previously defined when using this `GNUmakefile`. +Note that using the first option will overwrite any +environment variables you might have previously defined when using this `GNUmakefile`. You're good to go ! @@ -212,11 +212,13 @@ Here, the model ``air``, only contains 2 species (O2 and N2). The user is referr Eos_Model := Fuego Transport_Model := Constant -Finally, `PeleLMeX` utilizes the chemical kinetic ODE integrator `CVODE `_. This Third Party Librabry (TPL) is not shipped with the `PeleLMeX` distribution but can be readily installed through the makefile system of `PeleLMeX`. Note that compiling Sundials is necessary even if the simualtion do not involve reactions. To do so, type in the following command: :: +Finally, `PeleLMeX` utilizes the chemical kinetic ODE integrator `CVODE `_. This +Third Party Librabry (TPL) is shipped as a submodule of the `PeleLMeX` distribution and can be readily installed through the makefile system +of `PeleLMeX`. To do so, type in the following command: :: make -j4 TPL -Note that the installation of `CVODE` requires CMake 3.12.1 or higher. +Note that the installation of `CVODE` requires CMake 3.23.1 or higher. You are now ready to build your first `PeleLMeX` executable !! Type in: :: diff --git a/Docs/source/manual/Tutorials_HotBubble.rst b/Docs/source/manual/Tutorials_HotBubble.rst new file mode 100644 index 00000000..4ba5d541 --- /dev/null +++ b/Docs/source/manual/Tutorials_HotBubble.rst @@ -0,0 +1,356 @@ +.. highlight:: rst + +.. _sec:tutorialHB: + +Rising light bubble +=================== + +.. _sec:TUTO_HB::Intro: + +Introduction +------------ + +At the core of `PeleLMeX` algorithm is a variable-density, low Mach number Navier-Stokes solver based +on a fractional step approach. This short tutorial presents the case of a rising light bubble under +the influence of a gravity field, which exercice this algorithm without the added complexity +of chemical reactions or embedded boundaries. + +.. _sec:TUTO_HB::PrepStep: + +Setting-up your environment +--------------------------- + +Getting a functioning environment in which to compile and run `PeleLMeX` is the first step of this tutorial. +Follow the steps listed below to get to this point: + +#. The first step is to get `PeleLMeX` and its dependencies. To do so, use a recursive *git clone*: :: + + git clone --recursive https://github.com/AMReX-Combustion/PeleLMeX.git + +#. Move into the Exec folder containing the ``HotBubble``. To do so: :: + + cd PeleLMeX/Exec/RegTests/HotBubble + +Note that the makefile system is set up such that default paths are automatically set to the +submodules obtained with the recursive *git clone*, however the user can set their own dependencies +in the `GNUmakefile` by updating the top-most lines as follows: :: + + PELELMEX_HOME = + AMREX_HOME = + AMREX_HYDRO_HOME = + PELE_PHYSICS_HOME = + SUNDIALS_HOME = + +or directly through shell environment variables (using *bash* for instance): :: + + export PELELMEX_HOME= + export AMREX_HOME= + export AMREX_HYDRO_HOME= + export PELE_PHYSICS_HOME= + export SUNDIALS_HOME= + +Note that using the first option will overwrite any +environment variables you might have previously defined when using this `GNUmakefile`. + +You're good to go ! + +Case setup +---------- + +A `PeleLMeX` case folder contains a minimal set of files to enable compilation, +and the reader is referred to the FlameSheet tutorial :doc:`Tutorials_FlameSheet` for a +more detailed description of `PeleLMeX` case setup. The case of interest for this +tutorial can be found in `PeleLMeX` Exec folder: :: + + Exec/RegTests/HotBubble + +Geometry, grid and boundary conditions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This simulation is performed on a 0.016x0.032 :math:`m^2` 2D computational domain, +with the bottom left corner located at (0.0:0.0) and the top right corner at (0.016:0.032). +Two versions of that domain will be considered: 1) a 2D-cartesian case, 2) a 2D-RZ +case where the :math:`x`-low is the symmetry axis of the azimuthally homogeneous case. In both cases, the +:math:`x`-low boundary is treated as symmetry BC while :math:`x`-high is treated as an adiabatic slip wall, +the :math:`y`-low boundary is treated as an adiabatic slip wall, while the :math:`y`-high boundary is an ``Outflow``. + +All of the geometrical information can be specified the first two blocks of the input file (`input.2d-regt_sym`): :: + + #----------------------DOMAIN DEFINITION------------------------ + geometry.is_periodic = 0 0 # For each dir, 0: non-perio, 1: periodic + geometry.coord_sys = 0 # 0 => cart, 1 => RZ + geometry.prob_lo = 0.0 0.0 # x_lo y_lo (z_lo) + geometry.prob_hi = 0.016 0.032 # x_hi y_hi (z_hi) + + # >>>>>>>>>>>>> BC FLAGS <<<<<<<<<<<<<<<< + # Interior, Inflow, Outflow, Symmetry, + # SlipWallAdiab, NoSlipWallAdiab, SlipWallIsotherm, NoSlipWallIsotherm + peleLM.lo_bc = Symmetry SlipWallAdiab + peleLM.hi_bc = SlipWallAdiab Outflow + + +In the 2D-RZ case (`input.2d-regt_symRZ`), the ``geometry.coord_sys`` is updated to ``1`` to trigger the use of a polar +coordinates system. + +.. note:: + Note that when running 2D simulations, it is not necessary to specify entries for the third dimension. + +The base grid is decomposed into a 64x128 cell array with AMR triggered up to level 2. + +The refinement ratio between each level is set to 2 and `PeleLMeX` currently does not support +refinement ratio of 4. Regrid operation will be performed every 5 steps. ``amr.n_error_buf`` specifies, +for each level, the number of buffer cells used around the cell tagged for refinement, while ``amr.grid_eff`` +describes the grid efficiency, i.e. how much of the new grid contains tagged cells. Higher values lead +to tighter grids around the tagged cells. For more information on how these parameters affect grid generation, +see the `AMReX documentation `_. + +All of those parameters are specified in the `AMR CONTROL` block: :: + + #------------------------- AMR CONTROL ---------------------------- + amr.n_cell = 64 128 # Level 0 number of cells in each direction + amr.v = 1 # AMR verbose + amr.max_level = 2 # maximum level number allowed + amr.ref_ratio = 2 2 2 2 # refinement ratio + amr.regrid_int = 5 # how often to regrid + amr.n_error_buf = 1 1 2 2 # number of buffer cells in error est + amr.grid_eff = 0.7 # what constitutes an efficient grid + amr.blocking_factor = 16 # block factor in grid generation (min box size) + amr.max_grid_size = 128 # max box size + +Problem specifications +^^^^^^^^^^^^^^^^^^^^^^ + +.. _sec:TUTO_HB::Problem: + +The problem setup is mostly contained in the three C++ source/header files described in :doc:`Tutorials_FlameSheet`. +The user parameters are gathered in the struct defined in ``pelelm_prob_parm.H``: :: + + struct ProbParm + { + amrex::Real P_mean = 101325.0_rt; + amrex::Real T_mean = 300.0_rt; + amrex::Real T_bubble = 600.0_rt; + amrex::Real bubble_rad = 0.005_rt; + amrex::Real bubble_y0 = 0.01; + int bubble_is_mix = 0; + int is_sym = 0; + }; + +* ``P_mean`` : initial thermodynamic pressure + +* ``T_mean`` : the ambient gas temperature + +* ``T_bubble`` : the bubble gas temperature + +* ``bubble_rad`` : the radius of the light bubble + +* ``bubble_y0`` : the initial position of the bubble in the :math:`y` direction + +* ``bubble_is_mix`` : a flag to switch to a mixture-based density change in the bubble + +* ``is_sym`` : a flag to indicate that the initial conditions are for a :math:`x`-low symmetric case. + +The initial solution consists of air at the pressure/temperature specified by the user, with a bubble +of a different temperature/mixture intended to be lighter such that the bubble will rise under the +influence of gravity. Note that the user can easily reverse the problem with a heavier bubble. +The default parameters provided above are overwritten using AMReX ParmParse in ``pelelm_prob.cpp`` +and the initial/boundary conditions implemented in ``pelelm_prob.H``. Because this case does not feature +any dirichlet BC on the state variables, the ``bcnormal`` function in ``pelelm_prob.H`` will not be called +and can thus be left empty (but its definition is still required). + +The interesting aspect of this case is the inclusion of buoyancy effects in the presence of gravity. +To trigger gravity the following input key is required: :: + + peleLM.gravity = 0.0 -9.81 0.0 + + +which in this case defines an usual Earth-like gravity oriented towards :math:`y`-low. + +.. note:: + At the moment, the hydrostatic outflow boundqry conditions are not available in PeleLMeX, so Outflow should not be + employed in the direction transverse to the gravity vector ! + + +Numerical parameters +^^^^^^^^^^^^^^^^^^^^ + +The ``PeleLM CONTROL`` block contains a few of the `PeleLMeX` algorithmic parameters. Many more +unspecified parameters are relying on their default values which can be found in :doc:`LMeXControls`. +Of particular interest are the ``peleLM.sdc_iterMax`` parameter controlling the number of +SDC iterations (see :doc:`Model` for more details on SDC in `PeleLMeX`) and the +``peleLM.num_init_iter`` one controlling the number of initial iteration the solver will do +after initialization to obtain a consistent pressure and velocity field. + +Building the executable +----------------------- + +Now that we have reviewed the basic ingredients required to setup the case, it is time to build the `PeleLMeX` executable. +Although both GNUmake and CMake are available, it is advised to use GNUmake. The ``GNUmakefile`` file provides some compile-time options +regarding the simulation we want to perform. +The first few lines specify the paths towards the source codes of `PeleLMeX`, `AMReX`, `AMReX-Hydro` and `PelePhysics`, overwritting +any environment variable if necessary, and might have been already updated in :ref:`sec:TUTO_HB::PrepStep` earlier. + +The next few lines specify AMReX compilation options and compiler selection: :: + + # AMREX + DIM = 2 + DEBUG = FALSE + PRECISION = DOUBLE + VERBOSE = FALSE + TINY_PROFILE = FALSE + + # Compilation + COMP = gnu + USE_MPI = TRUE + USE_OMP = FALSE + USE_CUDA = FALSE + USE_HIP = FALSE + USE_SYCL = FALSE + +It allows users to specify the number of spatial dimensions (2D), +trigger debug compilation and other AMReX options. The compiler (``gnu``) and the parallelism paradigm +(in the present case only MPI is used) are then selected. Note that on OSX platform, one should update the compiler to ``llvm``. + +In `PeleLMeX`, the chemistry model (set of species, their thermodynamic and transport properties as well as the description +of their of chemical interactions) is specified at compile time. Chemistry models available in +`PelePhysics` can used in `PeleLMeX` by specifying the name of the folder in `PelePhysics/Support/Mechanisms/Models` containing +the relevant files, for example: :: + + Chemistry_Model = air + +Here, the model ``air`` contains only 2 species (O2 and N2) without any reactions. A constant transport model is used +and transport properties are set to zero in the input files which is effectively equivalent to solving the variable-density +Euler equations. +The user is referred to the `PelePhysics `_ documentation for a +list of available mechanisms and more information regarding the EOS, chemistry and transport models specified: :: + + Eos_Model := Fuego + Transport_Model := Constant + +Finally, `PeleLMeX` utilizes the chemical kinetic ODE integrator `CVODE `_. This +Third Party Librabry (TPL) is shipped as a submodule of the `PeleLMeX` distribution and can be readily installed through the makefile system +of `PeleLMeX`. To do so, type in the following command: :: + + make -j4 TPL + +Note that the installation of `CVODE` requires CMake 3.23.1 or higher. + +You are now ready to build your first `PeleLMeX` executable!! Type in: :: + + make -j4 + +The option here tells `make` to use up to 4 processors to create the executable (internally, `make` follows a dependency graph to ensure any required ordering in the build is satisfied). This step should generate the following file (providing that the build configuration you used matches the one above): :: + + PeleLMeX2d.gnu.MPI.ex + +You're good to go! + +Checking the initial conditions +------------------------------- + +It is always a good practice to check the initial conditions. To do so, run the simulation specifying +an ``amr.max_step`` of 0. Open the ``input.2d-regt_sym`` with your favorite editor and update the following parameters :: + + #---------------------- Time Stepping CONTROL -------------------- + amr.max_step = 0 # Maximum number of time steps + + +Since we've set the maximum number of steps to 0, the solver will exit after +the initial solution is obtained. Let's run the simulation with the default problem parameter +listed in the input file. To do so, use: :: + + ./PeleLMeX2d.gnu.MPI.ex input.2d-regt + +A variety of information is printed to the screen: + +#. AMReX/SUNDIALs initialization along with the git hashes of the various subrepositories + +#. A summary of the `PeleLMeX` state components + +#. Initial projection and initial iterations. + +#. Saving the initial solution to `plt00000` file. + +Use Amrvis, Paraview or yt to visualize the plot file. Using Amrvis, the solution should look +similar to :numref:`HB_InitSol`. + +.. figure:: images/tutorials/HB_InitSol.png + :name: HB_InitSol + :align: center + :figwidth: 95% + + : Contour plots of density, temperature, cell-averaged pressure after initialization. + +The cell-averaged pressure (the perturbational pressure in node-centered in the projection-based scheme +employed in PeleLMeX, see `Almgren `_ for more details) +clearly shows the effect of the gravity field with the presence of an hydrostatic pressure gradient. + +Advance the solution +-------------------- + +Let's now advance the solution for 400 steps, using the base grid and 2 AMR level and the default time stepping +parameters. To do so, ensure that: :: + + amr.max_step = 400 + +Additionally, make sure that ``amr.check_int`` is set to a positive value to trigger writing a +checkpoint file from which to later restart the simulation. If available, use more than one MPI +rank to run the simulation and redirect the standard output to a log file using: :: + + mpirun -n 4 ./PeleLMeX2d.gnu.MPI.ex input.2d-regt_sym > logInit.dat & + +A typical `PeleLMeX` stdout for a time step now looks like: :: + + ==================== NEW TIME STEP ==================== + Est. time step - Conv: 7.249645299e-05, divu: 1e+20 + STEP [384] - Time: 0.05931322581, dt 7.249645299e-05 + SDC iter [1] + >> PeleLMeX::Advance() --> Time: 0.2141339779 + +clearly showing the use of 1 SDC iteration. The first line at each step provides +the time step contraint from the CFL +condition (``Conv:``) and from the density change condition (``divu:``). +In the absence of reaction and diffusion, the ``divu:`` constraint is irrelevant and set to a +large value. + +Visualizing the `plt00400` file, we can see that the solution has evolved. The light bubble started rising +under the effect of buoyancy, resulting in a shear layer at the interface of between the hot and cold gases. +Vorticies appears in the shear layer, wrinking the interface. Smearing of the temperature gradient at the interface +is induced by the numerical scheme diffusion, but appearances of local extrema are very limited. + +.. figure:: images/tutorials/HB_400steps2lvl.png + :name: HB_400steps2lvl + :align: center + :figwidth: 95% + + : Contour plots of density, both velocity components and vorticity after 400 steps. + +In order to compare 2D-cartesian and 2D-RZ, you can now start another simulation using `input.2d-regt_symRZ`. To insure +both simulations evolved for the same physical time, set the final time of the 2D-RZ simulation to that of the first run: :: + + amr.max_step = 1000 + amr.stop_time = 0.060458236391426 + + +and change the prefix of the plotfile output for clarity: :: + + amr.plot_file = "pltRZ" + +then start the 2D-RZ run: :: + + mpirun -n 4 ./PeleLMeX2d.gnu.MPI.ex input.2d-regt_symRZ > logInitRZ.dat & + +The 2D-RZ simulation is found to have a smaller time step size resulting from the stronger acceleration of the bubble. Indeed, +in the 2D-cartesian case, the hot region is actually an infinitely long cylinder which has inertia larger than that of the +bubble effectively represented in the 2D-RZ case. + +This is end of the guided section of this tutorial. Interested users can explore the effects of the following parameters +on the simulation results since the computational time is minimal: + +* Spatial resolution: increase the maximum number of AMR levels, ensuring that the simulation final time matches that of the initial run. What is the effect on the rising bubble's velocity and shape ? + +* Switch to a mixture composition change instead of a temperature one or reverse the problem by using a bubble temperature lower than that of the ambient air. + +* Switch to a different advection scheme (see the :doc:`LMeXControls` page for a list of available schemes). What is the effect on the interface wrinkling and smearing ? + +* If your computational resources allows, build the 3D version of the case and compare the 2D-RZ and 3D results. diff --git a/Docs/source/manual/Tutorials_TripleFlame.rst b/Docs/source/manual/Tutorials_TripleFlame.rst index 65a0aa28..7718309d 100644 --- a/Docs/source/manual/Tutorials_TripleFlame.rst +++ b/Docs/source/manual/Tutorials_TripleFlame.rst @@ -53,20 +53,25 @@ Follow the steps listed below to get to this point: cd PeleLMeX/Exec/RegTests/TripleFLame Note that the makefile system is set up such that default paths are automatically set to the -submodules obtained with the recursive *git clone*, however the user can set its own dependencies +submodules obtained with the recursive *git clone*, however the user can set their own dependencies in the `GNUmakefile` by updating the top-most lines as follows: :: PELELMEX_HOME = AMREX_HOME = AMREX_HYDRO_HOME = PELE_PHYSICS_HOME = + SUNDIALS_HOME = -or directly through shell environement variables (using *bash* for instance): :: +or directly through shell environment variables (using *bash* for instance): :: export PELELMEX_HOME= export AMREX_HOME= export AMREX_HYDRO_HOME= export PELE_PHYSICS_HOME= + export SUNDIALS_HOME= + +Note that using the first option will overwrite any +environment variables you might have previously defined when using this `GNUmakefile`. You're good to go ! @@ -259,12 +264,12 @@ mechanisms and more information regarding the EOS, chemistry and transport model Transport_Model := Simple Finally, `PeleLMeX` utilizes the chemical kinetic ODE integrator `CVODE `_. This -Third Party Librabry (TPL) is not shipped with the `PeleLMeX` distribution but can be readily installed through the makefile system +Third Party Librabry (TPL) is shipped as a submodule of the `PeleLMeX` distribution and can be readily installed through the makefile system of `PeleLMeX`. To do so, type in the following command: :: make TPL -Note that the installation of `CVODE` requires CMake 3.17.1 or higher. +Note that the installation of `CVODE` requires CMake 3.23.1 or higher. You are now ready to build your first `PeleLMeX` executable !! Type in: :: @@ -560,4 +565,3 @@ At this point, the simulation is considered complete. .. [PCI2007] S. Chung, Stabilization, propagation and instability of tribrachial triple flames, Proceedings of the Combustion Institute 31 (2007) 877–892 .. [CF1990] R. Bilger, S. Starner, R. Kee, On reduced mechanisms for methane-air combustion in nonpremixed flames, Combustion and Flames 80 (1990) 135-149 .. [CAMCS2006] J. Bell, M. Day, J. Grcar, M. Lijewski, Active Control for Statistically Stationary Turbulent PremixedFlame Simulations, Communications in Applied Mathematics and Computational Science 1 (2006) 29-51 - diff --git a/Docs/source/manual/images/tutorials/HB_400steps2lvl.png b/Docs/source/manual/images/tutorials/HB_400steps2lvl.png new file mode 100644 index 00000000..cc05bc6b Binary files /dev/null and b/Docs/source/manual/images/tutorials/HB_400steps2lvl.png differ diff --git a/Docs/source/manual/images/tutorials/HB_InitSol.png b/Docs/source/manual/images/tutorials/HB_InitSol.png new file mode 100644 index 00000000..af362073 Binary files /dev/null and b/Docs/source/manual/images/tutorials/HB_InitSol.png differ diff --git a/Exec/RegTests/EB_BackwardStepFlame/README.md b/Exec/RegTests/EB_BackwardStepFlame/README.md new file mode 100644 index 00000000..77350143 --- /dev/null +++ b/Exec/RegTests/EB_BackwardStepFlame/README.md @@ -0,0 +1,3 @@ +## EB\_BackwardStepFlame + +This folder implements the case of a laminar flame stabilized behind a backward facing step. It is intended to test integrated EB representation with reactive flow. More details on this case and step-by-step instructions can be found in this [tutorial](https://amrex-combustion.github.io/PeleLMeX/manual/html/Tutorials_BFSFlame.html) diff --git a/Exec/RegTests/EB_EnclosedFlame/README.md b/Exec/RegTests/EB_EnclosedFlame/README.md new file mode 100644 index 00000000..1e223d16 --- /dev/null +++ b/Exec/RegTests/EB_EnclosedFlame/README.md @@ -0,0 +1,2 @@ +## EB\_EnclosedFlame +A 2D (or 3D) cylindrical flame enclosed in a cylindrical EB chamber. Testing the closed chamber algorithm with EB. diff --git a/Exec/RegTests/EB_EnclosedVortex/README.md b/Exec/RegTests/EB_EnclosedVortex/README.md new file mode 100644 index 00000000..ac52e0fa --- /dev/null +++ b/Exec/RegTests/EB_EnclosedVortex/README.md @@ -0,0 +1,2 @@ +## EB\_EnclosedVortex +A 2D (or 3D) vortex enclosed in a cylindrical EB chamber. Testing EB for pure incompressible flows. diff --git a/Exec/RegTests/EB_FlowPastCylinder/README.md b/Exec/RegTests/EB_FlowPastCylinder/README.md new file mode 100644 index 00000000..47be75c4 --- /dev/null +++ b/Exec/RegTests/EB_FlowPastCylinder/README.md @@ -0,0 +1,4 @@ +## EB\_FlowPastCylinder +A 2D (or 3D) inflow/outflow setup with either an EB cylinder in the middle of the flow or an EB +bump on a WallNoSlipAdiab domain boundary. Testing EB for incompressible flows and pure mixing cases (no reactions). +More details and step-by-step instructions can be found in this [tutorial](https://amrex-combustion.github.io/PeleLMeX/manual/html/Tutorials_FlowPastCyl.html). diff --git a/Exec/RegTests/EB_PipeFlow/README.md b/Exec/RegTests/EB_PipeFlow/README.md new file mode 100644 index 00000000..2c2a1de9 --- /dev/null +++ b/Exec/RegTests/EB_PipeFlow/README.md @@ -0,0 +1,5 @@ +## EB\_PipeFlow +2D and 3D setup of both channel and pipe flow, either in an inflow/outflow setup or a +periodc setup driven by a background pressure gradient. + +Poiseuille flow results with PeleLMeX can be found [here](https://amrex-combustion.github.io/PeleLMeX/manual/html/Validation.html#laminar-poiseuille-flow). diff --git a/Exec/RegTests/EnclosedFlame/README.md b/Exec/RegTests/EnclosedFlame/README.md new file mode 100644 index 00000000..6d26c5d4 --- /dev/null +++ b/Exec/RegTests/EnclosedFlame/README.md @@ -0,0 +1,2 @@ +## EnclosedFlame +A 2D (or 3D) cylindrical flame enclosed in a square (rectangular) domain. Testing the closed chamber algorithm. diff --git a/Exec/RegTests/EnclosedInjection/README.md b/Exec/RegTests/EnclosedInjection/README.md new file mode 100644 index 00000000..71dbd903 --- /dev/null +++ b/Exec/RegTests/EnclosedInjection/README.md @@ -0,0 +1,2 @@ +## EnclosedInjection +A simple jet injection into an EB cylindrical chamber. Testing the closed chamber algorithm with mass injection and EB. diff --git a/Exec/RegTests/FlameSheet/README.md b/Exec/RegTests/FlameSheet/README.md new file mode 100644 index 00000000..4ccad320 --- /dev/null +++ b/Exec/RegTests/FlameSheet/README.md @@ -0,0 +1,7 @@ +## FlameSheet +A 2D (or 3D) harmonically perturbed flame sheet, initial solution from a Cantera simulation provided for 3 mechanisms +(drm19, dodecane\_lu and dodecane\_lu\_qss). This is the basis for [weak scaling studies](https://amrex-combustion.github.io/PeleLMeX/manual/html/Performances.html) in PeleLMeX and tests all the +reactive pieces of the algorithm as well as transport options (Unity Lewis number, Soret effect, ...). +More details on the case setup and step-by-step instructions can be found in this [tutorial](https://amrex-combustion.github.io/PeleLMeX/manual/html/Tutorials_FlameSheet.html). + +Comparisons of PeleLMeX results for a methane/air flame against Cantera at several resolutions are reported [here](https://amrex-combustion.github.io/PeleLMeX/manual/html/Validation.html#laminar-premixed-flame). diff --git a/Exec/RegTests/FlameSheet/input.2d-regt b/Exec/RegTests/FlameSheet/input.2d-regt index b8a60b79..f1f12d6e 100644 --- a/Exec/RegTests/FlameSheet/input.2d-regt +++ b/Exec/RegTests/FlameSheet/input.2d-regt @@ -28,7 +28,7 @@ prob.pertmag = 0.00045 prob.pertlength = 0.016 pmf.datafile = "drm19_pmf.dat" -#---------------------- PeleLM CONTROL --------------------------- +#---------------------- PeleLMeX CONTROL ------------------------- peleLM.v = 1 # PeleLMeX version peleLM.use_wbar = 1 # Include Wbar term in species diffusion fluxes peleLM.sdc_iterMax = 2 # Number of SDC iterations diff --git a/Exec/RegTests/HITDecay/README.md b/Exec/RegTests/HITDecay/README.md new file mode 100644 index 00000000..d6fab358 --- /dev/null +++ b/Exec/RegTests/HITDecay/README.md @@ -0,0 +1,3 @@ +## HITDecay +A 3D decaying Homogeneous Isotropic Turbulence (HIT) case, where the initial solution is generated with a Passot-Pouquet spectrum. Testing the basic incompressible +integration algorithm and Large Eddy Simulation implementation. diff --git a/Exec/RegTests/HotBubble/GNUmakefile b/Exec/RegTests/HotBubble/GNUmakefile index f57c6023..e654758f 100644 --- a/Exec/RegTests/HotBubble/GNUmakefile +++ b/Exec/RegTests/HotBubble/GNUmakefile @@ -26,6 +26,4 @@ Chemistry_Model = air Eos_Model = Fuego Transport_Model = Constant -USE_SUNDIALS_PP = TRUE - include $(PELELMEX_HOME)/Utils/Make.PeleLMeX diff --git a/Exec/RegTests/HotBubble/README.md b/Exec/RegTests/HotBubble/README.md new file mode 100644 index 00000000..bf982161 --- /dev/null +++ b/Exec/RegTests/HotBubble/README.md @@ -0,0 +1,3 @@ +## HotBubble +Case of a 2D/2D-RZ/3D bubble of light gases (either hotter or lighter mixture composition) lifted under the effect of +a gravity field. Testing of RZ algorthim and gravity forces. diff --git a/Exec/RegTests/HotBubble/input.2d-regt_sym b/Exec/RegTests/HotBubble/input.2d-regt_sym index 978e355f..381c66ca 100644 --- a/Exec/RegTests/HotBubble/input.2d-regt_sym +++ b/Exec/RegTests/HotBubble/input.2d-regt_sym @@ -1,80 +1,66 @@ -#----------------------DOMAIN DEFINITION------------------------ -geometry.is_periodic = 0 0 # For each dir, 0: non-perio, 1: periodic -geometry.coord_sys = 0 # 0 => cart, 1 => RZ -geometry.prob_lo = 0.0 0.0 0.0 # x_lo y_lo (z_lo) -geometry.prob_hi = 0.016 0.032 0.016 # x_hi y_hi (z_hi) +#---------------------- DOMAIN DEFINITION ------------------------ +geometry.is_periodic = 0 0 # For each dir, 0: non-perio, 1: periodic +geometry.coord_sys = 0 # 0 => cart, 1 => RZ +geometry.prob_lo = 0.0 0.0 # x_lo y_lo (z_lo) +geometry.prob_hi = 0.016 0.032 # x_hi y_hi (z_hi) -# >>>>>>>>>>>>> BC FLAGS <<<<<<<<<<<<<<<< +#---------------------- BC FLAGS --------------------------------- # Interior, Inflow, Outflow, Symmetry, # SlipWallAdiab, NoSlipWallAdiab, SlipWallIsotherm, NoSlipWallIsotherm peleLM.lo_bc = Symmetry SlipWallAdiab peleLM.hi_bc = SlipWallAdiab Outflow - -#-------------------------AMR CONTROL---------------------------- -amr.n_cell = 32 64 32 # Level 0 number of cells in each direction -amr.n_cell = 64 128 32 # Level 0 number of cells in each direction -amr.v = 1 # AMR verbose -amr.max_level = 1 # maximum level number allowed -amr.ref_ratio = 2 2 2 2 # refinement ratio -amr.regrid_int = 5 # how often to regrid -amr.n_error_buf = 1 1 2 2 # number of buffer cells in error est -amr.grid_eff = 0.7 # what constitutes an efficient grid -amr.blocking_factor = 16 # block factor in grid generation (min box size) +#---------------------- AMR CONTROL ------------------------------ +amr.n_cell = 64 128 # Level 0 number of cells in each direction +amr.v = 1 # AMR verbose +amr.max_level = 2 # maximum level number allowed +amr.ref_ratio = 2 2 2 2 # refinement ratio +amr.regrid_int = 5 # how often to regrid +amr.n_error_buf = 1 1 2 2 # number of buffer cells in error est +amr.grid_eff = 0.7 # what constitutes an efficient grid +amr.blocking_factor = 16 # block factor in grid generation (min box size) amr.max_grid_size = 128 # max box size -#--------------------------- Problem ------------------------------- +#---------------------- Problem ---------------------------------- prob.P_mean = 101325.0 prob.T_bubble = 450.0 prob.bubble_radius = 0.003 prob.bubble_y0 = 0.01 prob.use_symmetry = 1 -#-------------------------PeleLM CONTROL---------------------------- +#---------------------- PeleLMeX CONTROL ------------------------- peleLM.v = 1 peleLM.gravity = 0.0 -9.81 0.0 peleLM.use_wbar = 0 peleLM.sdc_iterMax = 1 -peleLM.floor_species = 0 peleLM.num_init_iter = 2 -#peleLM.num_divu_iter = 0 -#peleLM.advection_scheme = "Godunov_PLM" -peleLM.deltaT_verbose = 1 +#---------------------- Time Stepping CONTROL -------------------- +amr.max_step = 400 +amr.stop_time = 0.07 +amr.cfl = 0.2 +amr.dt_shrink = 1.0 + +#---------------------- IO CONTROL ------------------------------- #amr.restart = chk00005 #amr.check_int = 2000 amr.plot_int = 2000 -amr.max_step = 2000 -amr.dt_shrink = 1.0 -#amr.init_dt = 0.0001 -amr.stop_time = 0.07 -amr.cfl = 0.2 amr.derive_plot_vars = avg_pressure mag_vort mass_fractions -cvode.solve_type = dense # CVODE Linear solve type (for Newton direction) -ode.analytical_jacobian = 0 # Provide analytical jacobian (from Fuego) -nodal_proj.verbose = 0 -#nodal_proj.rtol = 1e-13 -#mac_proj.rtol = 1e-12 -mac_proj.verbose = 0 - -#------------ INPUTS TO CONSTANT TRANSPORT ----------------- -transport.const_viscosity = 0.0 -transport.const_bulk_viscosity = 0.0 -transport.const_conductivity = 0.0 -transport.const_diffusivity = 0.0 +#---------------------- Reactor CONTROL -------------------------- +peleLM.chem_integrator = "ReactorNull" -#--------------------REFINEMENT CONTROL------------------------ -#amr.refinement_indicators = temp -#amr.temp.max_level = 1 -#amr.temp.value_greater = 305 -#amr.temp.field_name = temp +#---------------------- Constant Transport ----------------------- +transport.const_viscosity = 0.0 +transport.const_bulk_viscosity = 0.0 +transport.const_conductivity = 0.0 +transport.const_diffusivity = 0.0 -#amr.refinement_indicators = magVort -#amr.magVort.max_level = 1 -#amr.magVort.value_greater = 500.0 -#amr.magVort.field_name = mag_vort +#---------------------- Linear solver CONTROL -------------------- +mac_proj.verbose = 0 +nodal_proj.verbose = 0 +#---------------------- Refinement CONTROL------------------------ amr.refinement_indicators = temp gtemp amr.temp.max_level = 2 amr.temp.value_greater = 350.0 @@ -84,6 +70,7 @@ amr.gtemp.max_level = 4 amr.gtemp.adjacent_difference_greater = 10.0 amr.gtemp.field_name = temp +#---------------------- Debug/HPC CONTROL------------------------- #amrex.fpe_trap_invalid = 1 #amrex.fpe_trap_zero = 1 #amrex.fpe_trap_overflow = 1 diff --git a/Exec/RegTests/HotBubble/input.2d-regt_symRZ b/Exec/RegTests/HotBubble/input.2d-regt_symRZ index 515d34e5..04c2cd33 100644 --- a/Exec/RegTests/HotBubble/input.2d-regt_symRZ +++ b/Exec/RegTests/HotBubble/input.2d-regt_symRZ @@ -1,81 +1,66 @@ -#----------------------DOMAIN DEFINITION------------------------ -geometry.is_periodic = 0 0 # For each dir, 0: non-perio, 1: periodic -geometry.coord_sys = 1 # 0 => cart, 1 => RZ -geometry.prob_lo = 0.000 0.0 0.0 # x_lo y_lo (z_lo) -geometry.prob_hi = 0.016 0.032 0.016 # x_hi y_hi (z_hi) +#---------------------- DOMAIN DEFINITION ------------------------ +geometry.is_periodic = 0 0 # For each dir, 0: non-perio, 1: periodic +geometry.coord_sys = 1 # 0 => cart, 1 => RZ +geometry.prob_lo = 0.0 0.0 # x_lo y_lo (z_lo) +geometry.prob_hi = 0.016 0.032 # x_hi y_hi (z_hi) -# >>>>>>>>>>>>> BC FLAGS <<<<<<<<<<<<<<<< +#---------------------- BC FLAGS --------------------------------- # Interior, Inflow, Outflow, Symmetry, # SlipWallAdiab, NoSlipWallAdiab, SlipWallIsotherm, NoSlipWallIsotherm peleLM.lo_bc = Symmetry SlipWallAdiab peleLM.hi_bc = SlipWallAdiab Outflow - -#-------------------------AMR CONTROL---------------------------- -amr.n_cell = 32 64 32 # Level 0 number of cells in each direction -amr.n_cell = 64 128 32 # Level 0 number of cells in each direction -amr.v = 1 # AMR verbose -amr.max_level = 1 # maximum level number allowed -amr.ref_ratio = 2 2 2 2 # refinement ratio -amr.regrid_int = 5 # how often to regrid -amr.n_error_buf = 1 1 2 2 # number of buffer cells in error est -amr.grid_eff = 0.7 # what constitutes an efficient grid -amr.blocking_factor = 16 # block factor in grid generation (min box size) +#---------------------- AMR CONTROL ------------------------------ +amr.n_cell = 64 128 # Level 0 number of cells in each direction +amr.v = 1 # AMR verbose +amr.max_level = 2 # maximum level number allowed +amr.ref_ratio = 2 2 2 2 # refinement ratio +amr.regrid_int = 5 # how often to regrid +amr.n_error_buf = 1 1 2 2 # number of buffer cells in error est +amr.grid_eff = 0.7 # what constitutes an efficient grid +amr.blocking_factor = 16 # block factor in grid generation (min box size) amr.max_grid_size = 128 # max box size -#--------------------------- Problem ------------------------------- +#---------------------- Problem ---------------------------------- prob.P_mean = 101325.0 prob.T_bubble = 450.0 prob.bubble_radius = 0.003 prob.bubble_y0 = 0.01 prob.use_symmetry = 1 -#-------------------------PeleLM CONTROL---------------------------- +#---------------------- PeleLMeX CONTROL ------------------------- peleLM.v = 1 peleLM.gravity = 0.0 -9.81 0.0 peleLM.use_wbar = 0 peleLM.sdc_iterMax = 1 -peleLM.floor_species = 0 peleLM.num_init_iter = 2 -#peleLM.num_divu_iter = 0 -#peleLM.advection_scheme = "Godunov_PLM" -peleLM.deltaT_verbose = 1 +#---------------------- Time Stepping CONTROL -------------------- +amr.max_step = 400 +amr.stop_time = 0.07 +amr.cfl = 0.2 +amr.dt_shrink = 1.0 + +#---------------------- IO CONTROL ------------------------------- #amr.restart = chk00005 #amr.check_int = 2000 amr.plot_int = 2000 -amr.max_step = 2000 -amr.dt_shrink = 1.0 -#amr.init_dt = 0.0001 -amr.stop_time = 0.07 -amr.cfl = 0.2 amr.derive_plot_vars = avg_pressure mag_vort mass_fractions -cvode.solve_type = dense # CVODE Linear solve type (for Newton direction) -ode.analytical_jacobian = 0 # Provide analytical jacobian (from Fuego) -nodal_proj.verbose = 0 -nodal_proj.rtol = 1e-13 -mac_proj.rtol = 1e-12 -mac_proj.atol = 1e-16 -mac_proj.verbose = 0 - -#------------ INPUTS TO CONSTANT TRANSPORT ----------------- -transport.const_viscosity = 0.0 -transport.const_bulk_viscosity = 0.0 -transport.const_conductivity = 0.0 -transport.const_diffusivity = 0.0 +#---------------------- Reactor CONTROL -------------------------- +peleLM.chem_integrator = "ReactorNull" -#--------------------REFINEMENT CONTROL------------------------ -#amr.refinement_indicators = temp -#amr.temp.max_level = 1 -#amr.temp.value_greater = 305 -#amr.temp.field_name = temp +#---------------------- Constant Transport ----------------------- +transport.const_viscosity = 0.0 +transport.const_bulk_viscosity = 0.0 +transport.const_conductivity = 0.0 +transport.const_diffusivity = 0.0 -#amr.refinement_indicators = magVort -#amr.magVort.max_level = 1 -#amr.magVort.value_greater = 500.0 -#amr.magVort.field_name = mag_vort +#---------------------- Linear solver CONTROL -------------------- +mac_proj.verbose = 0 +nodal_proj.verbose = 0 +#---------------------- Refinement CONTROL------------------------ amr.refinement_indicators = temp gtemp amr.temp.max_level = 2 amr.temp.value_greater = 350.0 @@ -85,9 +70,7 @@ amr.gtemp.max_level = 4 amr.gtemp.adjacent_difference_greater = 10.0 amr.gtemp.field_name = temp -diffusion.verbose = 0 -tensor_diffusion.verbose = 0 - +#---------------------- Debug/HPC CONTROL------------------------- #amrex.fpe_trap_invalid = 1 #amrex.fpe_trap_zero = 1 #amrex.fpe_trap_overflow = 1 diff --git a/Exec/RegTests/PeriodicCases/README.md b/Exec/RegTests/PeriodicCases/README.md new file mode 100644 index 00000000..aaebf7e9 --- /dev/null +++ b/Exec/RegTests/PeriodicCases/README.md @@ -0,0 +1,4 @@ +## PeriodicCases +A set of 2D/3D periodic cases: convected vortex in different directions, convected temperature/mixture Gaussian bump, and +pure diffusion of species/temperature. This is the base case to test PeleLMeX accuracy and order of convergence of +the spatial and temporal schemes. diff --git a/Exec/RegTests/README.md b/Exec/RegTests/README.md index b40acc37..4f0bc170 100644 --- a/Exec/RegTests/README.md +++ b/Exec/RegTests/README.md @@ -1,53 +1,6 @@ # PeleLMeX testing This folder contains a set of small-scale cases testing the various functionalities of PeleLMeX. -A number of these cases are automatically compiled or compiled & run upon pull request into the development branch, -The following provide a brief summary of each case and its intended purpose. - -## EB\_EnclosedFlame -A 2D (3D) cylindrical flame enclosed in a cylindrical EB chamber. Testing the closed chamber algorithm with EB. - -## EB\_EnclosedVortex -A 2D (3D) vortex enclosed in a cylindrical EB chamber. Testing EB for pure incompressible flows. - -## EB\_FlowPastCylinder -A 2D (3D) inflow/outflow setup with either an EB cylinder in the middle of the flow or an EB -bump on a WallNoSlipAdiab domain boundary. Testing EB for incompressible flows and pure mixing cases (no reactions). - -## EB\_PipeFlow -2D and 3D setup of both channel and pipe flow, either in an inflow/outflow setup or a -periodc setup driven by a background pressure gradient. - -## EnclosedFlame -A 2D (3D) cylindrical flame enclosed in a square (rectangular) domain. Testing the closed chamber algorithm. - -## EnclosedInjection -A simple jet injection into an EB cylindrical chamber. Testing the closed chamber algorithm with mass injection. - -## FlameSheet -A 2D (3D) harmonically perturbed flame sheet, initial solution from a Cantera simulation provided for 3 mechanisms -(drm19, dodecane\_lu and dodecane\_lu\_qss). This is the basis for weak scaling studies in PeleLMeX and test all the -reactive pieces of the algorithm as well as transport options (Unity Lewis number, Soret effect, ...) - -## HITDecay -A 3D decaying HIT case, where the initial solution is generated with a Passot-Pouquet spectrum. Testing basic incompressible -integration algorithm and Large Eddy Simulation implementation. - -## HotBubble -Case of a 2D/2D-RZ/3D bubble of light gases (either hotter or lighter mixture composition) lifted under the effect of -a gravity field. Testing of RZ algorthim and gravity forces. - -## PeriodicCases -A set of 2D/3D periodic cases: convected vortex in different directions, convected temperature/mixture Gaussian bump, -pure diffusion of species/temperature. This is the base case to test PeleLMeX accuracy and order of convergence of -the spatial and temporal schemes. - -## SprayTest -Basic testing of the coupling between PeleLMeX and PeleMP Lagrangian particle module. - -## TaylorGreen -3D Taylor-Green vortex case. Enable testing of the different advection schemes available in PeleLMeX. - -## TurbInflow -Injection of a 3D precomputed turbulent velocity field at the boundary of a PeleLMeX simulation. Testing the -turbulence injection functionality. +A number of these cases are automatically compiled or compiled & run upon pull requests into the development branch. +A short description of all the case is provided in the README file in each folder, stating the tested PeleLMeX capabilities and, +when available, pointing towards a tutorial or documentation page. diff --git a/Exec/RegTests/SprayTest/README.md b/Exec/RegTests/SprayTest/README.md new file mode 100644 index 00000000..53cb8c2b --- /dev/null +++ b/Exec/RegTests/SprayTest/README.md @@ -0,0 +1,2 @@ +## SprayTest +Basic testing of the coupling between PeleLMeX and PeleMP Lagrangian particle module. diff --git a/Exec/RegTests/TaylorGreen/README.md~ b/Exec/RegTests/TaylorGreen/README.md~ new file mode 100644 index 00000000..cbf22376 --- /dev/null +++ b/Exec/RegTests/TaylorGreen/README.md~ @@ -0,0 +1,21 @@ +# Taylor-Green Vortex (pulled from PeleC) + +This setup is one of the test problems outlined by [the High-Order CFD workshop](https://www.grc.nasa.gov/hiocfd/). + +A complete description of the problem can be found +[here](https://www.grc.nasa.gov/hiocfd/wp-content/uploads/sites/22/case_c3.3.pdf) and +the reference data is found +[here](https://www.grc.nasa.gov/wp-content/uploads/sites/22/C3.3_datafiles.zip). More +details of the problem and methods used to obtain the reference data +can be found in Bull and Jameson (2014) 7th AIAA Theoretical Fluid +Mechanics Conference (doi: 10.2514/6.2014-3210) and DeBonis (2013) +51st AIAA Aerospace Sciences Meeting (doi:10.2514/6.2013-382). + +To directly plot the evolution of integrated kinetic energy and enstrophy, use +the data provided in temporals/tempState with the help of the processTGdata.py +script. + +Note that these commands provide adimentional results, with t\* = t/(L\_0/V\_0), +E\_k\* = E\_k / ( rho\_0 * V\_0 * V\_0) and psi\* = psi / (V\_0/L\_0)^2 + +PeleLMeX results can be found in [here](https://amrex-combustion.github.io/PeleLMeX/manual/html/Validation.html#taylor-green-vortex-breakdown). diff --git a/Exec/RegTests/TaylorGreen/Readme.md b/Exec/RegTests/TaylorGreen/Readme.md index 2294c207..fcbc3736 100644 --- a/Exec/RegTests/TaylorGreen/Readme.md +++ b/Exec/RegTests/TaylorGreen/Readme.md @@ -2,7 +2,7 @@ This setup is one of the test problems outlined by [the High-Order CFD workshop](https://www.grc.nasa.gov/hiocfd/). -A complete description of the problem can be found +A complete description of the problem can be found [here](https://www.grc.nasa.gov/hiocfd/wp-content/uploads/sites/22/case_c3.3.pdf) and the reference data is found [here](https://www.grc.nasa.gov/wp-content/uploads/sites/22/C3.3_datafiles.zip). More @@ -15,5 +15,7 @@ To directly plot the evolution of integrated kinetic energy and enstrophy, use the data provided in temporals/tempState with the help of the processTGdata.py script. -Note that these commands provide adimentional results, with t\* = t/(L\_0/V\_0), +Note that these commands provide dimensional results, with t\* = t/(L\_0/V\_0), E\_k\* = E\_k / ( rho\_0 * V\_0 * V\_0) and psi\* = psi / (V\_0/L\_0)^2 + +PeleLMeX results can be found [here](https://amrex-combustion.github.io/PeleLMeX/manual/html/Validation.html#taylor-green-vortex-breakdown). diff --git a/Exec/RegTests/TripleFlame/README.md b/Exec/RegTests/TripleFlame/README.md new file mode 100644 index 00000000..537c68c9 --- /dev/null +++ b/Exec/RegTests/TripleFlame/README.md @@ -0,0 +1,3 @@ +## Triple flame +2D case of a methane/air triple flame propagating into a mixing layer and stabilized using PeleLMeX active control capabilities. +More details and step-by-step instructions on this case are available in this [tutorial](https://amrex-combustion.github.io/PeleLMeX/manual/html/Tutorials_TripleFlame.html). diff --git a/Exec/RegTests/TurbInflow/README.md b/Exec/RegTests/TurbInflow/README.md new file mode 100644 index 00000000..d01cbf2d --- /dev/null +++ b/Exec/RegTests/TurbInflow/README.md @@ -0,0 +1,3 @@ +## TurbInflow +Injection of a 3D precomputed turbulent velocity field at the boundary of a PeleLMeX simulation. Testing the +turbulence injection functionality. diff --git a/README.md b/README.md index b07b4904..aedc5110 100644 --- a/README.md +++ b/README.md @@ -5,13 +5,19 @@ [![Language: C++17](https://img.shields.io/badge/language-C%2B%2B17-blue)](https://isocpp.org/) [![JOSS](https://joss.theoj.org/papers/6142eb838783b07fce14450fefe21e07/status.svg)](https://joss.theoj.org/papers/6142eb838783b07fce14450fefe21e07) -## Overview - ![CUDA build](https://github.com/AMReX-Combustion/PeleLMeX/workflows/PeleLMeX_Cuda/badge.svg) ![HIP build](https://github.com/AMReX-Combustion/PeleLMeX/workflows/PeleLMeX_Hip/badge.svg) ![SYCL build](https://github.com/AMReX-Combustion/PeleLMeX/workflows/PeleLMeX_Intel/badge.svg) -*PeleLMeX* is a non-subcycling version of [*PeleLM*](https://github.com/AMReX-Combustion/PeleLM) based on AMReX's [AmrCore](https://amrex-codes.github.io/amrex/docs_html/AmrCore.html) and borrowing from the incompressible solver [incflo](https://github.com/AMReX-Codes/incflo). It is designed to run efficiently on small workstations as well as the largest ExaScale platforms currently available. +## Overview + +*PeleLMeX* is a solver for high fidelity reactive flow simulations, namely direct numerical simulation (DNS) and large eddy simulation (LES). +The solver combines a low Mach number approach, adaptive mesh refinement (AMR), embedded boundary (EB) geometry treatment and high performance +computing (HPC) to provide +a flexible tool to address research questions on platforms ranging from small workstations to the world's largest GPU-accelerated supercomputers. +*PeleLMeX* has been used to study complex flame/turbulence interactions in [RCCI engines](https://gfm.aps.org/meetings/dfd-2022/63236765199e4c2c0873f9f6) and +[hydrogen combustion](https://www.sciencedirect.com/science/article/pii/S001021802300192X) or [the effect of sustainable aviation fuel +on gas turbine combustion](https://www.osti.gov/biblio/1995457). *PeleLMeX* is part of the [Pele combustion Suite](https://amrex-combustion.github.io/). @@ -19,7 +25,14 @@ ![Documentation](https://github.com/AMReX-Combustion/PeleLMeX/workflows/PeleLMeX-Docs/badge.svg) -*PeleLMeX* solves of the multispecies reactive Navier-Stokes equations in the low Mach number limit as described in the [documentation](https://amrex-combustion.github.io/PeleLMeX/manual/html/index.html). It inherits most of *PeleLM* algorithmic features, but differs significantly in its implementation stemming from the non-subcycling approach. +*PeleLMeX* is a non-subcycling version of [*PeleLM*](https://github.com/AMReX-Combustion/PeleLM) based on AMReX's +[AmrCore](https://amrex-codes.github.io/amrex/docs_html/AmrCore.html) and borrowing from the incompressible +solver [incflo](https://github.com/AMReX-Codes/incflo). It solves of the multispecies reactive Navier-Stokes equations +in the low Mach number limit as described in the [documentation](https://amrex-combustion.github.io/PeleLMeX/manual/html/index.html). +It inherits most of *PeleLM* algorithmic features, but differs significantly in its implementation stemming from the non-subcycling approach. +*PeleLM* is no longer under active development; *PeleLMeX* should be used for simulations of low Mach number reacting flows and +[*PeleC*](https://github.com/AMReX-Combustion/PeleC) for simulations of flows with higher Mach numbers where compressibility effects are +significant. A overview of *PeleLMeX* controls is provided in the [documentation](https://amrex-combustion.github.io/PeleLMeX/manual/html/LMeXControls.html). @@ -39,18 +52,30 @@ https://amrex-combustion.github.io/PeleLMeX/manual/html/Tutorials.html ## Installation +### Requirements + +The compilations of *PeleLMeX* requires a C++17 compatible compiler (GCC >= 8 or Clang >= 3.6) as +well as [CMake](https://cmake.org/download/) >= 3.23 for compiling the [SUNDIALS](https://github.com/LLNL/sundials) third party library. + +Most of the examples provided hereafter and in the [tutorials](https://amrex-combustion.github.io/PeleLMeX/manual/html/Tutorials.html) +will use MPI to run in parallel. Although not mandatory, it is advised to build *PeleLMeX* with MPI support from the get go if +more than a single core is available to you. Any of [mpich](https://www.mpich.org/downloads/) or +[open-mpi](https://www.open-mpi.org/software/ompi/v4.1/) is a suitable option if MPI is not already available on your platform. + +Finally, when building with GPU support, CUDA >= 11 is required with NVIDIA GPUs and ROCm >= 5.2 is required with AMD GPUs. + ### Download -The prefered method consist in cloning the *PeleLMeX* and its submodules using a recursive `git clone`: -([amrex](https://github.com/AMReX-Codes/amrex), [AMReX-Hydro](https://github.com/AMReX-Codes/AMReX-Hydro) and [PelePhysics](https://github.com/AMReX-Combustion/PelePhysics), [PeleMP](https://github.com/AMReX-Combustion/PeleMP)) using: +The prefered method consists of cloning *PeleLMeX* and its submodules +([amrex](https://github.com/AMReX-Codes/amrex), [AMReX-Hydro](https://github.com/AMReX-Codes/AMReX-Hydro), [PelePhysics](https://github.com/AMReX-Combustion/PelePhysics), [PeleMP](https://github.com/AMReX-Combustion/PeleMP)), and [SUNDIALS](https://github.com/LLNL/sundials) using a recursive `git clone`: ``` git clone --recursive https://github.com/AMReX-Combustion/PeleLMeX.git ``` -Alternatively, you can use separate `git clone` of the each of the submodules. -The default location for *PeleLMeX* dependencies is the `Submodule` folder but you optionnally -setup the following environment variables (e.g. using bash) to an other location: +Alternatively, you can use a separate `git clone` of each of the submodules. +The default location for *PeleLMeX* dependencies is the `Submodules` folder but you optionnally +setup the following environment variables (e.g. using bash) to any other location: ``` export PELELMEX_HOME= @@ -58,24 +83,25 @@ export AMREX_HOME=${PELELMEX_HOME}/Submodules/amrex export AMREX_HYDRO_HOME=${PELELMEX_HOME}/Submodules/AMReX-Hydro export PELE_PHYSICS_HOME=${PELELMEX_HOME}/Submodules/PelePhysics export PELEMP_HOME=${PELELMEX_HOME}/Submodules/PeleMP +export SUNDIALS_HOME=${PELELMEX_HOME}/Submodules/sundials ``` ### Compilation Both GNUmake and CMake can be used to build a *PeleLMeX* executable, but GNUmake is the prefered choice. The code handling the initial condition and boundary conditions is unique to each case, -and subfolders in the `Exec` directory provide a number of example. +and subfolders in the `Exec` directory provide a number of examples. For instance, to compile the executable for the case of a rising hot bubble, move into the `HotBubble` folder: ``` -cd PeleLMeX/Exec/RegTest/HotBubble +cd PeleLMeX/Exec/RegTests/HotBubble ``` If this is a clean install, you will need to make the third party libraries with: `make TPL` (note: if on macOS, you might need to specify `COMP=llvm` in the `make` statements). -Finally, make with: `make -j`, or if on macOS: `make -j COMP=llvm`. To clean the installation, use either `make clean` or `make realclean`. If running into compile errors after changing compile time options in PeleLMeX (e.g., the chemical mechanism), the first thing to try is to clean your build by running `make TPLrealclean && make realclean`, then try to rebuild the third party libraries and PeleLMeX with `make TPL && make -j`. +Finally, make with: `make -j`, or if on macOS: `make -j COMP=llvm`. To clean the installation, use either `make clean` or `make realclean`. If running into compile errors after changing compile time options in PeleLMeX (e.g., the chemical mechanism), the first thing to try is to clean your build by running `make TPLrealclean && make realclean`, then try to rebuild the third party libraries and PeleLMeX with `make TPL && make -j`. See the [Tutorial](https://amrex-combustion.github.io/PeleLMeX/manual/html/Tutorials_HotBubble.html) for this case for instructions on how to compile with different options (for example, to compile without MPI support or to compile for GPUs) and how to run the code once compiled. When using CMake (cmake version >= 3.23 is required), first configure CMake directly in *PeleLMeX* root folder: @@ -89,7 +115,13 @@ Then build the executable: cmake --build buildHotBubble --parallel 4 ``` -## Contributing +## Getting help, contributing + +Do you have a question ? Found an issue ? Please use the [GitHub Discussions](https://github.com/AMReX-Combustion/PeleLMeX/discussions) to engage +with the development team or open a new [GitHub issue](https://github.com/AMReX-Combustion/PeleLMeX/issues) to report a bug. The developmemt team +also encourages users to take an active role in respectfully answering each other's questions in these spaces. When reporting a bug, it is helpful +to provide as much detail as possible, including a case description and the major compile and runtime options being used. Though not required, +it is most effective to create a fork of this repository and share a branch of that fork with a case that minimally reproduces the error. New contributions to *PeleLMeX* are welcome ! Contributing Guidelines are provided in [CONTRIBUTING.md](CONTRIBUTING.md). diff --git a/Source/Efield/PeleLMEFNLSolve.cpp b/Source/Efield/PeleLMEFNLSolve.cpp index c9fbd3a9..cdcfb3df 100644 --- a/Source/Efield/PeleLMEFNLSolve.cpp +++ b/Source/Efield/PeleLMEFNLSolve.cpp @@ -20,7 +20,7 @@ void PeleLM::implicitNonLinearSolve(int sdcIter, std::unique_ptr &diffData, std::unique_ptr &advData) { - BL_PROFILE_VAR("PeleLM::implicitNonLinearSolve()", implicitNonLinearSolve); + BL_PROFILE_VAR("PeleLMeX::implicitNonLinearSolve()", implicitNonLinearSolve); const Real strt_time = ParallelDescriptor::second(); @@ -196,7 +196,7 @@ void PeleLM::implicitNonLinearSolve(int sdcIter, Real avgGMRES = (float)GMRES_tot_count/(float)NK_tot_count; amrex::Print() << " [" << sdcIter << "] dt: " << a_dt << " - Avg GMRES/Newton: " << avgGMRES << "\n"; } - amrex::Print() << " >> PeleLM::implicitNLSolve() " << run_time << "\n"; + amrex::Print() << " >> PeleLMeX::implicitNLSolve() " << run_time << "\n"; } //VisMF::Write(advData->Forcing[0],"ForcingNE"); @@ -813,7 +813,7 @@ void PeleLM::getAdvectionFluxes(int lev, void PeleLM::setUpPrecond(const Real &a_dt, const Vector &a_nE) { - BL_PROFILE("PeleLM::setUpPrecond()"); + BL_PROFILE("PeleLMeX::setUpPrecond()"); // Udpate LinearOps defs if needed -> done internally by the getPrecondOp() func @@ -1064,7 +1064,7 @@ void PeleLM::jTimesV(const Vector &a_v, void PeleLM::applyPrecond(const Vector &a_v, const Vector &a_Pv) { - BL_PROFILE("PeleLM::applyPrecond()"); + BL_PROFILE("PeleLMeX::applyPrecond()"); // Setup aliases and temps Vector nE_al; diff --git a/Source/Efield/PeleLMEFPoisson.cpp b/Source/Efield/PeleLMEFPoisson.cpp index 1d3d3e11..fab50506 100644 --- a/Source/Efield/PeleLMEFPoisson.cpp +++ b/Source/Efield/PeleLMEFPoisson.cpp @@ -7,7 +7,7 @@ using namespace amrex; void PeleLM::poissonSolveEF(const TimeStamp &a_time) { - BL_PROFILE_VAR("PeleLM::poissonSolveEF()", poissonSolveEF); + BL_PROFILE("PeleLMeX::poissonSolveEF()"); if (ef_verbose) { Print() << " EF Poisson solve \n"; } diff --git a/Source/Efield/PeleLMEFTransport.cpp b/Source/Efield/PeleLMEFTransport.cpp index 055cf390..70325da9 100644 --- a/Source/Efield/PeleLMEFTransport.cpp +++ b/Source/Efield/PeleLMEFTransport.cpp @@ -4,7 +4,7 @@ using namespace amrex; void PeleLM::calcEFTransport(const TimeStamp &a_time) { - BL_PROFILE("PeleLM::calcEFTransport()"); + BL_PROFILE("PeleLMeX::calcEFTransport()"); for (int lev = 0; lev <= finest_level; ++lev) { diff --git a/Source/PeleLMAdvance.cpp b/Source/PeleLMAdvance.cpp index 73d78b1d..68499e67 100644 --- a/Source/PeleLMAdvance.cpp +++ b/Source/PeleLMAdvance.cpp @@ -5,7 +5,7 @@ using namespace amrex; void PeleLM::Advance(int is_initIter) { - BL_PROFILE("PeleLM::Advance()"); + BL_PROFILE("PeleLMeX::Advance()"); #ifdef AMREX_MEM_PROFILING // Memory profiler if compiled @@ -16,7 +16,7 @@ void PeleLM::Advance(int is_initIter) { Real strt_time = ParallelDescriptor::second(); //---------------------------------------------------------------- - BL_PROFILE_VAR("PeleLM::advance::setup", PLM_SETUP); + BL_PROFILE_VAR("PeleLMeX::advance::setup", PLM_SETUP); //---------------------------------------------------------------- // Deal with ambient pressure @@ -121,11 +121,11 @@ void PeleLM::Advance(int is_initIter) { floorSpecies(AmrOldTime); //---------------------------------------------------------------- - BL_PROFILE_VAR("PeleLM::advance::mac", PLM_MAC); + BL_PROFILE_VAR("PeleLMeX::advance::mac", PLM_MAC); setThermoPress(AmrOldTime); BL_PROFILE_VAR_STOP(PLM_MAC); //---------------------------------------------------------------- - BL_PROFILE_VAR("PeleLM::advance::diffusion", PLM_DIFF); + BL_PROFILE_VAR("PeleLMeX::advance::diffusion", PLM_DIFF); computeDifferentialDiffusionTerms(AmrOldTime,diffData); BL_PROFILE_VAR_STOP(PLM_DIFF); //---------------------------------------------------------------- @@ -192,7 +192,7 @@ void PeleLM::Advance(int is_initIter) { //---------------------------------------------------------------- //---------------------------------------------------------------- - BL_PROFILE_VAR("PeleLM::advance::velocity", PLM_VEL); + BL_PROFILE_VAR("PeleLMeX::advance::velocity", PLM_VEL); // Velocity advance Real VelAdvStart = 0.0; if (m_verbose > 1) { @@ -235,7 +235,7 @@ void PeleLM::Advance(int is_initIter) { { Real run_time = ParallelDescriptor::second() - strt_time; ParallelDescriptor::ReduceRealMax(run_time, ParallelDescriptor::IOProcessorNumber()); - amrex::Print() << " >> PeleLM::Advance() --> Time: " << run_time << "\n"; + amrex::Print() << " >> PeleLMeX::Advance() --> Time: " << run_time << "\n"; } } @@ -243,7 +243,7 @@ void PeleLM::oneSDC(int sdcIter, std::unique_ptr &advData, std::unique_ptr &diffData) { - BL_PROFILE("PeleLM::oneSDC()"); + BL_PROFILE("PeleLMeX::oneSDC()"); m_sdcIter = sdcIter; if (m_verbose > 0) { @@ -290,7 +290,7 @@ void PeleLM::oneSDC(int sdcIter, //---------------------------------------------------------------- // Get u MAC //---------------------------------------------------------------- - BL_PROFILE_VAR("PeleLM::advance::mac", PLM_MAC); + BL_PROFILE_VAR("PeleLMeX::advance::mac", PLM_MAC); Real MACStart = 0.0; if (m_verbose > 1) { MACStart = ParallelDescriptor::second(); @@ -318,7 +318,7 @@ void PeleLM::oneSDC(int sdcIter, //---------------------------------------------------------------- // Scalar advections //---------------------------------------------------------------- - BL_PROFILE_VAR("PeleLM::advance::scalars_adv", PLM_SADV); + BL_PROFILE_VAR("PeleLMeX::advance::scalars_adv", PLM_SADV); Real ScalAdvStart = 0.0; if (m_verbose > 1) { ScalAdvStart = ParallelDescriptor::second(); @@ -349,7 +349,7 @@ void PeleLM::oneSDC(int sdcIter, //---------------------------------------------------------------- // Scalar diffusion //---------------------------------------------------------------- - BL_PROFILE_VAR("PeleLM::advance::diffusion", PLM_DIFF); + BL_PROFILE_VAR("PeleLMeX::advance::diffusion", PLM_DIFF); Real ScalDiffStart = 0.0; if (m_verbose > 1) { ScalDiffStart = ParallelDescriptor::second(); @@ -378,7 +378,7 @@ void PeleLM::oneSDC(int sdcIter, //---------------------------------------------------------------- // Reaction //---------------------------------------------------------------- - BL_PROFILE_VAR("PeleLM::advance::reactions", PLM_REAC); + BL_PROFILE_VAR("PeleLMeX::advance::reactions", PLM_REAC); Real ScalReacStart = 0.0; if (m_verbose > 1) { ScalReacStart = ParallelDescriptor::second(); diff --git a/Source/PeleLMBC.cpp b/Source/PeleLMBC.cpp index 7154a522..031512d3 100644 --- a/Source/PeleLMBC.cpp +++ b/Source/PeleLMBC.cpp @@ -239,7 +239,7 @@ PeleLM::fetchBCRecArray(int scomp, int ncomp) { // Fill the entire class state at once void PeleLM::fillPatchState(const TimeStamp &a_time) { - BL_PROFILE("PeleLM::fillPatchState()"); + BL_PROFILE("PeleLMeX::fillPatchState()"); for (int lev = 0; lev <= finest_level; lev++) { fillPatchState(lev,a_time); } @@ -247,7 +247,7 @@ void PeleLM::fillPatchState(const TimeStamp &a_time) { // Fill the a given level class state void PeleLM::fillPatchState(int lev, const TimeStamp &a_time) { - BL_PROFILE("PeleLM::fillPatchStateLev()"); + BL_PROFILE("PeleLMeX::fillPatchStateLev()"); auto ldata_p = getLevelDataPtr(lev,a_time); Real time = getTime(lev, a_time); @@ -262,7 +262,7 @@ void PeleLM::fillPatchState(int lev, const TimeStamp &a_time) { // Fill a state components void PeleLM::fillPatchDensity(const TimeStamp &a_time) { - BL_PROFILE("PeleLM::fillPatchDensity()"); + BL_PROFILE("PeleLMeX::fillPatchDensity()"); for (int lev = 0; lev <= finest_level; lev++) { auto ldata_p = getLevelDataPtr(lev,a_time); Real time = getTime(lev, a_time); @@ -271,7 +271,7 @@ void PeleLM::fillPatchDensity(const TimeStamp &a_time) { } void PeleLM::fillPatchSpecies(const TimeStamp &a_time) { - BL_PROFILE("PeleLM::fillPatchSpecies()"); + BL_PROFILE("PeleLMeX::fillPatchSpecies()"); for (int lev = 0; lev <= finest_level; lev++) { auto ldata_p = getLevelDataPtr(lev,a_time); Real time = getTime(lev, a_time); @@ -280,7 +280,7 @@ void PeleLM::fillPatchSpecies(const TimeStamp &a_time) { } void PeleLM::fillPatchTemp(const TimeStamp &a_time) { - BL_PROFILE("PeleLM::fillPatchTemp()"); + BL_PROFILE("PeleLMeX::fillPatchTemp()"); for (int lev = 0; lev <= finest_level; lev++) { auto ldata_p = getLevelDataPtr(lev,a_time); Real time = getTime(lev, a_time); @@ -290,7 +290,7 @@ void PeleLM::fillPatchTemp(const TimeStamp &a_time) { #ifdef PELE_USE_EFIELD void PeleLM::fillPatchPhiV(const TimeStamp &a_time) { - BL_PROFILE("PeleLM::fillPatchPhiV()"); + BL_PROFILE("PeleLMeX::fillPatchPhiV()"); for (int lev = 0; lev <= finest_level; lev++) { auto ldata_p = getLevelDataPtr(lev,a_time); Real time = getTime(lev, a_time); @@ -305,7 +305,7 @@ void PeleLM::fillPatchPhiV(const TimeStamp &a_time) { // Fill the entire state at once std::unique_ptr PeleLM::fillPatchState(int lev, Real a_time, int nGrow) { - BL_PROFILE("PeleLM::fillPatchState()"); + BL_PROFILE("PeleLMeX::fillPatchState()"); std::unique_ptr mf; if ( m_incompressible ) { @@ -320,7 +320,7 @@ PeleLM::fillPatchState(int lev, Real a_time, int nGrow) { std::unique_ptr PeleLM::fillPatchReact(int lev, Real a_time, int nGrow) { - BL_PROFILE("PeleLM::fillPatchReact()"); + BL_PROFILE("PeleLMeX::fillPatchReact()"); int IRsize = NUM_SPECIES; #ifdef PELE_USE_EFIELD @@ -797,7 +797,7 @@ void PeleLM::fillcoarsepatch_chemFunctCall(int lev, void PeleLM::setInflowBoundaryVel(MultiFab &a_vel, int lev, TimeStamp a_time) { - BL_PROFILE_VAR("PeleLM::setInflowBoundaryVel()", setInflowBoundaryVel); + BL_PROFILE("PeleLMeX::setInflowBoundaryVel()"); Real time = getTime(lev, a_time); diff --git a/Source/PeleLMDiagnostics.cpp b/Source/PeleLMDiagnostics.cpp index 7037c35b..46a0f7e1 100644 --- a/Source/PeleLMDiagnostics.cpp +++ b/Source/PeleLMDiagnostics.cpp @@ -62,7 +62,7 @@ PeleLM::updateDiagnostics() void PeleLM::doDiagnostics() { - BL_PROFILE("PeleLM::doDiagnostics()"); + BL_PROFILE("PeleLMeX::doDiagnostics()"); // Assemble a vector of MF containing the requested data Vector > diagMFVec(finestLevel()+1); for (int lev{0}; lev <= finestLevel(); ++lev) { diff --git a/Source/PeleLMDiffusion.cpp b/Source/PeleLMDiffusion.cpp index 0ba212bd..8299a180 100644 --- a/Source/PeleLMDiffusion.cpp +++ b/Source/PeleLMDiffusion.cpp @@ -36,7 +36,7 @@ void PeleLM::computeDifferentialDiffusionTerms(const TimeStamp &a_time, std::unique_ptr &diffData, int is_init) { - BL_PROFILE("PeleLM::computeDifferentialDiffusionTerms()"); + BL_PROFILE("PeleLMeX::computeDifferentialDiffusionTerms()"); //---------------------------------------------------------------- // Checks @@ -198,7 +198,7 @@ void PeleLM::computeDifferentialDiffusionFluxes(const TimeStamp &a_time, const Vector > &a_wbarfluxes, const Vector > &a_soretfluxes) { - BL_PROFILE("PeleLM::computeDifferentialDiffusionFluxes()"); + BL_PROFILE("PeleLMeX::computeDifferentialDiffusionFluxes()"); #ifdef AMREX_USE_EB int have_EBfluxes = (a_EBfluxes.empty()) ? 0 : 1; @@ -614,7 +614,7 @@ void PeleLM::adjustSpeciesFluxes(const Vector > Vector const &a_spec) { - BL_PROFILE("PeleLM::adjustSpeciesFluxes()"); + BL_PROFILE("PeleLMeX::adjustSpeciesFluxes()"); // Get the species BCRec auto bcRecSpec = fetchBCRecArray(FIRSTSPEC,NUM_SPECIES); @@ -710,7 +710,7 @@ void PeleLM::computeSpeciesEnthalpyFlux(const Vector const &a_temp) { - BL_PROFILE("PeleLM::computeSpeciesEnthalpyFlux()"); + BL_PROFILE("PeleLMeX::computeSpeciesEnthalpyFlux()"); // Get the species BCRec auto bcRecSpec = fetchBCRecArray(FIRSTSPEC,NUM_SPECIES); @@ -798,7 +798,7 @@ void PeleLM::computeSpeciesEnthalpyFlux(const Vector &advData, std::unique_ptr &diffData) { - BL_PROFILE("PeleLM::differentialDiffusionUpdate()"); + BL_PROFILE("PeleLMeX::differentialDiffusionUpdate()"); //------------------------------------------------------------------------ // Setup fluxes @@ -1374,7 +1374,7 @@ void PeleLM::computeDivTau(const TimeStamp &a_time, int use_density, Real scale) { - BL_PROFILE("PeleLM::computeDivTau()"); + BL_PROFILE("PeleLMeX::computeDivTau()"); // Get the density component BCRec to get viscosity on faces auto bcRec = fetchBCRecArray(DENSITY,1); @@ -1395,7 +1395,7 @@ void PeleLM::computeDivTau(const TimeStamp &a_time, void PeleLM::diffuseVelocity() { - BL_PROFILE("PeleLM::diffuseVelocity()"); + BL_PROFILE("PeleLMeX::diffuseVelocity()"); // Get the density component BCRec to get viscosity on faces auto bcRec = fetchBCRecArray(DENSITY,1); diff --git a/Source/PeleLMEB.cpp b/Source/PeleLMEB.cpp index 237f64fa..2720a019 100644 --- a/Source/PeleLMEB.cpp +++ b/Source/PeleLMEB.cpp @@ -13,7 +13,7 @@ using namespace amrex; void PeleLM::makeEBGeometry() { - BL_PROFILE("PeleLM::makeEBGeometry()"); + BL_PROFILE("PeleLMeX::makeEBGeometry()"); int max_coarsening_level = 100; int req_coarsening_level = geom.size()-1; @@ -59,7 +59,7 @@ void PeleLM::redistributeAofS(int a_lev, const BCRec * d_bc, const Geometry &a_geom) { - BL_PROFILE("PeleLM::redistributeAofS()"); + BL_PROFILE("PeleLMeX::redistributeAofS()"); AMREX_ASSERT(a_tmpDiv.nComp() >= div_comp+ncomp); AMREX_ASSERT(a_AofS.nComp() >= aofs_comp+ncomp); AMREX_ASSERT(a_state.nComp() >= state_comp+ncomp); @@ -183,7 +183,7 @@ void PeleLM::redistributeDiff(int a_lev, const BCRec * d_bc, const Geometry &a_geom) { - BL_PROFILE("PeleLM::redistributeDiff()"); + BL_PROFILE("PeleLMeX::redistributeDiff()"); AMREX_ASSERT(a_tmpDiv.nComp() >= div_comp+ncomp); AMREX_ASSERT(a_diff.nComp() >= diff_comp+ncomp); AMREX_ASSERT(a_state.nComp() >= state_comp+ncomp); @@ -279,7 +279,7 @@ void PeleLM::initCoveredState() void PeleLM::setCoveredState(const TimeStamp &a_time) { - BL_PROFILE("PeleLM::setCoveredState()"); + BL_PROFILE("PeleLMeX::setCoveredState()"); for (int lev = 0; lev <= finest_level; lev++) { setCoveredState(lev,a_time); } @@ -379,7 +379,7 @@ void PeleLM::initialRedistribution() void PeleLM::getEBDistance(int a_lev, MultiFab &a_signDistLev) { - BL_PROFILE("PeleLM::getEBDistance()"); + BL_PROFILE("PeleLMeX::getEBDistance()"); if (a_lev == 0) { MultiFab::Copy(a_signDistLev,*m_signedDist0,0,0,1,0); @@ -566,7 +566,7 @@ void PeleLM::correct_vel_small_cells (Vector const& a_vel, Vector > const& a_umac) { - BL_PROFILE("PeleLM::correct_vel_small_cells"); + BL_PROFILE("PeleLMeX::correct_vel_small_cells"); for (int lev = 0; lev <= finest_level; lev++) { diff --git a/Source/PeleLMEos.cpp b/Source/PeleLMEos.cpp index 445d340d..dfd6857e 100644 --- a/Source/PeleLMEos.cpp +++ b/Source/PeleLMEos.cpp @@ -4,7 +4,7 @@ using namespace amrex; void PeleLM::setThermoPress(const TimeStamp &a_time) { - BL_PROFILE("PeleLM::setThermoPress()"); + BL_PROFILE("PeleLMeX::setThermoPress()"); AMREX_ASSERT(a_time == AmrOldTime || a_time == AmrNewTime); @@ -38,7 +38,7 @@ void PeleLM::calcDivU(int is_init, const TimeStamp &a_time, std::unique_ptr &diffData) { - BL_PROFILE("PeleLM::calcDivU()"); + BL_PROFILE("PeleLMeX::calcDivU()"); AMREX_ASSERT(a_time == AmrOldTime || a_time == AmrNewTime); @@ -181,7 +181,7 @@ void PeleLM::setRhoToSumRhoY(int lev, const TimeStamp &a_time) { } void PeleLM::setTemperature(const TimeStamp &a_time) { - BL_PROFILE_VAR("PeleLM::setTemperature()", setTemperature); + BL_PROFILE_VAR("PeleLMeX::setTemperature()", setTemperature); AMREX_ASSERT(a_time == AmrOldTime || a_time == AmrNewTime); @@ -213,7 +213,7 @@ void PeleLM::setTemperature(int lev, const TimeStamp &a_time) { void PeleLM::calc_dPdt(const TimeStamp &a_time, const Vector &a_dPdt) { - BL_PROFILE("PeleLM::calc_dPdt()"); + BL_PROFILE("PeleLMeX::calc_dPdt()"); AMREX_ASSERT(a_time == AmrOldTime || a_time == AmrNewTime); @@ -253,7 +253,7 @@ void PeleLM::calc_dPdt(int lev, Real PeleLM::adjustPandDivU(std::unique_ptr &advData) { - BL_PROFILE("PeleLM::adjustPandDivU()"); + BL_PROFILE("PeleLMeX::adjustPandDivU()"); Vector> ThetaHalft(finest_level+1); diff --git a/Source/PeleLMEvaluate.cpp b/Source/PeleLMEvaluate.cpp index a24b90e9..9a728a71 100644 --- a/Source/PeleLMEvaluate.cpp +++ b/Source/PeleLMEvaluate.cpp @@ -4,7 +4,7 @@ using namespace amrex; void PeleLM::Evaluate() { - BL_PROFILE("PeleLM::Evaluate()"); + BL_PROFILE("PeleLMeX::Evaluate()"); //---------------------------------------------------------------- // Check that requested evaluate entries exist and determine the size diff --git a/Source/PeleLMEvolve.cpp b/Source/PeleLMEvolve.cpp index 5da81fc3..0b5de7fe 100644 --- a/Source/PeleLMEvolve.cpp +++ b/Source/PeleLMEvolve.cpp @@ -3,7 +3,7 @@ using namespace amrex; void PeleLM::Evolve() { - BL_PROFILE("PeleLM::Evolve()"); + BL_PROFILE("PeleLMeX::Evolve()"); bool do_not_evolve = ( (m_max_step == 0) || ((m_stop_time >= 0.) && (m_cur_time > m_stop_time)) ); diff --git a/Source/PeleLMForces.cpp b/Source/PeleLMForces.cpp index af4cad8d..ecd456af 100644 --- a/Source/PeleLMForces.cpp +++ b/Source/PeleLMForces.cpp @@ -12,7 +12,7 @@ void PeleLM::getVelForces(const TimeStamp &a_time, int nGrowForce, int add_gradP) { - BL_PROFILE("PeleLM::getVelForces()"); + BL_PROFILE("PeleLMeX::getVelForces()"); int has_divTau = (!a_divTau.empty()); for (int lev = 0; lev <= finest_level; ++lev) { diff --git a/Source/PeleLMInit.cpp b/Source/PeleLMInit.cpp index ca7884ea..dc4d9194 100644 --- a/Source/PeleLMInit.cpp +++ b/Source/PeleLMInit.cpp @@ -7,7 +7,7 @@ using namespace amrex; void PeleLM::Init() { - BL_PROFILE_VAR("PeleLM::Init()", Init); + BL_PROFILE("PeleLMeX::Init()"); // Open temporals file openTempFile(); @@ -23,7 +23,7 @@ void PeleLM::MakeNewLevelFromScratch( int lev, amrex::Real time, const amrex::BoxArray& ba, const amrex::DistributionMapping& dm) { - BL_PROFILE_VAR("PeleLM::MakeNewLevelFromScratch()", MakeNewLevelFromScratch); + BL_PROFILE("PeleLMeX::MakeNewLevelFromScratch()"); if (m_verbose > 0) { amrex::Print() << " Making new level " << lev << " from scratch" << std::endl; @@ -147,7 +147,7 @@ void PeleLM::MakeNewLevelFromScratch( int lev, } void PeleLM::initData() { - BL_PROFILE_VAR("PeleLM::initData()", initData); + BL_PROFILE("PeleLMeX::initData()"); if (m_restart_chkfile.empty()) { //---------------------------------------------------------------- @@ -289,7 +289,7 @@ void PeleLM::initData() { } void PeleLM::initLevelData(int lev) { - BL_PROFILE_VAR("PeleLM::initLevelData()", initLevelData); + BL_PROFILE("PeleLMeX::initLevelData()"); // Get level data auto ldata_p = getLevelDataPtr(lev,AmrNewTime); @@ -434,7 +434,7 @@ void PeleLM::projectInitSolution() } void PeleLM::initialIterations() { - BL_PROFILE_VAR("PeleLM::initialIterations()", initialIterations); + BL_PROFILE("PeleLMeX::initialIterations()"); if (m_verbose > 0 && m_init_iter > 0) { amrex::Print() << "\n Doing initial pressure iteration(s) \n"; diff --git a/Source/PeleLMPlot.cpp b/Source/PeleLMPlot.cpp index 5d1e3c8a..ecfe0df9 100644 --- a/Source/PeleLMPlot.cpp +++ b/Source/PeleLMPlot.cpp @@ -48,7 +48,7 @@ void PeleLM::WriteDebugPlotFile(const Vector &a_MF, } void PeleLM::WritePlotFile() { - BL_PROFILE("PeleLM::WritePlotFile()"); + BL_PROFILE("PeleLMeX::WritePlotFile()"); const std::string& plotfilename = amrex::Concatenate(m_plot_file, m_nstep, m_ioDigits); @@ -443,7 +443,7 @@ void PeleLM::WriteHeader(const std::string& name, bool is_checkpoint) const void PeleLM::WriteCheckPointFile() { - BL_PROFILE("PeleLM::WriteCheckPointFile()"); + BL_PROFILE("PeleLMeX::WriteCheckPointFile()"); const std::string& checkpointname = amrex::Concatenate(m_check_file, m_nstep, m_ioDigits); @@ -492,7 +492,7 @@ void PeleLM::WriteCheckPointFile() void PeleLM::ReadCheckPointFile() { - BL_PROFILE("PeleLM::ReadCheckPointFile()"); + BL_PROFILE("PeleLMeX::ReadCheckPointFile()"); amrex::Print() << "Restarting from checkpoint " << m_restart_chkfile << "\n"; @@ -501,7 +501,7 @@ void PeleLM::ReadCheckPointFile() /*************************************************************************** ** Load header: set up problem domain (including BoxArray) * - ** allocate PeleLM memory (PeleLM::AllocateArrays) * + ** allocate PeleLMeX memory (PeleLM::AllocateArrays) * ** (by calling MakeNewLevelFromScratch) * ****************************************************************************/ diff --git a/Source/PeleLMProjection.cpp b/Source/PeleLMProjection.cpp index 58c3db8d..4f97330f 100644 --- a/Source/PeleLMProjection.cpp +++ b/Source/PeleLMProjection.cpp @@ -4,7 +4,7 @@ using namespace amrex; void PeleLM::initialProjection() { - BL_PROFILE_VAR("PeleLM::initialProjection()", initialProjection); + BL_PROFILE("PeleLMeX::initialProjection()"); if (m_verbose) { Vector velMax(AMREX_SPACEDIM); @@ -111,7 +111,7 @@ void PeleLM::initialProjection() void PeleLM::initialPressProjection() { - BL_PROFILE_VAR("PeleLM::initialPressProjection()", initialProjection); + BL_PROFILE("PeleLMeX::initialPressProjection()"); if (m_verbose) { amrex::Print() << " Initial pressure projection \n"; @@ -167,7 +167,7 @@ void PeleLM::velocityProjection(int is_initIter, const TimeStamp &a_rhoTime, const Real &a_dt) { - BL_PROFILE_VAR("PeleLM::velocityProjection()", velocityProjection); + BL_PROFILE("PeleLMeX::velocityProjection()"); int nGhost = 0; int incremental = (is_initIter) ? 1 : 0; diff --git a/Source/PeleLMReactions.cpp b/Source/PeleLMReactions.cpp index 33439954..2c16d99c 100644 --- a/Source/PeleLMReactions.cpp +++ b/Source/PeleLMReactions.cpp @@ -8,7 +8,7 @@ using namespace amrex; void PeleLM::advanceChemistry(std::unique_ptr &advData) { - BL_PROFILE("PeleLM::advanceChemistry()"); + BL_PROFILE("PeleLMeX::advanceChemistry()"); for (int lev = finest_level; lev >= 0; --lev) { if (lev != finest_level) { @@ -30,7 +30,7 @@ void PeleLM::advanceChemistry(int lev, const Real &a_dt, MultiFab &a_extForcing) { - BL_PROFILE("PeleLM::advanceChemistry_Lev"+std::to_string(lev)+"()"); + BL_PROFILE("PeleLMeX::advanceChemistry_Lev"+std::to_string(lev)+"()"); auto ldataOld_p = getLevelDataPtr(lev,AmrOldTime); auto ldataNew_p = getLevelDataPtr(lev,AmrNewTime); @@ -175,7 +175,7 @@ void PeleLM::advanceChemistryBAChem(int lev, const Real &a_dt, MultiFab &a_extForcing) { - BL_PROFILE("PeleLM::advanceChemistry_Lev"+std::to_string(lev)+"()"); + BL_PROFILE("PeleLMeX::advanceChemistry_Lev"+std::to_string(lev)+"()"); auto ldataOld_p = getLevelDataPtr(lev,AmrOldTime); auto ldataNew_p = getLevelDataPtr(lev,AmrNewTime); @@ -374,7 +374,7 @@ void PeleLM::computeInstantaneousReactionRate(int lev, const TimeStamp &a_time, MultiFab* a_I_R) { - BL_PROFILE("PeleLM::computeInstantaneousReactionRate()"); + BL_PROFILE("PeleLMeX::computeInstantaneousReactionRate()"); auto ldata_p = getLevelDataPtr(lev,a_time); #ifdef AMREX_USE_EB diff --git a/Source/PeleLMRegrid.cpp b/Source/PeleLMRegrid.cpp index 7cf8bb35..d6ea054b 100644 --- a/Source/PeleLMRegrid.cpp +++ b/Source/PeleLMRegrid.cpp @@ -6,7 +6,7 @@ void PeleLM::regrid(int lbase, amrex::Real time, bool initial) { - BL_PROFILE("PeleLM::regrid()"); + BL_PROFILE("PeleLMeX::regrid()"); if (!m_doLoadBalance && lbase >= max_level) return; @@ -260,7 +260,7 @@ void PeleLM::MakeNewLevelFromCoarse( int lev, amrex::Real time, const amrex::BoxArray& ba, const amrex::DistributionMapping& dm) { - BL_PROFILE_VAR("PeleLM::MakeNewLevelFromCoarse()", MakeNewLevelFromCoarse); + BL_PROFILE("PeleLMeX::MakeNewLevelFromCoarse()"); if (m_verbose > 0) { Print() << " Making new level " << lev << " from coarse\n"; @@ -362,7 +362,7 @@ void PeleLM::RemakeLevel( int lev, amrex::Real time, const amrex::BoxArray& ba, const amrex::DistributionMapping& dm) { - BL_PROFILE_VAR("PeleLM::RemakeLevel()", RemakeLevel); + BL_PROFILE("PeleLMeX::RemakeLevel()"); if (m_verbose > 0) { Print() << " Remaking level " << lev << "\n"; @@ -461,7 +461,7 @@ void PeleLM::RemakeLevel( int lev, } void PeleLM::ClearLevel(int lev) { - BL_PROFILE_VAR("PeleLM::ClearLevel()", ClearLevel); + BL_PROFILE("PeleLMeX::ClearLevel()"); m_leveldata_old[lev].reset(); m_leveldata_new[lev].reset(); diff --git a/Source/PeleLMSetup.cpp b/Source/PeleLMSetup.cpp index 4661c8e4..b8a71ee0 100644 --- a/Source/PeleLMSetup.cpp +++ b/Source/PeleLMSetup.cpp @@ -23,7 +23,7 @@ static Box grow_box_by_one (const Box& b) { return amrex::grow(b,1); } static Box grow_box_by_two (const Box& b) { return amrex::grow(b,2); } void PeleLM::Setup() { - BL_PROFILE("PeleLM::Setup()"); + BL_PROFILE("PeleLMeX::Setup()"); m_wall_start = amrex::ParallelDescriptor::second(); @@ -48,7 +48,7 @@ void PeleLM::Setup() { soot_model = new SootModel{}; #endif - // Read PeleLM parameters + // Read PeleLMeX parameters readParameters(); #ifdef AMREX_USE_EB @@ -149,7 +149,7 @@ void PeleLM::Setup() { } void PeleLM::readParameters() { - BL_PROFILE("PeleLM::readParameters()"); + BL_PROFILE("PeleLMeX::readParameters()"); readIOParameters(); @@ -559,7 +559,7 @@ void PeleLM::readParameters() { } void PeleLM::readIOParameters() { - BL_PROFILE_VAR("PeleLM::readIOParameters()", readIOParameters); + BL_PROFILE_VAR("PeleLMeX::readIOParameters()", readIOParameters); ParmParse pp("amr"); @@ -602,7 +602,7 @@ void PeleLM::readIOParameters() { } void PeleLM::variablesSetup() { - BL_PROFILE("PeleLM::variablesSetup()"); + BL_PROFILE("PeleLMeX::variablesSetup()"); //---------------------------------------------------------------- // Variables ordering is defined through compiler macro in PeleLM_Index.H @@ -788,7 +788,7 @@ void PeleLM::readGridFile(std::string grid_file, void PeleLM::derivedSetup() { - BL_PROFILE("PeleLM::derivedSetup()"); + BL_PROFILE("PeleLMeX::derivedSetup()"); if (!m_incompressible) { @@ -934,7 +934,7 @@ void PeleLM::derivedSetup() void PeleLM::evaluateSetup() { - BL_PROFILE("PeleLM::evaluateSetup()"); + BL_PROFILE("PeleLMeX::evaluateSetup()"); // Get species names Vector spec_names; @@ -1025,7 +1025,7 @@ void PeleLM::evaluateSetup() void PeleLM::taggingSetup() { - BL_PROFILE("PeleLM::taggingSetup()"); + BL_PROFILE("PeleLMeX::taggingSetup()"); std::string amr_prefix = "amr"; ParmParse ppamr(amr_prefix); diff --git a/Source/PeleLMTagging.cpp b/Source/PeleLMTagging.cpp index 9cd3f6e3..0fb54f5c 100644 --- a/Source/PeleLMTagging.cpp +++ b/Source/PeleLMTagging.cpp @@ -12,7 +12,7 @@ PeleLM::ErrorEst( int lev, Real time, int /*ng*/) { - BL_PROFILE_VAR("PeleLM::ErrorEst()", ErrorEst); + BL_PROFILE("PeleLMeX::ErrorEst()"); #ifdef AMREX_USE_EB // Tag EB up to m_EB_refine_LevMax-1 if Static or diff --git a/Source/PeleLMTimestep.cpp b/Source/PeleLMTimestep.cpp index 3fed3b23..dd1f2801 100644 --- a/Source/PeleLMTimestep.cpp +++ b/Source/PeleLMTimestep.cpp @@ -7,7 +7,7 @@ Real PeleLM::computeDt(int is_init, const TimeStamp &a_time) { - BL_PROFILE("PeleLM::computeDt()"); + BL_PROFILE("PeleLMeX::computeDt()"); Real estdt = 1.0e200; @@ -220,7 +220,7 @@ void PeleLM::checkDt(const TimeStamp &a_time, const Real &a_dt) { - BL_PROFILE("PeleLM::checkDt()"); + BL_PROFILE("PeleLMeX::checkDt()"); if (m_fixed_dt > 0.0 || !m_divu_checkFlag) { return; diff --git a/Source/PeleLMTransportProp.cpp b/Source/PeleLMTransportProp.cpp index 1949f6f8..4851376d 100644 --- a/Source/PeleLMTransportProp.cpp +++ b/Source/PeleLMTransportProp.cpp @@ -9,7 +9,7 @@ using namespace amrex; void PeleLM::calcTurbViscosity(const TimeStamp &a_time) { - BL_PROFILE("PeleLM::calcTurbViscosity()"); + BL_PROFILE("PeleLMeX::calcTurbViscosity()"); // We shouldn't be here unless we're doing LES AMREX_ALWAYS_ASSERT(m_do_les); @@ -218,7 +218,7 @@ void PeleLM::calcTurbViscosity(const TimeStamp &a_time) { } void PeleLM::calcViscosity(const TimeStamp &a_time) { - BL_PROFILE("PeleLM::calcViscosity()"); + BL_PROFILE("PeleLMeX::calcViscosity()"); for (int lev = 0; lev <= finest_level; ++lev) { @@ -250,7 +250,7 @@ void PeleLM::calcViscosity(const TimeStamp &a_time) { } void PeleLM::calcDiffusivity(const TimeStamp &a_time) { - BL_PROFILE("PeleLM::calcDiffusivity()"); + BL_PROFILE("PeleLMeX::calcDiffusivity()"); for (int lev = 0; lev <= finest_level; ++lev) { @@ -325,7 +325,7 @@ PeleLM::getDiffusivity(int lev, int beta_comp, int ncomp, int doZeroVisc, MultiFab const& beta_cc, int addTurbContrib) { - BL_PROFILE("PeleLM::getDiffusivity()"); + BL_PROFILE("PeleLMeX::getDiffusivity()"); AMREX_ASSERT(bcrec.size() >= ncomp); AMREX_ASSERT(beta_cc.nComp() >= beta_comp+ncomp); diff --git a/Source/PeleLMUMac.cpp b/Source/PeleLMUMac.cpp index 8cf9b9d2..2d551aea 100644 --- a/Source/PeleLMUMac.cpp +++ b/Source/PeleLMUMac.cpp @@ -8,7 +8,7 @@ using namespace amrex; void PeleLM::predictVelocity(std::unique_ptr &advData) { - BL_PROFILE("PeleLM::predictVelocity()"); + BL_PROFILE("PeleLMeX::predictVelocity()"); // set umac boundaries to zero if ( advData->umac[0][0].nGrow() > 0 ) { @@ -69,7 +69,7 @@ void PeleLM::predictVelocity(std::unique_ptr &advData) void PeleLM::createMACRHS(std::unique_ptr &advData) { - BL_PROFILE("PeleLM::createMACRHS()"); + BL_PROFILE("PeleLMeX::createMACRHS()"); for (int lev = 0; lev <= finest_level; ++lev) { Real halftime = 0.5 * (m_t_old[lev] + m_t_new[lev]); @@ -81,7 +81,7 @@ void PeleLM::addChiIncrement(int a_sdcIter, const TimeStamp &a_time, std::unique_ptr &advData) { - BL_PROFILE("PeleLM::addChiIncrement()"); + BL_PROFILE("PeleLMeX::addChiIncrement()"); int nGrow = m_nGrowAdv; Vector chiIncr(finest_level+1); @@ -125,7 +125,7 @@ void PeleLM::macProject(const TimeStamp &a_time, std::unique_ptr &advData, const Vector &a_divu) { - BL_PROFILE("PeleLM::macProject()"); + BL_PROFILE("PeleLMeX::macProject()"); int has_divu = (!a_divu.empty()); diff --git a/Source/PeleLMUtils.cpp b/Source/PeleLMUtils.cpp index c1896591..cb2b78c9 100644 --- a/Source/PeleLMUtils.cpp +++ b/Source/PeleLMUtils.cpp @@ -15,7 +15,7 @@ void PeleLM::fluxDivergence(const Vector &a_divergence, int intensiveFluxes, Real scale) { - BL_PROFILE("PeleLM::fluxDivergence()"); + BL_PROFILE("PeleLMeX::fluxDivergence()"); if (intensiveFluxes) { // Fluxes are intensive -> need area scaling in div for (int lev = 0; lev <= finest_level; ++lev) { intFluxDivergenceLevel(lev,*a_divergence[lev], div_comp, a_fluxes[lev], flux_comp, @@ -39,7 +39,7 @@ void PeleLM::fluxDivergence(const Vector &a_divergence, int intensiveFluxes, Real scale) { - BL_PROFILE("PeleLM::fluxDivergence()"); + BL_PROFILE("PeleLMeX::fluxDivergence()"); if (intensiveFluxes) { // Fluxes are intensive -> need area scaling in div for (int lev = 0; lev <= finest_level; ++lev) { intFluxDivergenceLevelEB(lev,*a_divergence[lev], div_comp, a_fluxes[lev], flux_comp, @@ -67,7 +67,7 @@ void PeleLM::fluxDivergenceRD(const Vector &a_state, const Real &scale, const Real &a_dt) { - BL_PROFILE("PeleLM::fluxDivergenceRD()"); + BL_PROFILE("PeleLMeX::fluxDivergenceRD()"); #ifdef AMREX_USE_EB int have_ebfluxes = (a_EBfluxes.empty()) ? 0 : 1; for (int lev = 0; lev <= finest_level; ++lev) { @@ -437,7 +437,7 @@ void PeleLM::advFluxDivergence(int a_lev, amrex::Real scale, bool fluxes_are_area_weighted) { - BL_PROFILE("PeleLM::advFluxDivergence()"); + BL_PROFILE("PeleLMeX::advFluxDivergence()"); AMREX_ASSERT(a_divergence.nComp() >= div_comp+ncomp); AMREX_ASSERT(a_fluxes[0]->nComp() >= flux_comp+ncomp); @@ -561,7 +561,7 @@ void PeleLM::advFluxDivergence(int a_lev, void PeleLM::floorSpecies(const TimeStamp &a_time) { - BL_PROFILE("PeleLM::floorSpecies()"); + BL_PROFILE("PeleLMeX::floorSpecies()"); AMREX_ASSERT(a_time == AmrOldTime || a_time == AmrNewTime); if (!m_floor_species) { return; @@ -602,7 +602,7 @@ PeleLM::floorSpecies(const TimeStamp &a_time) void PeleLM::resetCoveredMask() { - BL_PROFILE("PeleLM::resetCoveredMask()"); + BL_PROFILE("PeleLMeX::resetCoveredMask()"); if (m_resetCoveredMask) { if (m_verbose) Print() << " Resetting fine-covered cells mask \n"; @@ -785,7 +785,7 @@ PeleLM::derive(const std::string &a_name, int lev, int nGrow) { - BL_PROFILE("PeleLM::derive()"); + BL_PROFILE("PeleLMeX::derive()"); AMREX_ASSERT(nGrow >= 0); std::unique_ptr mf; @@ -841,7 +841,7 @@ PeleLM::deriveComp(const std::string &a_name, int lev, int nGrow) { - BL_PROFILE("PeleLM::derive()"); + BL_PROFILE("PeleLMeX::derive()"); AMREX_ASSERT(nGrow >= 0); std::unique_ptr mf; @@ -1002,7 +1002,7 @@ PeleLM::parseVars(const Vector &a_varsNames, Real PeleLM::MLNorm0(const Vector &a_MF) { - BL_PROFILE("PeleLM::MLNorm0()"); + BL_PROFILE("PeleLMeX::MLNorm0()"); Real r = 0.0; for (int lev = 0; lev < a_MF.size(); ++lev) { if (lev != finest_level) { @@ -1019,7 +1019,7 @@ Vector PeleLM::MLNorm0(const Vector &a_MF, int startcomp, int ncomp) { - BL_PROFILE("PeleLM::MLNorm0()"); + BL_PROFILE("PeleLMeX::MLNorm0()"); AMREX_ASSERT(a_MF[0]->nComp() >= startcomp+ncomp); Vector r(ncomp); for (int n = 0; n < ncomp; n++) { @@ -1125,7 +1125,7 @@ PeleLM::fetchDiffTypeArray(int scomp, int ncomp) Real PeleLM::MFSum (const Vector &a_mf, int comp) { - BL_PROFILE("PeleLM::MFSum()"); + BL_PROFILE("PeleLMeX::MFSum()"); // Get the integral of the MF, not including the fine-covered and // EB-covered cells @@ -1307,7 +1307,7 @@ PeleLM::MFmax(const MultiFab *a_MF, const iMultiFab &a_mask, int comp) { - BL_PROFILE("PeleLM::MFmax()"); + BL_PROFILE("PeleLMeX::MFmax()"); Real mx = std::numeric_limits::lowest(); #ifdef AMREX_USE_EB @@ -1397,7 +1397,7 @@ PeleLM::MFmin(const MultiFab *a_MF, const iMultiFab &a_mask, int comp) { - BL_PROFILE("PeleLM::MFmin()"); + BL_PROFILE("PeleLMeX::MFmin()"); Real mn = std::numeric_limits::max(); #ifdef AMREX_USE_EB @@ -1486,7 +1486,7 @@ Vector PeleLM::MLmax(const Vector &a_MF, int scomp, int ncomp) { - BL_PROFILE("PeleLM::MLmax()"); + BL_PROFILE("PeleLMeX::MLmax()"); AMREX_ASSERT(a_MF[0]->nComp() >= scomp+ncomp); Vector nmax(ncomp,AMREX_REAL_LOWEST); @@ -1512,7 +1512,7 @@ Vector PeleLM::MLmin(const Vector &a_MF, int scomp, int ncomp) { - BL_PROFILE("PeleLM::MLmin()"); + BL_PROFILE("PeleLMeX::MLmin()"); AMREX_ASSERT(a_MF[0]->nComp() >= scomp+ncomp); Vector nmin(ncomp,AMREX_REAL_MAX); @@ -1717,7 +1717,7 @@ PeleLM::parseComposition(Vector compositionIn, void PeleLM::extendSignedDistance(MultiFab *a_signDist, Real a_extendFactor) { - BL_PROFILE("PeleLM::extendSignedDistance()"); + BL_PROFILE("PeleLMeX::extendSignedDistance()"); // This is a not-so-pretty piece of code that'll take AMReX cell-averaged // signed distance and propagates it manually up to the point where we need to have it // for derefining. diff --git a/Source/Spray/PeleLMSprayParticles.cpp b/Source/Spray/PeleLMSprayParticles.cpp index 34b9cee3..6077fb5a 100644 --- a/Source/Spray/PeleLMSprayParticles.cpp +++ b/Source/Spray/PeleLMSprayParticles.cpp @@ -38,7 +38,7 @@ PeleLM::SprayEstDt() if (!do_spray_particles || SprayPC == nullptr) { return estdt; } - BL_PROFILE("PeleLM::SprayEstDt()"); + BL_PROFILE("PeleLMeX::SprayEstDt()"); for (int lev = 0; lev <= finest_level; ++lev) { Real estdt_lev = SprayPC->estTimestep(lev); if (estdt_lev > 0. && estdt_lev < estdt) { @@ -95,7 +95,7 @@ PeleLM::SpraySetup() void PeleLM::setupVirtualParticles(const int level) { - BL_PROFILE("PeleLM::setupVirtualParticles()"); + BL_PROFILE("PeleLMeX::setupVirtualParticles()"); if (SprayPC != nullptr) { if (level < finest_level) { SprayParticleContainer::AoS virts; @@ -120,7 +120,7 @@ PeleLM::removeVirtualParticles(const int level) void PeleLM::setupGhostParticles(const int ngrow, const int level) { - BL_PROFILE("PeleLM::setupGhostParticles()"); + BL_PROFILE("PeleLMeX::setupGhostParticles()"); AMREX_ASSERT(level < finest_level); if (SprayPC != nullptr) { SprayParticleContainer::AoS ghosts; @@ -149,7 +149,7 @@ PeleLM::SprayCreateData() void PeleLM::SprayInit() { - BL_PROFILE("PeleLM::SprayInit()"); + BL_PROFILE("PeleLMeX::SprayInit()"); if (!do_spray_particles) { return; @@ -272,7 +272,7 @@ PeleLM::SprayMKD(const Real time, const Real dt) if (spray_verbose) { Print() << "moveKickDrift ... updating particle positions and velocity\n"; } - BL_PROFILE("PeleLM::SprayMKD()"); + BL_PROFILE("PeleLMeX::SprayMKD()"); // Setup the virtual particles that represent particles on finer levels setupVirtualParticles(0); for (int lev = 0; lev <= finest_level; ++lev) { @@ -367,7 +367,7 @@ PeleLM::SprayPostRegrid() void PeleLM::SprayInjectRedist() { - BL_PROFILE("PeleLM::SprayInjectRedist"); + BL_PROFILE("PeleLMeX::SprayInjectRedist"); Long prev_count = 0; if (spray_verbose >= 3) { prev_count = SprayPC->TotalNumberOfParticles(true, false); diff --git a/Source/main.cpp b/Source/main.cpp index e62d8a97..ab050af0 100644 --- a/Source/main.cpp +++ b/Source/main.cpp @@ -23,31 +23,31 @@ int main(int argc, char* argv[]) { } // timer for profiling - BL_PROFILE_VAR("PeleLM::main()", main); + BL_PROFILE_VAR("PeleLMeX::main()", main); // wallclock time const Real strt_total = ParallelDescriptor::second(); { - // declare an PeleLM object to manage multilevel data - PeleLM pelelm; + // declare an PeleLMeX object to manage multilevel data + PeleLM pelelmex; // Description Setup - pelelm.Setup(); + pelelmex.Setup(); // Description Init - pelelm.Init(); + pelelmex.Init(); // Switch between Evolve and UnitTest mode - if ( pelelm.runMode() == "normal" ) { + if ( pelelmex.runMode() == "normal" ) { // Advance solution to final time - pelelm.Evolve(); + pelelmex.Evolve(); - } else if ( pelelm.runMode() == "evaluate" ) { + } else if ( pelelmex.runMode() == "evaluate" ) { // - pelelm.Evaluate(); + pelelmex.Evaluate(); } else { Abort(" Wrong peleLM.run_mode ! It can only be 'normal' (D) or 'evaluate'"); diff --git a/Submodules/AMReX-Hydro b/Submodules/AMReX-Hydro index f24f4a6f..bc4e5f64 160000 --- a/Submodules/AMReX-Hydro +++ b/Submodules/AMReX-Hydro @@ -1 +1 @@ -Subproject commit f24f4a6f00b93d79e9772100e620d219be7787cb +Subproject commit bc4e5f64c9ac3072793ebf426b422be91231652c diff --git a/Submodules/amrex b/Submodules/amrex index 9fc63542..22908d8a 160000 --- a/Submodules/amrex +++ b/Submodules/amrex @@ -1 +1 @@ -Subproject commit 9fc635426caeef7d2c0b6bf7ae7700bd5e6600ec +Subproject commit 22908d8ad2141ed096401e9be9f067e8e589ec31 diff --git a/Utils/RunScripts/Scaling/WeakScaling/ExtractScalingData.py b/Utils/RunScripts/Scaling/WeakScaling/ExtractScalingData.py index e5b26776..9c841467 100755 --- a/Utils/RunScripts/Scaling/WeakScaling/ExtractScalingData.py +++ b/Utils/RunScripts/Scaling/WeakScaling/ExtractScalingData.py @@ -74,27 +74,27 @@ def parse_args(arg_string=None): currRunTime = procRunTime.communicate()[0].decode("utf-8").strip() runTimes.append(currRunTime) # Get component times: reaction, diffusion, MacProj, NodalProj, ScalAdv, VelAdv, Sync - cmd = "cat {}".format(logfile)+" | grep 'PeleLM::advance::reactions' | awk 'NR%2==0' | awk 'NR%2!=0' | awk '{print $4}'" + cmd = "cat {}".format(logfile)+" | grep 'PeleLMeX::advance::reactions' | awk 'NR%2==0' | awk 'NR%2!=0' | awk '{print $4}'" procRunTime = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE) currReactTime = procRunTime.communicate()[0].decode("utf-8").strip() reactionTimes.append(currReactTime) - cmd = "cat {}".format(logfile)+" | grep 'PeleLM::advance::diffusion' | awk 'NR%2==0' | awk 'NR%2!=0' | awk '{print $4}'" + cmd = "cat {}".format(logfile)+" | grep 'PeleLMeX::advance::diffusion' | awk 'NR%2==0' | awk 'NR%2!=0' | awk '{print $4}'" procRunTime = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE) currDiffTime = procRunTime.communicate()[0].decode("utf-8").strip() diffusionTimes.append(currDiffTime) - cmd = "cat {}".format(logfile)+" | grep 'PeleLM::advance::mac' | awk 'NR%2==0' | awk 'NR%2!=0' | awk '{print $4}'" + cmd = "cat {}".format(logfile)+" | grep 'PeleLMeX::advance::mac' | awk 'NR%2==0' | awk 'NR%2!=0' | awk '{print $4}'" procRunTime = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE) currMACTime = procRunTime.communicate()[0].decode("utf-8").strip() MacProjTimes.append(currMACTime) - cmd = "cat {}".format(logfile)+" | grep 'PeleLM::velocityProjection()' | awk 'NR%2==0' | awk 'NR%2!=0' | awk '{print $4}'" + cmd = "cat {}".format(logfile)+" | grep 'PeleLMeX::velocityProjection()' | awk 'NR%2==0' | awk 'NR%2!=0' | awk '{print $4}'" procRunTime = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE) currProjTime = procRunTime.communicate()[0].decode("utf-8").strip() NodalProjTimes.append(currProjTime) - cmd = "cat {}".format(logfile)+" | grep 'PeleLM::advance::scalars_adv' | awk 'NR%2==0' | awk 'NR%2!=0' | awk '{print $4}'" + cmd = "cat {}".format(logfile)+" | grep 'PeleLMeX::advance::scalars_adv' | awk 'NR%2==0' | awk 'NR%2!=0' | awk '{print $4}'" procRunTime = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE) currScalAdvTime = procRunTime.communicate()[0].decode("utf-8").strip() ScalAdvTimes.append(currScalAdvTime) - cmd = "cat {}".format(logfile)+" | grep 'PeleLM::advance::velocity' | awk 'NR%2==0' | awk 'NR%2!=0' | awk '{print $4}'" + cmd = "cat {}".format(logfile)+" | grep 'PeleLMeX::advance::velocity' | awk 'NR%2==0' | awk 'NR%2!=0' | awk '{print $4}'" procRunTime = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE) currVelAdvTime = procRunTime.communicate()[0].decode("utf-8").strip() VelAdvTimes.append(currVelAdvTime)