Improve timer output for the fluid applications

CHANGELOG.md

@@ -3,6 +3,12 @@
 All notable changes to the Lethe project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/).
 
+## [Master] - 2024-07-08
+
+### Changed
+
+- MINOR  The timer output for the lethe-fluid and lethe-fluid-matrix-free applications contains now more detailed elements. Including post-processing capabilities that were not timed, the time spent in reading mesh and manifolds, and setting the initial conditions. [#1187](https://github.com/chaos-polymtl/lethe/pull/1187) 
+
 ## [Master] - 2024-07-05
 
 ### Changed

doc/source/parameters/cfd/timer.rst

@@ -18,29 +18,33 @@ The following table shows an example of the output of the timer:
 
 .. code-block:: text
 
-   +---------------------------------------------+------------+------------+
-   | Total wallclock time elapsed since start    |      11.5s |            |
-   |                                             |            |            |
-   | Section                         | no. calls |  wall time | % of total |
-   +---------------------------------+-----------+------------+------------+
-   | assemble_rhs                    |         4 |      2.28s |        20% |
-   | assemble_system                 |         4 |      7.78s |        68% |
-   | output                          |         2 |      1.21s |        11% |
-   | setup_ILU                       |         4 |     0.164s |       1.4% |
-   | solve_linear_system             |         4 |    0.0812s |      0.71% |
-   +---------------------------------+-----------+------------+------------+
+    +---------------------------------------------------------+------------+------------+
+    | Total wallclock time elapsed since start                |      1.11s |            |
+    |                                                         |            |            |
+    | Section                                     | no. calls |  wall time | % of total |
+    +---------------------------------------------+-----------+------------+------------+
+    | Assemble RHS                                |         9 |     0.275s |        25% |
+    | Assemble matrix                             |         7 |     0.293s |        26% |
+    | Calculate and output norms after Newton its |         8 |    0.0135s |       1.2% |
+    | Output VTU                                  |         2 |    0.0491s |       4.4% |
+    | Read mesh and manifolds                     |         1 |   0.00733s |      0.66% |
+    | Setup DOFs                                  |         1 |    0.0144s |       1.3% |
+    | Setup ILU                                   |         7 |    0.0801s |       7.2% |
+    | Solve linear system                         |         7 |     0.369s |        33% |
+    +---------------------------------------------+-----------+------------+------------+
 
-For every block of functions, Lethe reports the number of calls, the wall time spent and the percentage of time spent in the function. It is important to monitor this when running a simulation.
+For every block of functions, Lethe reports the number of calls, the wall time spent and the percentage of time spent in the function. It is important to monitor this when running a simulation. The most important ones from a user perspective in most of the simulations are:
 
-* ``assemble_rhs`` refers to the assembly of the right-hand side of the equation, that is the residual of the equation.
+* ``Assemble RHS`` refers to the assembly of the right-hand side of the equation, that is the residual of the equation.
 
-* ``assemble_system`` refers to the time spent assembling the matrix associated with the equation.
+* ``Assemble matrix`` refers to the time spent assembling the matrix associated with the equation.
 
-* ``output`` refers to the output of the ``.vtu``, ``.pvtu`` and ``.pvd`` files.
+* ``Output VTU`` refers to the output of the ``.vtu``, ``.pvtu`` and ``.pvd`` files.
 
-* ``setup_ILU`` or ``setup_AMG`` describe the time spent to set-up the preconditioner.
+* ``Read mesh and manifolds`` refers to the time it takes to create a ``dealii`` triangulation or read a ``gmsh`` one, and attach corresponding manifolds.
 
-* ``solve_linear_system`` refers to the time spent solving the linear system of equations, without the time required to assemble the preconditioner.
+* ``Setup ILU`` describe the time spent to set-up the preconditioner. Other options are ``Setup AMG`` and ``Setup GMG``.
 
+* ``Solve linear system`` refers to the time spent solving the linear system of equations, without the time required to assemble the preconditioner.
 
 Depending on the type of simulation, there may be other categories that appear in the timer. This is useful to monitor the functions responsible for taking up most of the simulation time. 

source/dem/read_checkpoint.cc

@@ -15,7 +15,7 @@ read_checkpoint(
   std::shared_ptr<Insertion<dim>>                         &insertion_object,
   std::vector<std::shared_ptr<SerialSolid<dim - 1, dim>>> &solid_surfaces)
 {
-  TimerOutput::Scope timer(computing_timer, "read_checkpoint");
+  TimerOutput::Scope timer(computing_timer, "Read checkpoint");
   std::string        prefix = parameters.restart.filename;
   simulation_control->read(prefix);
   particles_pvdhandler.read(prefix);

source/dem/write_checkpoint.cc

@@ -17,7 +17,7 @@ write_checkpoint(
   const ConditionalOStream                                &pcout,
   MPI_Comm                                                &mpi_communicator)
 {
-  TimerOutput::Scope timer(computing_timer, "write_checkpoint");
+  TimerOutput::Scope timer(computing_timer, "Write checkpoint");
 
   pcout << "Writing restart file" << std::endl;
 

source/fem-dem/cfd_dem_coupling.cc

@@ -207,7 +207,7 @@ template <int dim>
 void
 CFDDEMSolver<dim>::write_checkpoint()
 {
-  TimerOutput::Scope timer(this->computing_timer, "write_checkpoint");
+  TimerOutput::Scope timer(this->computing_timer, "Write checkpoint");
 
   std::string prefix =
     this->simulation_parameters.simulation_control.output_folder +
@@ -290,7 +290,7 @@ template <int dim>
 void
 CFDDEMSolver<dim>::read_checkpoint()
 {
-  TimerOutput::Scope timer(this->computing_timer, "read_checkpoint");
+  TimerOutput::Scope timer(this->computing_timer, "Read checkpoint");
   std::string        prefix =
     this->simulation_parameters.simulation_control.output_folder +
     this->simulation_parameters.restart_parameters.filename;

source/solvers/gls_navier_stokes.cc

@@ -1123,6 +1123,9 @@ GLSNavierStokesSolver<dim>::set_initial_condition_fd(
   Parameters::InitialConditionType initial_condition_type,
   bool                             restart)
 {
+  if (this->simulation_parameters.timer.type == Parameters::Timer::Type::end)
+    TimerOutput::Scope t(this->computing_timer, "Set initial conditions");
+
   if (restart)
     {
       this->pcout << "************************" << std::endl;
@@ -1608,7 +1611,15 @@ GLSNavierStokesSolver<dim>::solve_system_GMRES(const bool   initial_step,
                   << solver_control.last_value() << std::endl;
               }
           }
+
+          this->computing_timer.enter_subsection(
+            "Distribute constraints after linear solve");
+
           constraints_used.distribute(completely_distributed_solution);
+
+          this->computing_timer.leave_subsection(
+            "Distribute constraints after linear solve");
+
           auto &newton_update = this->newton_update;
           newton_update       = completely_distributed_solution;
           success             = true;
@@ -1784,13 +1795,17 @@ template <int dim>
 void
 GLSNavierStokesSolver<dim>::solve()
 {
+  this->computing_timer.enter_subsection("Read mesh and manifolds");
+
   read_mesh_and_manifolds(
     *this->triangulation,
     this->simulation_parameters.mesh,
     this->simulation_parameters.manifolds_parameters,
     this->simulation_parameters.restart_parameters.restart,
     this->simulation_parameters.boundary_conditions);
 
+  this->computing_timer.leave_subsection("Read mesh and manifolds");
+
   this->setup_dofs();
   this->enable_dynamic_zero_constraints_fd();
   this->box_refine_mesh(this->simulation_parameters.restart_parameters.restart);

source/solvers/mf_navier_stokes.cc

@@ -1456,18 +1456,25 @@ template <int dim>
 void
 MFNavierStokesSolver<dim>::solve()
 {
+  this->computing_timer.enter_subsection("Read mesh and manifolds");
+
   read_mesh_and_manifolds(
     *this->triangulation,
     this->simulation_parameters.mesh,
     this->simulation_parameters.manifolds_parameters,
     this->simulation_parameters.restart_parameters.restart,
     this->simulation_parameters.boundary_conditions);
 
+  this->computing_timer.leave_subsection("Read mesh and manifolds");
+
   this->setup_dofs();
   this->set_initial_condition(
     this->simulation_parameters.initial_condition->type,
     this->simulation_parameters.restart_parameters.restart);
-  this->update_multiphysics_time_average_solution();
+
+  // Only needed if other physics apart from fluid dynamics are enabled.
+  if (this->multiphysics->get_active_physics().size() > 1)
+    this->update_multiphysics_time_average_solution();
 
   while (this->simulation_control->integrate())
     {
@@ -1503,19 +1510,27 @@ MFNavierStokesSolver<dim>::solve()
 
       if (is_bdf(this->simulation_control->get_assembly_method()))
         {
+          this->computing_timer.enter_subsection(
+            "Calculate time derivative previous solutions");
+
           calculate_time_derivative_previous_solutions();
           this->time_derivative_previous_solutions.update_ghost_values();
           this->system_operator->evaluate_time_derivative_previous_solutions(
             this->time_derivative_previous_solutions);
 
+          this->computing_timer.leave_subsection(
+            "Calculate time derivative previous solutions");
+
           if (this->simulation_parameters.flow_control.enable_flow_control)
             this->system_operator->update_beta_force(
               this->flow_control.get_beta());
         }
 
       // Provide the fluid dynamics dof_handler, present solution and previous
-      // solution to the multiphysics interface
-      update_solutions_for_multiphysics();
+      // solution to the multiphysics interface only if other physics
+      // apart from fluid dynamics are enabled
+      if (this->multiphysics->get_active_physics().size() > 1)
+        update_solutions_for_multiphysics();
 
       this->iterate();
       this->postprocess(false);
@@ -1663,8 +1678,10 @@ MFNavierStokesSolver<dim>::setup_dofs_fd()
                     this->locally_relevant_dofs,
                     this->mpi_communicator);
 
-  // Provide relevant solution to multiphysics interface
-  update_solutions_for_multiphysics();
+  // Provide relevant solution to multiphysics interface only if other physics
+  // apart from fluid dynamics are enabled.
+  if (this->multiphysics->get_active_physics().size() > 1)
+    update_solutions_for_multiphysics();
 }
 
 template <int dim>
@@ -1702,6 +1719,9 @@ MFNavierStokesSolver<dim>::set_initial_condition_fd(
   Parameters::InitialConditionType initial_condition_type,
   bool                             restart)
 {
+  if (this->simulation_parameters.timer.type == Parameters::Timer::Type::end)
+    TimerOutput::Scope t(this->computing_timer, "Set initial conditions");
+
   if (restart)
     {
       this->pcout << "************************" << std::endl;
@@ -1933,9 +1953,7 @@ void
 MFNavierStokesSolver<dim>::assemble_system_matrix()
 {
   // Required for compilation but not used for matrix free solvers.
-  this->computing_timer.enter_subsection("Assemble matrix");
-
-  this->computing_timer.leave_subsection("Assemble matrix");
+  TimerOutput::Scope t(this->computing_timer, "Assemble matrix");
 }
 
 template <int dim>
@@ -1959,6 +1977,9 @@ template <int dim>
 void
 MFNavierStokesSolver<dim>::update_multiphysics_time_average_solution()
 {
+  TimerOutput::Scope t(this->computing_timer,
+                       "Update multiphysics average solution");
+
   if (this->simulation_parameters.post_processing.calculate_average_velocities)
     {
       TrilinosWrappers::MPI::Vector temp_average_velocities(
@@ -1996,7 +2017,7 @@ template <int dim>
 void
 MFNavierStokesSolver<dim>::setup_GMG()
 {
-  this->computing_timer.enter_subsection("Setup GMG");
+  TimerOutput::Scope t(this->computing_timer, "Setup GMG");
 
   if (!gmg_preconditioner)
     gmg_preconditioner = std::make_shared<MFNavierStokesPreconditionGMG<dim>>(
@@ -2013,15 +2034,13 @@ MFNavierStokesSolver<dim>::setup_GMG()
                                  this->flow_control,
                                  this->present_solution,
                                  this->time_derivative_previous_solutions);
-
-  this->computing_timer.leave_subsection("Setup GMG");
 }
 
 template <int dim>
 void
 MFNavierStokesSolver<dim>::setup_ILU()
 {
-  this->computing_timer.enter_subsection("Setup ILU");
+  TimerOutput::Scope t(this->computing_timer, "Setup ILU");
 
   int current_preconditioner_fill_level =
     this->simulation_parameters.linear_solver.at(PhysicsID::fluid_dynamics)
@@ -2039,8 +2058,6 @@ MFNavierStokesSolver<dim>::setup_ILU()
 
   ilu_preconditioner->initialize(system_operator->get_system_matrix(),
                                  preconditionerOptions);
-
-  this->computing_timer.leave_subsection("Setup ILU");
 }
 
 
@@ -2085,6 +2102,9 @@ template <int dim>
 void
 MFNavierStokesSolver<dim>::update_solutions_for_multiphysics()
 {
+  TimerOutput::Scope t(this->computing_timer,
+                       "Update solutions for multiphysics");
+
   // Provide the fluid dynamics dof_handler to the multiphysics interface
   this->multiphysics->set_dof_handler(PhysicsID::fluid_dynamics,
                                       &this->dof_handler);
@@ -2301,9 +2321,14 @@ void
 MFNavierStokesSolver<dim>::setup_preconditioner()
 {
   this->present_solution.update_ghost_values();
+
+  this->computing_timer.enter_subsection("Evaluate non linear term and tau");
+
   this->system_operator->evaluate_non_linear_term_and_calculate_tau(
     this->present_solution);
 
+  this->computing_timer.leave_subsection("Evaluate non linear term and tau");
+
   if (this->simulation_parameters.linear_solver.at(PhysicsID::fluid_dynamics)
         .preconditioner == Parameters::LinearSolver::PreconditionerType::ilu)
     setup_ILU();
@@ -2444,7 +2469,13 @@ MFNavierStokesSolver<dim>::solve_system_GMRES(const bool   initial_step,
                   << solver_control.last_value() << std::endl;
     }
 
+  this->computing_timer.enter_subsection(
+    "Distribute constraints after linear solve");
+
   constraints_used.distribute(this->newton_update);
+
+  this->computing_timer.leave_subsection(
+    "Distribute constraints after linear solve");
 }
 
 template class MFNavierStokesSolver<2>;