article.hid2-ub-cicd.ppam24.tex

\documentclass[runningheads]{llncs}
\usepackage{fullpage}
\usepackage{graphicx} % for including images
\usepackage{float}
\usepackage[T1]{fontenc}
\usepackage{wrapfig}
\usepackage[]{subcaption}
\usepackage{booktabs}
\usepackage{tabularx}
\usepackage{siunitx}
\usepackage{ragged2e}
\usepackage{multicol}
\usepackage{url}
\usepackage{tikz}
\usepackage{xcolor}
\usepackage{paralist}
\usepackage{amsmath}
\definecolor{darkred}{rgb}{0.6, 0, 0}
\definecolor{darkgreen}{rgb}{0, 0.5, 0}
\definecolor{darkblue}{rgb}{0, 0, 0.5}
\definecolor{darkmagenta}{rgb}{0.5, 0, 0.5}
\usepackage{hyperref}

\IfFileExists{.git/gitHeadInfo.gin}{
    \usepackage[pcount,grumpy,mark,markifdirty]{gitinfo2}
}{%
    \usepackage[local,pcount,grumpy,mark,markifdirty]{gitinfo2}
}

%% By default, llncs tocdepth value is 0, so only chapters and parts are shown in the ToC, but no sections and deeper levels of structure.
%% If \setcounter{tocdepth}{1} is used, the sections show up in the ToC.
%% I suggest to use \usepackage{tocbibind} in order to add the LoF and LoT in the ToC as well, if the ToC should not be listed itself, use \usepackage[nottoc]{tocbibind} instead.
%% As long as no figure or table environments with \caption commands or explicit \addcontentsline{lof}{...}{...} etc. are used, the LoF and LoT are empty of course.
\setcounter{secnumdepth}{3}
\setcounter{tocdepth}{1}
\usepackage[section,notlof,notlot,nottoc,]{tocbibind}


\title{Ktirio Urban Building: A Computational Framework for City Energy Simulations Enhanced by CI/CD Innovations on EuroHPC Systems}
\author{Luca Berti\inst{1} \and 
Vincent Chabannes \inst{1}\orcidID{0009-0005-3602-3524} \and
Gwennolé Chappron \inst{1}\orcidID{0009-0003-7588-3370} \and
Javier Cladellas \inst{1}\orcidID{0009-0003-8687-7881} \and
Abdoulaye Diallo \inst{1}\orcidID{0009-0006-8731-0547}\ \and
Maryam Maslek Elayam \inst{1}\orcidID{0000-0003-0880-5180} \and
Philippe Pinçon \inst{1}\orcidID{0009-0009-7724-3055 } \and
Christophe Prud'homme\inst{1}\orcidID{0000-0003-2287-2961}}
\institute{Cemosis, IRMA UMR 7501, University of Strasbourg, CNRS\\
\email{\{vincent.chabannes,christophe.prudhomme\}@cemosis.fr}}
\date{\gitReln\  \gitAuthorDate\ (\gitAbbrevHash)}
\authorrunning{V. Chabannes et al.}

% Define custom color
\definecolor{CustomBlue}{rgb}{0.25, 0.41, 0.88} % RoyalBlue
% Set up hyperref with the custom citecolor
\hypersetup{
    pdftitle={\@title},
    pdfauthor={\@author},
    pdfsubject={\@subject},
    pdfkeywords={HPC, HPCOps, Urban building, City Energy Simulation},
    bookmarksnumbered,bookmarksopen,linktocpage,
    colorlinks=true,
    citecolor=CustomBlue,
    linkcolor=CustomBlue,
    urlcolor=blue
}
\usepackage{cleveref}

\begin{document}





\maketitle



\begin{abstract}
The building sector in the European Union significantly impacts energy consumption and greenhouse gas emissions. The EU's Horizon 2050 initiative sets ambitious goals to reduce these impacts through enhanced building renovation rates. The CoE HiDALGO2 supports this initiative by developing high-performance computing solutions, specifically through the Urban Building pilot application, which utilizes advanced CI/CD methodologies to streamline simulation and deployment across various computational platforms, such as the EuroHPC JU supercomputers. The present work provides an overview of the Ktirio Urban Building framework (KUB), starting with its workflow and a description of some of the main ingredients of the software stack, it then discusses some results performed on EuroHPC JU supercomputers using an innovative CI/

\keywords{HPC, HPCOps, Urban building, City Energy Simulation.}

\end{abstract}

%\tableofcontents
%\listoffigures
%\listoftables


\section{Introduction}
\label{sec:introduction}

The building sector accounts for approximately 40\% of final energy consumption and 36\% of greenhouse gas emissions within the European Union~\cite{european_commision_energy_2020}. In response, the EU has established ambitious targets under the Horizon 2050 framework to double energy renovation rates over the next decade~\cite{european_commision_stakeholder_2021}, highlighting the need for innovative solutions to drive these initiatives forward. The Centre of Excellence (CoE) HiDALGO2 project, focusing on high-performance computing and advanced simulations, is at the forefront of tackling this challenge, mainly through its Urban Building pilot application.

The Ktirio Urban Building (KUB) pilot in CoE HiDALGO2 aims to leverage high-performance computing to enhance city energy simulation for better energy management and air quality assessment. Advanced simulation tools predict energy consumption, thermal comfort, and indoor air quality across both the building and urban scales. These simulations support detailed individual building-level analysis and extend to broader urban environments, influencing urban planning and policy-making. KUB is part of the platform Ktirio~\cite{cemosis_ktirio_2024} which itself is based on Feel++~\cite{christophe_prudhomme_feelppfeelpp_2024}.

Building on the foundations of the Feel++ framework, this paper introduces an innovative CI/CD environment specifically designed for deployment on EuroHPC JU supercomputers. 
This environment enhances our ability to develop and test simulations rapidly and provides a robust platform for deploying and running these simulations at multiple scales. 
By leveraging containerized environments, we ensure that all dependencies are encapsulated, making the simulations portable and reproducible across different supercomputing resources. 
These containers are deployed on EuroHPC JU infrastructures, which represents a significant advancement in making high-performance computing resources accessible for urban simulation studies. 
The use of containers also simplifies software management and ensures consistency in the deployment process, enabling seamless scalability and reliability across EuroHPC systems.

Thanks to synergies with other CoE HiDALGO2 pilots, the KUB project is set to integrate building simulations with Urban Air Pollution (UAP) pilot models to comprehensively assess the environmental impact of building stocks soon. This integration improves the predictive accuracy of the simulations by incorporating data such as wind speed and solar radiation, enhancing the models' responsiveness and forecasts to environmental conditions.

Implementing the proposed KUB workflow will facilitate more informed urban planning decisions, support policy development for energy efficiency, and contribute to reducing urban greenhouse gas emissions. The project also focuses on enhancing the interactions between different environmental models to provide a holistic view of urban ecosystems.

This paper is organized as follows: 
\begin{inparaenum}[\it (i)]
    \item \cref{sec:kub-workflow} presents the KUB workflow, detailing the data handling and simulation processes, and the final analysis and urban scale energy evaluation;
    \item \cref{sec:urban-building} provides an overview of the geometrical and physical modeling components of the urban building application;
    \item \cref{sec:cicd-framework} delves into the CI/CD processes tailored for EuroHPC environments;
    \item \cref{sec:conclusion} concludes the paper.

\end{inparaenum}
%Feel++\cite{christophe_prudhomme_feelppfeelpp_2024}

\section{Ktirio Urban Building Workflow}
\label{sec:kub-workflow}

The Ktirio urban building workflow, see \cref{fig:kub-workflow}, integrates various data sources and computational tools to simulate and analyze urban building energy and its impact on urban environments. The process encompasses data acquisition, processing, simulation, and analysis, eventually coupled with Urban Air Pollution (UAP) models.

\begin{figure}
    \centering
    \includegraphics[width=.9\textwidth,page=1]{img-kub-workflow.pdf}
    \caption{Current Urban Building Workflow from localization to city energy simulation report.}
    \label{fig:kub-workflow}
\end{figure}


\subsection{Data Handling and Simulation Process}
\label{sec:data-handling}

The workflow begins with collecting and preparing GIS and weather data and transforming them into a usable format for simulations. The transformed data are then partitioned for scalable processing and converted into Modelica and Feel++~\cite{christophe_prudhomme_feelppfeelpp_2024} compatible formats through the \textit{UBEM Generator}.

This processed data is employed to simulate energy consumption and indoor environmental quality using the \textit{Urban Building Energy Model (UBEM)}. The simulation focuses on a \textit{radiative heat transfer model}, enabling enhancement of the accuracy of the energy models. It also computes \textit{view factors} and \textit{shading masks}, assessing how buildings affect each other's exposure to natural light and heat and overall building energy needs, and the outdoor environment by influencing the urban heat island effect.

The building simulation outputs are then optionally fed into the \textit{Urban Air Pollution (UAP) simulator} to evaluate the impact of building emissions on urban air quality. A feedback loop refines the building simulation scenarios based on intermediate outputs from the \textit{UAP simulator}, ensuring that the models accurately reflect the complex interdependencies between urban building energy usage and urban air quality.

\subsection{Final Analysis and Urban Scale Energy Evaluation}
\label{sec:final-analysis}

The final step involves the \textit{UBEM Simulator}, which generates large-scale outputs that summarize the overall energy consumption and environmental impact of buildings on an urban scale. This comprehensive urban scale analysis merges data from the building energy and air quality models to provide a holistic view of urban environmental quality.

In addition to these exports, the simulator condenses, aggregates and saves relevant quantities on the fly to avoid having to manipulate large data volumes after the simulation. These metrics concern weather information, energy consumption and comfort assessment, and are exported in JSON format. The produced files are uploaded to a data management platform via the CI/CD pipeline (see \cref{fig:kub-devops}), who is then in charge of notifying a different workflow in order to render insightful figures that showcase the simulation results, as detailed in \cref{sec:cicd-framework}.

Insights are presented in the form of a comprehensive report and are automatically deployed on a website built using the \textit{Antora} framework. End users are finally able to explore the simulation outputs in a detailed but malleable manner, for both the city and the building scales.

\Cref{fig:ktirio-cases-report} presents a gallery of city reports that are taken as test cases for the Urban Building pilot, as well as an example of a city report section showing the occlusion factor and heat flow plot.
\begin{figure}[htbp]
  \centering
  \subfloat[Gallery of test case cities reports\label{fig:city-gallery}]{%
    \includegraphics[width=0.45\textwidth]{img-compressed-city-gallery.png}
  }
  \hfill
  \subfloat[Section of a city report\label{fig:city-report-section}]{%
    \includegraphics[width=0.45\textwidth]{img-compressed-city-report-section.png}
  }
  \caption{Screenshots of the \url{https://cases.ktirio.fr} reporting website.}
  \label{fig:ktirio-cases-report}
\end{figure}



\section{Overview of Urban Building Modeling and Simulation}
\label{sec:urban-building}

This section provides an overview of the geometrical and physical modeling and simulation components of our urban building application.

\subsection{Geometry Reconstruction of the KUB Urban Model}

The geometric reconstruction of urban environments within KUB involves a sophisticated approach to creating multi-fidelity representations of buildings, terrain, vegetation, roads, and other urban elements. This section outlines the methodologies employed and the various Levels of Detail (LOD) used in the different models of kUB workflow.


The primary challenge lies in accurately representing the complex urban landscape to support various simulations. A tiled web map approach is adopted, which allows for distributed data management and adapts the Level of Detail (LOD) based on specific needs. However, this approach requires carefully integrating tiles to ensure seamless representation.


We describe our definition of Levels of Detail for Buildings such as:
\begin{description}
    \item[LOD-0] The most straightforward form, representing buildings as oriented bounding boxes. This level is typically used for large-scale preliminary analyses and quick visual assessments.
    \item[LOD-1] Buildings are represented as polygonal extrusions, with added roof structures to improve the visual accuracy and utility in simulations that do not require detailed internal features.
    \item[LOD-2] At this level, buildings are detailed using Industry Foundation Classes (IFC) standards, supporting detailed thermal and structural analyses. This includes detailed geometries for each building entity, often using complex surface models like B-REP, swept solids, or CSG techniques.
\end{description}


\begin{figure}[ht]%{R}{1\textwidth} % 'R' or 'L' for right or left, and width
\centering
\subfloat[LOD-0: a building is represented by its bounding box]{%
  \includegraphics[width=0.45\linewidth]{img-compressed-buildings-lod0.png}
  \label{fig:building-lod0}
}\hspace{0.05\linewidth}
%\hfill % This ensures that the images are placed side by side
\subfloat[LOD-1: a building is represented by its ground footprint elevated to its height]{%
  \includegraphics[width=0.45\linewidth]{img-compressed-buildings-lod1.png}
  \label{fig:building-lod1}
}\\ % Ends the line for the first row of figures

\subfloat[LOD-2: a building in full detail using BIM. Note that LOD-2 and LOD-1 are mixed.]{%
  \includegraphics[width=0.45\linewidth]{img-compressed-buildings-lod2.png}
  \label{fig:building-lod2}
}\hspace{0.05\linewidth}
%\hfill
\subfloat[LOD-2: A zoom on the LOD-2 building.]{%
  \includegraphics[width=0.45\linewidth]{img-compressed-buildings-lod2-zoom.png}
  \label{fig:building-lod2-zoom}
}

\caption{Different representations of a building using our LOD definition}
\label{fig:buildings}

\end{figure}

In \cref{fig:buildings}, we illustrate the different levels of details. 
\Cref{fig:building-lod0} displays the LOD-0 of a building with its bounding box. 
\Cref{fig:building-lod1} displays the LOD-1 of a building using its footprint elevated to its height. 
\Cref{fig:building-lod2} and~\cref{fig:building-lod2-zoom} display the LOD-2 representation using BIM. 
In \Cref{fig:city-strasbourg}, we display an illustration of the center of the city of Strasbourg with LOD-0, see \cref{fig:city-strasbourg-lod0}, and LOD-1, see \cref{fig:city-strasbourg-lod1}, representations.




\subsubsection{Building Modeling}
Building meshes are generated from metadata fetched from web services like OpenStreetMap~\cite{openstreetmap_contributors_planet_2017}, which provides multi-polygons with holes representing the complex urban fabric. For \textbf{LOD-0 and LOD-1}, buildings are modeled from 2D footprints extruded to form three-dimensional volumes. These volumes are then combined or subtracted to represent the district or the entire city, applying union operations on buildings that touch or intersect. In \textbf{LOD-2}, the focus shifts to creating conformal and watertight meshes suitable for detailed simulation tasks. These meshes are generated from Building Information Modeling (BIM) data in IFC format, enabling a detailed representation of each building component.



\subsubsection{Terrain Modeling}
The terrain modeling process utilizes elevation data extracted from raster images. The initial step involves creating a uniform mesh based on the size of the raster image. Following this, the elevation at each node is evaluated.
However, this method provides a very refined mesh, even when the terrain is flat. For this reason, we plan to apply a mesh adaptation technique to reduce mesh density where feasible, using the following processing:
\begin{inparaenum}[\it (i)]
    \item Compute an arbitrary number of isolines for elevation,
    \item Build a new terrain mesh conform with the isolines nodes and adapt to terrain elevation gradient.
\end{inparaenum}
%Subsequently, isolines for elevation are computed, and mesh adaptation techniques are applied to reduce mesh density where feasible. Finally, each tile is glued together to ensure the continuity of the terrain mesh.

\subsubsection{Vegetation Modeling}

Including vegetation (trees) in our geometric model is essential for the simulation of buildings. The shading provided by trees and the cooling provided by wooded areas had a significant impacts on building environment  . We use the OpenStreetMap database to obtain the metadata associated with the vegetation. In addition to the position of the trees, we can also obtain other information, such as the tree's height and species. The current strategy is as follows:
\begin{inparaenum}[\it (i)]
\item Definition of a reference tree library that lists a certain number of tree species at different levels of precision (LOD). These geometric entities are parameterized so that a transformation can be applied.
\item Fetching vegetation metadata via OpenStreetMap. The attributes (height, species) are not always available, so we try to define them as best as we can (by searching around).
\item Creating a tree geometric model using an affine transformation of the reference tree model.
\end{inparaenum}

\subsubsection{Integration of all urban geometric components}

This process involves creating a conforming mesh that includes all components (buildings, terrains, vegetation). Moreover, this step requires accomplishing some challenges by using complex geometric algorithms to realize the following points: \textit{(i)} Ensure that buildings on slopes are accurately modeled by adapting their height and embedding them into the terrain mesh. 
\Cref{fig:city-grenoble-terrain} illustrates this aspect; \textit{(ii)} applies the intersection of buildings and consequently defines contact zones that can be included in building thermal models (coupling). \textit{(iii)} Apply the intersection of vegetation with buildings and terrain; and finally \textit{(iv)} Improve mesh quality after the previous geometric operations (mesh adaption).


\subsubsection{Visual Representation}
Advanced rendering techniques visualize the multi-fidelity urban models, supporting detailed analysis and general urban planning discussions.
These visualizations are crucial for assessing the impact of urban changes and for stakeholder engagement.

This approach enhances urban models' accuracy, utility, and scalability, making them vital for comprehensive urban analysis and planning in the HiDALGO2 project.

%\begin{wrapfigure}{R}{0.6\textwidth}
\begin{figure}[ht]
\centering
\subfloat[Representation of Strasbourg center with LOD-0\label{fig:city-strasbourg-lod0}]{%
  \includegraphics[width=0.48\textwidth]{img-compressed-city-strasbourg-lod-0.png}
}
\hfill % Ensures that the images are placed side by side
\subfloat[Representation of Strasbourg center with LOD-1\label{fig:city-strasbourg-lod1}]{%
  \includegraphics[width=0.48\textwidth]{img-compressed-city-strasbourg-lod-1.png}
}\\ % Ends the line for the first row of figures

\subfloat[Representation of Grenoble LOD-1 terrain elevation\label{fig:city-grenoble-terrain}]{%
  \includegraphics[width=0.48\textwidth]{img-compressed-city-grenoble-terrain.png}
}

\caption{Various representations of cities and terrains}
\label{fig:city-strasbourg}
\vspace{-0.5cm}
\end{figure}
%\end{wrapfigure}

\subsubsection{Computational Tools for Meshes Generation}

In the urban modeling process, particularly in generating building meshes, the Computational Geometry Algorithms Library (CGAL) plays a pivotal role. CGAL, \cite{the_cgal_project_cgal_2024} is renowned for its robust and efficient algorithms, which are crucial for handling complex geometric data and generating high-quality meshes. This library is currently used for the following operations: boolean operation of polygon, multi-polygon repairs~\cite{loriot_polygon_2024}, mesh generation, mesh intersection, mesh adaptation, and building roof skeleton.

All the features described above are not yet operational in our framework. We can reconstruct a geometric model of an urban area whose location is arbitrary so that the user can select an area anywhere in the world. The final mesh comprises all buildings LOD-0 or LOD-1 and the terrain with elevation. The integration of all the urban components is currently being investigated. This part requires fairly costly algorithms (mesh intersection), for which it will be necessary to form groups intelligently to reduce the computational cost.

\paragraph{Current Mesh Generation Strategy}
The current strategy for mesh generation employs multi-threading (MT) to handle various stages of the mesh construction process. Initially, the used tiles are determined based on the specified location, a task performed sequentially. Following this, GIS data, including buildings and elevation information, is downloaded in parallel. Next, polygons are repaired~\cite{loriot_polygon_2024} to ensure they are suitable for mesh generation, with this process executed in parallel using MT. The terrain mesh is then generated in parallel, employing CGAL's algorithms to ensure precision and efficiency. Union operations at tile junctions are performed to ensure continuity, a step that is currently sequential. Finally, building meshes are generated using a parallel MT approach, leveraging CGAL for its advanced mesh generation capabilities.

\paragraph{Advancing Towards Full Parallelism}
The following steps in enhancing the mesh generation process involve moving towards a fully parallel approach using both Multi-threading (MT) and Message Passing Interface (MPI). The goal is to scale the process to handle entire cities or larger urban areas. This scale-up involves utilizing MPI to manage distributed computing resources effectively. To achieve the latter objective, partitions of tiles with overlapping regions are created to ensure complete and accurate building descriptions without requiring extensive MPI communications. Furthermore, each process uses an MT strategy to generate meshes independently, with overlapping zones allowing for the creation of complete building structures.

\subsubsection{Partitioning Strategies Depending on Simulation Use Cases}

Data partitioning is a crucial stage in the deployment of the supercomputer simulator. We need to partition the geometry of the city in such a way as to distribute the simulation computation correctly.
Different partitioning strategies are considered depending on the specific requirements of the simulation use cases :
\begin{inparadesc}
\item[Case 0] simulates simple scenarios where buildings do not interact with their environment, a basic listing and weighting strategy is sufficient;
\item[Case 1] simulates buildings interact with environmental elements, a more complex partitioning strategy is necessary that considers both the buildings and their immediate non-building surroundings; the build meshes and environment meshes (terrain, vegetation) are integrated conformably; each component can be partitioned separately;
\item[Case 2] simulates full interaction models where the partitioning strategy starts with the buildings and extends to the entire urban mesh, ensuring all elements are considered to minimize communication overhead;
\item[Case 3] simulates scenarios with extreme partitioning and employ a multi-grid approach to  define coarse and fine meshes and manage computational resources efficiently; this multi-fidelity approach enhances the accuracy and applicability of urban models in simulations and ensures scalability across different computational platforms, making it a cornerstone of urban analysis.
\end{inparadesc}

The current version of our simulation framework has implemented case 0 and case 1. Cases 2 and 3 require a full conformal mesh, which is not yet available. Moreover, these partitioning strategies are costly, so we have planned to use other third-party tools, such as Zoltan2~\cite{the_zoltan2_team_zoltan2_nodate}, to improve efficiency.
\begin{wrapfigure}{R}{0.7\textwidth}
\centering
\subfloat[Partitioning Case 0\label{fig:city-strasbourg-lod0-parts}]{%
  \includegraphics[width=0.46\linewidth]{img-compressed-city-strasbourg-lod0-parts.png}
}\hspace{0.02\linewidth}
%\hfill % Ensures that the images are placed side by side
\subfloat[Partitioning Case 1\label{fig:city-strasbourg-lod1-parts}]{%
  \includegraphics[width=0.46\linewidth]{img-compressed-city-strasbourg-lod1-parts.png}
}
%\subfloat[Partitioning Case 3: Large scale\label{fig:city-ny-largescale}]{%
%  \includegraphics[width=0.4\textwidth]{img-compressed-city-newyork-largescale.png}
%}
\caption{Mesh partitioning illustrations}
\label{fig:partitioning}
\end{wrapfigure}

\Cref{fig:partitioning} illustrates the different strategies discussed previously and presents a reconstruction of New York City in \cref{fig:city-ny-largescale}. This geometric model has an area of 400 $km^2$ (20 km square side of 20 km) and has generated around 450000 buildings. This example requires the large-scale approach discussed but not yet implemented or tested.

\begin{figure}[htbp]
\centering
\subfloat[View on whole 3D mesh\label{fig:city-ny-largescale-whole}]{%
  \includegraphics[width=0.45\textwidth]{img-compressed-city-newyork-largescale-whole.png}
}
\hfill
\subfloat[Zoom on Manhattan buildings\label{fig:city-ny-largescale-zoomB}]{%
  \includegraphics[width=0.45\textwidth]{img-compressed-city-newyork-largescale-zoomB.png}
}\\
\subfloat[Focus on Manhattan\label{fig:city-ny-largescale-zoomZ}]{%
  \includegraphics[width=0.45\textwidth]{img-compressed-city-newyork-largescale-zoomA.png}
}
\hfill
\subfloat[Central Park\label{fig:city-ny-largescale-zoomC}]{%
  \includegraphics[width=0.45\textwidth]{img-compressed-city-newyork-largescale-zoomC.png}
}
\caption{$20 \times 20 \mathrm{km}^2\ $ geometric reconstruction of New York City (LOD-1)}
\label{fig:city-ny-largescale}
\end{figure}


\subsubsection{Conclusion on mesh construction}

Our strategy aims to improve the efficiency and scalability of the mesh generation process and enhance the fidelity and accuracy of the urban models used in simulations.

\subsection{Modeling and simulations}

We now turn to describing the physical modeling of city energy simulation.
To do this, we built on two main tools: Feel++ and Modelica.

\subsubsection{Heat transfer model in buildings}
\label{sec:heat-transfer-in-buildings}

The physical model of city buildings is currently based on a multizone model.
One zone is allocated per building floor. 
The energy balance for each zone is calculated by taking into account energy exchanges with adjacent zones and the environment.

The energy balance is written according to the first law of thermodynamics: in an isolated system, the sum of all forms of energy is constant.

\begin{equation}
    C \frac{dT}{dt} = \phi_{\mathrm{HVAC}} + \phi_{\mathrm{gain}} - \phi_{\mathrm{losses}}
\end{equation}
where
\begin{inparaenum}[\it (i)]
    \item $C \dfrac{dT}{dt}$ represents the thermal inertia of the construction, quantifying how the temperature of the zone changes over time due to its thermal mass. Specifically,
    $C$ is a measure of how much heat a material can store and release, combining the heat capacity of the construction (how much energy is required to change the temperature) with the rate of temperature change and 
    $\dfrac{dT}{dt}$ is the rate of change of temperature with respect to time. A higher thermal inertia means the construction will have a slower rate of temperature change in response to energy inputs and losses, effectively buffering temperature fluctuations;
    \item $\phi_{\mathrm{HVAC}}$ represents the energy input from the Heating, Ventilation, and Air Conditioning (HVAC) systems. It includes energy added to the zone to maintain desired temperature levels;
    \item $\phi_{\mathrm{gain}}$, referred to as free energy gain, includes solar gains (heat from sunlight) and internal gains (heat generated by occupants and equipment);
    \item $\phi_{\mathrm{losses}}$ represents the energy lost through walls, roofs, floors, and air exchanges with the environment.
\end{inparaenum}

\begin{description}
  \item[HVAC heat flow] In the version of the model at the time of writing, the HVAC system is treated as an ideal heat source, which means it adjusts its output to maintain a set temperature within the building.
  \item[Heat gain] Heat gain is the addition of energy to the building, derived from two sources: \textbf{internal heat gains} and \textbf{solar heat gains}.
  \item[Internal heat gain] The internal heat gain captures the heat produced by occupants inside the building. 
  It is modeled as a function of metabolic rate $\mathrm{Met}$ and the number of inhabitants, $d_i$, which varies over time $t$:
  \begin{equation}
    \phi_{\mathrm{int}} = \mathrm{Met} \cdot d_i(t)
  \end{equation}

\item[Solar heat gain] The total solar heat gain includes contributions from direct, diffuse, and reflected solar radiation, taking into account the building’s surface characteristics and shading coefficient:
\begin{equation}
  \phi_{\mathrm{solar}} = \alpha \cdot S \cdot \left( E_{\mathrm{dir}} + E_{\mathrm{diff}} + E_{\mathrm{ref}} \right) \cdot S_C
\end{equation}
Direct solar radiation is the component of solar radiation adjusted for the angle of incidence of the sunlight:
\begin{equation}
  E_{\mathrm{dir}} = F_{\mathrm{dir}} \cdot \cos(I)
\end{equation}
Here, $F_{\mathrm{dir}}$ is the direct solar radiation, which is a function of the horizontal solar radiation measured by a weather station:
\begin{equation}
  F_{\mathrm{dir}} = \dfrac{F_{\mathrm{dir,Hz}}}{\sin(H_s)}
\end{equation}
The term $\cos(I)$ calculates the cosine of the angle of incidence of sunlight, incorporating the sun's altitude and azimuth angles as well as the building's tilt angle:
\begin{equation}
  \cos(I) = \cos(H_s) \cdot \sin(\beta) \cdot \cos(A_{zs} - A_z) + \sin(H_s) \cdot \cos(\beta)
\end{equation}
Diffuse solar radiation includes contributions from the sky dome and is adjusted for the building's tilt:
\begin{equation}
  E_{\mathrm{diff}} = F_{\mathrm{dif,Hz}} \cdot \left( \dfrac{1 + \cos(\beta)}{2} \right)
\end{equation}
Reflected solar radiation is calculated using the total horizontal radiation and the environment's albedo:
\begin{equation}
  E_{\mathrm{ref}} = \left[ F_{\mathrm{dir,Hz}} + F_{\mathrm{dif,Hz}} \right] \cdot \left( \dfrac{1 + \cos(\beta)}{2} \right) \cdot \text{albedo}
\end{equation}

\item[Conduction losses] Energy loss through walls is computed by summing the losses across all wall segments. It depends on the heat transfer coefficient, the wall area, and the temperature difference between the indoor and outdoor environments. In our current model, we simplify the calculations by not including the effects of thermal bridges, as they depend on specific information about the building's envelope, which we do not have:
\begin{equation}
  \phi_{\mathrm{wall}} = \sum_{j} U_j \cdot S_j \cdot \Delta T
\end{equation}
The convection losses are computed differently depending on the surface of the wall to which they are applied. On the interior surface, we apply a constant:
\begin{equation}
  \phi_{\mathrm{conv,int}} = \alpha_{\mathrm{conv}} \cdot \Delta T
\end{equation}
On the exterior surface, we use a forced convection model:
\begin{equation}
  \phi_{\mathrm{conv,ext}} = \left( 1.8 + 4.8 \cdot v_{\mathrm{air}} \right) \cdot \Delta T
\end{equation}

\item[Radiation losses] Radiation losses describe the heat energy lost due to thermal radiation, impacting both energy efficiency and thermal comfort.
We differentiate between the exterior of the wall that exchanges with the sky and the interior where the walls exchange with each other.
namely, for the exterior radiation exchange:
\begin{equation}
  \phi_{\mathrm{net,sky}} = \varepsilon_i \cdot \sigma \cdot \left( T_i^4 - T_{\mathrm{sky}}^4 \right)
\end{equation}
and for the interior radiation exchange:
\begin{equation}
  J[i] = \varepsilon_i \cdot \sigma \cdot T_i^4 + \left( 1.0 - \varepsilon_i \right) \cdot \sum_{j=1}^{n} F[i,j] \cdot J[j]
\end{equation}
where the radiative heat exchange $J[i]$ includes two terms:
\begin{inparaenum}[\it (i)]
    \item $\varepsilon_i \cdot \sigma \cdot T_i^4$, which calculates the heat radiated by surface $i$ where $\varepsilon_i$ measures how effectively the surface emits radiation and $T_i^4$ reflects that radiative heat loss increases rapidly with temperature; and
    \item $\left( 1.0 - \varepsilon_i \right) \cdot \sum_{j=1}^{n} F[i,j] \cdot J[j]$, which accounts for the heat absorbed from other surfaces $j$ within the zone. The factor $\left( 1.0 - \varepsilon_i \right)$ adjusts for the fraction of incoming radiation not emitted by surface $i$.
\end{inparaenum}
The net radiation flow is given by:
\begin{equation}
  \phi_{\mathrm{net,int}}[i] = S \cdot \dfrac{\varepsilon_i}{1.0 - \varepsilon_i} \cdot \left( \sigma \cdot T_i^4 - J[i] \right)
\end{equation}
Net radiation flow $Q_{\mathrm{flow}}[i]$ considers:
\begin{inparaenum}[\it (i)]
    \item $\dfrac{\varepsilon_i}{1.0 - \varepsilon_i} \cdot \left( \sigma \cdot T_i^4 - J[i] \right)$, which calculates the net heat flow due to radiation for surface $i$, considering the difference between the heat emitted by the surface and the heat received from other surfaces. The ratio $\dfrac{\varepsilon_i}{1.0 - \varepsilon_i}$ scales the net flow based on the emissivity of the surface.
\end{inparaenum}

\item[Air change] This term represents the energy loss due to air exchange between the zone and the exterior:
\begin{equation}
  \phi_{\mathrm{air}} = 0.34 \cdot Q_v \cdot \Delta T
\end{equation}
\end{description}
The described thermal models are implemented using the Modelica language, which is an object-oriented, equation-based language well-suited for modeling complex physical systems. 
These models are then transformed into Functional Mock-up Units (FMUs) compliant with the Functional Mock-up Interface (FMI) standard. 
FMUs encapsulate the models and enable them to be shared and integrated across different simulation environments.


\paragraph{Conclusion} By integrating these models within a high-performance computing framework, it is possible to simulate the thermal behavior of city-scale building clusters.
This facilitates the assessment of energy consumption, thermal comfort, and the impact of various energy-saving strategies under different environmental conditions and operational scenarios.

\subsubsection{KUB is based on the Feel++ toolchain}

Feel++ is a comprehensive framework designed to tackle problems based on Ordinary Differential Equations (ODEs) and Partial Differential Equations (PDEs). 
Using modern C++ (C++17 and C++20) standards coupled with a Python layer through Pybind11, Feel++ enables seamless parallelism and is equipped with default communicators that simplify handling complex computational tasks.
The framework's versatility is evident in its deployment across various platforms, including research and educational environments and cloud services tailored for high-performance computing needs.

Key features of the Feel++ framework encompass an extensive range of numerical methods designed to address Partial Differential Equations (PDEs), see~\cite{christophe_prudhomme_feelppfeelpp_2024}. 
These methods include continuous Galerkin (cG), discontinuous Galerkin (dG), hybrid discontinuous Galerkin (hdG), and reduced basis methods (rb/mor). 
A Domain Specific Language (DSL) for Galerkin methods significantly enhances the ease of implementing and experimenting with new numerical methods. 
The de Rham complex provides a comprehensive toolkit for constructing finite element spaces of arbitrary order, facilitating precise mathematical modeling. 
The framework's automatic differentiation and symbolic integration capabilities effectively bridge the gap between mathematical expressions and their computational implementation.
Feel++ supports diverse applications, from fluid dynamics and structural mechanics to heat transfer and electrostatics, demonstrating its flexibility and broad applicability.
Furthermore, its integration with Specx for task-based parallel execution optimizes performance and scalability on modern computational architectures.

%\paragraph{Simulation Tools and Applications}
%Feel++ comes equipped with various toolboxes tailored for specific application domains, offering pre-built scenarios for fluid dynamics, solid mechanics, and electromagnetism, among others. These toolboxes are designed to provide robust and efficient solutions, leveraging the capabilities of Feel++ to address complex multi-physics problems in real-world scenarios.

Documentation and further details can be accessed through Feel++ Toolboxes Documentation\footnote{\url{https://docs.feelpp.org/toolboxes/latest/}}.
This powerful toolchain is essential for KUB.

\subsubsection{Computing Shading Masks and View Factors with Feel++}

In city energy simulations, the computation of shading masks and view factors is crucial for accurately modeling the impact of solar radiation on building surfaces. Shading masks quantify the percentage of blocked solar radiation for each building surface (including walls and roofs) depending on the sun's direction. This is influenced by nearby structures such as other buildings, vegetation, and urban furniture. The view factors describe the fraction of radiation that leaves one surface and strikes another, essential for calculating radiative heat exchanges between building surfaces.

\paragraph{Numerical Methods and Challenges}
Both shading masks and view factors are computed using Monte Carlo and ray tracing techniques, which allow for handling complex geometries with various obstructions. Despite being purely geometric quantities, these calculations face significant challenges such as:
\begin{inparaenum}[\it (i)]
    \item Efficient computation of integrals for view factors, especially when considering specular surfaces that require multiple ray bounces.
    \item Managing large-scale mesh computations and data storage, particularly when detailed urban environments are modeled.
\end{inparaenum}

\paragraph{Implementation in Feel++}
Feel++ facilitates these computations through its robust numerical methods optimized for high performance and parallel execution. For each face of a building, Feel++ computes solar masks using a Monte Carlo approach for various sun positions, ensuring efficient and scalable processing across multiple CPU cores. This enables the integration of dynamic solar shading effects into the simulation of building energy performance, providing a more accurate representation of real-world conditions.

\begin{figure}[ht]
\centering
\subfloat[LOD-0\label{fig:sm-building-east}]{%
  \includegraphics[width=0.45\textwidth]{img-compressed-solar-masks-east-facing.png}
}
\hfill
\subfloat[LOD-1\label{fig:sm-whole-building}]{%
  \includegraphics[width=0.45\textwidth]{img-compressed-solar-masks-whole-building.png}
}\\
\subfloat[LOD-1 Large scale\label{fig:sm-strasbourg}]{%
  \includegraphics[width=0.45\textwidth]{img-compressed-solar-masks-strasbourg.png}
}
\hfill
\subfloat[Heat transfer benchmark in 2D including view factors~\cite{van_eck_surface_2016}\label{fig:view-factor}]{%
  \includegraphics[width=0.45\textwidth]{img-compressed-view-factors-benchmark.png}
}
\caption{Solar masks and view factors computations}
\label{fig:solar-masks-vf}
\end{figure}


In \cref{fig:solar-masks-vf}, we illustrate several aspects of solar masks: \textit{(i)} the solar masks of a building face oriented eastward, showing a discretization of the sun's position in \cref{fig:sm-building-east}; \textit{(ii)} the solar masks for an entire building, representing a detailed discretization of the sun's position throughout the day in \cref{fig:sm-whole-building}; \textit{(iii)} a visualization of the solar mask in the early morning for the city center of Strasbourg in \cref{fig:sm-strasbourg}; and \textit{(iv)} an image of a standard benchmark~\cite{van_eck_surface_2016} used for the computation of view factors and solving heat transfer problems between three building blocks in 2D in \cref{fig:view-factor}.

The grayscale values in each solar mask represent varying levels of solar obstruction for different azimuth and altitude angles.
In these masks, the scale ranges from 0 to 1, where 0 (white) indicates no obstruction and full exposure to sunlight, and 1 (black) represents full obstruction, meaning no sunlight reaches that part of the building face for the given sun position. 
Intermediate shades of gray indicate partial obstruction, where a portion of sunlight is blocked by surrounding buildings or other urban features.
The level of detail in the solar mask depends on the chosen level of discretization (LOD), as illustrated in \cref{fig:sm-building-east} and \cref{fig:sm-whole-building}.

%In \cref{fig:sm-building-east} (LOD-0), we see a coarser discretization with larger, generalized angular segments, leading to more abrupt transitions between obstructed and non-obstructed areas. In contrast, \cref{fig:sm-whole-building} (LOD-1) provides a finer level of detail, capturing more granular variations in solar accessibility across different sun angles. These masks help quantify the solar exposure of building surfaces over time, which is critical for accurately modeling heat gains and understanding the thermal dynamics of urban environments.

\subsubsection{Heat Transfer Modeling with Feel++ and Modelica}
In the Urban Building Energy Model (UBEM), heat transfer analysis is crucial for predicting energy consumption, internal air temperature variations, and overall building energy performance. This analysis is facilitated by a combination of Feel++ and Modelica, enabling detailed simulations of heat dynamics within urban buildings.

\paragraph{Modelica for Multizone Heat Transfer}
Modelica offers extensive capabilities for multizone building energy simulations, utilizing models that range from simple (LOD-0) to more complex (LOD-1) representations. The multizone approach in Modelica is beneficial for modular and scalable simulations, where each building zone can be modeled with different fidelity based on the simulation requirements. This method leverages the generation of Functional Mock-up Units (FMUs), which integrate seamlessly into larger C/C++ applications, providing a robust framework for handling complex simulations involving multiple interacting systems.
The FMUs built from \cref{sec:heat-transfer-in-buildings} are loaded by the \texttt{Feel++} environment.
Input data for the models are sourced from available building information, such as geometric dimensions and material properties.
When specific data are unavailable, default values are assigned based on the building's construction year, referencing standard building codes or typical practices from that period.
Occupancy schedules are given as input to the models using reglementary occupancy.
We are currently working on parametrized occupancy profiles, which are adjusted based on the building's use type (e.g., residential, office, commercial).
This approach ensures that the models can be applied to a wide range of buildings, even when detailed information is limited.


\paragraph{Finite Element Analysis with Feel++}
Complementing Modelica's capabilities, Feel++ provides robust tools for finite element analysis, particularly in handling the detailed aspects of heat transfer within urban environments. It uses advanced numerical methods like reduced basis methods for rapid scenario testing and parallel-in-time algorithms for efficient simulations. This is particularly important for assessing the impact of solar radiation and external shading, which are modeled using geometric and dynamic shading masks derived from solar paths.

\paragraph{Integrated Approach}
The integration of Feel++ and Modelica is exemplified in their use of shading masks and view factors, which are critical for accurate solar heat gain calculations. These masks are computed using Monte Carlo simulations and ray tracing methods to assess the percentage of solar radiation impacting various building surfaces. This data feeds into the Modelica simulations, enhancing the accuracy of the thermal load predictions.

\paragraph{Challenges and Solutions}
One of the main challenges in urban building simulation is managing the computational load, which is addressed through computing strategies that leverage currently CPUs but in the GPUs but, in the future, will enable hybrid computing with both CPUs and GPUs. 
This approach ensures that large-scale simulations, necessary for city-wide energy analysis, remain feasible and efficient. 
Additionally, the mesh partitioning techniques discussed earlier are employed to optimize the data handling and processing times, further integrating the spatial data management with the thermal modeling processes.

%By utilizing the combined strengths of Feel++ for detailed finite element analysis and Modelica for system-level energy simulation, the UBEM framework is a tool capable of %%addressing the complex dynamics of urban heat transfer and energy management.%

%Continuous Integration (CI) and Continuous Deployment (CD) are pivotal in developing and operating the Urban Building pilot. These practices enable efficient management of complex simulation software, ensuring seamless integration and deployment across various computational infrastructures.

%CI practices handle complex dependencies and computational requirements specific to urban building simulations. Key features include automated testing, code quality checks, and automated builds, ensuring the software remains robust and maintainable.

%CD practices are tailored to manage deployment complexities across different HPC environments using containerization through Apptainer. This approach facilitates consistent, reproducible deployments, optimizing performance and resource utilization.

\subsubsection{Input data}

To run a KUB simulation, several input files need to be generated.
We have several metadata files (in JSON format), mesh data files (in MSH format), and meteorological measurement files (in CSV format).
\Cref{tab:input-data-size-comparison} lists these files with their corresponding sizes for New York City, see~\cref{fig:city-ny-largescale}, and a duration of one year for the simulation.
In addition, we compared the size of the input data with the size of the zone.

\begin{table}[ht]
  \centering
  \caption{Size comparison of input data files for different geographical extents in New York City}
  \label{tab:input-data-size-comparison}
  \begin{tabularx}{.8\linewidth}{l *{3}{>{\RaggedLeft\arraybackslash}X}} 
      \toprule
      \textbf{Files} & \textbf{\text{4 $\times$ 4} \si{\kilo\meter\squared}} & \textbf{\text{10 $\times$ 10} \si{\kilo\meter\squared}} & \textbf{\text{20 $\times$ 20} \si{\kilo\meter\squared}} \\
      \midrule
      \texttt{geographicdata\_setup.json} & \SI{4.0}{\kilo\byte} & \SI{4.0}{\kilo\byte} & \SI{4.0}{\kilo\byte} \\ 
      \texttt{gis.json} & \SI{23}{\mega\byte} & \SI{117}{\mega\byte} & \SI{470}{\mega\byte} \\
      \texttt{mesh\_lod0.msh} & \SI{17}{\mega\byte} & \SI{112}{\mega\byte} & \SI{496}{\mega\byte} \\
      \texttt{mesh\_lod1.msh} & \SI{44}{\mega\byte} & \SI{240}{\mega\byte} & \SI{896}{\mega\byte} \\
      \texttt{weather-data-0.hourly-variables.csv} & \SI{384}{\kilo\byte} & \SI{384}{\kilo\byte} & \SI{384}{\kilo\byte} \\
      \texttt{weather-data.json} & \SI{4.0}{\kilo\byte} & \SI{4.0}{\kilo\byte} & \SI{4.0}{\kilo\byte} \\
      \bottomrule
  \end{tabularx}
  \end{table}

In the future, the input data is expected to grow even larger with the addition of new details such as roofs, building characteristics, vegetation (e.g., trees), and terrain information.
These enhancements will increase the complexity and accuracy of the simulation and the amount of data required to accurately model the urban environment.

\section{CI/CD Framework for the Urban Building Pilot}
\label{sec:cicd-framework}

The Urban Building pilot utilizes the Feel++ framework, supported by a robust CI/CD framework that facilitates efficient development and deployment.
A critical component for both Feel++ and KUB is the use of containerized environments, which encapsulate all necessary dependencies and configurations.


%This section outlines how GitHub Actions and Docker streamline the entire process, ensuring the robustness and productivity of the software development lifecycle.

\subsection{Standard CI/CD DevOps}
The development and deployment of KUB builds on top of the Feel++ CI/CD framework. It employs GitHub Actions and Docker: GitHub Actions automate workflows to compile, test, and validate code changes, facilitating rapid development cycles and ensuring code quality.
On the other hand, Docker provides a containerized environment that encapsulates Feel++ and its dependencies, ensuring consistent operations across diverse computing environments.
These Docker images, customized for various system requirements, are maintained on the GitHub Container Registry (ghcr.io) to accommodate multiple deployment scenarios.

\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{img-feelpp-devops.pdf}
    \caption{CI/CD DevOps for Feel++}
    \label{fig:feelpp-devops}
\end{figure}
The CI/CD workflow, see \cref{fig:feelpp-devops}, is crucial for efficiently integrating and deploying updates across all projects that utilize the Feel++ framework. 
The workflow leverages various main ingredients of GitHub Actions features.
\begin{inparaenum}[\it (1)]
    \item \textbf{Pull Requests and Merges:} Triggering CI to verify that new code integrations meet all tests and standards.
    \item \textbf{Graphical User Interface (GUI) thanks to \texttt{workflow\_dispatch}:} Enabling developers to manually trigger pipelines through a GUI, which facilitates rapid deployment or testing.
    \item \textbf{Scheduled Runs:} Conducting regular updates and maintenance checks to ensure continuous system integrity and responsiveness.
\end{inparaenum}




\subsection{HPC DevOps (HPCOps)}
Feel++ CI/CD workflow for high-performance computing applications incorporates specialized HPCOps (HPC DevOps) practices that ensure the software performs consistently across various HPC systems.

%This advanced CI/CD strategy uses Apptainer (formerly Singularity) for secure and reproducible deployments across HPC environments without root privileges.

\Cref{fig:feelpp-hpcops} illustrates the HPC CI/CD or HPCOps workflow for Feel++.
\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{img-feelpp-hpcops.pdf}
    \caption{CI/CD HPCOps for Feel++}
    \label{fig:feelpp-hpcops}
\end{figure}

The tools and strategies for HPCOps are  \textit{(i) }\textbf{Reframe-HPC:} Utilized to define and manage systematic benchmarks that are reproducible across different HPC environments, facilitating the testing of performance and scalability, see~\cite{karakasis_reframe-hpcreframe_2024};
\textit{(ii)}\textbf{SLURM:} Employs its REST API if available, \textit{e.g.} on MeLuXiNa, otherwise scripted SLURM usage for CI/CD for scheduling and managing jobs on integrated HPC systems, allowing programmable job submission and monitoring directly from CI workflows, see~\cite{slurm_development_team_slurm_2024}; and \textit{(iii)} \textbf{Apptainer:} Ensures that Docker containers can be deployed securely and efficiently in HPC settings, supporting portability and consistency, see~\cite{apptainer_contributors_apptainer_2024}.

Integration with EuroHPC JU supercomputers such as LUMI, Karolina, Meluxina, Discoverer, Vega, and Leonardo enhances the capability to perform large-scale simulations and check the parallel properties and correctness of Feel++. 
The operations include automatic testing that triggers larger-scale tests on designated HPC nodes once new changes are integrated and verified by standard CI/CD pipelines.

Regarding monitoring and reporting, performance results from these operations are automatically captured and uploaded to the data storage system, such as the performance reports

\begin{wrapfigure}{R}{.7\linewidth}
    \centering
    \includegraphics[width=\linewidth,page=1]{img-ub-devops-hpcops.pdf}
    \caption{KUB standard DevOps}
    \label{fig:kub-devops}
\end{wrapfigure}
This framework leverages cutting-edge computational technologies, ensuring the high performance and accuracy of the Feel++ framework and Feel++-based applications.
It sets a benchmark for integrating modern software frameworks with advanced HPC infrastructure to significantly advance computational research and applications.
\Cref{fig:kub-devops} shows that the CI/CD standard DevOps framework for KUB uses similar steps.
\Cref{fig:kub-hpcops} finally shows the HPCOps deployed on EuroHPC systems to check the parallel properties and correctness of the pilot on large-scale cases. It extends Feel++ HPC ops with additional steps to handle our complex pre- and post-processing steps.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth,page=2]{img-ub-devops-hpcops.pdf}
    \caption{Ktirio Urban Building HPCOps workflow}
    \label{fig:kub-hpcops}
\end{figure}

\subsection{Benchmarking KUB}
\label{sec:benchmarking}

Finally, we display some results of our benchmarking activities~\cite{hidalgo2_d31_2024} for the KUB application regarding HPC performance.
Running these experiments regularly is essential for maintaining the program's efficiency when developing the application and after machine updates.
Hence, we have used the KUB application in the Strasbourg city center, with a square area of $4 \mathrm{km}^2$ (approximately 17K buildings).
The scalability results were realized on EuroHPC JU systems obtained with our HPCOps pipeline; see \cref{fig:kub-hpcops}.
\Cref{fig:scalability} depicts the speedup achieved on Discoverer, Karolina, and MeluXina.
They present results for the total execution (end-to-end) of the KUB application and the simulation component.

\begin{wrapfigure}{R}{0.6\textwidth}  % Adjust 'R' and width as needed
  \centering
  \begin{subfloat}[Scalability tests on a few EuroHPC systems from 1 to 32 nodes of 128 cores per node]{
    \includegraphics[width=0.9\linewidth]{img-compressed-kub-scalability.png}
    \label{fig:scalability}
  }\end{subfloat}
  \\
  \begin{subfloat}[Execution breakdown on a few EuroHPC systems from 1 to 32 nodes of 128 cores per node]{
    \includegraphics[width=0.9\linewidth]{img-compressed-kub-bencharkings-execution.png}
    \label{fig:execution-breakdown}
  }\end{subfloat}
  \caption{Detailed performance metrics for scalability and execution on EuroHPC systems}
  \label{fig:combined-metrics}
\end{wrapfigure}
The pipeline's simulation part scales almost linearly, which is expected as the buildings are not coupled together in the model currently used for the simulation. On the other hand, the total execution of the pilot's pipeline does not scale; as more nodes are employed, the performance degrades.

To better understand what causes this degradation, we measure the computing times of the different stages of the pipeline and present the execution breakdown in \cref{fig:execution-breakdown}. 
\cref{fig:execution-breakdown} reports the portion of the total execution taken by:
\begin{description}
\item[Pre-processing (Pre-proc)] The time elapsed in initialization before entering the time loop of the simulation
\item[Simulation (Simulation)] The cumulative time spent calculating the new solution at each time step of the simulation described in \cref{sec:heat-transfer-in-buildings}.
\item[Post-processing (Post-proc)] The cumulative time spent exporting results, i.e., generating files containing the output of the UB model.
\end{description}

In~\cref{fig:combined-metrics}, the total elapsed time varies from 100 seconds to 1000 seconds in function of the number of nodes.
All machines tested have an elapsed time of the same magnitude order.


Pre-processing does not scale efficiently, but it occupies only a small portion of the total execution time and is therefore not performance-critical.
As more nodes are employed and the actual simulation time decreases, the post-processing stage becomes the dominant component of execution, posing a potential bottleneck and causing the previously observed performance degradation.
This behavior is primarily due to multiple files being written in parallel to the shared file system, with a significant portion of time spent opening and closing files simultaneously.

However, our end-user approach focuses not on the complete solution fields, unlike this benchmark, but rather on outputs of interest and automatically generated analysis reports, which occupy only a small fraction of the typical post-processing time.
By targeting specific metrics and reports, we minimize the post-processing overhead and reduce data transfer and storage demands (see \cref{sec:final-analysis} for more information).

We are nevertheless actively exploring potential solutions to further streamline the processing of solution fields, such as asynchronous writes, data caching, and other strategies to mitigate the impact of file operations. 
As the project progresses, we anticipate that the urban building models used in the simulation will grow in complexity, leading to increased time spent in the simulation itself (\cref{sec:heat-transfer-in-buildings}) and subsequently reducing the relative impact of post-processing on the overall execution time of the pilot.


\section{Conclusion}
\label{sec:conclusion}

We are developing the computational Ktirio Urban Building(KUB) framework: assembling this very compelling application encompasses challenges in mathematics --- scalable modeling and simulation, large-scale watertight robust mesh generation, advanced analysis including data simulation --- and computer science --- scalable framework, software architecture, modern development, testing and packaging environment including standard DevOps and now HPCOps.---
The overall programming, integration, delivery, and deployment environment is critical to develop such an application. To our knowledge, this is the first application that can be automatically benchmarked and executed on EuroHPC JU systems thanks to CI/CD or HPCOps. Moreover, the workflow from Feel++ to KUB provides a considerable gain in terms of development and testing time, automated as much as possible, and enabling researchers and developers to focus their work better.

Our next steps include \textit{(i)} enabling tasks-based parallelism using the C++ framework Specx, see~\cite{cardosi_specx_2022}; \textit{(ii)} improving the modeling and simulation components, including mesh generation, handling of vegetation and urban furniture, and enabling view factors as well as providing a variety of configurable building energy models; \textit{(iii)} enhanced parallel strategies particularly in terms of partitioning and improved large scale I/O; and of course \textit{(iv)} pursue our benchmarking activities on EuroHPC JU supercomputers.


\begin{credits}
\subsubsection{\ackname} Funded by the  European  Union. This work has received funding from the European High-Performance Computing Joint Undertaking (EuroHPC JU)  and  Poland, Germany,  Spain,  Hungary,  France, and Greece under grant agreement number 101093457. This  publication  expresses  the  opinions  of  the  authors  and  not  necessarily those of the EuroHPC JU and Associated Countries which are not responsible for any use of the information contained in this publication

Part of this work was also funded by \textit{(i)} the France 2030 NumPEx Exa-MA (ANR-22-EXNU-0002) project managed by the French National Research Agency (ANR), \textit{(ii)} AMIES, the french agency for interaction between mathematics and enterprises and \textit{(iii)} CNRS through its prematuration programme.

We acknowledge the EuroHPC Joint Undertaking for awarding this project access through  EuroHPC Development Access grants EHPC-DEV-2024D05-025 and EHPC-DEV-2023D08-047 to the EuroHPC JU supercomputers : \textit{(i)} Kumi, hosted by CSC (Finland) and the Lumi consortium, \textit{(ii)} MeluXina hosted by LuxProvide, Luxembourg, \textit{(iii)} Karolina hosted by IT4Innovations National Supercomputing Center, Czechia, \textit{(iv)} Discoverer hosted by Sofia Tech Park, Bulgaria, \textit{(v)} Vega hosted by IZUM, Slovenia, and \textit{(vi)} Leonardo hosted by CINECA, Italy.


Finally the authors would like to acknowledge the many fruitful discussions with our partners Luc Kern from Synapse Concept and Leopold Fischer from Cisco Meraki, our colleagues \textit{(i)} from Hidalgo2 ICCS Kostis Nikas, Aristomenis Theodoridis and Petros Anastasiadis for the discussions on Reframe and joining their EuroHPC access grant, \textit{(ii)} Hidalgo2 HLRS Sameer Haroon for the discussions on CI/CD, \textit{(iii)} Pierre Alliez from INRIA Titane and Andreas Fabri from Geometry Factory regarding the discussions on CGAL and using Polygon Repair, and finally \textit{(iv)} our former colleague Zohra Djatouti, now at Kipsum, which whom we initiated this endeavor.

% \subsubsection{\discintname}
% It is now necessary to declare any competing interests or to specifically
% state that the authors have no competing interests. Please place the
% statement with a bold run-in heading in small font size beneath the
% (optional) acknowledgments\footnote{If EquinOCS, our proceedings submission
% system, is used, then the disclaimer can be provided directly in the system.},
% for example: The authors have no competing interests to declare that are
% relevant to the content of this article. Or: Author A has received research
% grants from Company W. Author B has received a speaker honorarium from
% Company X and owns stock in Company Y. Author C is a member of committee Z.
\end{credits}


\section*{Nomenclature}

\begin{multicols}{2}
$\alpha$ : Absorption of a material.\hfill[-]\\
$\alpha_{conv}$=10 : Heat conductance of convection.\hfill[-]\\
$\beta$ : Angle of inclination of the wall to the horizontal.\hfill[-]\\
$\Delta T$ : Temperature difference.\hfill[K]\\
$\epsilon$ : Emissivity of the surface.\hfill[-]\\
$\phi_{air}$ : Air heat flow.\hfill[W]\\
$\phi_{net}$ : Radiative heat flow.\hfill[W]\\
$\phi_{solar}$ : Solar heat flow from short wavelength.\hfill[W]\\
$\phi_{wall}$ : Heat flow through a wall.\hfill[W]\\
$\sigma$ : Stefan-Boltzmann constant $5,67\cdot10^{-8}$.\hfill[W$\cdot m^{-2}\cdot K^{-4}$]\\
$albedo$ : Reflective properties of the material.\hfill[-]\\
$A_z$ : Azimuth of the surface.\hfill[-]\\
$A_{zs}$ : Azimuth of the sun.\hfill[-]\\
$C$ : Heat capacity of the zone.\hfill[W$\cdot K^{-1}$]\\
$d_{inhabitant}(timetable)$ : Density of inhabitants as a function of the timetable set for the building type.\hfill[-]\\
$E_{dir}$ : Direct solar heat flow.\hfill[W$\cdot m^{-2}$]\\
$E_{diff}$ : Diffuse solar heat flow.\hfill[W$\cdot m^{-2}$]\\
$E_{ref}$ : Reflected solar heat flow.\hfill[W$\cdot m^{-2}$]\\
$F_{dir}$ : Direct solar radiation.\hfill[W$\cdot m^{-2}$]\\
$F_{dir,Hz}$ : Direct solar radiation on a horizontal surface given by climate data.\hfill[W$\cdot m^{-2}$]\\
$F_{ij}$ : View factor between $surface_{(i)}$ and $surface_{(j)}$.\hfill[-]\\
$H_s$ : Angular height of the sun.\hfill[-]\\
$J$ : Radiosity of a surface.\hfill[W$\cdot m^{-2}$]\\
$Met$ : Metabolic rate.\hfill[W]\\
$0.34\cdot Q_v$ : Airflow as a fraction of the zone's volume per hour multiplied by the air volumetric heat capacity.\hfill[W$\cdot K^{-1}$]\\
$S$ : Surface area.\hfill[m$^2$]\\
$S_C$ : Shading coefficient $\in$ [0,1].\hfill[-]\\
$U_j$ : Conductance of a $wall_j$.\hfill[W$\cdot m^{-2}\cdot K^{-1}$]\\
$V_{air}$ : Wind speed.\hfill[m$\cdot s^{-1}$]
\end{multicols}

\bibliographystyle{splncs04}
\bibliography{references}

\end{document}