A question about process and thread

[zuoyanzhang]

Hi, HPX Team, I want to know when and in what scenarios HPX will use processes and create processes, or whether it always uses lightweight threads. Thanks.

[Pansysk75]

Hello @zuoyanzhang,

Typically HPX spawns its worker threads (by default one per CPU core) when the HPX runtime is initialized. These are OS threads (ie "pthreads" on Linux) and their number stays constant for the duration of the program. The number of worker threads and how they correspond to CPU cores can be controlled by various command line options.

After that, no more OS threads or processes need to be created, as all parallelism and asynchrony within HPX are done using the HPX lightweight threads, which can be large in number, and are distributed among the existing OS worker threads. For example, parallel/asynchronous algorithms, hpx::thread, and hpx::futures all utilize lightweight threads.

In addition, a developer using HPX should use the hpx:: variants of anything that might involve asynchrony or threading. Using std:: constructs may spawn OS threads, which can negatively affect the performance of an HPX application.

Thanks for your question and let me know if anything is unclear

[zuoyanzhang]

Hi @Pansysk75 ,
The node creates an OS thread to execute the user thread of HPX. The cross-node task uses multiple processes. Am I right?

[Pansysk75]

The cross-node task uses multiple processes. Am I right?

Yes, multiple processes are needed for cross-node tasks, while a single multi-threaded process suffices for running on a single node.

A typical setup is having a single HPX process (albeit multi-threaded) on each node. This can suffice for utilizing all the node's computing resources. We refer to one such instance of the HPX runtime as a "locality".

One can also spawn multiple HPX processes/localities on the same node, but separate processes don't have shared memory, so usually it's best to have one process per node.

The creation of multiple HPX processes/localities is usually managed by SLURM or MPI (using srun, mpiexec etc.), and these localities communicate with each other to facilitate distributed work.