Possibly enabling oversubscribe/cpu-pinning in framework

[Crivella]

Putting here some thoughts following the work I did on easybuilders/easybuild-easyblocks#3917

I think we in general do not want mpi related commands to fail due to lack of resources.

EG: it is already the case that many EC files sets environment variables to allow oversubscription of resources at steps that invoke mpirun or similar.

I've given this some thoughts and i feel like implementing this would require 2 decision:

• Oversubscription vs(or in conjunction) to cpu-pinning
• Have this behavior on by default or on-demand

Oversubscription vs(or in conjunction) to cpu-pinning

I think both would have their pros and cons.
Oversubscription would allow us to defer choosing the cores to the resource manager but it might not be possible to enforce max_parallel
For example in HPL the test will not run with less than 4 processes.
Let's say we set max_parallel = 2 and allow for oversubscription. If the system running EB has more than 2 CPU running

mpirun -n 4 --map-by :oversubscribe ...

will actually end up using more cpus than what was requested with max_parallel

I am still exploring the capabilities of --map-by and --hostfiles for OpenMPI, but i am not sure if it is possible to tell mpirun to never use more than a specific number of processors even if more are available without explicitly pinning the cores

On the other hand we could do something similar to what i did in easybuilders/easybuild-easyblocks#3917 so if one of the following is true we default to pinning

• mpi is not available
• self.cfg.parallel is lower than a specified requested value

There are a few cons here:

• We have to know how CPUs are numbered. I am assuming we defer to hwloc so we might always be able to do a bind-to core and than give a sequential number (EG req=7 max_parallel = 3 --> --cpu-set 0,1,2,0,1,2,0) (in the HPL PR at the time of writing this i am binding all processes to 0 as i am not sure this is always reliable)
• We risk binding to a CPU that is already in use by the machine instead of one that is currently free

I will try to investigate this more and try to come up with (hopefully not more than 2) viable implementation for this, but if there is someone more experienced that could give some ideas/feedback it would be really appreaciated

Where could this be implemented

• If we decide this behavior should always be on this could be done at TC loading by setting the appropriate env variables depending on the MPI family.
• An alternative could be to modify https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/tools/toolchain/mpi.py#L273 to add an extra parameter oversubscribe, which means we would than need to properly enforce that every MPI command is generated through this helper function

[Crivella]

Attempts with system OpenMPI outside of using mpirun specific settings

crivella@crivella-desktop:~/test/mpirun$ mpirun -V
mpirun (Open MPI) 4.1.2

hwloc-bind

crivella@crivella-desktop:~/test/mpirun$ hwloc-bind --cpubind --strict --single --  mpirun --report-bindings --map-by core:oversubscribe --bind-to core:overload-allowed -np 4 echo -n ""
[crivella-desktop:753241] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../.././././././././././././././././.]
[crivella-desktop:753241] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]: [../BB/../../../../../.././././././././././././././././.]
[crivella-desktop:753241] MCW rank 2 bound to socket 0[core 2[hwt 0-1]]: [../../BB/../../../../.././././././././././././././././.]
[crivella-desktop:753241] MCW rank 3 bound to socket 0[core 3[hwt 0-1]]: [../../../BB/../../../.././././././././././././././././.]

taskset

crivella@crivella-desktop:~/test/mpirun$ taskset -c 0,1 mpirun --report-bindings --map-by core:oversubscribe --bind-to core:overload-allowed -np 4 echo -n ""
[crivella-desktop:753799] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../.././././././././././././././././.]
[crivella-desktop:753799] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]: [../BB/../../../../../.././././././././././././././././.]
[crivella-desktop:753799] MCW rank 2 bound to socket 0[core 2[hwt 0-1]]: [../../BB/../../../../.././././././././././././././././.]
[crivella-desktop:753799] MCW rank 3 bound to socket 0[core 3[hwt 0-1]]: [../../../BB/../../../.././././././././././././././././.]

cgroups/cpuset

Not tested yet. I think it could work but it would rely on having the respective capability available on the system

[Crivella]

Considering the discussion in https://github.com/easybuilders/easybuild/wiki/Conference-call-notes-20250910, since it is still impossible for EB to perfectly enforce that more CPUs than what has been requested with max_parallel are used (we could do it for EB stuff but not for all the commands that are called during every build), probably relying on oversubscription only without pinning should be enough.

I think the best place for this would be at the TC level with a toolchainopt flag

[ocaisa]

I think the right place for that is in

easybuild-framework/easybuild/tools/toolchain/mpi.py

Line 273 in 73f25e3

def mpi_cmd_for(self, cmd, nr_ranks):

with an oversubscribe key (True by default?) and then pass that through to

easybuild-framework/easybuild/tools/toolchain/mpi.py

Line 51 in 73f25e3

def get_mpi_cmd_template(mpi_family, params, mpi_version=None):

where you precede the command with the appropriate environment variable for the MPI type (or add the relevant option to the command).

For MPICH I can't find much information on how that is enabled (LLM says that is there by default and you should disable binding with export MPICH_CPU_BIND=none), for the others:

Intel MPI: I_MPI_ALLOW_OVERSUBSCRIPTION=1
Open MPI: OMPI_MCA_rmaps_base_oversubscribe=1

[ocaisa]

Putting it in the template, allows for people to totally customise the behaviour (as they can create their own template from scratch)

[Crivella]

Putting it in the template, allows for people to totally customize the behavior (as they can create their own template from scratch)

I think mpi_cmd_template would not currently allow to customize this unless we redefine the base template to have an %(oversubscribe) template variable which might be not very user friendly in case it could be both an environment variable or a command parameter

For sure we can set the appropriate environment (possibly modifying an existing one) or pass a comand parameter through mpi_cmd_for + an extra oversubscribe optional variable

PS

OMPI_MCA_rmaps_base_oversubscribe=1

Will also needs the PRTE variant for OpenMPI >= 5
https://docs.open-mpi.org/en/main/mca.html#mca-parameter-changes-between-open-mpi-4-x-and-5-x

[ocaisa]

Putting it in the template, allows for people to totally customize the behavior (as they can create their own template from scratch)

I think mpi_cmd_template would not currently allow to customize this unless we redefine the base template to have an %(oversubscribe) template variable which might be not very user friendly in case it could be both an environment variable or a command parameter

If you were in a position where you need to define your own template, I would suggest that they bake in the oversubscription rather than have it configurable. The templates are for things that are universal (like the cmd and the nr_ranks), it doesn't really work when it comes to something that may even be specific to the system.

PS

OMPI_MCA_rmaps_base_oversubscribe=1

Will also needs the PRTE variant for OpenMPI >= 5 https://docs.open-mpi.org/en/main/mca.html#mca-parameter-changes-between-open-mpi-4-x-and-5-x

That's interesting, I suspect this will break a few things. There is a version parameter to get_mpi_cmd_template so we should be able to handle that.

[Crivella]

For MPICH I can't find much information on how that is enabled (LLM says that is there by default and you should disable binding with export MPICH_CPU_BIND=none), for the others:

I would say yes oversubscription seems to be on by default, but binding is not and i am not sure it is related to oversubscription regardless
It seems mpirun from MPICH allows to do mpirun -OPITON_NAME --help to get more details so i grepped the binary itself for stuff like over and subscribe without finding details.
Also not sure MPICH_CPU_BIND is actually used by mpich

For now I would probably treat it as on by default and do nothing in that case

[Crivella]

It seems that also intel oversubscribes by default.
I was actually not able to find a way to turn it off, unless there is some combo of the pinning env variables that replace the OpenMPI version of bind/map/cpuset that I am missing