scr: add init/finalize and start/complete output

[adammoody]

@franzpoeschel, @psychocoderHPC

This is getting ahead of things, since it's not yet apparent that SCR would be needed/useful. However, I had started the integration work just in case in might be. I wanted to open this PR, since I thought it'd be easier to discuss things than an email thread.

First, I've got a PR open that adds SCR calls into ADIOS2. That includes the commands to download and build ADIOS2 with SCR. I need to work with the ADIOS team to iron things out there, but it does work well enough that we can do testing.

ornladios/ADIOS2#3294

I think there are also spots in PiConGPU where we'll want to add some SCR calls.

First, we'll want to call SCR_Init and SCR_Finalize. These calls typically are placed near MPI_Init() and MPI_Finalize(), respectively. Really, I often end up placing SCR_Init after any application option processing that the user might specify. It's common to configure SCR according to user settings, like the output directory for checkpoints or to allow the user to name a particular checkpoint to read during restart. Most of those settings need to be applied before one calls SCR_Init. For now, I've just added it near the MPI_Init().

Second, we need to add SCR_Start_output and SCR_Complete_output calls. These two calls should bookend the checkpoint logic such that all files belonging to a given checkpoint are created and written in between those two calls. The SCR_Start_output call also takes a name that is meaningful to the application/user, and some bit flags that the application would set.

All four of these calls are implicitly collective over MPI_COMM_WORLD.

Are there good spots that you'd recommend for adding these?

I've found a few potential spots that look interesting, but I don't know whether I'm headed down the right path.

If you have time, would you please take a look and let me know?

Thanks!

[franzpoeschel]

Do I understand it correctly that this PR and the PR at ADIOS2 would both be necessary? So we would still write our IO/checkpointing logic in terms of openPMD/ADIOS2, but use SCR-enabled ADIOS2? And these changes inside PIConGPU are then necessary to have SCR running in the background?

I can have a look at the best place to put these calls inside the openPMD plugin, for the rest of PIConGPU René will be the better person to say this.

[franzpoeschel]

See adammoody#1
I moved the lines inside the openPMD plugin to where I think you intended for them to go logically.

(Note: I'm on holidays for the coming two weeks)

[adammoody]

Do I understand it correctly that this PR and the PR at ADIOS2 would both be necessary? So we would still write our IO/checkpointing logic in terms of openPMD/ADIOS2, but use SCR-enabled ADIOS2? And these changes inside PIConGPU are then necessary to have SCR running in the background?

I can have a look at the best place to put these calls inside the openPMD plugin, for the rest of PIConGPU René will be the better person to say this.

Thanks @franzpoeschel !

Correct. Both PRs are necessary, and this PR has a dependency on having an SCR-enabled ADIOS2 to work properly. The existing SCR API is such that some calls go in ADIOS2 and some go in openPMD/PIConGPU. The SCR calls in PIConGPU are there mainly to distinguish which datasets are checkpoints and which datasets are not checkpoints. The SCR calls that are added to ADIOS provide it with the file system path to open each physical file that ADIOS creates to store the dataset. This might be a temporary path like a node-local SSD. SCR will then migrate the dataset from that temporary path to the parallel file system.

[psychocoderHPC]

Thanks for your PR. I changed the destination branch to dev (our branch to push new developments too) and set the PR into draft mode.

[psychocoderHPC]

All four of these calls are implicitly collective over MPI_COMM_WORLD.
Are there good spots that you'd recommend for adding these?

These are the places where we call MPI_Init and MPI_Finalize

picongpu/include/pmacc/Environment.hpp

Lines 416 to 438 in 9b83a0e

	void EnvironmentContext::init()
	{
	m_isMpiInitialized = true;
	// MPI_Init with NULL is allowed since MPI 2.0
	MPI_CHECK(MPI_Init(nullptr, nullptr));
	}
	void EnvironmentContext::finalize()
	{
	if(m_isMpiInitialized)
	{
	pmacc::Environment<>::get().Manager().waitForAllTasks();
	// Required by scorep for flushing the buffers
	cuplaDeviceSynchronize();
	m_isMpiInitialized = false;
	/* Free the MPI context.
	* The gpu context is freed by the `StreamController`, because
	* MPI and CUDA are independent.
	*/
	MPI_CHECK(MPI_Finalize());
	}
	}

output

To mark output of our plugins this should be the right place. We have different output "plugins" all these will be called from this lines:

picongpu/include/pmacc/simulationControl/SimulationHelper.hpp

Line 298 in 9b83a0e

dumpTimes(tSimCalculation, tRound, roundAvg, currentStep);

[chillenzer]

What's the status of this, @franzpoeschel, @adammoody, @psychocoderHPC?

[psychocoderHPC]

I like the way how SCR describs which data must be stored in a checkpoint. Never the less there is no time from our side to try SCR :-(