MaestroWF and Gitlab CI

[adrienbernede]

I work on RADIUSS, and to make is short, I would like to help sharing practices around CI/CD on LC systems. As such, I am exploring how MaestroWF could be used to manage a test workflow in Gitlab CI.

My (short) experience:

My issue was that Gitlab CI yaml pipelines sort of competes with MaestroWF.

It is totally possible to launch a MaestroWF in a gitlab pipeline, but in some way that’s ignoring Gitlab pipeline mechanism, since everything would end up in 1 Gitlab job, with limited feedback integration.

Also, MaestroWF is interactive by default (I had to use “yes |” ) and ends up without returning error will the study is running (Normal! But again, not directly usable in Gitlab).

Now the idea:

A recent feature in Gitlab is Parent/Child pipelines, and it comes with pipeline generation.

I think it would be awesome if we found a way to generate a Gitlab pipeline using MaestroWF, or at least from an existing MaestroWF pipeline.

This is a “short cut” formulation, but it’s based on the idea that teams wouldn’t have to re-define there testing workflow for every single tool.

[FrankD412]

@adrienbernede -- This sounds like a nifty idea. I'd have to explore GitLab pipelines a bit more, but it definitely would be cool to have a platform agnostic workflow that could just be ported over.

In regards to how you answered yes for the Maestro prompt, there is the -y under the maestro run sub-command that will honor return codes. The issue there is that the return code won't be returned to you since that would still be piped through nohup. However, you could still use the -fg to force the conductor in the foreground and have GitLab wait on it.

That still does not solve the fact that the steps don't map to pipeline steps, so definitely would be fun to explore that.

[adrienbernede]

Thanks for the tips. I probably checked maestro options and not maestro run ones.

[FrankD412]

No worries -- I figured I'd pass that info along. At least it'll make Maestro a little more useful in the meantime. :)

Do you happen to have a link to some good GitLab documentation on pipelines? I'm curious to read more.

[jsemler]

This is an interesting idea.

Here are the links to the documentation:

• GitLab CI
• GitLab CI Pipelines

[adrienbernede]

@jsemler Thanks for that, @FrankD412 sorry for not answering, I thought I had.

In practice, I use the pipeline reference page a lot. And do not hesitate to ask questions.

You can have a look to LLNL/Umpire and LLNL/Serac for some example of pipelines on LC systems. And also this PR on MFEM.