Tidy up where files come from

[gilesknap]

@coretl @GDYendell

So I'm finding it confusing that the files that are crucial to getting a fully working IOC instance are coming from a variety of places.

• ibek itself
• scripts and templates from epics-base
• scripts and templates from ioc-XXX
• YAML and helm from beamline repos
• more helm from the helm library
• things from ibek-support
• CI scripts in all of the repos involved

When you are in a container and you find you have the wrong version of something or you want to reset something it may not be easy or clear where it comes from. I say this as the primary author of this framework! Goodness knows what a neophyte containers engineer will make of it all.

Some ideas for things that can be embedded in ibek itself are listed below. These would then always be a consistent set. (adding to this as I think of things)

• global configure/RELEASE initial state epics-base
• Makefile Jinja template epics-base
• boilerplate helm chart for beamline/domain repos blxxy
• boilerplate ioc folder structure epics-base
• these could become functions of ibek
  • start.sh, liveness.sh stop.sh ioc-xxx
  • script for preparing the runtime target by copying module output files only epics-base
  • many of things that we do in CI (because this repeats knowledge about URLS and file locations etc.) all over the place

Now you might say this is hiding many things inside a python module. At present the scripts can be inspected and possibly customised. I think good customisation hooks would mitigate this.

But the benefit is you should get a consistent, working set of things versioned along with ibek itself.

It would give us clear(ish) separation of responsibility as follows:

• epics-base is just a compilation of epics in a container: it will need to set a bunch of env vars to say where things are (/epics/epics-base /epics/support etc.) but that is all and hopefully only ibek needs to consume those env vars
• ibek manifests all the rest of the things that enable epics-containers framework, EXCEPT:-
  • beamline repos supply a list of IOC.YAML only , uses ibek to publish them to registry in CI
  • ioc-XXX repos list a sequence of support modules to install uses ibek/ibek-support to compile them (mostly done already)
  • ibek-support provides
    • means to compile the support modules above
    • means to instantiate the IOC.YAML from beamline repos
  • helm chart library contains the common functions in the helm charts because its helm (but you could even embed that in ibek if you wanted to go the whole way)

[gilesknap]

Maybe on reflection we should keep ibek only for inside container things. So that eliminates the CI bits and helm.

Perhaps we get ec to do some of the CI work though.

I'm also pretty sure we will find a few areas of overlap of configuration inside and outside the containers.

[gilesknap]

But another possibility is we go all in on ibek and accept outside of container into it - roll in ec, helm and CI things.

[gilesknap]

Here are some proposed changes which tighten things up.
Its all ibek inside and ec outside.

1. Remove all IOC things from epics-base. It still creates /epics/support and builds sncseq in there.
   • base still sets up the following which are used by ibek
   • TARGET_ARCHITECTURE, EPICS_HOST_ARCH, EPICS_BASE path, SUPPORT path, IOC path, python venv which is in the PATH, EPICS_VERSION, VIRTUAL_ENV path, installs ibek. ioc-xxx may update ibek to a new version if needed and we only need to rebuild base if the features it uses in ibek change (ibek ibek ioc copy-to-runtime is the only one so far)
2. Make the IOC template in ioc-XXX using new command ibek ioc generate-source which will do nothing if you have checked in your /ioc folder - therefore allowing manual custom override.
3. Above to include the Makefile Jinja
4. ibek support register or ibek support add-release-macro will create a new /epics/source/configure/RELEASE global file if it does not yet exist. Again if this already exists then that overrides ibeks built in template.
5. new command ibek ioc copy-to-runtime will do the runtime prep stage in the dockerfile. Will try to reduce this down to two steps - one call to ibek in the prep stage and a single COPY in the runtime stage - to be configurable to bring in runtime libraries that the developer stage has compiled for example.
6. new command ibek startup launch will perform the function of start.sh (looking in config folder and determining how to start the IOC instance)
7. new command ibek runtime check-live will verify that the IOC is running (BUT this involves adding channel access to ibek)
8. new command ibek runtime stop will attempt a clean shutdown of the IOC
9. review the ec command namespace and use ec in
   • CI at least for beamline repos but maybe others (will have a think about this)
   • e.g. ec helm create-ioc will make the boilerplate helm chart from a 'config' folder - but again you can check in the boiler plate and manually customise to your needs.

[gilesknap]

ALSO: we currently mount the ioc-adaravis repo at /workspaces/ioc-adaravis in the devcontainer which is the default for devcontainers.

This is annoying for the scirpts and Dockerfile because the location of the source and in particular the ibek-support sub module varies from one GIOC to the next.

I propose that we always mount to:

• this_repo_root:/ioc-src
• this_repo_root/ioc:/epics/ioc

WIth the changes in the above comment epics/ioc would then be a compilable thing that you can fix/run and have your changes propgated back to the repo automnatically.

[gilesknap]

@coretl I propose that we change the ibek 'startup' command namespace to 'runtime'

• runtime generate
• runtime launch
• runtime check-live
• runtime stop

[coretl]

sounds fine

[gilesknap]

PROGRESS pm ibek change:

• Remove all IOC things from epics-base
• ibek ioc make-source-template
• create a new /epics/source/configure/RELEASE
• ibek ioc extract-runtime-assets
• ibek runtime launch
• ibek runtime check-live
• ibek runtime stop
• fix mount locations to simplify runtime / devcontainer compatibility

[gilesknap]

I've addressed all of the above in #115.

Note that start.sh, stop.sh and check-live.sh are still scripts. But they now live as templates inside ibek at
https://github.com/epics-containers/ibek/tree/dev/src/ibek/templates/ioc

Looking at them I'm not sure of the benefit of converting to python. But this is an easy thing to add later if we think it makes sense.

[gilesknap]

ibek is done.

NEXT: sort out ec CLI and CI. Leaving this open to review if runtime launch / stop / check-live should get converted to python. This is because these files are a link between outside and inside of container.

[gilesknap]

I'm pretty happy with ec as well now.

Final tidy up is:

• CI framework for process.run based commands in ibek
• as above for ec (all the commands in ec in other words)

[gilesknap]

EC CI is done and its a reasonable framework for testing process.run functions via YAML!

TODO - add the same framework for testing the ibek process.run based commands.

[gilesknap]

broken the last remaining issue in here out into #137