EPIC: Plan Development User Experience

[callumforrester]

This is an epic that can be used to create constituent issues. It will also be used to outline the rough plan for improving the UX.

Current State of Play and Problems

Currently we have a scratch area on /dls_sw where we check out the plans and devices so we can edit them on the fly. Blueapi mounts this area from its container and editably installs the checked out repositories. There are a few problems with this approach:

• Convention: You need to just know that the scratch area is required, where it should go and what repos should be in it
• Permissions: It is easy to run into filesystem permissions issues, such as someone owning a file you want to delete or the container wants to read
• Change management: It is easy to leave changes on the beamline uncommitted, there is a risk that they may be lost or not progressed, or that it is difficult to integrate new changes from upstream without handling them first
• Onboarding new people/Introducing scientists: This is a system with lots of moving parts and it is difficult to onboard people, a lot of it relies on procedure and it is complicated for occasional programmers.

Proposed System

We plan to introduce:

• Coder a.k.a. vscode in a browser, which provides a ubiquitous, easily managed development environment as well as quick-and-easy access to the right plans. Currently if someone wants to edit the data we have to tell them what to use and where to go on the filesystem. The goal here is that all they should need is a URL
• A lightweight git server for storing raw edits (options include gitolite and gogs). Ideally there will be a script or coder plugin that automatically commits and pushes changes. This reduces the risk of uncommitted changes being lost and increases the ability to rollback to a known good state; There is a record of every piece of code ever run by blueapi.
• Github as a mirror: Github should not be the source of plans during operational runtime, but we want the "known good state" of the beamline to be stored there because . There will be some automated process that opens a draft PR from the beamline and keeps it up-to-date. The idea is that the PR can be reviewed/improved by a DAQ engineer as well as providing a general view of the state of the beamline at-a-glance. DAQ's job is to continuously monitor/improve these PRs and keep main up-to-date with a nice version with a squashed commit history.
• An initContainer to actually set up the scratch area (clone repos etc.) rather than require a DAQ engineer to follow a long manual guide to installing blueapi on their beamline

Incremental Approach

The greatest value comes from introducing coder and the initContainer quickly because it removes manual processes and frees up developer time.

Moving away from the shared filesystem is the next logical step, removing the permissions problems we have been seeing on the beamlines.

After that, we introduce git more formally.

[callumforrester]

Issues documented here: https://github.com/DiamondLightSource/blueapi/milestone/18

[callumforrester]

One major concern here is security: We want to make sure that sensitive information isn't pushed to Github and that edits to the plans are tracked and controlled via auth/auth

[stan-dot]

gogs is much better - 44k stars vs 8k stars and in Golang vs Perl

update: it looks like we'd need to configure the k8s and helm ourselves (no official docs on this https://gogs.io/docs/installation/install_from_binary)

[stan-dot]

After that, we introduce git more formally.

an alternative would be something like gitdoc

re: security
why not just use gitlab instead?

[callumforrester]

@stan-dot Gitlab is an option, yes, I would like as many things as possible to be open source, but I accept that is mostly for ideological reasons.

[stan-dot]

I mean the private scripts of scientists they use to get their data is private and confidential before a paper is published AFAIK, so it's highly important that we make them feel secure about their experiment code.

[callumforrester]

Well then, the whole Hyperion project is a write-off already in that case, which it clearly isn't. You raise a good point though, I think I'm going to look into this.

[stan-dot]

Hyperion is more of an industrial project afaik

[stan-dot]

got some exploratory work towards using gogs in a gitlab repo plans-git-server-instance.