Define plugin testing process

[skabashnyuk]

Description

The issue is we need to provide a place to host meta.yaml and plugin.tar.gz that is under development.

Proposed process.

Define http-server plugin as a sidecar. Define volume /project to this sidecar.
From Environment variable we will configre a root folder to serve for example HTTP_SERVE_FOLDER=web.
During plugin development, a user will have to put his meta.yaml and plugin.tar.gz under /project/web folder
To be able to specify plugin served by this server we need to fix this Add an ability to use a custom registry for WS.NEXT plugins #11265

[garagatyi]

Looks good

[l0rd]

@skabashnyuk

• When you say Define http-server plugin as a sidecar you mean Add a che-plugin-registry as a workspace plugin? Or Add an http-server as a workspace plugin?
• Define volume /project you mean Mount folder /projects as a volume?

[l0rd]

The problem with having an http-server or che-plugin-registry in a workspace is that it won't work if we need to restart the workspace. It should be a separate pod.

[skabashnyuk]

When you say Define http-server plugin as a sidecar you mean Add a che-plugin-server as a workspace plugin? Or Add an http-server as a workspace plugin?

Not sure I understand a difference. I meant https://www.npmjs.com/package/http-server as a server-side Che-plugin

Define volume /project you mean Mount folder /projects as a volume?

yes

The problem with having an http-server or che-plugin-registry in a workspace is that it won't work if we need to restart the workspace. It should be a separate pod.

What for? I think if it's not related to workspace it's closer to prod mode then dev.

[garagatyi]

The problem with having an http-server or che-plugin-registry in a workspace is that it won't work if we need to restart the workspace. It should be a separate pod.

I forgot my own argument) Mario is correct! Since broker is working before workspace is up we can not use registry as part of the workspace.

[skabashnyuk]

@l0rd @garagatyi

The problem with having an http-server or che-plugin-registry in a workspace is that it won't work if we need to restart the workspace. It should be a separate pod.

Restart is not a problem since all data will be saved on the /project volume.
Or may be you want to consider availability of the data when workspace are STOPPED?

[skabashnyuk]

Small addition after a talk with @garagatyi . I thought that the newly developed plugin will be tested as part of a separate workspace. Where in workspace configuration we will define a reference to resources served by http-server sidecar of the first workspace.

[garagatyi]

If we put a meta.yaml file to the plugins or projects volume we can implement fetching the stuff from it in the broker.

[l0rd]

I would avoid using volumes among pods as much as possible for the well known problems we have on real clusters.

• The tar.gz can be copied with oc cp from the dev container to the httpd server container
• meta.yaml data can be provided as a configmap to the plugin-registry deployment config and restarted: when the che-plugin-registry starts it looks at the environment variables and if it founds a variable CHE_PLUGIN_DATA_<plugin_id>="id: \nversion: ... it parses it and create/update the plugin folder.

[skabashnyuk]

I think we need to make a step back and talk about prerequisites.

1. Do we want to have a generic workflow or just something quick and dirty just to see how it works?
2. Do we want to have something similar for OpenShift, kubernetes, docker?
3. Do we want to have this process limited to only one workspace or we can use two. I think
   might be some corner cases then developing and testing in the same workspace can be uncomfortable/impossible.
4. Request not to use volumes is nice to have or must have? A broker is using it, so why don't we use the same strategy. For instance, I like @garagatyi 's idea Define plugin testing process #11555 (comment) to read meta.yaml and plugins from /projects volume. We can define it something like this

  "attributes": {
    "editor": "volume:/projects/org.eclipse.che.editor.theia/1.0.0/meta.yaml",
    "plugins": "che-dummy-plugin:0.0.1,che-machine-exec-plugin:0.0.1"
  }

[l0rd]

The problem of using volumes to share the meta.yaml among pods is that it doesn't work in ephemeral mode. In ephemeral mode we use EmptyDir volumes and these are not shared between 2 pods. Look at this issue with the plugin broker volume.

Currently ephemeral mode looks like the best way to run Che on public OpenShift/Kube clusters like OSO. That's why I am considering it important. Even more important than thinking about Docker support.

I find @garagatyi proposal brilliant but it should work on ephemeral mode too. And I don't know how to do it. I hope that this answers some of your questions.

[skabashnyuk]

There is no ideal solution at this moment. We would like to take into account the technical difficulties and limitations that introduced by using PV. That is why for Workshop we would like to develop a set of "oc" command that will allow.

• Start some server in user namespace to host plugin metadata and plugin content
• Copy metadata and plugin content from user workspace to this servers.
• Metadata should be accessible for ChePlugin broker.

[benoitf]

is that this feature will be available on che.openshift.io on Tuesday next week ?

[skabashnyuk]

Yes. We are trying to do that. @mshaposhnik is working on it.

[benoitf]

ok thanks for the feedback

[mshaposhnik]

It seems I have some networking troubles on this. What i do:

1. Login:

oc login ${CHE_OSO_CLUSTER} --insecure-skip-tls-verify=${CHE_OSO_TRUST_CERTS} --token=${CHE_OSO_USER_TOKEN} && oc project ${CHE_OSO_PROJECT}

2. Run registry:

oc new-app eclipse/che-plugin-registry

3 . Check service:

oc describe svc che-plugin-registry

Output:

Name:              che-plugin-registry
Namespace:         mshaposh
Labels:            app=che-plugin-registry
Annotations:       openshift.io/generated-by=OpenShiftNewApp
Selector:          app=che-plugin-registry,deploymentconfig=che-plugin-registry
Type:              ClusterIP
IP:                172.30.188.176
Port:              8080-tcp  8080/TCP
TargetPort:        8080/TCP
Endpoints:         10.129.95.30:8080
Port:              8443-tcp  8443/TCP
TargetPort:        8443/TCP
Endpoints:         10.129.95.30:8443
Session Affinity:  None
Events:            <none>

And i cannot get access using curl to any of 10.129.95.30:8080 or 172.30.188.176 etc.
In the same time, if i create route, i can easily access the service via URL looking like
https://che-plugin-registry-mshaposh.8a09.starter-us-east-2.openshiftapps.com/. This is proofs that service is up & running, but this is useless since we want to copy files between local nodes and not use routes. Any thoughts?

[skabashnyuk]

Use oc copy to copy files between workspace and che-plugin-registry

# Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace <some-namespace>
  oc cp /tmp/foo <some-namespace>/<some-pod>:/tmp/bar

from oc cp --help

[mshaposhnik]

To make oc cp, we need a folder to be already present. To run oc exec mkdir .... seems we need chmod the /var/www/html/plugins in the image

[garagatyi]

As far as I remember k8s services are isolated in per-namepsace. So it might be expected that you can't access a service from another namespace.

[mshaposhnik]

With light modification in registry dockerfile (add chmod for plugins folder), i was able to hot-deploy the plugin yaml.
Sequence:

1. Login (as above)
2. Create service (as above)
3. Create route oc create route edge --service=che-plugin-registry
4. Get route path oc get route
5. Get pod name oc get pod
6. Create subfolder for plugin oc exec che-plugin-registry-1-qqzzs -- mkdir -p /var/www/html/plugins/che-makis-plugin/0.0.1
7. Copy meta.yaml from project
   oc cp ./blankproj/meta.yaml che-plugin-registry-1-qqzzs:/var/www/html/plugins/che-makis-plugin/0.0.1

Result: yaml is accessible at
https://che-plugin-registry-mshaposh.8a09.starter-us-east-2.openshiftapps.com/plugins/che-makis-plugin/0.0.1/meta.yaml

[mshaposhnik]

last 2 commands can be simplified into one (implying all folders for plugin exists in project):
oc cp ./blankproj/plugin-makis-makis/ che-plugin-registry-1-qqzzs:/var/www/html/plugins/

[l0rd]

The goal of this issue shoudl be to add a script (publish.sh for example) in the che-plugin-dev-tooling docker image (https://github.com/ws-skeleton/che-plugin-dev-tooling) that:

• checks if the plugin registry has been started, otherwise start it.
• retrieves the plugin registry pod name and URL
• copies the .theia or .tar.gz file of the plugin registry
• generates the meta.yaml file with the URL of .theia or .tar.gz files
• copies meta.yaml file in the plugin registry right folder

The examples of plugins that you can use for testing are https://github.com/ws-skeleton/che-dummy-plugin (for a .theia frontend plugin) and https://github.com/ws-skeleton/che-service-plugin/ (for a tar.gz che plugin).

Note that publish.sh should be run after the user has build the plugin (.theia and .tar.gz files should have been already generated).

[mshaposhnik]

So appropriate script was added into dev-tooling image, under path /home/user/publish.sh. It accepts plugin name and version as parameters. Script installs a local registry and performs deploy of plugin binary &meta on it. Currenlty only .tar.gz binaries are supported.
Short demo is at: https://www.youtube.com/watch?v=-EoFKr9sLkg