Dockerize dependency service

[humphd]

In #2797 we are adding a dependency service, and we need to dockerize it, and connect it to our Traefik API gateway for routing.

There are lots of examples of how to do this with other services.

[JerryHue]

The dependency-discovery service is a little bit special:

it requires a text file called deps.txt, which gets generated by toosl/pnpm-deps-gen.sh. However, to run this file you need:

• pnpm installed
• the whole Telescope project
• node_modules for each one of these projects

to run it successfully.

The way I am thinking to generate this is:

• copy the whole Telescope project into the Docker container
• run the script to generate the file in src/api/dependency-discovery

However, this might create a humongous container (if my understanding of how Docker works is right). There might be a better way to do this.

[humphd]

This is an interesting problem. I think we should solve it in a simple way to begin: run the command outside the Docker container and copy the deps.txt file in.

I'd run your script in our deploy.sh, maybe here https://github.com/Seneca-CDOT/telescope/blob/master/tools/autodeployment/deploy.sh#L12, and put the file in the right location so that it can be found when we build the dependency service image.

cc'ing @manekenpix and @raygervais in case they have a better solution.

[JerryHue]

That approach would be the most ideal, but sadly I think it only fits for deploying.

For development, we would have change other files to generate deps.txt prior to the docker building: maybe changing tools/services.js to call this script.

The only problem would be that you won't be able to build the image separately like in the other services.

[humphd]

Another option we could consider is adding this script to a preversion step, and store the file in git when we tag a new version. So we'd focus on deps for our latest deployment, but you can always re-generate it when we deploy. That way it would be in the tree, but also constantly updated, too.

I think that running this any more often than when we tag might be too much.

[JerryHue]

That sounds wonderful. I don't think the depedendency-discovery service has to keep track of the dependencies added in between release, just right before the release, so this fits perfectly.

[humphd]

So the only issue I see with this approach, is that your script depends on being run in a POSIX shell. If students are trying to do the release on Windows, it will fail.

We could run your script in our release workflow, but by then the git tag is already created (it's too late).

However, we could write a little container that copies your script and the lock file in, runs it, and spits out the result. This would let us run it on any system with docker.

Or we could rewrite your script in JS and make it portable.

Or we could get all our devs to switch to Linux/macOS :)

[JerryHue]

I think we could write a JS script so that it is portable.

When I was writing #2797, I wanted to write a JS script, but back then I was using pnpm list --depth=Infinity, so I couldn't place the whole thing into RAM. However, now that the depth is just 0, I think I can write this in a simple way without worrying about performance.

[manekenpix]

I thought about this a bit and I came up with a different approach.

We could try to take advantage of Satellite and the upcoming dockerfile template to achieve part of the work that needs to be done for this.

Consider this at a service level (it'd happen inside of every microservice):

• A different version of the pnpm-deps-gen.sh could run locally in every service at launch and collect information about its own dependencies. This could be a shell/js script, part of the build process in the dockerfile template, or an automated process embedded in Satellite.

• Once all the information about dependencies is collected, a predefined endpoint in Satellite would make this information available, and the dependency service could fetch from all the running services and combine all this data to be displayed in whatever way we choose. Another possibility would be to make every service send the dependency data directly into supabase, and in that case we might not even need a dependency service.

The advantages of this approach:

• We won't depend on the autodeployment server to run the pnpm-dep-gen script for us (which would also be difficult to test for developers). If we end up moving telescope to another cloud, we might not need our autodeployment server, and we'd have to rething and reimplement this feature.

• Every newly created microservice will come with this feature out of the box, and minimal or no changes will have to be made to include its dependencies in our data.

I'd like to hear opinions, ideas, pros, cons, dreams, nightmares... about this approach.

[humphd]

The only issue I can see is that not all aspects of our system are based on Satellite: web and docs, for example. We'd have to figure out how those would get added.

[JerryHue]

• We won't depend on the autodeployment server to run the pnpm-dep-gen script for us (which would also be difficult to test for developers).

The way we decided while writing #2838 was a script that gets run before the pnpm run version command. So, we wouldn't really have to depend on the autodeployment tool to run the script.

However, the idea itself does sound promising. One genuine concern is the one that @humphd mentions: the front-end and the docs are not really based off of Satellite, and I am not sure how easy it would be to make them to include it, especially the docs. There is also the dependencies listed in the root package.json: since that package.json belongs to the monorepo and not to a service itself, we would skip all of the dependencies shown there (major ones that we use such as prettier, eslint, turbo, etc.). However, I think these dependencies may be installed on every microservice, so I suppose it is not that big of a problem.

Either way, I cannot think of many "cons", and the "pros" are very appealing. I am open to this kind of idea and I would prefer to discuss it more during next week and in the meetings. Thank you @manekenpix!