Plugins devcontainer

[usingtechnology]

Add a devcontainer for running/debugging plugin source code.

To begin the discussion for devcontainers...

This is a Proof of Concept for running/debugging local plugin source code. I elected to place this in the /plugins folder as there are .devcontainer and .vscode directories in the root. Do we need those?

To test this code out, see if it works for you.

1. open VS code
2. Open Folder... navigate to /plugins
3. If prompted to "Reopen in Container", do so...
4. Else open your command pallette and select "Dev Containers: Open Folder in Container..."
5. This should begin building the container.

This will add docker-in-docker (self contained docker environment in the devcontainer) and will install poetry. I have elected to pip install to the default python for the devcontainer. We should do something to make it easier to keep devcontainer in sync with poetry. (The editor windows won't use the poetry virtual environment so it can't find the imports when you are editing the files...)

Included a Run/Debug (that uses poetry!) to start up acapy with the plugins loaded. I just created specific config and compose files for the devcontainer until we decide the path forward. The compose only starts db and proxy.

To test it all out, add a breakpoint to oca_record_list method in traction_innkeeper/v1_0/oca/routes.py, open Run/Debug view, select "Run/Debug Plugin" and start/F5.

This will kick off the docker compose build/up to start db and proxy, then will run acapy + plugins from source.

Open browser to get_oca, Try it out, execute and you should hit your breakpoint in your IDE.

Once we get this stuff squared away, I think we can remove the top level poetry and change the docker images. The image that should be used for "production" can pull the plugins from github (use the PR sha or a tag). It would be nice to get a single devcontainer config that will load up (and debug?) tenant UI as well. But start with the plugins...

[WadeBarnes]

I think it's best to keep any devContainer for the project at the root level that is suitable for working on the project as a whole. It looks like @amanji added a docker-outside-of-docker container for frontend development there.

Just a note on the different docker options. docker-in-docker provides an isolated docker environment for development. docker-outside-of-docker provides a connection to the host's docker environemnt to provide a shared docker environment. Which you use depends whether you need to interact with external services also running in docker.

[swcurran]

Just a note on the different docker options. docker-in-docker provides an isolated docker environment for development. docker-outside-of-docker provides a connection to the host's docker environemnt to provide a shared docker environment. Which you use depends whether you need to interact with external services also running in docker.

Is the tradeoff as easy as that - pick one or the other? If so, I would say go with docker-outside so that docker-based external services can be easily used — von-network, redid, tails server, even a database as needed. Keep those independent and just have them running, and stop/start Traction as needed. External versions of those services couyld be used, but the mixing and matching might be tough?

Definitely think the two instances of DevContainers need to be rationalized.

[usingtechnology]

yes, devcontainer should be at the top, this was mostly a PoC and then rationalize the other containers/vscode (not sure if they are used?)

i can experiment with docker outside, i do run external docker services (von-network) and use that inside the devcontainer (just need to use host.docker.internal).

[loneil]

The existing devContainer is just for Node for the TenantUI. Really that's just a standalone application that points at a traction instance in it's configuration.
We of course start it up with Traction as part of the whole ecosystem that runs up when you start up the docker compose. But it's its own stack and separate development from the plugins themselves so I don't know what's best practice for combining all those things or not when they're all in one repo.

Really the Tenant UI could be it's own repository without much difference.

I don't use the devcontainer for TenantUI dev (have the version of node I need locally right now) but it's a good practice to have around so people don't need to have specific Node versions locally to dev. Can do whatever is needed to try out the plugin devcontainer here.

[github-actions]

Deployment	Resource	Location
Traction
	Tenant UI	https://pr-684-tenant-ui-dev.apps.silver.devops.gov.bc.ca
	Innkeeper UI	https://pr-684-tenant-ui-dev.apps.silver.devops.gov.bc.ca/innkeeper
Aca-Py
	Admin Swagger	https://pr-684-traction-acapy-admin-dev.apps.silver.devops.gov.bc.ca/api/doc
	Tenant Proxy Swagger	https://pr-684-traction-tenant-proxy-dev.apps.silver.devops.gov.bc.ca/api/doc

Deployment URLs ready for review.

[loneil]

Updated docker compose startup to use askar-only ghcr.io/hyperledger/aries-cloudagent-python:py3.9-0.8.2 image as well

[loneil]

Updating progress on this ticket. Will keep looking into but would be good to see if anyone else on this PR has feedback.
There are a number of fronts here:

Update ACA-Py to 0.8.2, non-Indy image

Local Docker testing

Can start fresh and build/up the compose ecosystem from /scripts
Can do full Traction regression (tenant create, schema/cred create, connect to BC Wallet, issue, revoke, message)
So everything appears all good there

Pull request deployment

I've updated values needed in the helm vals for this and got it to deploy.
I can create a Tenant, make it an issuer, connect to a wallet (or other tenant), make schemas/creds
However, if I issue a cred, then accept it (I am getting the offer pop up in the wallet and can accept it), it is always ending up in abandoned state. I'm not sure if this is related to endorser or something?

Tenant here:
4b750fc9-70c1-4345-afcc-01d1e4d8aa3c
3ee16e18-c16f-4879-9e71-eef0f1d2a1ad

Dev container

Same results below on Windows and with WSL (ubuntu)

Can seem to build up everyhting when starting in container, but on Run/Debug it crashes out with a DB password error

2023-07-10 21:56:38,447 aries_cloudagent.core.profile INFO Create profile manager: askar
2023-07-10 21:56:38,534 aries_cloudagent.commands.start ERROR Exception during startup:
Traceback (most recent call last):
  File "/workspaces/traction/plugins/.venv/lib/python3.9/site-packages/aries_cloudagent/askar/store.py", line 157, in open_store
    store = await Store.open(
…
aries_askar.error.AskarError: Backend error
Caused by: error returned from database: password authentication failed for user "postgres"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspaces/traction/plugins/.venv/lib/python3.9/site-packages/aries_cloudagent/commands/start.py", line 72, in init
…
    opened = await store_config.open_store(provision=False)
  File "/workspaces/traction/plugins/.venv/lib/python3.9/site-packages/aries_cloudagent/askar/store.py", line 171, in open_store
    raise ProfileError("Error opening store") from err
aries_cloudagent.core.error.ProfileError: Error opening store

Shutting down

I do also see these problems listing and a popup warning me there's errors prior to that when

[usingtechnology]

postgres may need to be updated - try image: postgres:14

[usingtechnology]

also... the plugins/docker plugins/demo stuff is not nearly as configured as the stuff in scripts. so you will probably have to some re-engineering and additions if you want to run it with all those configurations. I'm assuming the addition of an env file and env vars (can that be done in vscode files?) for the startup of launch.json acapy.

maybe we should also take out pylint. those classes should be in the classpath at least on a rebuild of the devcontainer.

[loneil]

Will need to see what to do about this PR. @esune
The 0.8.2 update (actually on 0.8.3pre) has been merged in with additional tweaks from another PR now.

The devcontainer stuff I could not get to work, see comment above I believe from discussing with Sherman there's additional configuration or environment work or something that would need doing to be able to debug a plugin using the instructions documented above. I get the Caused by: error returned from database: password authentication failed for user "postgres" error shown in that comment when trying to run.
@usingtechnology might have other thoughts on additional needs for the dev container. Not sure if worth merging and working on further or if a different approach is needed?

[usingtechnology]

i don't see those messages in the Problems tab, on Mac but shouldn't really be different? also, i can launch acapy + plugins using the "Run/Debug plugin" command (launch.json) and then call the API through swagger in a browser and it acts ok (saves to the db). I don't know how you guys want to develop or what you want out of the devcontainers, so that would be for you to tweak (like if you want tenant UI started in the devcontainer, or if you want to run it externally but against plugins running in the IDE).

If you've got the main branch up to date (python 3.9, acapy 0.8.x) then just torch this PR, start fresh when you have time to figure out what the team wants for developers.

[loneil]

i don't see those messages in the Problems tab, on Mac but shouldn't really be different? also, i can launch acapy + plugins using the "Run/Debug plugin" command (launch.json) and then call the API through swagger in a browser and it acts ok (saves to the db). I don't know how you guys want to develop or what you want out of the devcontainers, so that would be for you to tweak (like if you want tenant UI started in the devcontainer, or if you want to run it externally but against plugins running in the IDE).

If you've got the main branch up to date (python 3.9, acapy 0.8.x) then just torch this PR, start fresh when you have time to figure out what the team wants for developers.

Yeah it's not the problems listing that's blocking, but that if I start out and pull this repo and open everything up following the instructions listed I get that Caused by: error returned from database: password authentication failed for user "postgres" error when trying to launch. But if that works out for you I'm not sure if I'm missing running something else along with it or it it's an issue with my system.

[usingtechnology]

just checked this PR out clean, new location on my machine. reopened in plugins in container, everything setup nicely. launched in debug and proxy, db and acapy stood up fine, able to use swagger at localhost:8032 to create reservations, log in as innkeeper etc. so no db connection issues.

don't know what is going on with your machine, but could try the command palette: "Dev Containers: Rebuild Container without Cache". maybe that will get you back to clean?

in another vscode window, opened up tenant-ui in a container, again, all built nicely. launch 'backend - run dev', then 'frontend- run dev', then 'frontend - chrome'. was able to set breakpoints in vue and in plugins and all worked fine. able to create reservations, approve etc. so db connectivity was good.

the problems, i guess it depends on what files are open? anyway, if you don't need pylint, just remove it and figure out what helper plugins you want.

given what is in this PR, I can debug plugins and tenant UI vue code with breakpoints in an IDE. be nice if other traction devs could try it out and see if you want to keep the PR and/or what tweaks you want/need to make. but if you guys can't get it to work or it that flow doesn't work, I'd just kill the PR. been around a while and is dead-weight if no one makes use of it.

[loneil]

just checked this PR out clean, new location on my machine. reopened in plugins in container, everything setup nicely. launched in debug and proxy, db and acapy stood up fine, able to use swagger at localhost:8032 to create reservations, log in as innkeeper etc. so no db connection issues.

Ok, thanks @usingtechnology !
Must be something with my setup, I will mess around more when I have time. If you are able to get debugging going from clean that's great.

Yeah the devcontainer is definitely something we'd want to work with IMO so if this can work for people to get debugging going then we should pull it in for sure.

I've rebased in the changes upgrading ACA-py to 0.8.3pre (from the other PR) and resolved any conflicts. The only remaining diffs here are entirely siloed to the devcontainer stuff.
So the changes in here are good to merge in.