seed data from a private repo

[pnadolny13]

Closes https://github.com/meltano/internal-data/issues/56

Imports the private dbt package for seeds. I tested that it worked locally but I still need to give CI access to the repo for testing and deployment.

I decided to use the webfactory/ssh-agent github action to add ssh keys easier but theres one issue that I want to confirm we wont have problems with webfactory/ssh-agent#129 and webfactory/ssh-agent#106, which relates to having self hosted runners. I dont think we use self hosted runners but wanted to confirm before merging.

[pnadolny13]

@WillDaSilva @magreenbaum two questions:

1. related to my question in description about webfactory/ssh-agent, can you confirm that we dont use self hosted runners for github actions? Any objections to me using this action
2. This PR adds a private dbt package that needs access to the internal-data repo. In CI I installed a deploy key so it works as expected. I'm not sure how to handle this for meltano cloud though. My first thought was to revert back to doing dbt:deps in CI (where I have access) once while building the docker image, that way the docker image ships with the deps installed and doesnt need to be shipped with access to the private repo. In meltano cloud we dont use my docker image though so it will need access the credentials to access the private repo at runtime. How can I do this? Is there a hook (assuming not yet) for me to install ssh keys similar to webfactory/ssh-agent for meltano cloud prior to kicking off my job? Is this something we can add or is there another solution?

For 2 I dont think this will be an uncommon pattern to install private packages so its good to come up with a pattern now. cc @tayloramurphy @aaronsteers

[magreenbaum]

1. We're not using self-hosted GitHub action runners so all good on that front.
2. Related: https://github.com/meltano/infra/issues/189. We currently don't have support for this but it is something we plan on supporting. As far as other options, potentially we could make it possible to use a personal fine-grained access token restricted to a specific repository but I think we would likely look into ssh auth first. If the token auth method works out, it might be more desirable for us since that wouldn't require us installing and configuring an ssh-agent on the cloud runners. But I'm open to thoughts on that.

[pnadolny13]

@magreenbaum awesome thanks for the info!

I just tested and I can use a fine grain access token for my dbt package git url so this will work for me. It seemed from stack overflow comments that deploy keys were preferable to access token because they have limited scope, but I guess with fine grain access tokens now you get the same feature, so thats fine with me.

For this approach I'll need someone to create me an access token from an account that has ownership of the internal-data repos, maybe meltybot? When I create fine grain tokens I can only select from my own pnadolny13 repos.

Is that something you can help me with?

[magreenbaum]

@pnadolny13 let me take a look at what's involved in the ssh implementation and get back to you on this. I think we might want to go that route to start since it will probably solve more than one use case.

[WillDaSilva]

Related:

• https://github.com/meltano/infra/pull/277

Thanks @magreenbaum!

[magreenbaum]

One more update, but after this PR we should be good.

We will be able to use the GIT_SSH_PRIVATE_KEY env var and encrypt it into secrets.yml to clone private repos so using deployment keys is the way to go.

When you're copying the private key to the .env file for encrypting (with kms_ext), the way to set it would be like this for example:

GIT_SSH_PRIVATE_KEY='-----BEGIN OPENSSH PRIVATE KEY-----
thereslotsofprivatekeymaterialstuffhere
onvariouslineslikethis
wecanjustcopypasteasitappearsinthefile
andusesinglequotesaroundthewholething
-----END OPENSSH PRIVATE KEY-----'

And that'll encrypt it into secrets.yml how we need it.

I'll do an instructional write-up in https://github.com/meltano/cloud-docs.

Edit: We should be good now. Update PR was merged.

[pnadolny13]

@magreenbaum thanks for that! I updated this PR with the secrets changes, can you verify that its normal to have all of them change even though I'm really just adding the git private key and cloudwatch keys?

[magreenbaum]

@magreenbaum thanks for that! I updated this PR with the secrets changes, can you verify that its normal to have all of them change even though I'm really just adding the git private key and cloudwatch keys?

@pnadolny13 yep, all the encrypted values get updated when using kms-ext to encrypt your .env file. That's normal. 👍