Feat/grpc tls #462

fabriziomoscon · 2021-12-08T19:16:07Z

Adding TLS for grpc executor.

Remove unused option plaintext.
Add move options to grpc executor for TLS connections:

tls_root_ca
tls_client_cert
tls_client_key
ignore_verify_ssl

Is there any existing test for grpc? I didn't find it.

fsamin · 2021-12-10T09:34:01Z

Hello,
Thanks for your pull request 👍
Sadly, we don't have tests for the GRPC executor, could we ask you to add a few tests ?

ovh-cds · 2021-12-10T09:40:23Z

CDS Report build-venom-a#706.0 ✘

Build
- Build ✔
Tests
- Acceptance Tests ✘

fabriziomoscon · 2021-12-10T11:58:00Z

@fsamin yes I will add tests. Thanks.

fabriziomoscon · 2021-12-17T10:00:43Z

@fsamin I had a look at the other tests, and it is not clear to me how I would add this test, could you please suggest your preferred way.
Usually, we test this with a go unit test, setup creates local files for pki, the test spawn a TLS server and connects a TLS client with mutual TLS.

fabriziomoscon · 2021-12-21T20:00:57Z

@fsamin @yesnault I have added a docker grpc server with TLS as venom-grpc.cid, which works. But I don't understand how to run venom against it. I have added a target grpc-test which uses venom_wrapper.sh to run my test locally but I receive an error:

make: *** [grpc-test] Error 127

Please let me know how would I run this test

fsamin · 2021-12-28T09:13:15Z

Our integration tests follows the following process:

To starts all tests dependencies. We are using make start-test-stack to start all dependencies as docker containers. We are using .cid files to store containers ID. Your venom-grpc.cid target seems fine.

Once all the dependencies are up and running, we compile a specific binary of venom (with the golang tests intrumentation to be able to gather code coverage). We are using make build-test-binary-docker. It produces a venom.test file.

To run the tests we are using make run-test. Under the hood, it relies on venom_wrappers.sh. It runs all yaml files in the tests directory and if needed you can add --var-from-file flag to specify specific variables to run your tests against your grpc server.

kravcs · 2022-01-10T13:18:34Z

Thanks @fsamin for the instructions, the integration tests for grpc TLS are working fine from our perspective.
We also added the ability to setup TLS options with PEM files in similar way as you did for http executor.
Please let us know if everything is fine from your side as well.

executors/grpc/README.md

fabriziomoscon · 2022-01-10T14:07:38Z

tests/Makefile

-	VENOM_VAR_MY_ENVAR=foo ./venom_wrapper.sh run -vv --format=xml --output-dir=. --lib-dir=./lib_custom --var='array_from_var=["biz","buz"]' --var-from-file ./kafka/testVariables.yml  --var-from-file ./vars/vars.yml ./*.yml ./assertions/*.yml && \
+run-test: generate-venom-pki
+	VENOM_VAR_MY_ENVAR=foo  ./venom_wrapper.sh run \
+	-vv --format=xml --output-dir=. --lib-dir=./lib_custom --var='array_from_var=["biz","buz"]' --var-from-file ./kafka/testVariables.yml --var-from-file=$(PKI_VAR_FILE) --var-from-file ./vars/vars.yml ./*.yml ./assertions/*.yml && \


this --var-from-file=$(PKI_VAR_FILE) is needed to generate the yaml formatted PEM import file used in the test.

fabriziomoscon · 2022-01-18T11:34:06Z

@fsamin just wanted to confirm that, as far as we can tell, we completed all the fixes for this PR. We are looking forward to read your new instructions.
Thank you

ovh-cds · 2022-01-24T08:43:51Z

CDS Report build-venom-a#792.0 ✘

Build
- Build ✔
Tests
- Acceptance Tests ✘

fsamin · 2022-01-24T08:47:58Z

tests/pki.mk

+	openssl genrsa \
+		-out $(PKI_DIR)/server.key 2048
+
+	# create a server Certificate Signing Request


This step fails with the following error:

mkdir -p ./grpc/pki # create a certificate authority (CA) that both the client and server trust. # The CA is just a public and private key with the public key wrapped up in a self-signed X.509 certificate. openssl req \ -new \ -x509 \ -nodes \ -days 365 \ -subj '/C=GB/O=Example/OU=TeamA/CN=ca.example.com' \ -keyout ./grpc/pki/ca.key \ -out ./grpc/pki/ca.crt 3m54s234ms Generating a RSA private key ..........................................+++++ ...+++++ writing new private key to './grpc/pki/ca.key' ----- unable to find 'distinguished_name' in config problems making Certificate Request 139874421728576:error:0E06D06C:configuration file routines:NCONF_get_string:no value:../crypto/conf/conf_lib.c:273:group=req name=distinguished_name make: *** [pki.mk:5: grpc/pki] Error 1

Could you handle this ?

Is the CI on linux?

yes, ubuntu-20.04

I quickly searched and found this: https://superuser.com/questions/947061/openssl-unable-to-find-distinguished-name-in-config
It looks like openssl req is not able to find a config value...
could you please let me know which openSSL version you run on ubuntu?

Also, we don't use -subj in test, so perhaps that could be removed and see if the distinguished_name issue goes away

I have tried to:

Run in Docker ubuntu:20.04

Install openssl with apt update && apt install --yes openssl

Run 2 commands and they were successful:

root@7bd2261de939:~# mkdir -p ./grpc/pki root@7bd2261de939:~# openssl req \ > -new \ > -x509 \ > -nodes \ > -days 365 \ > -subj '/C=GB/O=Example/OU=TeamA/CN=ca.example.com' \ > -keyout ./grpc/pki/ca.key \ > -out ./grpc/pki/ca.crt Generating a RSA private key .........................+++++ ...................................................................................................................+++++ writing new private key to './grpc/pki/ca.key' -----

Question would be: Are you using vanilla Ubuntu 20.04 Docker image? Does it have openssl installed ? Or you use some other custom Docker image with openssl installed in some other way ?

It seems your openssl can't find /usr/lib/ssl/openssl.cnf thus question is - is this file available on your Docker image?

@sarunask @fabriziomoscon openssl fixed on your worker model. Tests are now ok. Can you rebase your PR please?

ovh-cds · 2022-01-24T14:21:58Z

CDS Report build-venom-a#792.1 ✘

Build
- Build ✔
Tests
- Acceptance Tests ✘

ovh-cds · 2022-01-24T14:27:42Z

CDS Report build-venom-a#795.0 ✘

Build
- Build ✔
Tests
- Acceptance Tests ✘

ovh-cds · 2022-01-24T14:48:15Z

CDS Report build-venom-a#792.2 ■

Build
- Build ■

ovh-cds · 2022-01-24T14:48:15Z

CDS Report build-venom-a#792.2 ■

Build
- Build ■

Fix: tidy up modules. Remove unused option plaintext. Add move options to grpc executor for TLS connections: - tls_root_ca - tls_client_cert - tls_client_key - ignore_verify_ssl - grpc TLS add integration tests - grpc TLS with PEM files support Signed-off-by: Fabrizio Moscon <mosconfabrizio@gmail.com>

fabriziomoscon · 2022-01-25T09:01:04Z

Thank you

fabriziomoscon force-pushed the feat/grpc-tls branch 2 times, most recently from ac64949 to fdb2d00 Compare December 8, 2021 19:18

fabriziomoscon force-pushed the feat/grpc-tls branch from fdb2d00 to 44b2bfe Compare December 10, 2021 16:01

fabriziomoscon force-pushed the feat/grpc-tls branch from 19c2ffd to 8569656 Compare December 21, 2021 19:56

fabriziomoscon force-pushed the feat/grpc-tls branch 3 times, most recently from 9a01d53 to 382c4dd Compare December 21, 2021 20:03

kravcs force-pushed the feat/grpc-tls branch from ecad215 to 63d0345 Compare January 10, 2022 13:22

fabriziomoscon force-pushed the feat/grpc-tls branch 3 times, most recently from 719d21d to be1aaf6 Compare January 10, 2022 14:04

fabriziomoscon commented Jan 10, 2022

View reviewed changes

executors/grpc/README.md Show resolved Hide resolved

fabriziomoscon commented Jan 10, 2022

View reviewed changes

kravcs force-pushed the feat/grpc-tls branch from be1aaf6 to 6aab2dd Compare January 10, 2022 14:45

fsamin requested changes Jan 24, 2022

View reviewed changes

yesnault approved these changes Jan 24, 2022

View reviewed changes

fabriziomoscon force-pushed the feat/grpc-tls branch from 6aab2dd to a08a45d Compare January 24, 2022 16:17

fsamin approved these changes Jan 25, 2022

View reviewed changes

fsamin merged commit b3ab202 into ovh:master Jan 25, 2022

fabriziomoscon deleted the feat/grpc-tls branch January 25, 2022 08:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/grpc tls #462

Feat/grpc tls #462

fabriziomoscon commented Dec 8, 2021 •

edited

Loading

fsamin commented Dec 10, 2021

ovh-cds commented Dec 10, 2021

fabriziomoscon commented Dec 10, 2021

fabriziomoscon commented Dec 17, 2021

fabriziomoscon commented Dec 21, 2021

fsamin commented Dec 28, 2021

kravcs commented Jan 10, 2022

fabriziomoscon Jan 10, 2022

fabriziomoscon commented Jan 18, 2022

ovh-cds commented Jan 24, 2022

fsamin Jan 24, 2022

fabriziomoscon Jan 24, 2022

yesnault Jan 24, 2022

fabriziomoscon Jan 24, 2022

fabriziomoscon Jan 24, 2022

sarunask Jan 24, 2022

sarunask Jan 24, 2022

yesnault Jan 24, 2022

fabriziomoscon Jan 24, 2022

ovh-cds commented Jan 24, 2022

ovh-cds commented Jan 24, 2022

ovh-cds commented Jan 24, 2022

ovh-cds commented Jan 24, 2022

fabriziomoscon commented Jan 25, 2022

Feat/grpc tls #462

Feat/grpc tls #462

Conversation

fabriziomoscon commented Dec 8, 2021 • edited Loading

fsamin commented Dec 10, 2021

ovh-cds commented Dec 10, 2021

fabriziomoscon commented Dec 10, 2021

fabriziomoscon commented Dec 17, 2021

fabriziomoscon commented Dec 21, 2021

fsamin commented Dec 28, 2021

kravcs commented Jan 10, 2022

Choose a reason for hiding this comment

fabriziomoscon commented Jan 18, 2022

ovh-cds commented Jan 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ovh-cds commented Jan 24, 2022

ovh-cds commented Jan 24, 2022

ovh-cds commented Jan 24, 2022

ovh-cds commented Jan 24, 2022

fabriziomoscon commented Jan 25, 2022

fabriziomoscon commented Dec 8, 2021 •

edited

Loading