[ResponseOps][connectors] investigate using http proxy service forwarding to https services

[pmuellr]

This issue was the collecting spot for what has turned into meta issue #138319. See that issue for the most recent, relevant sub-issues. The remainder of this issue just contains historic content.

We've had a number of issues over time where users have been using http proxies to connect to https servers. I think this often works, however there do appear to be some limitations like with custom TLS options.

Feels like it's time to do some investigation - I suspect something in our proxy code is not quite right.

We have an existing issue open to beef up the proxy and email tests, using a kinda new pattern we used when testing the custom host settings: #102935 - I'd say what we should do is add a bunch of tests in that form for proxy (defer on the email testing, for now), and hopefully we'll find out where we might have issues.

[mikecote]

May not need to do #124250 but to be confirmed when going through the code.

[pmuellr]

Peeking a bit more, I noticed that axios has a number of proxy-related issues opened against it - https://github.com/axios/axios/issues?q=is%3Aissue+proxy+is%3Aopen.

We're also back-level on the version - we're currently on 0.21.1.

I still think we'll want to start with some proper tests for the proxy support (email + webhook should be good enough to test with; email is different than the other connectors, webhook is the same as the other connectors, in how the proxy is setup). If we find configurations that aren't working, does upgrading axios, and perhaps the http/s agents fix things?

[pmuellr]

Spent some time trying to set up some tests similar to the custom host setting tests. Some success - looks like we'll want to test the following scenarios:

• proxy is http or https
• proxy requires auth or doesn't
• target is http or https
• target requires auth or doesn't

So, 16 basic combinations.

Will also need to support custom cert stuff, but that will only be for target https (I think). I think if we test for custom certs, we won't need to test for publicly available certs (like from elastic.co), but I've currently got some in my basic tests to tell if my code is just plain broken, or only broken for custom certs. Testing with public certs also implies such tests will be dependent on some external website - and which would we pick? Currently doing some testing with example.com, but elastic.co seems like a possibility as well.

Getting the tests to run has been a nightmare though. The proxy code I've implmented is from http-proxy, which we use in Kibana already. However, it doesn't seem to directly support custom certs on https targets - I've found some customization of that code that may work though. That seemed to be endemic amongst proxy code - yes, everything works, except connecting to https servers with custom certs. And, kinda makes sense - most of the time the proxy is a firewall, just to provide access to some subset of existing services on the public internet, for which you don't need custom certs. Our customers, on the other hand, have specific needs for proxying to servers that are using custom certs.

Debugging this stuff is really, really hard - so many moving pieces!

I briefly looked at squid, ha-proxy, and nginx as possibilities for the proxy for testing, which would be a little more "real life", however the primary use for these seems to be to support reverse proxies, and we need a forward proxy. The doc on setting such things up was spartan to non-existent. Probably something we do want to look into though. Probably squid. Which would need to be some kind of "manual" test.

Axios is being updated in PR #111655 . There's a chance the existing proxy code in it will "just work", however looking through the existing axios issues seem to indicate that it still doesn't handle proxying to https with custom certs.

One of the axios issues noted the package hpagent seemed to handle that situation with axios though - like our proxy code does - by passing in custom agents. Tried a quick test, and it did seem to work! Oh yeah, it's from @delvedor

So, current direction seems like:

• wait for axios upgrade
• try it's built-in proxy support
• if that doesn't work, swap out the current custom proxy agents we use for hpagent

Also note that the http agent we are using - https://www.npmjs.com/package/http-proxy-agent - is 3 major releases behind (but a few majors were skipped, I think it is aligned with https://www.npmjs.com/package/https-proxy-agent). I tried upgrading it to 5, which had some API breakage, and it didn't seem to help. This has been our biggest problem, I think, getting https proxied through http. And worse, fleet is also using that proxy agent, so if we upgrade the package in a PR, we'll have to fix the fleet code as well. Which is why I'm thinking switching to hpagent may be a good move.

[pmuellr]

The team requested that we put together some sort of "troubleshooting" guide for the proxy stuff - which makes sense. Not sure where we'll publish it yet. Presumably it would include instructions on running the "standalone" proxy script we already have in the repo, but I've gotten v2 of that in my WIP PR #131971

[pmuellr]

closing this as it was primarily used for collecting research, the new proxy meta issue is here: #138319