Concurrently Deploy Connectors and Plugins

[mfleader]

Changes introduced with this PR

• Enable a Python connector factory to create connectors that can execute concurrently.
• Enable a Python connector to deploy plugins that can execute concurrently.
• Remove broken tests.
• Fix data race in cliwrapper.go.
• Add fix and unit test for hang on pip install of non-existent git repo by adding the environment variable and value GIT_TERMINAL_PROMPT=0.
• Add unit test for module pull policy by injecting a Python cli stub.
• Add end-to-end functional test of connector and plugin concurrency using multiple connectors and multiple plugins.
• Change CI golang testing to use go 1.18.

---

By contributing to this repository, I agree to the contribution guidelines.

[jaredoconnell]

So much has been re-written, and it has been a while since we've dealt with this PR. So I have some questions, and a request.

I see some detailed explanations on why certain design choices were made, which is very good. But I think this could use some more high-level comments. Not everything about this code is self-documenting. So my request is that you address the high-level design with some comments so that it's easy for the future maintainers, or our future selves.

Questions:

1. Do the latest changes fix IfNotPresent pull policy? If not, when will that be addressed?
2. Did you have a chance to look into Webb's note about if changed pull policy?

[mfleader]

@jaredoconnell

1. Yes, the IfNotPresent behavior is still available with these changes. IfNotPresent is the default behavior of pip.
2. I'm not 100% sure which comment you mean. If you are referencing "Is there a pull-policy of 'newer' available?", then yes that has been addressed by removing our logic to determine if a module is present in favor of using pip's functionality to determine if a module is present.

Re: design and code documentation

At this point, the design is flawed because it assumed a 1 to 1 to 1 to 1 correspondence between the connector, python cli interface, python virtual environment, and the python plugin. The original assumption of the 1 to 1 to 1 correspondence of the connector to the python cli interface to the python plugin predates this PR, and it is the fundamental correspondence that leads to current design flaw. This part needs to be removed via refactoring instead of given an explanation in code comments. I believe a better assumption would be 1 connector to 1 python cli interface to v python virtual environments to n python plugins.

[webbnh]

OK! I think I've made it all the way through, now. I found a few more things which you might want to polish.

[jaredoconnell]

Tested with latest changes. Looks good to me.

[webbnh]

It seems odd to remove the FIO step from the test. (I don't think we requested that; are we really better off without it?) Nevertheless, I'm still not seeing anything for which I would block the merge, so I'm approving. But, I've commented on a couple of small things below for your consideration, and there's the spaces vs. tabs thing still.