setuptools fails in corporate MITM proxy with intermediate SSL cacerts

[bond-]

I was looking at the following piece of code in package_index.py:

        use_ssl = (
            verify_ssl
            and ssl_support.is_available
            and (ca_bundle or ssl_support.find_ca_bundle())
        )
        if use_ssl:
            self.opener = ssl_support.opener_for(ca_bundle)

I have added/appened the corporate CA certificates to /etc/ssl/cert.pem but setuptools never actually picks it up or uses it. Did anyone ever solve this?

FYI I am on a MacOS and using Anaconda with Python 3.6

Error that I get:

Download error on https://pypi.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833) -- Some packages may not be found!
No local packages or working download links found for textacy
error: Could not find suitable distribution for Requirement.parse('textacy')

[ssbarnea]

This is as true as possible and is breaking the package installation for everyone using a proxy.

PIP works well with the same proxy but in some cases it needs to use setuptools (easy_install) to install some setup_requires packages and at this point it will chocke because setuptools is totally clueless when it comes to the custom CA used by the proxy.

For example in addition to the standard http[s]_proxy environment variables on MacOS I do have to defined these:

CURL_CA_BUNDLE=/Users/ssbarnea/cacert.pem
REQUESTS_CA_BUNDLE=/Users/ssbarnea/cacert.pem
SSL_CERT_FILE=/Users/ssbarnea/cacert.pem

This tells to the SSL libraries to use the bundle that contains our own CA certificate.

Curl works, python requests (uses urllib3) works, and python in general works. But not setuptools.

I was not able to find any workaround for this, other than disabling the http_proxy which is a serious issue.

This is related to #1543 but I think that instead of adding new configuration items which would require changing the proxy enablement deployment, it would be better to reuse one of the already known environment variables, likely SSL_CERT_FILE being the best candidate.

[bond-]

@ssbarnea Disabling http_proxy is not a possibility for me it's a enterprise wide setting and there's no http(s)_proxy variable set on our local machines.

It might help if setuptools can help with some sort of configuration parameter.

SSL_CERT_FILE doesn't seem to be working in my case. I was able to get this running for conda(.condarc file) and pip(.piprc file) based package installations but not setuptools

[bond-]

FYI in ssl_support.py

    def opener_for(ca_bundle=None):
    """Get a urlopen() replacement that uses ca_bundle for verification"""
    return urllib.request.build_opener(
        VerifyingHTTPSHandler(ca_bundle or find_ca_bundle())
    ).open

setuptools seems to be using urllib

[mondkaefer]

Here's what worked for me (python3):
I had a look at the certificate bundle files listed in the variable cert_paths in ssl_support.py (part of setuptools package), and copied my CA cert (and intermediate CA certs in my case) into the first bundle file in that list that existed on my system. Emphasis on "first".

[hjoukl]

To fix this properly imho s.th. like such a patch would be necessary:

diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py
index 06c98271..32f2880a 100644
--- a/setuptools/command/easy_install.py
+++ b/setuptools/command/easy_install.py
@@ -336,9 +336,13 @@ class easy_install(Command):
         else:
             hosts = ['*']
         if self.package_index is None:
+            # Be compatible with requests and cURL ca bundle environment
+            # configuration (somewhat, only bundle files but not directories)
+            ca_bundle = (os.environ.get('REQUESTS_CA_BUNDLE') or
+                         os.environ.get('CURL_CA_BUNDLE'))
             self.package_index = self.create_index(
                 self.index_url, search_path=self.shadow_path, hosts=hosts,
-            )
+                ca_bundle=ca_bundle) 
         self.local_index = Environment(self.shadow_path + sys.path)
 
         if self.find_links is not None:
diff --git a/setuptools/ssl_support.py b/setuptools/ssl_support.py
index 226db694..fcdc987a 100644
--- a/setuptools/ssl_support.py
+++ b/setuptools/ssl_support.py
@@ -153,6 +153,10 @@ class VerifyingHTTPSHandler(HTTPSHandler):
     """Simple verifying handler: no auth, subclasses, timeouts, etc."""
 
     def __init__(self, ca_bundle):
+        if not os.path.isfile(ca_bundle):
+            raise IOError(
+                "Could not find a suitable TLS CA certificate bundle file, "
+                "invalid path: {}".format(ca_bundle))
         self.ca_bundle = ca_bundle
         HTTPSHandler.__init__(self)

IIUC understand easy_install is deprecated and will be replaced by pip but it looks like setup_requires still installs via easy_install, so this might still be helpful.

I could prepare a pull request but I'd need a hint at how to write a proper test for this. Do the unittests actually connect to pypi or do stuff locally, only?

[hjoukl]

On a closer look it would probably be better to concentrate this in the ssl_support module and leave easy_install be altogether:

diff --git a/setuptools/ssl_support.py b/setuptools/ssl_support.py
index 226db694..56e1c00d 100644
--- a/setuptools/ssl_support.py
+++ b/setuptools/ssl_support.py
@@ -243,11 +243,35 @@ def get_win_certfile():
     return _wincerts.name
 
 
+def get_env_ca_bundle():
+    """Return a CA bundle file path set through environment variables.
+    
+    Supports the REQUESTS_CA_BUNDLE and CURL_CA_BUNDLE env vars for setting
+    the path to a CA bundle (Current restriction: path must be a file path, 
+    not a directory).
+    """
+    # Be compatible with requests and cURL ca bundle environment configuration
+    # (somewhat, only bundle files but not directories)
+    ca_bundle = (os.environ.get('REQUESTS_CA_BUNDLE') or
+                 os.environ.get('CURL_CA_BUNDLE'))
+    if ca_bundle is not None:
+        if os.path.isdir(ca_bundle):
+            raise NotImplementedError(
+                "TLS CA certificate bundle directory paths are currently  "
+                "not supported.")
+        if not os.path.isfile(ca_bundle):
+            raise IOError(
+                "Could not find a suitable TLS CA certificate bundle file, "
+                "invalid path: {}".format(ca_bundle))
+    return ca_bundle
+
+
 def find_ca_bundle():
     """Return an existing CA bundle path, or None"""
     extant_cert_paths = filter(os.path.isfile, cert_paths)
     return (
-        get_win_certfile()
+        get_env_ca_bundle()
+        or get_win_certfile()
         or next(extant_cert_paths, None)
         or _certifi_where()
     )

This makes testing rather trivial as you can simply test if a given env setting will get respected by find_ca_bundle().
Now trying to prepare for a pull request and get CI working with my fork...

[eholic]

Temporal solution of #1543, #1821 is overwriting cert_paths in setup.py.

from setuptools import setup
import setuptools.ssl_support
setuptools.ssl_support.cert_paths = ['/path/to/cafile']

setup()

setuptools/setuptools/ssl_support.py

Line 21 in 17ad2b7

cert_paths = """

[bclodius]

@eholic thanks worked like a charm! In my case to make it more "portable" I used

import os
import setuptools.ssl_support
setuptools.ssl_support.cert_paths = [os.environ.get('SSL_CERT_FILE')]

Note to others: I had to use full path /home/<some_path> and not relative path for home directory ~.

[eholic]

@bclodius Thank you. It would be better according to @hjoukl's PR#1821.

from setuptools import setup
import os
import setuptools.ssl_support

ca_bundle = (os.environ.get('SSL_CERT_FILE') or
             os.environ.get('REQUESTS_CA_BUNDLE') or
             os.environ.get('CURL_CA_BUNDLE'))
if ca_bundle:
    setuptools.ssl_support.cert_paths = [ca_bundle]

setup()

[jaraco]

I believe the need for this has been obviated by #1830. Please comment if not.