'_implicit_environ.compute_engine_id()' may raise

[tseaver]

I just saw this while running tests:

ERROR: test_set_explicit_None_wo_env_var_set (gcloud.datastore.test___init__.Test_set_default_dataset_id)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/gcloud/datastore/test___init__.py", line 71, in test_set_explicit_None_wo_env_var_set
    self._callFUT(None)
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/gcloud/datastore/test___init__.py", line 31, in _callFUT
    return set_default_dataset_id(dataset_id=dataset_id)
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/gcloud/datastore/__init__.py", line 91, in set_default_dataset_id
    dataset_id = _implicit_environ.compute_engine_id()
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/gcloud/datastore/_implicit_environ.py", line 73, in compute_engine_id
    response, content = http.request(uri, method='GET', headers=headers)
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/.tox/cover/lib/python2.7/site-packages/httplib2/__init__.py", line 1593, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/.tox/cover/lib/python2.7/site-packages/httplib2/__init__.py", line 1335, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/.tox/cover/lib/python2.7/site-packages/httplib2/__init__.py", line 1291, in _conn_request
    response = conn.getresponse()
  File "/opt/Python-2.7.9/lib/python2.7/httplib.py", line 1061, in getresponse
    raise ResponseNotReady()

[dhermes]

Yes I'm currently wrestling with it. It seems to be related to:
https://github.com/jcgregorio/httplib2/issues/284

Self-assigning bug and will keep digging.

[dhermes]

FYI, equivalent snippet:

        host = '169.254.169.254'
        uri_path = '/computeMetadata/v1/project/project-id'
        headers = {'Metadata-Flavor': 'Google'}
        connection = httplib.HTTPConnection(host, timeout=0.1)
        try:
            connection.request('GET', uri_path, headers=headers)
            response = connection.getresponse()
            if response.status == 200:
                return response.read()
        except socket.error:  # socket.timeout and socket.error(64, 'Host is down')
            pass
        finally:
            connection.close()

It demonstrates a philosophical position of relying on httplib2 as little as possible, in light of the move to requests always being discussed (couldn't find a link) and #551

---

This has also made me realize we aren't setting the defaults (e.g. _implicit_environ.CONNECTION) in a threadsafe way. I ran into pydanny/cached-property#9 along the way.

[dhermes]

OK I ran

#!/bin/bash

set -e

for i in `seq 1 40`;
do
    .tox/py27/bin/nosetests gcloud/datastore/test___init__.py
done

to make sure the error is gone. (It fails within the first 10 loops on master right now.)

[tseaver]

I can see how this might work out: we aren't using HTTPS or basic auth, which are known-hard with httplib, especially in combination. Still, it does seem odd to be relying on what is basically a 10-year old stdlib module over a more modern, presumably-maintained one (although it hasn't been updated in 6 months either).

[dhermes]

I've been inside the beast of httplib2 and am not thrilled with it. It's a dep from oauth2client

I'd prefer to use requests by default and enable a BYO http as outlined in #551

[tseaver]

FWIW, I'm not impressed with the code quality / maintainership of requests, either.

[dhermes]

Haha I never considered that. I've dealt so much with httplib2 for Google stuff I just assumed the grass was greener (everyone certainly has great things to say about requests).