TestArtifactCache.test_restful_cache_failover is flaky

[baroquebobcat]

Ran into this, a re-run of the shard fixed it.

                     _ TestArtifactCache.test_restful_cache_failover __
                     
                     self = <pants_test.cache.test_artifact_cache.TestArtifactCache testMethod=test_restful_cache_failover>
                     
                         def test_restful_cache_failover(self):
                           bad_url = 'http://badhost:123'
                         
                           with temporary_dir() as artifact_root:
                             local = TempLocalArtifactCache(artifact_root, 0)
                         
                             # With fail-over, rest call second time will succeed
                             with self.setup_server() as good_server:
                               artifact_cache = RESTfulArtifactCache(artifact_root,
                                                                     BestUrlSelector([bad_url, good_server.url], max_failures=0),
                                                                     local)
                               with self.assertRaises(NonfatalArtifactCacheError) as ex:
                                 self.do_test_artifact_cache(artifact_cache)
                               self.assertIn('Failed to HEAD', str(ex.exception))
                         
                     >         self.do_test_artifact_cache(artifact_cache)
                     
                     pants_test/cache/test_artifact_cache.py:88: 
                     _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
                     pants_test/cache/test_artifact_cache.py:97: in do_test_artifact_cache
                         self.assertTrue(artifact_cache.has(key))
                     pants/cache/restful_artifact_cache.py:67: in has
                         return self._request('HEAD', cache_key) is not None
                     _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
                     
                     self = <pants.cache.restful_artifact_cache.RESTfulArtifactCache object at 0x7f01bcba8550>
                     method = 'HEAD'
                     cache_key = CacheKey(id=u'muppet_key', hash=u'fake_hash')
                     body = None
                     
                         def _request(self, method, cache_key, body=None):
                         
                           session = RequestsSession.instance()
                           with self.best_url_selector.select_best_url() as best_url:
                             url = self._url_for_key(best_url, cache_key)
                             logger.debug('Sending {0} request to {1}'.format(method, url))
                             try:
                               if 'PUT' == method:
                                 response = session.put(url,
                                                        data=body,
                                                        timeout=self._write_timeout_secs,
                                                        allow_redirects=True)
                               elif 'GET' == method:
                                 response = session.get(url,
                                                        timeout=self._read_timeout_secs,
                                                        stream=True,
                                                        allow_redirects=True)
                               elif 'HEAD' == method:
                                 response = session.head(url,
                                                         timeout=self._read_timeout_secs,
                                                         allow_redirects=True)
                               elif 'DELETE' == method:
                                 response = session.delete(url,
                                                           timeout=self._write_timeout_secs,
                                                           allow_redirects=True)
                               else:
                                 raise ValueError('Unknown request method {0}'.format(method))
                             except RequestException as e:
                               raise NonfatalArtifactCacheError('Failed to {0} {1}. Error: {2}'
                     >                                          .format(method, url, e))
                     E         NonfatalArtifactCacheError: Failed to HEAD http://badhost:123/muppet_key/fake_hash.tgz. Error: HTTPConnectionPool(host='badhost', port=123): Max retries exceeded with url: /muppet_key/fake_hash.tgz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f01b8084f50>: Failed to establish a new connection: [Errno -2] Name or service not known',))
                     
                     pants/cache/restful_artifact_cache.py:132: NonfatalArtifactCacheError
                     -------------- Captured stderr call --------------
                     127.0.0.1 - - [28/Nov/2018 22:38:00] "HEAD /muppet_key/fake_hash.tgz HTTP/1.1" 307 -
                     127.0.0.1 - - [28/Nov/2018 22:38:00] code 404, message File not found
                     127.0.0.1 - - [28/Nov/2018 22:38:00] "HEAD /muppet_key/fake_hash.tgz/__redir__ HTTP/1.1" 404 -
                     127.0.0.1 - - [28/Nov/2018 22:38:00] "GET /muppet_key/fake_hash.tgz HTTP/1.1" 307 -
                     127.0.0.1 - - [28/Nov/2018 22:38:00] code 404, message File not found
                     127.0.0.1 - - [28/Nov/2018 22:38:00] "GET /muppet_key/fake_hash.tgz/__redir__ HTTP/1.1" 404 -
                     127.0.0.1 - - [28/Nov/2018 22:38:00] "HEAD /muppet_key/fake_hash.tgz HTTP/1.1" 307 -
                     127.0.0.1 - - [28/Nov/2018 22:38:00] code 404, message File not found
                     127.0.0.1 - - [28/Nov/2018 22:38:00] "HEAD /muppet_key/fake_hash.tgz/__redir__ HTTP/1.1" 404 -
                     127.0.0.1 - - [28/Nov/2018 22:38:00] "PUT /muppet_key/fake_hash.tgz HTTP/1.1" 307 -
                     --------------- Captured log call ----------------
                     artifact_cache.py          103 ERROR    Error while writing to artifact cache: Failed to PUT http://localhost:50631/muppet_key/fake_hash.tgz. Error: ('Connection aborted.', error(32, 'Broken pipe'))
                      generated xml file: /home/travis/build/pantsbuild/pants/.pants.d/test/pytest/tests.python.pants_test.cache.artifact_cache/junitxml/TEST-tests.python.pants_test.cache.artifact_cache.xml 

[stuhood]

TestArtifactCache.test_restful_cache as well.

[stuhood]

And test_multiproc: see the linked ticket.

[stuhood]

Seen again with test_restful_cache in master.

[Eric-Arellano]

Have ran into these multiple times the past week.

[pierrechevalier83]

Hit again in TestArtifactCache.test_local_backed_remote_cache https://travis-ci.org/pantsbuild/pants/jobs/589387228

[Eric-Arellano]

Encountered again, even with the 1 retry. I'm going to bump to 2 retries (it's a cheap test) and then we should remove the test if it happens again.

[Eric-Arellano]

Test was removed.