Handling very long connection timeouts

[dev169]

Long story short

Having a big list of hosts to connect to, some of them timeout and aiohttp waits eternally. What is the proper way to handle this case?

Expected behaviour

aiohttp should have some kind of connection timeout.

Actual behaviour

aiohttp hangs

Steps to reproduce

An extreme example:

$ time curl -verbose https://soso.com
* Rebuilt URL to: https://soso.com/
* Hostname was NOT found in DNS cache
*   Trying 106.120.151.169...
* connect to 106.120.151.169 port 443 failed: Connection timed out
*   Trying 106.120.151.169...
* After 86339ms connect time, move on!
* connect to 106.120.151.169 port 443 failed: Connection timed out
* Failed to connect to soso.com port 443: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to soso.com port 443: Connection timed out

real    3m33.671s
user    0m0.027s
sys     0m0.023s

What is the proper way to handle this? Example code:

#!/usr/bin/env python

import aiohttp
import asyncio
import json
import time

async def fetch(session, url):
    try:
        async with session.get(url) as response:
        # rs = await response.text()
        print(str(response.status) + " " + url)
    except Exception as e:
        print("Oops: " + url + " " + str(type(e)) + str(e))

if __name__ == '__main__':

    t = time.monotonic()

    watchlist  = {
        "https://360.cn": {},
        "https://chinadaily.com.cn": {},
        "https://cntv.cn": {},
        "https://gmw.cn": {},
        "https://googleusercontent.com": {},
        "https://hao123.com": {},
        "https://imdb.com": {},
        "https://live.com": {},
        "https://naver.com": {},
        "https://nicovideo.jp": {},
        "https://pixnet.net": {},
        "https://qq.com": {},
        "https://rakuten.co.jp": {},
        "https://sina.com.cn": {},
        "https://sohu.com": {},
        "https://soso.com": {},
        "https://t.co": {},
        "https://tianya.cn": {},
        "https://xinhuanet.com": {},
        "https://xvideos.com": {},
        "https://yahoo.co.jp": {},
        "https://yahoo.com": {},
        "https://yandex.ru": {}
    }

    loop = asyncio.get_event_loop()
    conn = aiohttp.TCPConnector(verify_ssl=False)
    asyncio.set_event_loop(loop)

    futures = []
    with aiohttp.ClientSession(loop=loop, connector=conn) as session:
        for url, url_check in watchlist.items():
            futures.append(asyncio.ensure_future(fetch(session, url)))

        loop.run_until_complete(asyncio.gather(*futures))

    print("Done in ", str(time.monotonic() - t))

The above will wait many minutes until it timeouts.

Using aiohttp.Timeout seemed like a good candidate to take care of this, but I've found that once the timeout is reached, all other futures throw an concurrent.futures._base.TimeoutError, so it appears to work "globally". So if we modify fetch like so:

async def fetch(session, url):
    try:
        with aiohttp.Timeout(5):
            async with session.get(url) as response:
                # rs = await response.text()
                print(str(response.status) + " " + url)
    except Exception as e:
        print("Oops: " + url + " " + str(type(e)) + str(e))

We now get:

Oops: https://googleusercontent.com <class 'aiohttp.errors.ClientOSError'>[Errno -2] Cannot connect to host googleusercontent.com:443 ssl:True [Can not connect to googleusercontent.com:443 [Name or service not known]]
200 https://yandex.ru
200 https://t.co
200 https://sohu.com
200 https://yahoo.com
200 https://gmw.cn
Oops: https://live.com <class 'aiohttp.errors.ClientOSError'>[Errno 111] Cannot connect to host live.com:443 ssl:True [Can not connect to live.com:443 [Connect call failed ('65.55.206.154', 443)]]
Oops: https://360.cn <class 'aiohttp.errors.ClientOSError'>[Errno 1] Cannot connect to host 360.cn:443 ssl:True [Can not connect to 360.cn:443 [[SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:645)]]
200 https://qq.com
Oops: https://pixnet.net <class 'aiohttp.errors.ClientOSError'>[Errno 111] Cannot connect to host pixnet.net:443 ssl:True [Can not connect to pixnet.net:443 [Connect call failed ('103.23.108.107', 443)]]
Oops: https://tianya.cn <class 'concurrent.futures._base.TimeoutError'>
Oops: https://xinhuanet.com <class 'concurrent.futures._base.TimeoutError'>
Oops: https://soso.com <class 'concurrent.futures._base.TimeoutError'>
Oops: https://hao123.com <class 'concurrent.futures._base.TimeoutError'>
Oops: https://nicovideo.jp <class 'concurrent.futures._base.TimeoutError'>
Oops: https://xvideos.com <class 'concurrent.futures._base.TimeoutError'>
Oops: https://naver.com <class 'concurrent.futures._base.TimeoutError'>
Oops: https://imdb.com <class 'concurrent.futures._base.TimeoutError'>
Oops: https://chinadaily.com.cn <class 'concurrent.futures._base.TimeoutError'>
Oops: https://cntv.cn <class 'concurrent.futures._base.TimeoutError'>
Oops: https://rakuten.co.jp <class 'concurrent.futures._base.TimeoutError'>
Oops: https://yahoo.co.jp <class 'concurrent.futures._base.TimeoutError'>
Oops: https://sina.com.cn <class 'concurrent.futures._base.TimeoutError'>
Done in  5.03474450999056

I've also tried setting conn_timeout in aiohttp.TCPConnector, but since I assume it is reused, once the timeout comes every future after the cutoff throws.

The obvious alternative is concurrent.futures.ThreadPoolExecutor and requests, which I've used and works fine, but i'm sure there must be something wrong with my approach regarding asyncio and aiohttp.

Any suggestions?

Your environment

Python 3.5.1
Ubuntu 14.04

[asvetlov]

Sorry for delay, I've finally got a time for the issue.

aiohttp.Timeout doesn't work globally, it operates exactly as you expect.

P.S. please note you should wrap every fetch() call into a task as you did by ensure_future() calls.

P.P.S.

My output for your snippet is:

Oops: https://googleusercontent.com <class 'aiohttp.errors.ClientOSError'>[Errno -5] Cannot connect to host googleusercontent.com:443 ssl:True [Can not connect to googleusercontent.com:443 [No address associated with hostname]]
Oops: https://live.com <class 'aiohttp.errors.ClientOSError'>[Errno 111] Cannot connect to host live.com:443 ssl:True [Can not connect to live.com:443 [Connect call failed ('65.55.206.154', 443)]]
Oops: https://imdb.com <class 'aiohttp.errors.ClientOSError'>[Errno None] Cannot connect to host imdb.com:443 ssl:True [Can not connect to imdb.com:443 [None]]
Oops: https://pixnet.net <class 'aiohttp.errors.ClientOSError'>[Errno 111] Cannot connect to host pixnet.net:443 ssl:True [Can not connect to pixnet.net:443 [Connect call failed ('103.23.108.107', 443)]]
200 https://t.co
Oops: https://cntv.cn <class 'aiohttp.errors.ClientOSError'>[Errno 113] Cannot connect to host cntv.cn:443 ssl:True [Can not connect to cntv.cn:443 [Connect call failed ('202.108.8.82', 443)]]
Oops: https://360.cn <class 'aiohttp.errors.ClientOSError'>[Errno 1] Cannot connect to host 360.cn:443 ssl:True [Can not connect to 360.cn:443 [[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:645)]]
200 https://yandex.ru
Oops: https://naver.com <class 'aiohttp.errors.ClientOSError'>[Errno None] Cannot connect to host naver.com:443 ssl:True [Can not connect to naver.com:443 [None]]
200 https://yahoo.com
200 https://sohu.com
Oops: https://nicovideo.jp <class 'concurrent.futures._base.TimeoutError'>
Oops: https://xvideos.com <class 'concurrent.futures._base.TimeoutError'>
Oops: https://gmw.cn <class 'concurrent.futures._base.TimeoutError'>
Oops: https://qq.com <class 'concurrent.futures._base.TimeoutError'>
Oops: https://sina.com.cn <class 'concurrent.futures._base.TimeoutError'>
Oops: https://tianya.cn <class 'concurrent.futures._base.TimeoutError'>
Oops: https://chinadaily.com.cn <class 'concurrent.futures._base.TimeoutError'>
Oops: https://rakuten.co.jp <class 'concurrent.futures._base.TimeoutError'>
Oops: https://yahoo.co.jp <class 'concurrent.futures._base.TimeoutError'>
Oops: https://hao123.com <class 'concurrent.futures._base.TimeoutError'>
Oops: https://xinhuanet.com <class 'concurrent.futures._base.TimeoutError'>
Oops: https://soso.com <class 'concurrent.futures._base.TimeoutError'>
Done in  5.01745690300595

It obviously shows: cancellation by timeout is performed by per-task basis.

[yeryry]

@asvetlov I believe your output (with all those TimeoutError at the end) is showing a bug, which I have also been encountering. As soon as one URL hits a timeout (whether you set it using Timeout() or not), future get attempts will time out immediately.
I have a list of several hundred URLs to try, wasn't setting a timeout (so the default is 5 minutes), and have reduced the connection pool size from the default 20. After about 5 minutes of pages downloading fine, I get a flood of timeout errors for almost all remaining pages (with no time at all between, let alone 5 minutes). I believe a particular pooled connection "dies" when it times out, and remains dead, which allows it to quickly "handle" requests it is given (but by timing out immediately), resulting in that connection being passed more URLs than the other still-working ones.

I'm not sure I'm doing things right, and had to look for a get-multiple-pages example (having one of these in the docs would be nice), but I've used basically the same procedure as dev169, and ended up with a similar problem as you both appear to have had, so I don't think it's can just be me doing something wrong. Using 1.1.6.

[uogbuji]

I have the same problem under 1.2.0. Definitely still a bug. I don't even know if @asvetlov is still monitoring this, because he closed it in July. I'll check back in a while before opening a dupe.

[fafhrd91]

@uogbuji could #1524 solve this problem?

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.