Unclosed connection

[Noctem]

Long story short

My event loop exception handler gets Unclosed connection messages from one part of my code that's using async context managers for ClientSession.

Expected behaviour

The context manager would create the session and properly close the session and connector without any errors.

Actual behaviour

My loop's exception handler receives:

{'client_connection': Connection<('sso.pokemon.com', 443, True)>, 'message': 'Unclosed connection'}

Steps to reproduce

This is where it happens in my code. I'm using sessions elsewhere without issue, the main difference here is that I'm using an async context manager to close it as soon as the requests are finished, whereas elsewhere I'm reusing sessions for the duration of execution.

Or simplified:

from asyncio import get_event_loop
from functools import partial

from aiohttp import TCPConnector, ClientSession

loop = get_event_loop()
conn = partial(TCPConnector, loop=loop, verify_ssl=False)
session = partial(
    ClientSession,
    loop=loop,
    headers=(('User-Agent', 'pokemongo/0 CFNetwork/758.5.3 Darwin/15.6.0'),),
    raise_for_status=True,
    conn_timeout=5.0,
    read_timeout=10.0)

async with session(connector=conn()) as sess:
    async with sess.get(login_url) as resp:
        data = await resp.json(encoding='utf-8', content_type=None)

It seems that error occurs when a Connection's _protocol isn't None at the time of deletion, but why isn't the context manager preventing that?

Your environment

macOS 10.12.4, Python 3.6.1, uvloop 0.8.0, aiohttp 2.0.6

[Noctem]

Creating the connector in yet another context manager seems to fix the issue (Noctem/aiopogo@aefd128), but it doesn't seem like that should be necessary. Shouldn't a connector that is created in the kwargs of a context-managed ClientSession also be cleaned up when the session is cleaned?

edit: I'm actually getting a few even with the extra context manager, though it seems less frequent.

[fafhrd91]

I can not reproduce this. it might be related to proxy.

[Noctem]

I experience the issue without using a proxy. I'll try to throw together a complete script for triggering it tonight.

[AraHaan]

I could never setup a proxy for my bot to use even. If I could then I would use it so it does not use the same IP address I use (as some servers ban by account and IP address which would also mean RIP my non bot account).

[Noctem]

Not sure if it was something that changed on my end or yours but the problem isn't happening any more so I'll close this.

[ClericPy]

aiohttp/client.py

line_no: 328

try:
    resp = req.send(conn)
    try:
        await resp.start(conn, read_until_eof)
    except BaseException:
        resp.close()
        conn.close()
        raise
except ClientError:
    raise
except OSError as exc:
    raise ClientOSError(*exc.args) from exc

Here is the uncatched exception which is raised by req.send(conn):

Cannot write to closing transport <class 'concurrent.futures._base.CancelledError'>

and then the Unclosed connection warning occurred.

traceback by add

except Exception as e:
    print(e, type(e), id(conn))

to the bottom of the former code snippet

[KimSJ]

I think there's definitely a race problem of some sort; doing a GET followed too quickly by a POST gives various errors, typically Can not write request body for ..., or sometimes [Errno 32] Broken pipe.

Inserting a short await asyncio.sleep() — as short as 0.001 seconds, in some cases — between them fixes the problem. Doing an await request.text() doesn't seem to help, however.

[Running on Ubuntu 16.04, i7 1.6GHz. Python 3.6.6, with Django server running on the same machine.]

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].
[new issue]: https://github.com/aio-libs/aiohttp/issues/new