Define web.Application.on_startup() handler

[f0t0n]

What do these changes do?

According to discussion in #1092 next changes were made:

• Define web.Application.on_startup() signal handler.
  • Run app.on_startup() along with the request handler within an event
    loop in web.run_app() and in worker.GunicornWebWorker().

Are there changes in behavior for the user?

Related issue number

aio-libs#1092

Checklist

• I think the code is well written
• Unit tests for the changes exist
• Documentation reflects the changes

[f0t0n]

Existing Problems

• Tests in test_worker.py don't pass on the master branch and complain on pytest.skip():

―――――――――――――――――――――――――――――――――――――――――――― ERROR collecting tests/test_worker.py ―――――――――――――――――――――――――――――――――――――――――――――
Using @pytest.skip outside a test (e.g. as a test function decorator) is not allowed. Use @pytest.mark.skip or @pytest.mark.skipif instead.

• Tests in test_run_app.py don't pass because loop is mocked with wrap parameter and loop.create_task(), loop.create_server() return mock (If I mock loop.create_task.return_value._source_traceback = [], it starts to complain on loop.create_server()):

―――――――――――――――――――――――――――――――――――――――――――――――――――――― test_run_app_http ―――――――――――――――――――――――――――――――――――――――――――――――――――――――

loop = <Mock spec='AbstractEventLoop' id='140501275451744'>

    def test_run_app_http(loop):
        loop = mock.Mock(spec=asyncio.AbstractEventLoop, wrap=loop)
        loop.call_later(0.01, loop.stop)

        app = web.Application(loop=loop)

>       web.run_app(app)

tests/test_run_app.py:14: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
aiohttp/web.py:287: in run_app
    loop=loop))
/usr/local/lib/python3.5/asyncio/tasks.py:607: in gather
    fut = ensure_future(arg, loop=loop)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

coro_or_future = <generator object Application.startup at 0x7fc900ac0200>

    def ensure_future(coro_or_future, *, loop=None):
        """Wrap a coroutine or an awaitable in a future.

        If the argument is a Future, it is returned directly.
        """
        if isinstance(coro_or_future, futures.Future):
            if loop is not None and loop is not coro_or_future._loop:
                raise ValueError('loop argument must agree with Future')
            return coro_or_future
        elif coroutines.iscoroutine(coro_or_future):
            if loop is None:
                loop = events.get_event_loop()
            task = loop.create_task(coro_or_future)
            if task._source_traceback:
>               del task._source_traceback[-1]
E               TypeError: 'Mock' object doesn't support item deletion

/usr/local/lib/python3.5/asyncio/tasks.py:541: TypeError

 tests/test_run_app.pytest_run_app_http ⨯                                                                         54% █████▍    

―――――――――――――――――――――――――――――――――――――――――――――――――――――― test_run_app_https ――――――――――――――――――――――――――――――――――――――――――――――――――――――

loop = <Mock spec='AbstractEventLoop' id='140501275453648'>

    def test_run_app_https(loop):
        loop = mock.Mock(spec=asyncio.AbstractEventLoop, wrap=loop)
        loop.call_later(0.01, loop.stop)

        app = web.Application(loop=loop)

        ssl_context = ssl.create_default_context()

>       web.run_app(app, ssl_context=ssl_context)

tests/test_run_app.py:29: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
aiohttp/web.py:287: in run_app
    loop=loop))
/usr/local/lib/python3.5/asyncio/tasks.py:607: in gather
    fut = ensure_future(arg, loop=loop)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

coro_or_future = <generator object Application.startup at 0x7fc9009ee3b8>

    def ensure_future(coro_or_future, *, loop=None):
        """Wrap a coroutine or an awaitable in a future.

        If the argument is a Future, it is returned directly.
        """
        if isinstance(coro_or_future, futures.Future):
            if loop is not None and loop is not coro_or_future._loop:
                raise ValueError('loop argument must agree with Future')
            return coro_or_future
        elif coroutines.iscoroutine(coro_or_future):
            if loop is None:
                loop = events.get_event_loop()
            task = loop.create_task(coro_or_future)
            if task._source_traceback:
>               del task._source_traceback[-1]
E               TypeError: 'Mock' object doesn't support item deletion

/usr/local/lib/python3.5/asyncio/tasks.py:541: TypeError

 tests/test_run_app.pytest_run_app_https ⨯                                                                        54% █████▍    

―――――――――――――――――――――――――――――――――――――――――――――― test_run_app_nondefault_host_port ―――――――――――――――――――――――――――――――――――――――――――――――

loop = <Mock spec='AbstractEventLoop' id='140501274873136'>, unused_port = 49459

    def test_run_app_nondefault_host_port(loop, unused_port):
        port = unused_port
        host = 'localhost'

        loop = mock.Mock(spec=asyncio.AbstractEventLoop, wrap=loop)
        loop.call_later(0.01, loop.stop)

        app = web.Application(loop=loop)

>       web.run_app(app, host=host, port=port)

tests/test_run_app.py:45: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
aiohttp/web.py:287: in run_app
    loop=loop))
/usr/local/lib/python3.5/asyncio/tasks.py:607: in gather
    fut = ensure_future(arg, loop=loop)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

coro_or_future = <generator object Application.startup at 0x7fc900adce60>

    def ensure_future(coro_or_future, *, loop=None):
        """Wrap a coroutine or an awaitable in a future.

        If the argument is a Future, it is returned directly.
        """
        if isinstance(coro_or_future, futures.Future):
            if loop is not None and loop is not coro_or_future._loop:
                raise ValueError('loop argument must agree with Future')
            return coro_or_future
        elif coroutines.iscoroutine(coro_or_future):
            if loop is None:
                loop = events.get_event_loop()
            task = loop.create_task(coro_or_future)
            if task._source_traceback:
>               del task._source_traceback[-1]
E               TypeError: 'Mock' object doesn't support item deletion

/usr/local/lib/python3.5/asyncio/tasks.py:541: TypeError

 tests/test_run_app.pytest_run_app_nondefault_host_port ⨯                                                         54% █████▍    

――――――――――――――――――――――――――――――――――――――――――――――――― test_run_app_custom_backlog ――――――――――――――――――――――――――――――――――――――――――――――――――

loop = <Mock spec='AbstractEventLoop' id='140501275019416'>

    def test_run_app_custom_backlog(loop):
        loop = mock.Mock(spec=asyncio.AbstractEventLoop, wrap=loop)
        loop.call_later(0.01, loop.stop)

        app = web.Application(loop=loop)

>       web.run_app(app, backlog=10)

tests/test_run_app.py:57: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
aiohttp/web.py:287: in run_app
    loop=loop))
/usr/local/lib/python3.5/asyncio/tasks.py:607: in gather
    fut = ensure_future(arg, loop=loop)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

coro_or_future = <generator object Application.startup at 0x7fc9009ee258>

    def ensure_future(coro_or_future, *, loop=None):
        """Wrap a coroutine or an awaitable in a future.

        If the argument is a Future, it is returned directly.
        """
        if isinstance(coro_or_future, futures.Future):
            if loop is not None and loop is not coro_or_future._loop:
                raise ValueError('loop argument must agree with Future')
            return coro_or_future
        elif coroutines.iscoroutine(coro_or_future):
            if loop is None:
                loop = events.get_event_loop()
            task = loop.create_task(coro_or_future)
            if task._source_traceback:
>               del task._source_traceback[-1]
E               TypeError: 'Mock' object doesn't support item deletion

/usr/local/lib/python3.5/asyncio/tasks.py:541: TypeError

[asvetlov]

Please don't touch worker in this PR but implement on_startup signal support in web.Application and web.run_app.
Documentation and tests are required.
Worker support can be done in separate PR. This way makes review process easier.

[f0t0n]

@asvetlov Agreed. Just reverted.

[f0t0n]

tests are required.

I still can't get existing tests passing. And have no idea how to beat this problem.

[asvetlov]

Hmm.
What command do you use for test run?
I believe

pip install -U -r requirements-dev.txt
make test

should work.
At least it works on my local environment as well as on travis and appveyor.

[f0t0n]

I followed documentation: http://aiohttp.readthedocs.io/en/stable/contributing.html

pip install -r requirements-dev.txt 
make vtest

Then tried make clean && make install && make vtest.

Now I've ran

make clean
pip install -U -r requirements-dev.txt
make test

And got same errors output.

[f0t0n]

For the error with test_worker probably it's somehow related to the latest changes in pytest?

pytest.skip() now raises an error when used to decorate a test function, as opposed to its original intent (to imperatively skip a test inside a test function)...

pytest-dev/pytest#1519

[f0t0n]

For an experiment I've just changed a version of pytest in requrements to pytest==2.9.2 and the test_worker.py didn't fail.

Though the problem with that mocked loop still exists since it's not related.

[asvetlov]

I've changed run_app tests to don't use mocked loop but mock explicitly loop.create_server()

[f0t0n]

Cool! Thanks, Andrew!

I gonna rebase and continue to work on this.