Concurrency backends vs AnyIO

[florimondmanca]

Follow-up for @sethmlarson's comment here: #276 (comment)

Just wondering since I just found the project, would anyio work for us?

For context: we have an internal ConcurrencyBackend interface whose job is to provide async-related operations that aren't tied to a particular implementation, e.g. asyncio or trio.

AnyIO (docs, code, cc @agronholm) is an existing project that takes the exact similar approach to provide an async API that somewhat resembles that of trio.

Sample code taken from here:

import anyio


async def sometask(num):
    print('Task', num, 'running')
    await anyio.sleep(1)
    print('Task', num, 'finished')


async def main():
    async with anyio.create_task_group() as group:
        for num in range(5):
            await group.spawn(sometask, num)

    print('All tasks finished!')


anyio.run(main)

I think @sethmlarson's question was really: should we consider replacing our ConcurrencyBackend API with AnyIO?

[florimondmanca]

I think there are several unknowns relative to this question:

• How much feature ground does AnyIO cover compared to our concurrency backends?
• How well does it cover them? (Bugs?)
• How do we make sure AnyIO stays well-maintained for our purposes? (I think it's obvious we'd need to share the maintenance load.)

I think the first two questions can be answered by prototyping an AnyIOBackend that would implement our current API, and see if it passes the test suite.

I quite like the idea of HTTPX driving the construction of a generic async interface that can run in various environments. That's what we've been building with concurrency backends anyway, and I have the intuition it could serve the broader community as well. Similar to how pytest seems to have popularized pluggy, HTTPX could popularize AnyIO.

[agronholm]

I had not heard of HTTPX before, but there is an HTTP client library already depending on anyio: asks. What I do not understand is why people don't work together on a single HTTP client implementation. HTTP is hard to get right, as you may have noticed, and having several separate projects working towards the same goal is something I see as a colossal waste of developer manpower. And the established urllib3 codebase is getting async support here.

[florimondmanca]

Thanks for pitching in @agronholm! I personally did not know about asks. I think it's a blessing to have an existing medium-large project that shows real-life usage examples of AnyIO.

On your comments about the multiplication of efforts — I know there has been discussions about this for HTTPX and async-urllib3 in #78. See #78 (comment) and #78 (comment) in particular. I wasn't personally involved in those discussions, but from my understanding the convergence (or at least some sort of collaboration) between the two projects has never been made out of the way. :) Also, I believe the will to converge efforts is what lead to the creation of the Python HTTP Working Group. cc @tomchristie

(I personally don't think it's that bad to currently have multiple attempts at building a user-friendly, async-compatible, performant and reliable HTTP client — these are experimental times. Sans-IO libraries already help a lot in not having to reimplement HTTP again and again, and I think projects like AnyIO can help with the pan-async side of things as well.)

I don't think it's a good idea to revive a wider-scope community discussion right here though — maybe #78 or the python-http discussions are better places to share and discuss those ideas. Let's keep this issue focused on HTTPX + AnyIO?

[tomchristie]

should we consider replacing our ConcurrencyBackend API with AnyIO?

Given where we are now, I don't think I see a clear enough benefit for us there, no.
(Tho it looks like an ace project.)

[florimondmanca]

Agreed with your position @tomchristie, thanks for your input!

[sethmlarson]

I think there are clear benefits, we're basically re-implementing a subset of work that's already been proven and we end up getting Trio support along with Curio.

[tomchristie]

More finessed opinion - I'd think we could be okay with considering a PR for it, but I think it brings us some submerged complexity vs. explicit complexity, and we have been trying to prefer fewer dependencies where possible, so it's not super obvious either way which way we'd want to lean.

[tomchristie]

It'd cost us a bunch of work, and it doesn't get us any closer to 1.0, so putting time into it probably ought to be contingent on an interested party deciding to take a crack at it, right?

[agronholm]

One question: has anyone evaluated AnyIO to see if it could serve as a replacement for your concurrency backends? Does it have all the necessary features?

[tomchristie]

From a glance over the docs it looks like it's got everything we'd need, yup. (But that's about as much depth as I'd gone into)

[tomchristie]

Counter-intuitively, the low-impact way of exploring this would be to implement an alternate concurrency backend, that uses AnyIO.

[agronholm]

I would happily participate in the discussion at #335 but I cannot as it's limited to collaborators only :)