Questions on collaboration style

[tomchristie]

This is prompted in part by an insightful comment from @agronholm...

I've got a couple of open questions around our collaboration style, and trying to figure out if we can be working on this more efficiently. Although we did explore the space, I hadn't realized that work is still actively continuing on the urllib3 branch, or that the asks project existed.

Engaging with @agronholm's question: Is there space for us to be trying to combine our efforts more fruitfully? 🍐

For example:

Right now httpx is aiming at:

• sync + async support.
• HTTP/1.1 + HTTP/2 + HTTP/3.

There's a lot of ground to cover there - I'm really proud of the work we're all doing

My intuition is asking if we're moving too fast, and also that we might be better off trying to focus on building a tool that does one paritcular kind of thing well, rather than trying to build the swiss army penknife of HTTP in Python?

requests has threaded-concurrency Python decently covered already. It may be that we'll be doing more valuable work by building towards something more differentiated?

There are also open questions along these lines, such as:

• Are we serving the community better by having Python's stdlib async gradually incorporate trio's lessons, or by focusing on trio's safety-first (even if it means an ecosystem split), or both?
• I feel like there's some hugely valuable insights in ASGI that I feel needs more discussion and exploration with the trio team. It's not obvious to me if we've adequately bridged a gap there, yet.

I'm not sure I want to solicit public opinion on this just yet, so I'll keep this issue locked for the moment, at least for a bit. Right now this is just me getting my thoughts out in the open. 🤣 🤝

[florimondmanca]

Brain-dumping some ideas here…

I've looked a bit at the urllib3 branch on the python-trio org, and I can see how much work duplication is going on. urllib3 backends is similar to our concurrency backends. They seek SOCKS support, and are starting to look at a high-level HTTP interface. I think similar questions to those your highlit here (e.g. becoming the swiss army knifepen of HTTP) might be in the minds of the urllib3 folks.

we might be better off trying to focus on building a tool that does one paritcular kind of thing well, rather than trying to build the swiss army penknife of HTTP in Python?

I think positioning HTTPX as the go-to async-capable HTTP client with a friendly user interface is a compelling story. HTTPX should do HTTP and work well with async environments (not necessarily in core, though?). I think I share your intuition on the lessons learnt from ASGI, though. I know Requests tried to build on a modular architecture via adapters, essentially aiming at becoming a platform which can be built on. I think the middleware idea we've been playing with belongs to the same realm, but maybe we can do better. I see a rather clear distinction between lower-level stuff (I/O, HTTP parsing, connection handling… i.e. the protocol abstraction layer) and higher-level stuff (pre-processing/post-processing/intercepting of requests/responses), and the pivot seems to be the request/response model. Wasn’t ASGI born from a similar situation?

Are we serving the community better by having Python's stdlib async gradually incorporate trio's lessons, or by focusing on trio's safety-first (even if it means an ecosystem split), or both?

The funny thing about AnyIO (but also our own concurrency backend) is that it kind of replicates the Trio API and incorporates its ideas. I personally start to think that Trio is a superior async library to say asyncio, but I don't think we can/should decide on a hard branch-off to trio land, simply because there's so much that's already been built around asyncio. Think Starlette, Uvicorn, databases, ORM… Most if not all of these Encode projects rely on asyncio and won't work with anything else without the same effort we've been putting into abstracting away asyncio as an async implementation.

My personal opinion on all this is that we're kind of plagued by history and the fact that async Python has put the implementation at the developer front at all. My dream would be a sans-asynclib situation, in which we rely on a set of primitives that async libraries implements Right now it seems we're finding out those primitives from existing async libraries.

[florimondmanca]

Interesting discussion about async backends: dabeaz/curio#266

[tomchristie]

Thanks @florimondmanca - some great pointers there.