async request / socket ? #16590

pn2new · 2025-01-15T15:36:56Z

pn2new
Jan 15, 2025

I am trying to establish a connection to a server, and would then like to keep the connection open and read the response line by line, in a non-blocking (async) way. The url uses https://.

So far, I tried to use requests, which could work but does not provide an option for line by line reading, and is not async. Looking into the code of request, I think the line by line reading could be added. However, this still is not async.

Is there a library that would allow me to do that without having to re-invent the wheel? What I am trying to achieve is something like this (works on "non-micro" python), but as mentioned everything should be async:

def poll_ntfy_messages():
    url = f"{SERVER}/{TOPIC}/json?since=60m"
    try:
        response = requests.get(url, stream=True)
        for line in response.iter_lines():
            if line:
                print(line)
        response.close()
    except Exception as e:
        print("Error polling messages:", e)

Answered by Carglglz

Jan 16, 2025

I think you need to specify HTTP/1.1 version for the ClientSession e.g. get.py example returns Transfer-Encoding: chunked,

import sys

# ruff: noqa: E402
sys.path.insert(0, ".")
import aiohttp
import asyncio


URL = sys.argv.pop()

if not URL.startswith("http"):
    URL = "http://micropython.org"

print(URL)


async def fetch(client):
    async with client.get(URL) as resp:
        assert resp.status == 200
        return await resp.text()


async def main():
    async with aiohttp.ClientSession(version=aiohttp.HttpVersion11) as client:
        html = await fetch(client)
        print(html)


if __name__ == "__main__":
    asyncio.run(main())

printing headers

 $ micropython examples/get.p…

View full answer

bixb922 · 2025-01-15T17:47:55Z

bixb922
Jan 15, 2025

Have you taken s look at aiohttp in micropython-lib?

4 replies

pn2new Jan 16, 2025
Author

I am trying to get this running with aiohttp. Reading the response in one piece works:

async def async_ntfy():
    url = f"{SERVER}/{TOPIC}/json?since=600m&poll=1"
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            html = await response.text()
            print(html)

asyncio.run(async_ntfy())

However, I do not manage to read the response line-by-line, i.e. in a "chunked" way. On my linux box, with the same server, this works (code in my initial post). So this is not an issue with the server, but the client side.

Here is what I have tried (on my ESP32):

async def async_ntfy():
    url = f"{SERVER}/{TOPIC}/json?since=600m&poll=0"
    async with aiohttp.ClientSession() as session:
        pprint(session)
        async with session.get(url, headers = { "TE": "chunked" }) as response:
            print("Status:", response.status)
            print("Content-type:", response.headers['Content-Type'])
            pprint(response.headers)
            # async for chunk in response.content.???
            #     print(chunk)
            pprint(response)
            html = await response.text()
            print(html)

asyncio.run(async_ntfy())

The output I get is (I added some print statements to the aiohttp/__init__.py file to see what is going on):

## request headers
{'User-Agent': 'compat', 'TE': 'chunked', 'Connection': 'close'}

sline b'HTTP/1.1 200 OK\r\n'
line=b'Server: nginx\r\n'
line=b'Date: Thu, 16 Jan 2025 14:03:38 GMT\r\n'
line=b'Content-Type: application/x-ndjson; charset=utf-8\r\n'
line=b'Connection: close\r\n'
line=b'Access-Control-Allow-Origin: *\r\n'
line=b'\r\n'

chunked: False

Somehow the server does not respond with "Transfer-Encoding: chunked"... But as said, in the "desktop python" code above, this works, with the same server on the other end...

Carglglz Jan 16, 2025

I think you need to specify HTTP/1.1 version for the ClientSession e.g. get.py example returns Transfer-Encoding: chunked,

import sys

# ruff: noqa: E402
sys.path.insert(0, ".")
import aiohttp
import asyncio


URL = sys.argv.pop()

if not URL.startswith("http"):
    URL = "http://micropython.org"

print(URL)


async def fetch(client):
    async with client.get(URL) as resp:
        assert resp.status == 200
        return await resp.text()


async def main():
    async with aiohttp.ClientSession(version=aiohttp.HttpVersion11) as client:
        html = await fetch(client)
        print(html)


if __name__ == "__main__":
    asyncio.run(main())

printing headers

 $ micropython examples/get.py
http://micropython.org
b'GET / HTTP/1.1\r\nUser-Agent: compat\r\nHost: micropython.org\r\nConnection: close\r\n\r\n'
b'Server: nginx/1.10.3\r\n'
b'Date: Thu, 16 Jan 2025 19:01:28 GMT\r\n'
b'Content-Type: text/html; charset=utf-8\r\n'
b'Transfer-Encoding: chunked\r\n'
b'Connection: close\r\n'
b'Vary: Accept-Encoding\r\n'
b'X-Frame-Options: SAMEORIGIN\r\n'
b'Strict-Transport-Security: max-age=15768000\r\n'
b'\r\n'
<!DOCTYPE html>

...

Not sure if you can use Transfer-Encoding in the headers, see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding

Answer selected by pn2new

pn2new Jan 16, 2025
Author

Thanks, adding version=aiohttp.HttpVersion11 indeed was the solution to get a chunked response! I did not see this because the response I got was b'HTTP/1.1 200 OK\r\n'.

Maybe you can also help me with iterating over the result by chunk?
I tried some async for constructs, but these all did not work.
What worked was something like below, but this seems awkward.

async def async_ntfy():
    url = f"{SERVER}/{TOPIC}/json?since=60m"
    async with aiohttp.ClientSession(version=aiohttp.HttpVersion11) as session:
        async with session.get(url) as response:
            # this does not work
            # async for line in response.content:
            #     print(f"line={line}")
            while True:
                line = await response.content.readline() 
                # extend the above by getting length of response, and then response, and then skip '\r\n'
                print(f"line={line}")

Carglglz Jan 17, 2025

Maybe you can also help me with iterating over the result by chunk?

I think your solution is good, as long as the response content is not encoded.
The alternative is to use await response.read(chunk_size) see https://docs.aiohttp.org/en/stable/client_quickstart.html#streaming-response-content

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MicroPython

async request / socket ? #16590

{{title}}

Replies: 1 comment 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

MicroPython

async request / socket ? #16590

pn2new Jan 15, 2025

Replies: 1 comment · 4 replies

bixb922 Jan 15, 2025

pn2new Jan 16, 2025 Author

Carglglz Jan 16, 2025

pn2new Jan 16, 2025 Author

Carglglz Jan 17, 2025

pn2new
Jan 15, 2025

Replies: 1 comment 4 replies

bixb922
Jan 15, 2025

pn2new Jan 16, 2025
Author

pn2new Jan 16, 2025
Author