Support for buffered/chunked queries with Gino

[mattyweb]

A little background:
I have an API supporting a web app that occasionally needs to generate a big (6m+ record) CSV file. I'm running my app in wimpy containers and gunicorn kills the worker if I try to load all the data into memory.

I'm wondering is there a way to chuck my query so I can write it to file as I go and keep my resource utilization to a minimum?

What I've looked at

I'm hoping to do something like described in this SO post but am unsure if it's supported by Gino:

    with engine.begin() as conn:
        conn = conn.execution_options(stream_results=True)
        results = conn.execute("")
        while True:
            chunk = results.fetchmany(10000)
            if not chunk:
                break
            csv_buffer = StringIO()
            csv_writer = csv.writer(csv_buffer, delimiter=';')
            csv_writer.writerows(chunk)
            yield csv_buffer.getvalue().encode("utf-8")

I see that Gino will pass through execution options to SQL Alchemy, but I don't see any fetchmany equivalent. I see some examples that use transactions and cursors, but am hoping there's a better way.

Architecturally, I'm hoping to do all this on my web server and do a callback when it's done. If Gino can't support this I'll do something like an AWS lambda writing to an S3 bucket but I'd rather not have to take those steps if there is an easier option.

Anyone know if Gino can do what I'm looking for?