-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Posting big files causes OOM and has a very strange memory usage #2669
Comments
I don't know what MP{X} abbreviation is for and what your test does.
In case of non-ascii data memory consumption doubles at least. You file is 2.5 GB long. Multiply to 2.5. |
Measure points, in time axis when the post is processed. No I don't except low memory footprint. But as soon as the post is being processed the memory is hopping up and down until oom. This has nothing to do with the read of the file and the decoding. Until the post, memory usage is as expected. |
The only thing I can suggest is converting body back to bytes before calling aiohttp: P.S. |
Sadly, same behavior. Memory is going crazy and server runs out of memory. The API is used by a customer and many others application send big files, so they just say it's my problem when it's not working with python... Maybe I can profile it to see why this is happening. Still don't know the source of this after a specific limit is reached. |
@ManuelGysin what is memory usage before session.post() call but after formatting string? |
File is in real 2379 MB, memory usage before post is 5596 MB. So we have at least a tripling of the usage within the post code path. |
How about streaming http://aiohttp.readthedocs.io/en/stable/client.html#streaming-uploads ? |
aiohttp has internal buffers for sent data, the memory duplication at the moment of transmitting is unavoidable. Loading a huge data into a memory is a bad idea anyway, moving gigabyte sized buffers in memory forth and back acts as a blocking call. Streaming upload is the proper solution. |
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs. |
Long story short
Sending very large file with POST causes memory leak and OOM on server.
Expected behaviour
Does not causing a memory leak.
Actual behaviour
Eats memory in a very strange way:
MP1: 2GB
MP2: 8GB
MP3: 6GB
MP4: 5GB
MP5: 12GB
MP6: 10GB
MP7: 15GB
MP8: Crash because OOM
Memory is going up and down within seconds until the process crashes.
Steps to reproduce
If the file is around 2.4 GB things are starting to getting broken.
Your environment
Linux: Fedora 27
Python: 3.6.3
aiohttp: 2.3.8 Client
The text was updated successfully, but these errors were encountered: