ClientPayloadError on some website

[ZeusFSX]

🐞 Describe the bug

When I try to get some data from the website, I get error Response payload is not completed.
But when I use python-requests all is OK.

💡 To Reproduce

import aiohttp
import asyncio


async def fetch():
    async with aiohttp.ClientSession() as session:
        headers = {
            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) "
                          "Chrome/80.0.3987.132 Safari/537.36",
            "Accept-Language": "uk-UA,uk;q=0.9,ru-RU,ru;q=0.8,en-US;q=0.7,en;q=0.6,*;q=0.5"
        }
        remote_resp = await session.request("GET", "https://11tv.dp.ua/", ssl=False, headers=headers)
        print(remote_resp)
        await remote_resp.text()


loop = asyncio.get_event_loop()
task = asyncio.wait([fetch()])
loop.run_until_complete(task)

📋 Logs/tracebacks

<ClientResponse(https://11tv.dp.ua/) [200 OK]>
<CIMultiDictProxy('Date': 'Wed, 08 Apr 2020 08:08:51 GMT', 'Server': 'Apache', 'Vary': 'Accept-Encoding,Cookie', 'Upgrade': 'h2,h2c', 'Connection': 'Upgrade', 'Accept-Ranges': 'bytes', 'X-Mod-Pagespeed': '1.12.34.2-0', 'Cache-Control': 'max-age=0, no-cache, must-revalidate', 'Content-Encoding': 'gzip', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-XSS-Protection': '1; mode=block', 'Referrer-Policy': 'no-referrer-when-downgrade', 'X-Pingback': 'https://11tv.dp.ua/xmlrpc.php', 'Pragma': 'public', 'Link': '</wp-content/cache/minify/78a52.js?x64856>; rel=preload; as=script', 'Link': '</wp-content/cache/minify/545b0.js?x64856>; rel=preload; as=script', 'Link': '</wp-content/cache/minify/ba52a.js?x64856>; rel=preload; as=script', 'Link': '</wp-content/cache/minify/adbfe.js?x64856>; rel=preload; as=script', 'Link': '</wp-content/cache/minify/a0a57.js?x64856>; rel=preload; as=script', 'Link': '</wp-content/cache/minify/b7ead.js?x64856>; rel=preload; as=script', 'Link': '</wp-content/cache/minify/846ca.css?x64856>; rel=preload; as=style', 'Link': '</wp-content/cache/minify/aa7db.css?x64856>; rel=preload; as=style', 'Link': '</wp-content/cache/minify/39d39.css?x64856>; rel=preload; as=style', 'Transfer-Encoding': 'chunked', 'Content-Type': 'text/html; charset=UTF-8', 'Content-Language': 'uk')>


Task exception was never retrieved
future: <Task finished name='Task-2' coro=<fetch() done, defined at test_async.py:5> exception=ClientPayloadError('Response payload is not completed')>
Traceback (most recent call last):
  File "test_async.py", line 13, in fetch
    await remote_resp.text()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 1009, in text
    await self.read()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 973, in read
    self._body = await self.content.read()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aiohttp/streams.py", line 358, in read
    block = await self.readany()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aiohttp/streams.py", line 380, in readany
    await self._wait('readany')
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/aiohttp/streams.py", line 296, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

📋 Your version of the Python

$ python --version
3.8.2

📋 Your version of the aiohttp/yarl/multidict distributions

$ python -m pip show aiohttp
3.6.2

$ python -m pip show multidict
Version: 4.7.4

$ python -m pip show yarl
Version: 1.4.2

📋 Additional context

[kolovrashka]

Also encountered this.
A bug is played if the last chunk('0\r\n\r\n') was split into 2 data_received calls. '...0\r\n' & '\r\n'

mini test

import aiohttp
import asyncio


def wrap(data_received):
    def rec(data):
        if data[-5:] == b'0\r\n\r\n':
            data_received(data[:-5])
            data_received(b'0\r\n')
            data_received(b'\r\n')
            print('wait infinity')
        else:
            data_received(data)
    return rec


async def test(): 
    size = 0
    async with aiohttp.ClientSession() as s:
        async with s.get('https://google.com') as resp:
            resp.raise_for_status()
            protocol = resp.connection._protocol
            protocol.data_received = wrap(protocol.data_received)
            async for chunk in resp.content.iter_any():
                size += len(chunk)
    return size


asyncio.run(test())

[ppodolsky]

Second the issue. I have hundred requests per minute to download files using aiohttp at our backends and ClientPayloadError is stably reproduced for about 10% of requests.

[hh-h]

Unfortunately, aiohttp 3.6.2 doesn't FULLY support python3.8 yet, please, try with python 3.7

[kolovrashka]

@hh-h bag reproduced with python 3.7 and HttpResponseParserPy

[asvetlov]

Duplicate for #4630