Sometimes filter does not send the final piece

[eustas]

See eustas#37

[dkrutsko]

Okay, so I ran several more tests and determined that it has nothing to do with nginx-module-vts but instead, the Brotli compression itself. Disabling Brotli in Nginx or requesting an uncompressed version of the page works properly. It turns out that the download is being truncated. I tested with all available compression levels and found that some result in truncation while others don't. Below is a list of compression levels and the amount of bytes they returned:

brotli_comp_level 1: ALL
brotli_comp_level 2: 114667
brotli_comp_level 3: 114667
brotli_comp_level 4: ALL
brotli_comp_level 5: 49152
brotli_comp_level 6: 49152
brotli_comp_level 7: 49152
brotli_comp_level 8: 49152
brotli_comp_level 9: ALL

Used with the following Brotli settings to request a 128978 byte HTML file

brotli on;
brotli_buffers 32 4k;
brotli_comp_level 6;
brotli_window 512k;
brotli_min_length 8192;
brotli_types application/javascript application/json application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/x-javascript application/xml application/xml+rss font/eot font/opentype font/otf image/svg+xml image/vnd.microsoft.icon image/x-icon text/css text/javascript text/plain text/xml;

Requested using this cURL command: curl ${URL} -H 'accept-encoding: br' --compressed.

As this turns out to be a completly different issue from what it was before, perhaps the title needs to be changed or a new issue opened. But I'm not sure if this is a problem with Brotli itself or just the Nginx module.

[HansVanEijsden]

Oh, thanks so much. Perhaps this explains why I had incomplete javascript and css downloads sometimes (using compression level 6). I'll try level 4 when the 100% CPU bug has been fixed too.

[eustas]

Finally, managed to reproduce (on truncated war-and-piece with compression level 2) -> soon will be able to diagnose and fix =)

[eustas]

#83 should fix the problem.
Currently have no test for it, though.

Could you, check if it helps in your situation?

Best regards,
Eugene.

[dkrutsko]

Okay, I can confirm that my use case has been fixed across all compression levels. @HansVanEijsden would you be able to verify if this fixes your asset issue?

[HansVanEijsden]

@dkrutsko I will try, but I cannot keep it running in production because of #80 - the next 24 hours I will be away or sleeping so I cannot monitor the production instance in case of the 100% CPU problem. Let's see how it goes for the next 2 hours. I will report back my findings here.

[HansVanEijsden]

@dkrutsko the problem seems to be solved, no more truncation here at compression level 6 as far as I could see.
I had to disable ngx_brotli again unfortunately, because of the bug #80 - CPU usage went to 400% (4 cores) within 10 minutes of time and Nginx was hanging.

[dkrutsko]

@HansVanEijsden Do you know if this only happens with level 6? or level 6 and up? Is there one you recommend running in production for the time being?

[HansVanEijsden]

@dkrutsko I don't know yet. Currently I'm on the road but I am home in about 24 hours. I will experiment with it as soon as I'm home in a stable online situation. I'll let you know here!

[eustas]

Just landed #83.

[dkrutsko]

@eustas Thanks a lot for this! Are there any plans to do GitHub releases for these or do you advise just cloning master in production?

[eustas]

Going to release soon, after fixing a little bit more. Hopefully this week.