Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

make PrestoSerializer compression compatible with Presto and add stats #8851

Closed
wants to merge 1 commit into from
Closed

make PrestoSerializer compression compatible with Presto and add stats #8851

wants to merge 1 commit into from

Commits on Mar 14, 2024

  1. make PrestoSerializer compression compatible with Presto and add stats ( 

    facebookincubator#8851)

Summary:
Presto uses the uncompressed size as part of checksum, not the compressed size. Presto also does not use the compression result unless it is under 0,8 x the source size. Presto does not require the compressed bit to be set if a compressing exchange codec is in place.

Adds counters to serializers for tracking compression efficacity.

Adds a setting to OutputBufferManager to decide if compression should be attempted for exchange.

Compatibly with Presto, uses uncompressed output if compression yields less then 20% savings. In addition to Presto, skips attempting compression for a number of forthcoming batches if compression was not effective.

Reports the compression input bytes, compressed bytes and bytes for which compression was not attempted in runtimeStats.


Reviewed By: xiaoxmeng

Differential Revision: D54178975

Pulled By: oerling
    @facebook-github-bot
    Orri Erling authored and facebook-github-bot committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    0fc334f View commit details
    Browse the repository at this point in the history