Add parquet-layout binary #3269

tustvold · 2022-12-04T21:08:57Z

Which issue does this PR close?

Closes #.

Rationale for this change

Frequently when debugging an issue the first port of call is working out the physical layout of the data in the parquet file, what indexes are present, what encodings are being used, how large the pages are, etc...

I have been unable to find such a tool, so I quickly wrote one up to replace the ad-hoc code I keep having to write 😅

parquet-testing/data/nested_lists.snappy.parquet

{
  "row_groups": [
    {
      "columns": [
        {
          "path": "a.list.element.list.element.list.element",
          "has_offset_index": false,
          "has_column_index": false,
          "has_bloom_filter": false,
          "pages": [
            {
              "compression": "snappy",
              "encoding": "plain_dictionary",
              "page_type": "dictionary",
              "compressed_bytes": 30,
              "uncompressed_bytes": 30,
              "header_bytes": 13,
              "num_values": 6
            },
            {
              "compression": "snappy",
              "encoding": "plain_dictionary",
              "page_type": "data_page_v1",
              "compressed_bytes": 34,
              "uncompressed_bytes": 33,
              "header_bytes": 27,
              "num_values": 18
            }
          ]
        },
        {
          "path": "b",
          "has_offset_index": false,
          "has_column_index": false,
          "has_bloom_filter": false,
          "pages": [
            {
              "compression": "snappy",
              "encoding": "plain_dictionary",
              "page_type": "dictionary",
              "compressed_bytes": 6,
              "uncompressed_bytes": 4,
              "header_bytes": 13,
              "num_values": 1
            },
            {
              "compression": "snappy",
              "encoding": "plain_dictionary",
              "page_type": "data_page_v1",
              "compressed_bytes": 4,
              "uncompressed_bytes": 2,
              "header_bytes": 33,
              "num_values": 3
            }
          ]
        }
      ]
    }
  ]
}

parquet-testing/data/data_index_bloom_encoding_stats.parquet

{
  "row_groups": [
    {
      "columns": [
        {
          "path": "String",
          "has_offset_index": true,
          "has_column_index": true,
          "has_bloom_filter": true,
          "pages": [
            {
              "compression": "gzip",
              "encoding": "plain",
              "page_type": "data_page_v1",
              "compressed_bytes": 127,
              "uncompressed_bytes": 138,
              "header_bytes": 25,
              "num_values": 14
            }
          ]
        }
      ]
    }
  ]
}

parquet-testing/data/alltypes_dictionary.parquet

{
  "row_groups": [
    {
      "columns": [
        {
          "path": "id",
          "has_offset_index": false,
          "has_column_index": false,
          "has_bloom_filter": false,
          "pages": [
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "dictionary",
              "compressed_bytes": 8,
              "uncompressed_bytes": 8,
              "header_bytes": 13,
              "num_values": 2
            },
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "data_page_v1",
              "compressed_bytes": 9,
              "uncompressed_bytes": 9,
              "header_bytes": 17,
              "num_values": 2
            }
          ]
        },
        {
          "path": "bool_col",
          "has_offset_index": false,
          "has_column_index": false,
          "has_bloom_filter": false,
          "pages": [
            {
              "compression": null,
              "encoding": "plain",
              "page_type": "data_page_v1",
              "compressed_bytes": 7,
              "uncompressed_bytes": 7,
              "header_bytes": 17,
              "num_values": 2
            }
          ]
        },
        {
          "path": "tinyint_col",
          "has_offset_index": false,
          "has_column_index": false,
          "has_bloom_filter": false,
          "pages": [
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "dictionary",
              "compressed_bytes": 8,
              "uncompressed_bytes": 8,
              "header_bytes": 13,
              "num_values": 2
            },
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "data_page_v1",
              "compressed_bytes": 9,
              "uncompressed_bytes": 9,
              "header_bytes": 17,
              "num_values": 2
            }
          ]
        },
        {
          "path": "smallint_col",
          "has_offset_index": false,
          "has_column_index": false,
          "has_bloom_filter": false,
          "pages": [
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "dictionary",
              "compressed_bytes": 8,
              "uncompressed_bytes": 8,
              "header_bytes": 13,
              "num_values": 2
            },
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "data_page_v1",
              "compressed_bytes": 9,
              "uncompressed_bytes": 9,
              "header_bytes": 17,
              "num_values": 2
            }
          ]
        },
        {
          "path": "int_col",
          "has_offset_index": false,
          "has_column_index": false,
          "has_bloom_filter": false,
          "pages": [
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "dictionary",
              "compressed_bytes": 8,
              "uncompressed_bytes": 8,
              "header_bytes": 13,
              "num_values": 2
            },
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "data_page_v1",
              "compressed_bytes": 9,
              "uncompressed_bytes": 9,
              "header_bytes": 17,
              "num_values": 2
            }
          ]
        },
        {
          "path": "bigint_col",
          "has_offset_index": false,
          "has_column_index": false,
          "has_bloom_filter": false,
          "pages": [
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "dictionary",
              "compressed_bytes": 16,
              "uncompressed_bytes": 16,
              "header_bytes": 13,
              "num_values": 2
            },
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "data_page_v1",
              "compressed_bytes": 9,
              "uncompressed_bytes": 9,
              "header_bytes": 17,
              "num_values": 2
            }
          ]
        },
        {
          "path": "float_col",
          "has_offset_index": false,
          "has_column_index": false,
          "has_bloom_filter": false,
          "pages": [
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "dictionary",
              "compressed_bytes": 8,
              "uncompressed_bytes": 8,
              "header_bytes": 13,
              "num_values": 2
            },
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "data_page_v1",
              "compressed_bytes": 9,
              "uncompressed_bytes": 9,
              "header_bytes": 17,
              "num_values": 2
            }
          ]
        },
        {
          "path": "double_col",
          "has_offset_index": false,
          "has_column_index": false,
          "has_bloom_filter": false,
          "pages": [
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "dictionary",
              "compressed_bytes": 16,
              "uncompressed_bytes": 16,
              "header_bytes": 13,
              "num_values": 2
            },
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "data_page_v1",
              "compressed_bytes": 9,
              "uncompressed_bytes": 9,
              "header_bytes": 17,
              "num_values": 2
            }
          ]
        },
        {
          "path": "date_string_col",
          "has_offset_index": false,
          "has_column_index": false,
          "has_bloom_filter": false,
          "pages": [
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "dictionary",
              "compressed_bytes": 12,
              "uncompressed_bytes": 12,
              "header_bytes": 13,
              "num_values": 1
            },
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "data_page_v1",
              "compressed_bytes": 9,
              "uncompressed_bytes": 9,
              "header_bytes": 17,
              "num_values": 2
            }
          ]
        },
        {
          "path": "string_col",
          "has_offset_index": false,
          "has_column_index": false,
          "has_bloom_filter": false,
          "pages": [
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "dictionary",
              "compressed_bytes": 10,
              "uncompressed_bytes": 10,
              "header_bytes": 13,
              "num_values": 2
            },
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "data_page_v1",
              "compressed_bytes": 9,
              "uncompressed_bytes": 9,
              "header_bytes": 17,
              "num_values": 2
            }
          ]
        },
        {
          "path": "timestamp_col",
          "has_offset_index": false,
          "has_column_index": false,
          "has_bloom_filter": false,
          "pages": [
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "dictionary",
              "compressed_bytes": 24,
              "uncompressed_bytes": 24,
              "header_bytes": 13,
              "num_values": 2
            },
            {
              "compression": null,
              "encoding": "plain_dictionary",
              "page_type": "data_page_v1",
              "compressed_bytes": 9,
              "uncompressed_bytes": 9,
              "header_bytes": 17,
              "num_values": 2
            }
          ]
        }
      ]
    }
  ]
}

What changes are included in this PR?

Are there any user-facing changes?

alamb · 2022-12-05T00:34:28Z

I will review this tomorrow -- the other one I know of is https://github.com/manojkarthick/pqrs which shows promise

alamb · 2022-12-05T18:13:19Z

parquet/src/bin/parquet-layout.rs

+// specific language governing permissions and limitations
+// under the License.
+
+//! Binary that prints the physical layout of a parquet file


👍

As a general musing, I think some of these cli helpers are quite nice and maybe eventually we could make them more discoverable / nicer to people who are not working with the parquet source code.

I didn't find anything other than https://github.com/apache/arrow-rs/tree/master/parquet/src/bin for documentation

Perhaps to prqs or something similar 🤔

alamb · 2022-12-05T18:15:01Z

parquet/src/bin/parquet-layout.rs

+
+                    let end = start + column.compressed_size() as u64;
+                    while start != end {
+                        let (header_len, header) = read_page_header(reader, start)?;


this is neat

alamb · 2022-12-05T18:15:58Z

parquet/src/bin/parquet-layout.rs

+                                num_values: data_page.num_values,
+                            })
+                        } else if let Some(data_page) = header.data_page_header_v2 {
+                            let is_compressed = data_page.is_compressed.unwrap_or(true);


is it really compressed by default? I expected unwrap_or(false)

If the column is compressed, and the header doesn't specify, the default is compressed. I think this was a later extension to make it optional

ursabot · 2022-12-05T22:14:44Z

Benchmark runs are scheduled for baseline = b155461 and contender = 94d597e. 94d597e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Add parquet-layout binary

41587c7

tustvold requested a review from alamb December 4, 2022 21:08

github-actions bot added the parquet Changes to the parquet crate label Dec 4, 2022

tustvold added 3 commits December 4, 2022 21:11

Docs

0dc5331

Clippy

6194dee

Add RowGroup row count

ebca004

alamb approved these changes Dec 5, 2022

View reviewed changes

tustvold merged commit 94d597e into apache:master Dec 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add parquet-layout binary #3269

Add parquet-layout binary #3269

tustvold commented Dec 4, 2022

alamb commented Dec 5, 2022

alamb Dec 5, 2022

alamb Dec 5, 2022

alamb Dec 5, 2022

tustvold Dec 5, 2022

ursabot commented Dec 5, 2022

Add parquet-layout binary #3269

Add parquet-layout binary #3269

Conversation

tustvold commented Dec 4, 2022

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

alamb commented Dec 5, 2022

alamb Dec 5, 2022

Choose a reason for hiding this comment

alamb Dec 5, 2022

Choose a reason for hiding this comment

alamb Dec 5, 2022

Choose a reason for hiding this comment

tustvold Dec 5, 2022

Choose a reason for hiding this comment

ursabot commented Dec 5, 2022