Code

Feature: added parquet sampling#1070

Code

Merged

taylorfturner merged 11 commits intocapitalone:devcapitalone/DataProfiler:devfrom menglinw:parquet-samplingmenglinw/DataProfiler:parquet-samplingCopy head branch name to clipboardDec 12, 2023

+138-31

Commits

Commits on Nov 16, 2023

• parquet sampling function developed in data_utils.py; Added sample_nrows argument in ParquetData class; Added test_len_sampled_data in test_parquet_data.py

  menglinw

  committed
  09a868d

• resolved conflict with dev, added more tests

  menglinw

  committed
  a17b4df

Commits on Nov 17, 2023

• fixed sample empty column bug

  menglinw

  committed
  05b1940

Commits on Nov 21, 2023

• fixed comments in data_utils.py, including:

  Show description for 1fbc722
  menglinw

  committed
  1fbc722

• 1. renamed variable names in covert_unicode_col_to_utf8 function (data_utils.py) to be more descriptive (types -> input_column_types, col -> iter_column), other part unchanged

  Show description for ba7f7d7
  menglinw

  committed
  ba7f7d7

Commits on Dec 7, 2023

• checked the datatype and input file path before and after reload with sampling option enabled

  menglinw

  committed
  e60dda6

Commits on Dec 8, 2023

• test

  menglinw

  committed
  897ffb6

• delete test edit in avro_data.py, updated fastavro version in requirment.txt

  menglinw

  committed
  8c872bc

• remove fastavro.reader type

  menglinw

  committed
  07e755f

Commits on Dec 11, 2023

• change fastavro version back to original

  menglinw

  committed
  80993bc

• 1. sample_parquet function description

  Show description for fa38188
  menglinw

  committed
  fa38188