No need construct def_level_decoder and rep_level_decoder in each page.

[Ted-Jiang]

arrow-rs/parquet/src/arrow/record_reader/mod.rs

Lines 97 to 120 in 37dd037

	/// Set the current page reader.
	pub fn set_page_reader(&mut self, page_reader: Box<dyn PageReader>) -> Result<()> {
	let descr = &self.column_desc;
	let values_decoder = CV::new(descr);
	let def_level_decoder = (descr.max_def_level() != 0).then(|| {
	DefinitionLevelBufferDecoder::new(
	descr.max_def_level(),
	packed_null_mask(descr),
	)
	});
	let rep_level_decoder = (descr.max_rep_level() != 0)
	.then(|| ColumnLevelDecoderImpl::new(descr.max_rep_level()));
	self.column_reader = Some(GenericColumnReader::new_with_decoders(
	self.column_desc.clone(),
	page_reader,
	values_decoder,
	def_level_decoder,
	rep_level_decoder,
	));
	Ok(())
	}

def_level_decoder and rep_level_decoder are base one the the column_desc (column chunk level). No need to create them in each page.

[Ted-Jiang]

@tustvold we can create it when new ColChunk, am i right?

[tustvold]

PageReader is an iterator of Page within a ColumnChunk, so they are being created once for each column chunk.

Theoretically we could create them once per file, in practice I wouldn't expect it to make a difference. The columns chunks should be large enough that any overhead will be amortised to insignificance...

[tustvold]

Can this be closed?

[Ted-Jiang]

Sure