Creating parquet table partitioned on a boolean column throws error

[georgecwan]

Description

Creating a parquet table partitioned on a column of boolean type and then running j_object.getColumnSources() will cause an exception.

Steps to reproduce

1. Run the following code snippet:

from deephaven import empty_table

part = empty_table(4).update("II=ii")

from deephaven.parquet import write, read

write(part, "/tmp/bool-test/boolCol=true/part.parquet")
write(part, "/tmp/bool-test/boolCol=false/part.parquet")

bool_partition = read("/tmp/bool-test")

2. Run bool_partition.j_object.getColumnSources()

[rcaudy]

First step, we need to understand if the issue is with CSV-based type-inference or with the Python code for adapting definitions.

[niloc132]

I think this is a Java bug - readPartitionTableInferSchema always unboxes primitives instead of handling boolean specially:

deephaven-core/extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTools.java

Lines 688 to 689 in 0d17b64

	allColumns.add(ColumnDefinition.fromGenericType(partitionKey,
	getUnboxedTypeIfBoxed(partitionValue.getClass()), null, ColumnDefinition.ColumnType.Partitioning));

Contrast with ParquetMetadataFileLayout's adjustPartitionDefinition, which special cases boolean:

deephaven-core/extensions/parquet/table/src/main/java/io/deephaven/parquet/table/layout/ParquetMetadataFileLayout.java

Lines 222 to 239 in 0d17b64

	// Primitive booleans should be boxed
	final Class<?> dataType = columnDefinition.getDataType();
	if (dataType == boolean.class) {
	return ColumnDefinition.fromGenericType(
	columnDefinition.getName(), Boolean.class, null, ColumnDefinition.ColumnType.Partitioning);
	}
	// Non-boolean primitives and boxed Booleans are supported as-is
	if (dataType.isPrimitive() || dataType == Boolean.class) {
	return columnDefinition.withPartitioning();
	}
	// Non-boolean boxed primitives should be unboxed
	final Class<?> unboxedType = TypeUtils.getUnboxedTypeIfBoxed(dataType);
	if (unboxedType != dataType) {
	return ColumnDefinition.fromGenericType(
	columnDefinition.getName(), unboxedType, null, ColumnDefinition.ColumnType.Partitioning);
	}