Add support for null values #67

osopardo1 · 2022-01-25T16:26:50Z

This PR shows the current status on the solution for issue #10

The theory is to substitute/map null values in the indexed columns transformation with a Random number in the [min, max] range.

New API for LinearTransformation and HashTransformation case classes to include nullValue as part of the contained information
New API for the Transformer which includes an optionalNullValue. If none is given, then a random is calculated internally with the min/max values
New API for the user (optional):

df.write.format("qbeast").option("columnsToIndex", "column:type(nullValue),column2:type(nullValue2)...")

Adaptation of the tests to this new functionality

…exed-columns-support # Conflicts: # core/src/main/scala/io/qbeast/core/transform/HashTransformer.scala # core/src/main/scala/io/qbeast/core/transform/LinearTransformer.scala # core/src/main/scala/io/qbeast/core/transform/Transformer.scala # core/src/test/scala/io/qbeast/core/transform/TransformerTest.scala

# Conflicts: # src/test/scala/io/qbeast/spark/index/SparkPointWeightIndexerTest.scala

core/src/main/scala/io/qbeast/core/transform/LinearTransformer.scala

cugni

We are still missing the filtering push down for the null values, meaning that a query with x=null, should be converted to x>=randomNullValue and x<randomNullValue and executed using the index.

core/src/main/scala/io/qbeast/core/transform/HashTransformer.scala

core/src/main/scala/io/qbeast/core/transform/LinearTransformation.scala

core/src/main/scala/io/qbeast/core/transform/LinearTransformer.scala

…ith all null values Pending to change the problem of having all values at null

eavilaes

Just a few comments, but generally looks good! :)

core/src/main/scala/io/qbeast/core/transform/HashTransformation.scala

core/src/main/scala/io/qbeast/core/transform/LinearTransformation.scala

core/src/test/scala/io/qbeast/core/transform/LinearTransformationTest.scala

core/src/main/scala/io/qbeast/core/transform/LinearTransformer.scala

core/src/main/scala/io/qbeast/core/transform/LinearTransformation.scala

eavilaes · 2022-03-03T09:58:49Z

I've seen some typos on the new AdvancedConfiguration.md file, you could maybe check it with Grammarly! :)

osopardo1 · 2022-03-03T11:41:38Z

I've seen some typos on the new AdvancedConfiguration.md file, you could maybe check it with Grammarly! :)

Thank you, fixed!

osopardo1 · 2022-03-11T16:11:08Z

Hey! I think is ready to merge, changes are addressed and it's compatible with the current version in the main 👍

eavilaes

Looks good to me! Great job :)

use matchPattern to skip comparing nullValue in Transformation

core/src/main/scala/io/qbeast/core/transform/LinearTransformer.scala

core/src/main/scala/io/qbeast/core/transform/Transformation.scala

cugni

Perfect 👌🏻

core/src/main/scala/io/qbeast/core/transform/LinearTransformer.scala

osopardo1 added 9 commits January 25, 2022 17:17

WIP on Add random values as null substitutes

b0c5b46

Use scala Random and correct tests

e06fc36

Cast random value to the respective types

5a84c9d

Change random Int generation

6fc69e7

Fixed HashTransformation test

a72ed55

Resolved conflicts on Transformers and Transformations

3ef1880

Merge branch 'main' into 10-null-values-on-indexed-columns-support

0281a18

# Conflicts: # src/test/scala/io/qbeast/spark/index/SparkPointWeightIndexerTest.scala

Fixed SpecExtractor for nulls

83cb21f

osopardo1 marked this pull request as ready for review February 14, 2022 09:29

osopardo1 requested review from cugni and eavilaes and removed request for cugni February 14, 2022 09:30

osopardo1 added 2 commits February 14, 2022 14:07

Change Option to Any

cf79a61

Support for all null values

a5904ed

osopardo1 commented Feb 14, 2022

View reviewed changes

core/src/main/scala/io/qbeast/core/transform/LinearTransformer.scala Outdated Show resolved Hide resolved

osopardo1 added 3 commits February 14, 2022 15:23

Remove unnecesary match

4d82b3a

Fix JSONSerializationTests

1865af0

Change nullValue variable name

4014b7c

cugni requested changes Feb 15, 2022

View reviewed changes

Change Any to Option, add Filter Pushdown on isNull and throw error w…

cea3c64

…ith all null values Pending to change the problem of having all values at null

eavilaes suggested changes Feb 16, 2022

View reviewed changes

osopardo1 added 2 commits February 16, 2022 12:06

Minor changes for scaladocs and fixed test

e8cdf06

Add test for QuerySpecBuilder

c411ad6

Add support for all null/same values and adjust tests

863ec58

osopardo1 requested review from cugni and eavilaes February 18, 2022 15:22

Remove unused requirement

e8d0c7c

eavilaes approved these changes Feb 25, 2022

View reviewed changes

core/src/main/scala/io/qbeast/core/transform/LinearTransformation.scala Outdated Show resolved Hide resolved

osopardo1 added 2 commits March 2, 2022 12:03

Add documentation and throw an error for version incompatibility

b349d91

Add AdvancedConfiguration.md

1f13d4a

Fixed typos

0515a6c

osopardo1 added 8 commits March 7, 2022 12:40

Compatibility between Transformations

726c059

Code readability

b0ef345

Include seed to generate random number

a8844e1

Actually use seed

274a211

Weird assembly error

bec3aa7

Extract correct float type

ae63a35

Use Murmur3Hash and add comments to the code

fce6216

import Murmur

32c45dd

eavilaes approved these changes Mar 14, 2022

View reviewed changes

cugni and others added 6 commits March 14, 2022 16:22

Big WIP

f24b817

Fix test

6fe25e9

use matchPattern to skip comparing nullValue in Transformation

Change documentation

aecb0ed

Improve generateRandomNumber and add tests

d714938

Adjust to main branch

134f9e3

Return IdentityTransformation when min and max are equal

1cc1d20

cugni reviewed Mar 16, 2022

View reviewed changes

Add NullToZeroTransformation

98661be

osopardo1 requested a review from cugni March 16, 2022 09:16

cugni approved these changes Mar 16, 2022

View reviewed changes

core/src/main/scala/io/qbeast/core/transform/LinearTransformer.scala Show resolved Hide resolved

osopardo1 merged commit c3b23ea into Qbeast-io:main Mar 16, 2022

osopardo1 mentioned this pull request Mar 16, 2022

NULL values on indexed columns support #10

Closed

osopardo1 deleted the 10-null-values-on-indexed-columns-support branch April 22, 2022 12:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for null values #67

Add support for null values #67

osopardo1 commented Jan 25, 2022 •

edited

Loading

cugni left a comment

eavilaes left a comment

eavilaes commented Mar 3, 2022

osopardo1 commented Mar 3, 2022

osopardo1 commented Mar 11, 2022

eavilaes left a comment

cugni left a comment

Add support for null values #67

Add support for null values #67

Conversation

osopardo1 commented Jan 25, 2022 • edited Loading

cugni left a comment

Choose a reason for hiding this comment

eavilaes left a comment

Choose a reason for hiding this comment

eavilaes commented Mar 3, 2022

osopardo1 commented Mar 3, 2022

osopardo1 commented Mar 11, 2022

eavilaes left a comment

Choose a reason for hiding this comment

cugni left a comment

Choose a reason for hiding this comment

osopardo1 commented Jan 25, 2022 •

edited

Loading