Missing values

[AlenkaF]

No description provided.

[AlenkaF]

The protocol on 13/08/2021 works for missing values that are int and float dtype. It doesn't work if there are missing values in boolean dtype columns (same in pandas implementation) and it works partly for categorical dtypes.

Here the transition from Pandas dataframe to Vaex dataframe works. The problem is when constructing Vaex dataframe with categorical column that has missing values. I tested two options (categorize() doesn't work if there are missing values):

• with map() function changing value of categorical column to np.nan
• with func.where() function changing value of categorical column to None

both cases change the dtype:

• dtype becomes float
• dtpye becomes object

and so it errors with pandas or vaex implementation in both cases. The reason:

• when calling _VaexBuffer it errors due to 'pyarrow.lib.DoubleArray' object has no attribute '__array_interface__'
• object dtype not yet supported

Need to research the topic more.

[AlenkaF]

In the PR comment for Vaex library Maarten pointed me toward the correct way explaining "In vaex, we don't treat nan as missing, it's 'just another float', all columns are nullable. Arrow columns via bitmask, NumPy columns via bool/byte arrays."

I played around little bit more and saved the work in Notebook. The protocol seem to work even on nullable NumPy columns. Next step I should probably take is to implement mask handling.

[AlenkaF]

Protocol now passes for all numpy and arrow dtypes with missing data. The methods that needed change were:

• convert_column_to_ndarray
• convert_categorical_column
• describe_null
• null_count
• get_data_buffer
• get_mask

The Notebook test is available here. The code in the draft PR ofr Vaex will be updated to handle missing values this week.