Correct nullable string validity map treatment

[eddelbuettel]

Work on the TileDB SOMA package and its independent implementation of the read-path (relying on Arrow buffers) revealed that we encoded missingness incorrectly for nullable strings. (This happened for read and write so round-turn tests passed. This is also not a concern for numeric values were encoded correctly but only concerns strings.)

This PR corrects this. It also tightens one test on deletion by switching to a triple || condition.

Example from https://app.shortcut.com/tiledb-inc/story/25027/r-unable-to-write-arrow-field-with-nullable-strings below the fold.

#!/usr/bin/env Rscript

suppressMessages({
    library(arrow)
    library(tiledbsoma)
})

if (getwd() == "/home/edd") setwd("~/git/tiledb-adhoc/soma/sc25027")

#uri <- tempfile()
uri <- "r_soma_2"
if (dir.exists(uri)) unlink(uri, recursive=TRUE)
sch <- arrow::schema(arrow::field("soma_joinid", arrow::int64(), nullable = FALSE),
                     arrow::field("str_not_nullable", arrow::utf8(), nullable = FALSE),
                     arrow::field("str_nullable", arrow::utf8(), nullable = TRUE))

tbl0 <- arrow::arrow_table(soma_joinid = seq_along(letters[1:10]),
                           str_not_nullable = letters[1:10],
                           str_nullable = c(letters[1:4], NA, letters[6:8], NA, letters[10]),
                           schema = sch)
cat("== Before write:\n")
print(dplyr::collect(tbl0))
sdf <- SOMADataFrame$new(uri)
sdf$create(sch, index_column_names = "soma_joinid")
sdf$write(tbl0)
tbl1 <- sdf$read()
cat("== After read:\n")
print(dplyr::collect(tbl1))
all.equal(tbl1$str_not_nullable$as_vector(), tbl0$str_not_nullable$as_vector())
all.equal(tbl1$str_nullable$as_vector(), tbl0$str_nullable$as_vector())

generates

> suppressMessages({
+ library(arrow)
+ library(tiledbsoma)
+ })
> if (getwd() == "/home/edd") setwd("~/git/tiledb-adhoc/soma/sc25027")
> uri <- "r_soma_2"
> if (dir.exists(uri)) unlink(uri, recursive=TRUE)
> sch <- arrow::schema(arrow::field("soma_joinid", arrow::int64(), nullable = FALSE),
+ arrow::field("str_not_nullable", arrow::utf8(), nullable = FALSE),
+ arrow::field("str_nullable", arrow::utf8(), nullable = TRUE))
> tbl0 <- arrow::arrow_table(soma_joinid = seq_along(letters[1:10]),
+ str_not_nullable = letters[1:10],
+ str_nullable = c(letters[1:4], NA, letters[6:8], NA, letters[10]),
+ schema = sch)
> cat("== Before write:\n")
== Before write:
> print(dplyr::collect(tbl0))
# A tibble: 10 × 3
   soma_joinid str_not_nullable str_nullable
         <int> <chr>            <chr>       
 1           1 a                a           
 2           2 b                b           
 3           3 c                c           
 4           4 d                d           
 5           5 e                NA          
 6           6 f                f           
 7           7 g                g           
 8           8 h                h           
 9           9 i                NA          
10          10 j                j           
> sdf <- SOMADataFrame$new(uri)
> sdf$create(sch, index_column_names = "soma_joinid")
> sdf$write(tbl0)
> tbl1 <- sdf$read()
> cat("== After read:\n")
== After read:
> print(dplyr::collect(tbl1))
# A tibble: 10 × 3
   soma_joinid str_not_nullable str_nullable
         <int> <chr>            <chr>       
 1           1 a                a           
 2           2 b                b           
 3           3 c                c           
 4           4 d                d           
 5           5 e                NA          
 6           6 f                f           
 7           7 g                g           
 8           8 h                h           
 9           9 i                NA          
10          10 j                j           
> all.equal(tbl1$str_not_nullable$as_vector(), tbl0$str_not_nullable$as_vector())
[1] TRUE
> all.equal(tbl1$str_nullable$as_vector(), tbl0$str_nullable$as_vector())
[1] TRUE
> print(tbl0$str_nullable$as_vector())
 [1] "a" "b" "c" "d" NA  "f" "g" "h" NA  "j"
> print(tbl1$str_nullable$as_vector())
 [1] "a" "b" "c" "d" NA  "f" "g" "h" NA  "j"
> 

and Python is happy too:

$ ./arrow_nullable.py
Reading R back...
pyarrow.Table
soma_joinid: int64
str_not_nullable: large_string
str_nullable: large_string
----
soma_joinid: [[1,2,3,4,5,6,7,8,9,10]]
str_not_nullable: [["a","b","c","d","e","f","g","h","i","j"]]
str_nullable: [["a","b","c","d",null,"f","g","h",null,"j"]]
$

[shortcut-integration]

This pull request has been linked to Shortcut Story #25027: R unable to write arrow field with nullable strings.

[ihnorton]

Moved to draft pending discussion.

[ihnorton]

Per discussion, let's

• remove use of the copy-then-delete in the conversion script
• add copying of metadata from the old to new array

Otherwise LGTM. Let's get an ACK from @Shelnutt2 as well.