Nullability of Struct series

[dycw]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

from typing import Literal
from polars import Boolean, Series, Struct
from polars.testing import assert_series_equal
from pytest import mark, param


@mark.parametrize("case", [param("struct of Nones"), param("None")])
def test_nullability(*, case: Literal["struct of Nones", "None"]) -> None:
    match case:
        case "struct of Nones":
            element = {"a": None, "b": None}
        case "None":
            element = None
    sr = Series(
        [
            element,
            {"a": True, "b": None},
            {"a": None, "b": False},
            {"a": True, "b": False},
        ],
        dtype=Struct({"a": Boolean, "b": Boolean}),
    )
    assert_series_equal(
        sr.is_null(), Series([True, False, False, False], dtype=Boolean)
    )
    assert_series_equal(
        sr.is_not_null(), Series([False, True, True, True], dtype=Boolean)
    )

Log output

No response

Issue description

This used to work in Polars 1.1. I believe in Polars 1.2, the small bug fix "Properly implement struct (#17522)" is what changed this behavior. Now:

• the struct of Nones case fails, whilst
• the None case still passes

After all, this is the same test case found under here.

My questions are:

1. If this is the way forward, then how does one create this Series of type None rather than struct of Nones using when? What I believe is idiomatic code only gives me the latter:

from polars import Int64, when, col, struct, Series, Struct, Boolean
from polars.testing import assert_series_equal


def test_when() -> None:
    sr = Series([0, 1, 2, 3], dtype=Int64)
    df = sr.rename("int").to_frame()
    df = df.with_columns(
        result=when(col("int") >= 1).then(
            struct(
                a=when(col("int") % 2 == 1).then(True),
                b=when(col("int") >= 2).then(False),
            )
        )
    )
    sr = Series(
        [
            {"a": None, "b": None},
            {"a": True, "b": None},
            {"a": None, "b": False},
            {"a": True, "b": False},
        ],
        dtype=Struct({"a": Boolean, "b": Boolean}),
    )
    assert_series_equal(df["result"], sr, check_names=False)

2. Are there any idiomatic ways/suggestions/workarounds to inferring nullability as prescribed in the test above even for Series of type struct of Nones? A workaround for Series can be done because we can recursively inspect its type ; annoying at best. However, this doesn't seem feasible for Exprs.

P.S. Also see #3462 from 2022-05.

Expected behavior

As above

Installed versions

[ins] In [1]: pl.show_versions()
--------Version info---------
Polars:               1.4.0
Index type:           UInt32
Platform:             macOS-14.6-arm64-arm-64bit
Python:               3.12.2 (main, Apr 17 2024, 22:56:11) [Clang 15.0.0 (clang-1500.3.9.4)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               <not installed>
gevent:               <not installed>
great_tables:         <not installed>
hvplot:               <not installed>
matplotlib:           <not installed>
nest_asyncio:         1.6.0
numpy:                2.0.1
openpyxl:             <not installed>
pandas:               2.2.2
pyarrow:              17.0.0
pydantic:             <not installed>
pyiceberg:            <not installed>
sqlalchemy:           2.0.32
torch:                <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

[cmdlineluser]

I'm guessing the when behaviour is unintended - it seems like it should give None instead.

Just while experimenting for workarounds, it seems coalesce does something similar - which seems "wrong".

df = pl.DataFrame(
    data = {"x": [None, None]}, 
    schema = {"x": pl.Struct({"a": pl.Boolean})}
) 

(df.with_columns(y = pl.coalesce("x"))
   .with_columns(pl.all().is_null().name.suffix("_is_null"))
)

# shape: (2, 4)
# ┌───────────┬───────────┬───────────┬───────────┐
# │ x         ┆ y         ┆ x_is_null ┆ y_is_null │
# │ ---       ┆ ---       ┆ ---       ┆ ---       │
# │ struct[1] ┆ struct[1] ┆ bool      ┆ bool      │
# ╞═══════════╪═══════════╪═══════════╪═══════════╡
# │ null      ┆ {null}    ┆ true      ┆ false     │
# │ null      ┆ {null}    ┆ true      ┆ false     │
# └───────────┴───────────┴───────────┴───────────┘

[ritchie46]

I'm guessing the when behaviour is unintended

It is, that should return a None.

[dycw]

Thanks so much!