first() returns wrong result on empty Dataframe with struct column

[barak1412]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl


df = pl.DataFrame({
    'col1': []
}, schema={'col1': pl.Struct([
    pl.Field("n", dtype=pl.Int64),
    pl.Field("o", dtype=pl.Float64),
    pl.Field("l", dtype=pl.Utf8)
])})

print(df.select(pl.first('col1')))

Log output

No response

Issue description

On above example we get:

shape: (1, 1)
┌──────────────────┐
│ col1             │
│ ---              │
│ struct[3]        │
╞══════════════════╡
│ {null,null,null} │
└──────────────────┘

Instead of getting empy dataframe.

Expected behavior

Expected output:

shape: (0, 1)
┌──────────────────┐
│ col1             │
│ ---              │
│ struct[3]        │
╞══════════════════╡
└──────────────────┘

Installed versions

--------Version info---------
Polars:               0.20.21
Index type:           UInt32
Platform:             Windows-10-10.0.19041-SP0
Python:               3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          2.2.0
connectorx:           0.3.3a1
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               2023.10.0
gevent:               <not installed>
hvplot:               <not installed>
matplotlib:           3.5.1
nest_asyncio:         1.5.1
numpy:                1.24.4
openpyxl:             <not installed>
pandas:               2.0.3
pyarrow:              13.0.0
pydantic:             1.8.2
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           1.4.32
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

[cmdlineluser]

Can reproduce.

Also happens with .last()

>>> df.select(pl.col('col1').last())
shape: (1, 1)
┌──────────────────┐
│ col1             │
│ ---              │
│ struct[3]        │
╞══════════════════╡
│ {null,null,null} │
└──────────────────┘

Doesn't happen with .slice()

>>> df.select(pl.col('col1').slice(0, 1))
shape: (0, 1)
┌───────────┐
│ col1      │
│ ---       │
│ struct[3] │
╞═══════════╡
└───────────┘

[orlp]

I'll fix this tomorrow.

[orlp]

Wait a second, this is actually currently 'correct', until we add outer nullability for structs. .first() should return null if there's nothing to select:

>>> pl.DataFrame({"x": []}, schema={"x": pl.Int64}).select(pl.col.x.last())
shape: (1, 1)
┌──────┐
│ x    │
│ ---  │
│ i64  │
╞══════╡
│ null │
└──────┘

For structs this currently means returning an all-null row, since structs currently lack outer nullability.

I will close this issue as a duplicate of #3462.

[cmdlineluser]

Ah - I thought the issue was that it returned 1 row for a frame of height 0.

But that happens regardless of the dtype.

df = pl.DataFrame({"foo": []}).cast(pl.String)
# shape: (0, 1)
# ┌─────┐
# │ foo │
# │ --- │
# │ str │
# ╞═════╡
# └─────┘

df.select(pl.first("foo"))
# shape: (1, 1)
# ┌──────┐
# │ foo  │
# │ ---  │
# │ str  │
# ╞══════╡
# │ null │
# └──────┘

[barak1412]

@cmdlineluser

Yes, Iunderstood from @orlp answers.
However, it still seems to me pretty odd to get one row with null value.

[cmdlineluser]

Yes, the term that seems to be used for this is: introduces a "phantom row".

I'm not sure if this is expected or not - or if it has been previously discussed.

As the title/example was about structs - it seemed like you may have been talking about the other problem. (struct validity)

Perhaps you could create an issue without mentioning structs to discuss the first() behaviour.

[barak1412]

@cmdlineluser you are right.
The overall behaviour of first should be disscussed, unrelated to Struct type.