join fails because of an uninstructed cast from int to array[int, x] on 1.14.0

[TNieuwdorp]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl
import numpy as np

dtype = [("id", "<i4")]
data = np.array([[(1,), (2,)]], dtype=dtype)

df = pl.LazyFrame(data).explode(pl.col("*"))

other_df = pl.LazyFrame({"node": [1]})

result = df.join(other_df, left_on="id", right_on="node").collect()

print(result)

Log output

No response

Issue description

When joining two LazyFrames, one constructed from a numpy structured array, and then exploded, the schema reflects the change, but during a join operation this still goes wrong.

The state of the two LazyFrames before the join:

The error of the join:

This error also occurs when optimizations are turned off:

Explicitly casting the columns to int32 before the operation seems to work:

Expected behavior

I expect the data type to not be an array, and the join to succeed.

Installed versions

>>> pl.show_versions()
--------Version info---------
Polars:              1.14.0
Index type:          UInt32
Platform:            macOS-15.0.1-x86_64-i386-64bit
Python:              3.12.7 (main, Oct  1 2024, 02:05:46) [Clang 15.0.0 (clang-1500.3.9.4)]
LTS CPU:             False

----Optional dependencies----
adbc_driver_manager  <not installed>
altair               <not installed>
cloudpickle          <not installed>
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               2023.12.2
gevent               <not installed>
great_tables         0.10.0
matplotlib           3.8.4
nest_asyncio         1.6.0
numpy                1.26.4
openpyxl             3.1.2
pandas               2.2.2
pyarrow              14.0.2
pydantic             1.10.15
pyiceberg            <not installed>
sqlalchemy           <not installed>
torch                <not installed>
xlsx2csv             0.8.2
xlsxwriter           <not installed>
</details>

[alexander-beedie]

This should have been fixed by #19753, and we have made a new patch release available; take a look at 1.13.1? 👍

[TNieuwdorp]

@alexander-beedie Unfortunately that patch doesn't fix the problem, and it still occurs exactly as described.

[alexander-beedie]

Hmm, surprising! Are you able to create a small reproducible test-case that you can paste in to the Issue report?

[nameexhaustion]

Did you by any chance happen to have exploded the pgm_id column at some point?

I think I can produce the same error message with this -

q = pl.LazyFrame().select(
    pl.lit(pl.Series([[1, 1], [2, 2]], dtype=pl.Array(pl.Int64, 2)))
    .explode()
    .alias("k")
)

q = q.join(pl.LazyFrame({"k": [1, 2]}), on="k")

print(q.collect())

[TNieuwdorp]

@nameexhaustion Let me know if you get stuck on this, I might be able to dig a bit deeper in our code to try and figure out the source of the data and order of operations that are applied that leads up to this.

[TNieuwdorp]

Did you by any chance happen to have exploded the pgm_id column at some point?

I think I can produce the same error message with this -

q = pl.LazyFrame().select(
    pl.lit(pl.Series([[1, 1], [2, 2]], dtype=pl.Array(pl.Int64, 2)))
    .explode()
    .alias("k")
)

q = q.join(pl.LazyFrame({"k": [1, 2]}), on="k")

print(q.collect())

Checking...

[TNieuwdorp]

Yes, explode() is applied to the data!

The original data comes from a structured numpy array (although since you reproduced it without that, it might not be relevant)

[TNieuwdorp]

@nameexhaustion Unfortunately the error persists on 1.14.0 and it seems to be caused by something else...
I'm working on getting you guys a MRE

[TNieuwdorp]

@ritchie46 @nameexhaustion I've updated the original issue with an MRE