.list.index_of_in() architectural review PR

[itamarst]

This PR implements a sketch of .list.index_of_in() as described in #20626

The questions I would definitely like feedback on at the moment are:

1. Is using amortized_iter() the right approach?
2. Is using needles.iter() the right approach?

[itamarst]

It's also possible the casting logic in Polars will need to be extended; CastingRules::FirstArgLossless isn't what we want since it casts the needles e.g. to List[Int64], we'd need some new option like CastingRules::FirstArgInnerLossless or something that casts to the inner type of the first argument.

[codecov]

Codecov Report

Attention: Patch coverage is 6.41026% with 73 lines in your changes missing coverage. Please review.

Project coverage is 79.31%. Comparing base (958d00f) to head (85b26ff).
Report is 110 commits behind head on main.

Files with missing lines	Patch %	Lines
...s/polars-ops/src/chunked_array/list/index_of_in.rs	0.00%	49 Missing ⚠️
crates/polars-plan/src/dsl/list.rs	0.00%	10 Missing ⚠️
crates/polars-plan/src/dsl/function_expr/list.rs	0.00%	9 Missing ⚠️
crates/polars-python/src/expr/list.rs	0.00%	3 Missing ⚠️
py-polars/polars/expr/list.py	71.42%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #20733      +/-   ##
==========================================
- Coverage   79.75%   79.31%   -0.45%     
==========================================
  Files        1561     1580      +19     
  Lines      221785   224397    +2612     
  Branches     2530     2573      +43     
==========================================
+ Hits       176885   177970    +1085     
- Misses      44318    45839    +1521     
- Partials      582      588       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ritchie46]

I am not sure we want this. The compilation bloat of this seems very high to me and we've got list.eval, so it is able to get what you want.

[itamarst]

So the code in the PR doesn't have much in the way of generics or templates, at least at the moment, so it doesn't seem like it adds much to compilation? Or is it that you're worried about the cumulative costs of lots of small features adding up?

[itamarst]

AFAICT list.eval() doesn't help with the case where you want to match up across columns, per row:

df = pl.DataFrame({"lists": [[3, 1], [2, 4], [5, 3, 1]], "values": [1, 2, 6]})
result = df.select(pl.col("lists").list.index_of_in(pl.col("values")))

because you can't use pl.col("values") inside of .list.eval() expressions (case 1B in the original issue).

@orlp had a pure-Python suggestion in the issue, but it was certainly beyond my admittedly very basic Polars query-writing skills:

result2 = df.with_row_index().select(
    pl.col("lists").explode().index_of(pl.col("values").first()).over("index")
)
print(result2)

If this technique is acceptable another option is a pure Python .list.index_of_in() so it's easier for users.

[itamarst]

Actually, not clear you can actually apply that technique with index inside expression function, so that might not be a viable approach.

[itamarst]

@ritchie46 could you expand about your concerns re compilation? See above. Thank you!

[deanm0000]

I think this is a good feature, I would also like to see .list.arg_where but I digress.

You can use .list.eval with a little trick, concat the values to the lists then inside the eval block of that slice the last element off and trigger index_of from the slice and use pl.element().last as the values.

df.select("lists","values",index_of = pl.col("lists").list.concat(pl.col("values")).list.eval(pl.element().slice(0,pl.element().len()-1).index_of(pl.element().last())).list.first())

In a test with this df

df=(
    pl.DataFrame({"a":np.random.uniform(0,100,100_000),
                 "lists":np.random.uniform(0,10,100_000)})
    .with_columns(pl.all().cast(pl.Int64))
    .group_by("a")
    .agg('lists')
    .select("lists", values=pl.lit(pl.Series(np.random.uniform(0,10,100)).cast(pl.Int64)))
)

I get 4.55ms for my method and 8ms for the explode/over approach so having something native that does even better than a .list.eval would be cool.

[itamarst]

Looks like that method might be implementable as a method on the Python Expr class' list namespace? So I might try that out.

[itamarst]

So, trying the Python version using list.eval() on a one million row dataframe:

1. It seems to be doable as an implementation strategy.
2. The Python version based on list.concat() and list.eval() 10× slower (i.e. 900% slower) than the compiled version in this PR. Tested in release mode.
3. It was 80ms for the compiled version; as comparison .list.arg_max() was 50ms. So this actually seems right ballpark, though still pretty slow honestly.

Conclusion: this feature seems doable with just Python, but at the cost of a massive 10× slowdown.