Look into using polars

[paulf81]

The polars dataframe might be a way to increase the performance of flasc in the future, we would replace pandas dataframes with polars and perhaps get a significant speed-up. Can't tackle this now so just wanted to flag the idea

https://www.pola.rs/

[Bartdoekemeijer]

Good point. I have been wondering about Pandas alternatives, like Dask. We must consider not only speed, but also ease of implementation and userbase. Polars seems to have a substantial userbase and also seems to be the fastest option, based on this benchmark: https://h2oai.github.io/db-benchmark/.

[paulf81]

Or maybe we just see if pandas catches up anyway?
https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i

[Bartdoekemeijer]

@paulf81 Interesting article! It seems that polars will likely remain much faster than pandas. We can use Pandas for the less computationally intensive operations for simplicity sake while using Polars for computationally intensive operations with the new porting functionalities polars.from_pandas and polars.to_pandas.

At the time of writing this post polars.to_pandas has been updated just few days ago to support the use_pyarrow_extension_array and allow the Arrow data in Polars to be shared directly to pandas (without converting it to NumPy arrays. And polars.from_pandas has a bug that causes the data being copied, but it should be fixed soon. The above code will unnecessarily make a copy right now, but it should not making a copy after the release of pandas 2.0 and the next release of Polars.

Anyhow, something to keep in mind! Polars is a less mature library than Pandas so I am hesitant to completely replace Pandas with Polars for now. I haven't worked with high-resolution data in FLASC yet but I can imagine this becomes more important then. If we ever run into computational issues, let's look into using Polars for parts of the code!

[paulf81]

this makes sense to me, we'll have this discussion to flag the idea and then we can both wait for more maturity and a more urgent need on our end, thanks!

[paulf81]

closing this one now that #98 is open to discuss how to move forward