Update to Spark 3.5 and Delta 3.0 [Spark] [Delta]

[osopardo1]

Following the news on Delta and Spark releases we should update the qbeast-spark libraries to keep up to date with the latest features.

This is the summary of the new developments included in both releases.

Spark 3.5

• Scala and Go client support in Spark Connect.
• Structured Streaming support for Spark Connect in Python and Scala
• Introduce Arrow Python UDFs
• PyTorch-based distributed ML Support

Read the full notes here.

Delta 3.0

• Delta Universal Format: allow you to read Delta tables with Hudi and Iceberg clients.
• Delta Kernel: decoupling reading and writing for building Delta Connectors.
• Delta Spark: change on package name (before was delta-core)
• Support for Spark 3.5
• Better performance for Deletion Vectors

Read the full notes here.

[osopardo1]

For the record: this update would be for main-1.0.0

[osopardo1]

Some things I am experiencing with the versions upgrade:

• withNewTransaction method from Delta is deprecated. Now they enforce to pass a series of new arguments as an Option to commit to a table:
  • CatalogTable (if exists)
  • Snapshot (if exists)
  • Still need to read the code and understand it's role in the Metadata Writer.
• Properties/options are not passed as usual to the saveAsTable / Catalog code. A.k.a:columnsToIndex is not propagated properly and throws an AnalysisException when writing to Qbeast.
• Delta has a new trait called TableChanges that can be confused for our own TableChanges interface in package core. Be careful when importing.

[osopardo1]

This feature is merged in 1.0.0-main!

[osopardo1]

Merged on #284