Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SEDONA-703] Introducing the StructuredAdapter class #1780

Merged
merged 4 commits into from
Feb 4, 2025
Merged

Conversation

jiayuasu
Copy link
Member

@jiayuasu jiayuasu commented Jan 30, 2025

Did you read the Contributor Guide?

Is this PR related to a JIRA ticket?

What changes were proposed in this PR?

  1. Introducing the StructuredAdapter class to smooth the conversion among RDDs, DataFrames, and SpatialRDDs

The following stores InternalRow in geom.userData

  • SpatialRDD to DF
  • DF to SpatialRDD
  • SpatialPairRDD to DF
  • SpatialRDD to spatialPartitionedDF

The following stores Row in geom.userData

  • SpatialRDD to RowRDD
  • RowRDD to SpatialRDD
  1. Deprecating the old Adapter class which relies on a string field to serialize objects
  2. Introducing schema:StructType field for SpatialRDD
  3. Replace all Adapter in the tutorials with StructuredAdapter to deprioritize the old Adapter class.

How was this patch tested?

Passed new unit tests

Did this PR include necessary documentation updates?

  • Yes, I have updated the documentation.

@jiayuasu jiayuasu added this to the sedona-1.7.1 milestone Jan 30, 2025
Copy link
Member

@paleolimbot paleolimbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm new to the Spark internals here, but the partitioned part + test looks great!

@github-actions github-actions bot added the docs label Feb 4, 2025
@jiayuasu jiayuasu marked this pull request as ready for review February 4, 2025 05:52
@jiayuasu jiayuasu merged commit 79c1da8 into master Feb 4, 2025
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants