Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add java priority queue coder #5520

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add java priority queue coder #5520

wants to merge 3 commits into from

Conversation

kellen
Copy link
Contributor

@kellen kellen commented Oct 26, 2024

When using e.g. algebird PriorityQueueMonoid, scio needs to serialize java PriorityQueues. If Kryo is used, especially for small queues, there can be significant overhead.

Instead, this PR provides a PriorityQueue coder backed by a scala Ordering (as indeed the monoid is).

The coder must be created explicitly by the user so that they are asserting that the comparator of the original PriorityQueues and the reconstituted ones are the same. e.g.

implicit val pqCoder = Coders.jPriorityQueueCoder[T](ord)

@RustedBones Is there some trick I can use to get an error message when Coder[java.util.PriorityQueue[T]] fails to be derived (that's not just the generic implicitNotFound on Coder)?

Copy link

codecov bot commented Oct 26, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.44%. Comparing base (0a450e9) to head (e8c0616).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5520      +/-   ##
==========================================
+ Coverage   61.43%   61.44%   +0.01%     
==========================================
  Files         312      312              
  Lines       11103    11107       +4     
  Branches      762      778      +16     
==========================================
+ Hits         6821     6825       +4     
  Misses       4282     4282              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant