Refactor OpenAIEmbeddings #14334
Merged
Refactor OpenAIEmbeddings #14334
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* SPARKNLP-1036: Fix dev python kernel names * SPARKNLP-1036: Bump transformers version * SPARKNLP-1036: Fix Colab buttons * SPARKNLP-1036: Pin onnx version for compatibility * SPARKNLP-1036: Upgrade Spark version * SPARKNLP-1036: Minor Fixes * SPARKNLP-1036: Clean Metadata * SPARKNLP-1036: Add/Adjust Documentation - Note for supported Spark Version of Annotators - added missing Documentation for BGEEmbeddings
|
I created my branch from the master, So, There are some other commits, too. |
danilojsl
requested changes
Jun 24, 2024
| from pyspark.sql import DataFrame | ||
| from pyspark.sql import SparkSession | ||
|
|
||
| class OpenAIEmbeddingsTestCase(unittest.TestCase): |
There was a problem hiding this comment.
Please add @pytest.mark.slow annotation on top of class definition.
There was a problem hiding this comment.
Hi @danilojsl,
I have added the @pytest.mark.slow annotation with my new commit and updated the documentation.
e4a7c1a
danilojsl
approved these changes
Jun 24, 2024
maziyarpanahi
approved these changes
Jun 28, 2024
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactor OpenAIEmbeddings annotator
Description
Refactor OpenAIEmbeddings
1- Supported escape chars that break the Open AI json content.
2- Changed the output annotator type. DOCUMENT --> SENTENCE_EMBEDDINGS
NOTE: This approach is the reverse of the backward compatibility
3- Added metadata which comes from the document column to output embeddings
4- Added Python unit test class
5- Added a new submodule to support saving/loading the annotator
NOTE: The new submodule will fix saving/loading the annotator
Motivation and Context
How Has This Been Tested?
Tested via Python and Scala locally.
Additionally, I added new unit tests that cover my changes.
Screenshots (if appropriate):
Types of changes
Checklist: