Skip to content
This repository has been archived by the owner on Nov 13, 2024. It is now read-only.

Add dimension support for OpenAI embeddings #273

Merged

Conversation

izellevy
Copy link
Collaborator

@izellevy izellevy commented Feb 4, 2024

Problem

New OpenAI embedding models support controlling the dimension parameter for the embeddings vectors. We cannot currently pass a dynamic dimension to OpenAI models.

Solution

Updated the pinecone-text package version that supports passing dimension to the Encoder. Changed the default embedding model to the new one (text-embedding-3-small). Removed support for taking the dimension on create_canopy_index. From now on, we always rely on the RecordEncoder's dimension to create the index.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update
  • Infrastructure change (CI configs, etc)
  • Non-code change (docs, etc)
  • None of the above: (explain here)

Test Plan

Describe specific steps for validating this change.

Copy link
Contributor

@acatav acatav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@izellevy izellevy added this pull request to the merge queue Feb 4, 2024
Merged via the queue into pinecone-io:main with commit 2ca606f Feb 4, 2024
7 checks passed
@izellevy izellevy deleted the feature/encoder_dimension_changes branch February 4, 2024 11:29
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants