NaN Embedding Dimension Support

[NolanTrem]

Allows for specifying nan dimensions for models that do not support dimension parameters.

[embedding]
provider = "litellm"
base_model = "ollama/bge-large"
base_dimension = nan
batch_size = 128
add_title_as_prefix = true
concurrent_request_limit = 2

[ellipsis-dev]

👍 Looks good to me! Reviewed everything up to 2fc7761 in 1 minute and 43 seconds

More details

• Looked at 355 lines of code in 8 files
• Skipped 0 files when reviewing.
• Skipped posting 7 drafted comments based on config settings.

1. py/core/database/graphs.py:81

• Draft comment:
  The _get_vector_column_str function is correctly used here to handle vector column string generation, improving code readability and maintainability.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The PR introduces a utility function _get_vector_column_str to handle vector column string generation, which is used in multiple files. This function is correctly implemented to handle NaN dimensions by returning an empty string, allowing Postgres to handle any dimension. The function is used in various places to replace inline logic, improving code readability and maintainability.

2. py/core/database/graphs.py:529

• Draft comment:
  The _get_vector_column_str function is correctly used here to handle vector column string generation, improving code readability and maintainability.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The PR introduces a utility function _get_vector_column_str to handle vector column string generation, which is used in multiple files. This function is correctly implemented to handle NaN dimensions by returning an empty string, allowing Postgres to handle any dimension. The function is used in various places to replace inline logic, improving code readability and maintainability.

3. py/core/database/graphs.py:1017

• Draft comment:
  The _get_vector_column_str function is correctly used here to handle vector column string generation, improving code readability and maintainability.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The PR introduces a utility function _get_vector_column_str to handle vector column string generation, which is used in multiple files. This function is correctly implemented to handle NaN dimensions by returning an empty string, allowing Postgres to handle any dimension. The function is used in various places to replace inline logic, improving code readability and maintainability.

4. py/core/database/chunks.py:123

• Draft comment:
  The _get_vector_column_str function is correctly used here to handle vector column string generation, improving code readability and maintainability.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The PR introduces a utility function _get_vector_column_str to handle vector column string generation, which is used in multiple files. This function is correctly implemented to handle NaN dimensions by returning an empty string, allowing Postgres to handle any dimension. The function is used in various places to replace inline logic, improving code readability and maintainability.

5. py/core/database/chunks.py:155

• Draft comment:
  The _get_vector_column_str function is correctly used here to handle vector column string generation, improving code readability and maintainability.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The PR introduces a utility function _get_vector_column_str to handle vector column string generation, which is used in multiple files. This function is correctly implemented to handle NaN dimensions by returning an empty string, allowing Postgres to handle any dimension. The function is used in various places to replace inline logic, improving code readability and maintainability.

6. py/core/database/chunks.py:222

• Draft comment:
  The _get_vector_column_str function is correctly used here to handle vector column string generation, improving code readability and maintainability.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The PR introduces a utility function _get_vector_column_str to handle vector column string generation, which is used in multiple files. This function is correctly implemented to handle NaN dimensions by returning an empty string, allowing Postgres to handle any dimension. The function is used in various places to replace inline logic, improving code readability and maintainability.

7. py/core/database/chunks.py:327

• Draft comment:
  The _get_vector_column_str function is correctly used here to handle vector column string generation, improving code readability and maintainability.
• Reason this comment was not posted:
  Confidence changes required: 0%
  The PR introduces a utility function _get_vector_column_str to handle vector column string generation, which is used in multiple files. This function is correctly implemented to handle NaN dimensions by returning an empty string, allowing Postgres to handle any dimension. The function is used in various places to replace inline logic, improving code readability and maintainability.

Workflow ID: wflow_kGlO0vQo1UOynxr6

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

[qdrddr]

[embedding]
provider = "litellm"
concurrent_request_limit = 32          # Embedding concurrency limit

base_model = "openai/nebius/bge-en-icl"
base_dimension = nan

my .env

export OPENAI_API_KEY=sk-123
export OPENAI_API_BASE="https://litellm.mywebsite.com/v1"

export LITELLM_PROXY_API_KEY=sk-123
export LITELLM_PROXY_API_BASE="https://litellm.mywebsite.com/v1"

Does not work.
My steps: I have deployed docker image 3.3.24 and replaced/mapped into the container only the files from this commit:

• py/core/base/providers/embedding.py
• py/core/base/utils/init.py
• py/core/database/chunks.py
• py/core/database/documents.py
• py/core/database/graphs.py
• py/core/providers/embeddings/litellm.py
• py/shared/utils/init.py
• py/shared/utils/base_utils.py

When I try to ingest files, I again get the same error message that r2r is trying to set dimensionality parameters when posting to my LiteLLM Proxy.

2025-01-09 20:04:26 2025-01-10 02:04:26 - ERROR - Error getting embeddings: litellm.BadRequestError: OpenAIException - Error code: 400 - {'error': {'message': "litellm.UnsupportedParamsError: Setting dimensions is not supported for OpenAI `text-embedding-3` and later models. To drop it from the call, set `litellm.drop_params = True`.

[NolanTrem]

@qdrddr we haven't yet pushed a release with this change. You'd have to build off of main for it to take effect.

[qdrddr]

I tried to build from the main, but got errors with the container keep restarting.

[qdrddr]

FYI this is the patch that was working for me when applied to v3.3.22 (but doesn't work in 3.3.24 anymore):
Its a huck, I'm not a developer, but it was working for me. @NolanTrem

--- ./py/app/core/main/providers/embeddings/litellm.py.patch/original.py	2025-01-08 15:49:55
+++ ./py/app/core/main/providers/embeddings/litellm.py.patch/manually_updated.py	2025-01-08 15:53:39
@@ -59,13 +59,17 @@
         if "amazon" in self.base_model:
             logger.warn("Amazon embedding model detected, dropping params")
             litellm.drop_params = True
+        if "infinity" in self.base_model:
+            logger.warn("Infinity embedding model detected, dropping params")
+            litellm.drop_params = True
         self.base_dimension = config.base_dimension
 
     def _get_embedding_kwargs(self, **kwargs):
         embedding_kwargs = {
             "model": self.base_model,
-            "dimensions": self.base_dimension,
-        }
+        }
+        if "bge-en-icl" not in self.base_model:
+            embedding_kwargs["dimensions"] = self.base_dimension
         embedding_kwargs.update(kwargs)
         return embedding_kwargs