hybrid serach+ filters on AzureSearch as as vector store

[aradhanachaturvedi]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

embeddings: AzureOpenAIEmbeddings = AzureOpenAIEmbeddings(
    azure_deployment="model",
    openai_api_version=AZURE_API_VERSION,
    azure_endpoint=AZURE_EMBEDDING_ENDPOINT,
    api_key=AZURE_OPENAI_API_KEY)

vector_store: AzureSearch = AzureSearch(
    azure_search_endpoint=endpoint,
    azure_search_key=search_key,
    index_name=index_name,
    embedding_function=embeddings.embed_query
)


    doc = Document(
                page_content=file_data,
                gender=gender,
                country=country,
                metadata={"file": file_name}
            )
vector_store.add_documents([doc])

Description

I am creating a Hybrid search but I have a requirement where First I want to filter the content based on user query e.g. ("only male user in ABC Country") here first step should be to filter based on gender and country and out of those filtered document, hybrid search should be applied. I have indexed my data in content field of Azure search which has semi structured data and created embedding out of it . But for better response , I need to apply filtering so how can I store gender and country as a separate field in Azure search , When I used langchain , its not allowing me to index these values, I can see only null values in gender and country with below code.
Document(
page_content=file_data,
gender=gender,
country=country,
metadata='file)

do we need to store these fields on metadata? if yes, how can we apply different filtering based on user query

System Info

langchain==0.1.20
langchain-community==0.0.38
langchain-core==0.1.52
langchain-experimental==0.0.57
langchain-openai==0.1.3
langchain-text-splitters==0.2.2

[dosubot[bot]]

To store gender and country fields in AzureSearch using LangChain and apply filtering based on these fields for hybrid search, you need to define these fields in the fields parameter when initializing the AzureSearch class. You can then use these fields in the filter parameter when performing a search.

Here is an example of how you can achieve this:

Step 1: Define the gender and country fields

from azure.search.documents.indexes.models import SimpleField, SearchFieldDataType
from langchain_community.vectorstores.azuresearch import AzureSearch
from langchain_openai import OpenAIEmbeddings

# Define the fields including gender and country
fields = [
    SimpleField(name="id", type=SearchFieldDataType.String, key=True, filterable=True),
    SimpleField(name="content", type=SearchFieldDataType.String, searchable=True),
    SimpleField(name="content_vector", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), searchable=True),
    SimpleField(name="metadata", type=SearchFieldDataType.String, searchable=True),
    SimpleField(name="gender", type=SearchFieldDataType.String, filterable=True),
    SimpleField(name="country", type=SearchFieldDataType.String, filterable=True),
]

# Initialize the AzureSearch instance
azure_search = AzureSearch(
    azure_search_endpoint="your_endpoint",
    azure_search_key="your_key",
    index_name="your_index_name",
    embedding_function=OpenAIEmbeddings().embed_query,
    fields=fields
)

Step 2: Apply filtering based on gender and country fields

filters = "gender eq 'male' and country eq 'ABC Country'"

results = azure_search.client.search(
    search_text="your_query",
    vector_queries=[
        VectorizedQuery(
            vector=np.array(azure_search.embed_query("your_query"), dtype=np.float32).tolist(),
            k_nearest_neighbors=10,
            fields="content_vector",
        )
    ],
    filter=filters,
    query_type="semantic",
    semantic_configuration_name=azure_search.semantic_configuration_name,
    query_caption="extractive",
    query_answer="extractive",
    top=10,
)

This code snippet demonstrates how to define the gender and country fields and apply filtering based on these fields for hybrid search in AzureSearch using LangChain ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

Could you provide more details on how you are adding the documents to the Azure Search index? Specifically, the code or method you are using to add the documents might help identify why the gender and country fields are not being populated correctly. Additionally, are you using any specific SDK or library for interacting with Azure Search?

To continue talking to Dosu, mention @dosu.

[aradhanachaturvedi]

@dosu.

I am usign langchain to upload documents
from langchain_community.vectorstores.azuresearch import AzureSearch

vector_store: AzureSearch = AzureSearch(
azure_search_endpoint=vector_store_address,
azure_search_key=vector_store_password,
index_name=index_name,
embedding_function=embedding_function,
fields=fields,
)

vector_store.add_texts(["Test 1"],
[
{"gender": "male", "country": "A", },
],)
)

[dosubot[bot]]

The issue seems to be with how the documents are being added to the Azure Search index using the add_texts method. The add_texts method needs to correctly map the metadata fields to the corresponding fields in the Azure Search index.

Here is an example of how you can modify the add_texts method to ensure that the gender and country fields are correctly populated:

1. Ensure the fields configuration includes gender and country:

from azure.search.documents.indexes.models import SimpleField, SearchFieldDataType, SearchableField, SearchField
from langchain_community.vectorstores.azuresearch import AzureSearch
from langchain_openai import OpenAIEmbeddings

# Define the fields including gender and country
fields = [
    SimpleField(name="id", type=SearchFieldDataType.String, key=True, sortable=True),
    SearchableField(name="content", type=SearchFieldDataType.String, searchable=True, filterable=True),
    SimpleField(name="gender", type=SearchFieldDataType.String, searchable=True, filterable=True),
    SimpleField(name="country", type=SearchFieldDataType.String, searchable=True, filterable=True),
    SearchField(name="content_vector", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), searchable=True, vector_search_dimensions=1536, vector_search_profile_name="myHnswProfile"),
    SearchableField(name="metadata", type=SearchFieldDataType.String, searchable=True),
]

# Initialize AzureSearch with the custom fields
azure_search = AzureSearch(
    azure_search_endpoint="your_endpoint",
    azure_search_key="your_key",
    index_name="your_index_name",
    embedding_function=OpenAIEmbeddings(
        openai_api_key="your_openai_api_key",
        openai_api_version="2023-05-15",
        model="text-embedding-ada-002"
    ).embed_query,
    fields=fields
)

2. Modify the add_texts method to correctly handle the custom fields:

Ensure that the add_texts method in the AzureSearch class correctly maps the gender and country fields from the metadata to the respective fields in the Azure Search index.

def add_texts(self, texts, metadatas):
    data = []
    for i, text in enumerate(texts):
        metadata = metadatas[i] if metadatas else {}
        doc_data = {
            "id": str(uuid.uuid4()),
            "content": text,
            "content_vector": self.embedding_function(text),
            "gender": metadata.get("gender", None),
            "country": metadata.get("country", None),
            "metadata": json.dumps(metadata)
        }
        data.append(doc_data)
    
    # Upload data to AzureSearch index
    response = self.client.upload_documents(documents=data)
    if not all(r.succeeded for r in response):
        raise Exception(response)

3. Add documents to the Azure Search index:

vector_store.add_texts(
    ["Test 1"],
    [
        {"gender": "male", "country": "A"},
    ]
)

By ensuring that the gender and country fields are correctly mapped and included in the document data, you should be able to filter based on these fields in your Azure Search queries.

This approach ensures that gender and country are stored as separate fields in the Azure Search index and not within the metadata field ^[1].

To continue talking to Dosu, mention @dosu.

[aradhanachaturvedi]

@dosu. do I need to apply same logic if I want to use add_documents method instead of add_texts.

[dosubot[bot]]

Yes, you need to apply the same logic if you want to use the add_documents method instead of add_texts. The key is to ensure that the gender and country fields are correctly mapped from the metadata to the respective fields in the Azure Search index.

Here is how you can modify the add_documents method to handle the custom fields:

1. Ensure the fields configuration includes gender and country:

from azure.search.documents.indexes.models import SimpleField, SearchFieldDataType, SearchableField, SearchField
from langchain_community.vectorstores.azuresearch import AzureSearch
from langchain_openai import OpenAIEmbeddings

# Define the fields including gender and country
fields = [
    SimpleField(name="id", type=SearchFieldDataType.String, key=True, sortable=True),
    SearchableField(name="content", type=SearchFieldDataType.String, searchable=True, filterable=True),
    SimpleField(name="gender", type=SearchFieldDataType.String, searchable=True, filterable=True),
    SimpleField(name="country", type=SearchFieldDataType.String, searchable=True, filterable=True),
    SearchField(name="content_vector", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), searchable=True, vector_search_dimensions=1536, vector_search_profile_name="myHnswProfile"),
    SearchableField(name="metadata", type=SearchFieldDataType.String, searchable=True),
]

# Initialize AzureSearch with the custom fields
azure_search = AzureSearch(
    azure_search_endpoint="your_endpoint",
    azure_search_key="your_key",
    index_name="your_index_name",
    embedding_function=OpenAIEmbeddings(
        openai_api_key="your_openai_api_key",
        openai_api_version="2023-05-15",
        model="text-embedding-ada-002"
    ).embed_query,
    fields=fields
)

2. Modify the add_documents method to correctly handle the custom fields:

Ensure that the add_documents method in the AzureSearch class correctly maps the gender and country fields from the metadata to the respective fields in the Azure Search index.

def add_documents(self, documents):
    data = []
    for doc in documents:
        doc_data = {
            "id": doc.metadata.get("id", str(uuid.uuid4())),
            "content": doc.page_content,
            "content_vector": self.embedding_function(doc.page_content),
            "gender": doc.metadata.get("gender", None),
            "country": doc.metadata.get("country", None),
            "metadata": json.dumps(doc.metadata)
        }
        data.append(doc_data)
    
    # Upload data to AzureSearch index
    response = self.client.upload_documents(documents=data)
    if not all(r.succeeded for r in response):
        raise Exception(response)

3. Add documents to the Azure Search index:

from langchain.schema import Document

# Create Document instances with gender and country fields
docs = [
    Document(
        page_content="Test 1",
        metadata={
            "id": "abc",
            "gender": "Female",
            "country": "A"
        }
    ),
    Document(
        page_content="Test 2",
        metadata={
            "id": "def",
            "gender": "male",
            "country": "A"
        }
    )
]

# Add the documents to the AzureSearch index
azure_search.add_documents(docs)

By ensuring that the gender and country fields are correctly mapped and included in the document data, you should be able to filter based on these fields in your Azure Search queries. This approach ensures that gender and country are stored as separate fields in the Azure Search index and not within the metadata field ^[1].

To continue talking to Dosu, mention @dosu.