[Question]: The "score" in a NodeWithScore

[Kai-Chen00]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

What is the meaning of the "score" parameter within the NodeWithScore object?

[dosubot]

The "score" parameter within the NodeWithScore object is an optional float that represents a score associated with the node. If the score is not set, it defaults to None. The get_score method can be used to retrieve this score, returning 0.0 if the score is not set and raise_error is False. If raise_error is True and the score is not set, a ValueError is raised ^[1].

To continue talking to Dosu, mention @dosu.

[Kai-Chen00]

@dosu,

retriever = index.as_retriever(similarity_top_k=10)
nodes = retriever.retrieve("Effective ways of communicating\r")
nodes

This is my code, and there are score returned, what does it mean?

[dosubot]

The score returned in your code signifies the relevance or confidence level of the retrieved result. It indicates how well the retrieved content matches the query. The nodes are sorted by their similarity scores in descending order, meaning the nodes with the highest scores are considered the most relevant to your query ^[1][2].

To continue talking to Dosu, mention @dosu.

[Kai-Chen00]

@dosu, So is it the "score" cosine similarity?

[dosubot]

Yes, the "score" typically represents the cosine similarity between the query and the retrieved nodes. This score measures the cosine of the angle between two non-zero vectors, indicating how similar they are. A higher score means a higher similarity, suggesting that the node is more relevant to the query.

To continue talking to Dosu, mention @dosu.

[Kai-Chen00]

@dosu

settings.node_parser = parser # 使用自定义的node_parser来建立每个node，这里是每个entity是一个node即数据库的一条数据
# Settings.embed_model = embed_model #设定embed_model

# em_fn = embedding_functions.SentenceTransformerEmbeddingFunction(model_name="dunzhang/stella_en_1.5B_v5", device="cuda") # setting embedding function for collection

chroma_client = chromadb.PersistentClient()
try:
#   chroma_collection = chroma_client.create_collection("entity_collection", metadata={"hnsw:space": "cosine"}, embedding_function=em_fn) # eclidian distance是l2, cosine distance是cosine
    chroma_collection = chroma_client.create_collection("entity_collection", metadata={"hnsw:space": "l2"}) # euclidian distance是l2, cosine distance是cosine
except Exception as e:
  if "already exists" in str(e):
#     chroma_collection = chroma_client.get_collection("entity_collection", embedding_function=em_fn)
    chroma_collection = chroma_client.get_collection("entity_collection")
  else:
    raise e

vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(entity_doc, storage_context=storage_context, embed_model=embed_model, show_progress=True)
retriever = index.as_retriever(similarity_top_k=10)
nodes = retriever.retrieve("Effective ways of communicating\r")
nodes

These are my codes, the metric I use is cosine distance for chromadb's collection. So what's the meaning of the "score"?

[dosubot]

In your code, when using cosine distance for ChromaDB's collection, the "score" represents a similarity score calculated as math.exp(-distance). This score is derived from the distance returned by the ChromaDB query, where a lower distance indicates higher similarity. The similarity score is computed by taking the exponential of the negative distance, transforming the distance into a similarity measure ^[1].

To continue talking to Dosu, mention @dosu.

[Kai-Chen00]

@dosu
why exp(-distance) rather than distance itself?

[dosubot]

The similarity score is calculated as exp(-distance) rather than using the distance itself to transform the distance into a more intuitive and interpretable similarity measure. The exponential function exp(-distance) converts the distance into a similarity score that ranges between 0 and 1, where a smaller distance results in a higher similarity score. This transformation ensures that as the distance decreases (indicating more similarity), the similarity score increases exponentially, providing a more meaningful representation of similarity. In the context of LlamaIndex's BagelVectorStore, the similarity is calculated as 1.0 - math.exp(-distance), which further adjusts the score to fit within a specific range, ensuring that the similarity score is directly proportional to the closeness of the vectors ^[1][2].

To continue talking to Dosu, mention @dosu.