Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Knowledge Retrieval Node Incorrectly Reverts to RerankModel #8854

Closed
5 tasks done
Kevin9703 opened this issue Sep 27, 2024 · 1 comment · Fixed by #8842
Closed
5 tasks done

Knowledge Retrieval Node Incorrectly Reverts to RerankModel #8854

Kevin9703 opened this issue Sep 27, 2024 · 1 comment · Fixed by #8842
Labels
🐞 bug Something isn't working 🤔 cant-reproduce We can not reproduce it or the information is limited

Comments

@Kevin9703
Copy link
Contributor

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.8.3

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

  1. Configures a knowledge retrieval node to use WeightedScore
  2. Upon saving or reopening the workflow, the node is found to be using RerankModel instead.

✔️ Expected Behavior

The node should consistently maintain its WeightedScore configuration as set by the user.

Root Cause:
After investigating the API calls, we found that the frontend is inconsistently sending the required data fields to the backend via the draft API. Specifically, the 'reranking_mode' and 'weights' parameters are sometimes missing from the API requests, causing a mismatch between the node's intended settings and its actual configuration (reverting to the RerankModel).

❌ Actual Behavior

I have identified an intermittent bug in the workflow involving the knowledge retrieval node. When setting the node to use WeightedScore, it occasionally and unexpectedly reverts to using the RerankModel instead. While this issue is not consistently reproducible, it occurs frequently enough to cause concern.

Copy link

dosubot bot commented Sep 27, 2024

The intermittent reversion from WeightedScore to RerankModel in the knowledge retrieval node of Dify version 0.8.3 could be due to the retrieval strategy configuration. Specifically, if the retrieval mode is set to "Multi-way Recall," the system requires a Rerank model to be configured for result re-ranking. This could cause the node to revert to using the RerankModel even if it was initially set to use WeightedScore [1][2].

Additionally, the retrieval settings in Dify's high-quality indexing mode indicate that the Rerank model is essential for semantic reordering of retrieved document results. This model is required for the TopK and Score Threshold settings to take effect during the reranking step [3].

The code in knowledge_retrieval_node.py shows that the retrieval mode and reranking configuration are explicitly checked and applied during the retrieval process. If the retrieval mode is set to "MULTIPLE" and the reranking mode is "reranking_model," the system will use the specified reranking model. If the reranking mode is "weighted_score," it will use the provided weights instead [4].

This behavior aligns with the observed issue where the node reverts to using the RerankModel.

To continue talking to Dosu, mention @dosu.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐞 bug Something isn't working 🤔 cant-reproduce We can not reproduce it or the information is limited
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant