The Dataset infinite scroll functionality breaks when filters change

[nht1206]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

1.0.0

Cloud or Self Hosted

Self Hosted (Docker), Self Hosted (Source)

Steps to reproduce

1. Create or ensure there are more than 30 knowledge bases in the workspace.
2. Apply any filter
3. Attempt to scroll to load more filtered results

✔️ Expected Behavior

Should load more items matching the filter

❌ Actual Behavior

Shows the same page being requested repeatedly

[dosubot]

The issue with the infinite scroll functionality breaking when filters change in the self-hosted Dify version 1.0.0 is likely due to the useSWRInfinite hook's configuration. The getKey function, which generates the key for fetching data, depends on the tags, keywords, and includeAll parameters. When these parameters change, the hook should re-fetch the data with the new filters. However, the current setup might not automatically handle these changes to reset the pagination or re-fetch the data correctly.

To resolve this, ensure that the useSWRInfinite hook re-fetches data when the filter parameters change. You can achieve this by adding the filter parameters as dependencies in the useEffect hook that manages the scroll event. This ensures that the data is re-fetched when these parameters change:

useEffect(() => {
  const onScroll = debounce(() => {
    if (!loadingStateRef.current) {
      const { scrollTop, clientHeight } = containerRef.current!
      const anchorOffset = anchorRef.current!.offsetTop
      if (anchorOffset - scrollTop - clientHeight < 100)
        setSize(size => size + 1)
    }
  }, 50)

  containerRef.current?.addEventListener('scroll', onScroll)
  return () => containerRef.current?.removeEventListener('scroll', onScroll)
}, [tags, keywords, includeAll]) // Add filter parameters as dependencies

This change updates the scroll event listener whenever the filter parameters change, prompting the useSWRInfinite hook to re-fetch data with the new filters ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}