Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor and enhance issue indexer to support both searching, filtering and paging #26012

Merged
merged 108 commits into from
Jul 31, 2023
Merged

Refactor and enhance issue indexer to support both searching, filtering and paging #26012

merged 108 commits into from
Jul 31, 2023

Conversation

wolfogre
Copy link
Member

@wolfogre wolfogre commented Jul 20, 2023

Fix #24662.

Replace #24822 and #25708 (although it has been merged)

Background

In the past, Gitea supported issue searching with a keyword and conditions in a less efficient way. It worked by searching for issues with the keyword and obtaining limited IDs (as it is heavy to get all) on the indexer (bleve/elasticsearch/meilisearch), and then querying with conditions on the database to find a subset of the found IDs. This is why the results could be incomplete.

To solve this issue, we need to store all fields that could be used as conditions in the indexer and support both keyword and additional conditions when searching with the indexer.

Major changes

  • Redefine IndexerData to include all fields that could be used as filter conditions.
  • Refactor Search(ctx context.Context, kw string, repoIDs []int64, limit, start int, state string) to Search(ctx context.Context, options *SearchOptions), so it supports more conditions now.
  • Change the data type stored in issueIndexerQueue. Use IndexerMetadata instead of IndexerData in case the data has been updated while it is in the queue. This also reduces the storage size of the queue.
  • Enhance searching with Bleve/Elasticsearch/Meilisearch, make them fully support SearchOptions. Also, update the data versions.
  • Keep most logic of database indexer, but remove issues.SearchIssueIDsByKeyword in models to avoid confusion where is the entry point to search issues.
  • Start a Meilisearch instance to test it in unit tests.
  • Add unit tests with almost full coverage to test Bleve/Elasticsearch/Meilisearch indexer.

@wxiaoguang wxiaoguang added the status/blocked This PR cannot be merged yet, i.e. because it depends on another unmerged PR label Jul 28, 2023
Copy link
Member

@techknowlogick techknowlogick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ty :)

@techknowlogick techknowlogick added reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. and removed status/blocked This PR cannot be merged yet, i.e. because it depends on another unmerged PR labels Jul 31, 2023
@techknowlogick techknowlogick enabled auto-merge (squash) July 31, 2023 02:01
@lunny lunny disabled auto-merge July 31, 2023 04:36
@techknowlogick techknowlogick enabled auto-merge (squash) July 31, 2023 04:43
@techknowlogick techknowlogick merged commit 1e76a82 into go-gitea:main Jul 31, 2023
23 checks passed
@GiteaBot GiteaBot removed the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Jul 31, 2023
zjjhot added a commit to zjjhot/gitea that referenced this pull request Jul 31, 2023
* upstream/main: (26 commits)
  Add 'Show on a map' button to Location in profile, fix layout (go-gitea#26214)
  Use shared template for webhook icons (go-gitea#26242)
  Reduce margins on user settings page, introduce `flex-container` (go-gitea#26046)
  Refactor and enhance issue indexer to support both searching, filtering and paging (go-gitea#26012)
  Show image size on view page (go-gitea#25884)
  Fix pull request check list is limited (go-gitea#26179)
  Fix API leaking Usermail if not logged in (go-gitea#25097)
  [skip ci] Updated licenses and gitignores
  Fix typo in metadata (go-gitea#26207)
  Update js and py dependencies (go-gitea#26243)
  De-emphasize issue sidebar buttons (go-gitea#26171)
  Don't autosize textarea in diff view (go-gitea#26233)
  Add `/public/assets` to `.ignore` (go-gitea#26232)
  Fix attachment clipboard copy on insecure origin (go-gitea#26224)
  Fix commit compare style (go-gitea#26209)
  Fix unable to display individual-level project (go-gitea#26198)
  Fix access check for org-level project (go-gitea#26182)
  Fixed incorrect locale references (go-gitea#26218)
  Use calendar icon for `Joined on...` in profiles (go-gitea#26215)
  Add changelog for 1.20.2 (go-gitea#26208)
  ...
silverwind pushed a commit that referenced this pull request Aug 15, 2023
lunny pushed a commit that referenced this pull request Aug 16, 2023
Fix #26536

Follow #26012

Whatever the comment type is, always update the issue indexer. So the
issue indexer will be updated when there is a change in Status,
Assignee, Label, and so on.

I added the logic for `NotifyUpdateComment`, but missed it for
`NotifyCreateIssueComment` and `NotifyDeleteComment`.
lunny pushed a commit that referenced this pull request Aug 16, 2023
Follow #26012 #26490.

A detailed description has been added to the comment.
techknowlogick pushed a commit that referenced this pull request Aug 23, 2023
This PR has multiple parts, and I didn't split them because
it's not easy to test them separately since they are all about the
dashboard page for issues.

1. Support counting issues via indexer to fix #26361
2. Fix repo selection so it also fixes #26653
3. Keep keywords in filter links.

The first two are regressions of #26012.

After:

https://github.com/go-gitea/gitea/assets/9418365/71dfea7e-d9e2-42b6-851a-cc081435c946

Thanks to @CaiCandong  for helping with some tests.
@go-gitea go-gitea locked as resolved and limited conversation to collaborators Oct 29, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/enhancement An improvement of existing functionality type/refactoring Existing code has been cleaned up. There should be no new functionality.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Incomplete issue search results in repository with many issues
5 participants