Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: behavior of Elasticsearch query differs from mongoDB query #1184

Merged

Conversation

Junjiequan
Copy link
Contributor

@Junjiequan Junjiequan commented Apr 22, 2024

Description

This PR refines the Elasticsearch integration by updating the dataset field mappings, improving the query-building service, and ensuring that the logic is more intuitive and maintainable. The changes are aimed at enhancing the search functionality, ensuring better performance, and providing more accurate search results.

Motivation

The purpose of this PR is to enhance the Elasticsearch configuration and query-building logic to improve the flexibility and accuracy of search queries. This includes updating field mappings, refining the query-building service, and ensuring better alignment with the new requirements for data filtering and querying.

Fixes:

  • Items added

Changes:

Field Mapping Updates (datasetFieldMapping.ts):

  • Added ignore_above attribute to several keyword fields to ensure Elasticsearch handles large field values correctly.
  • Added new field ownerEmail to the mapping.

Elasticsearch Service (elastic-search.service.ts):

  • Corrected logging statement in onDrop method to use doc.document.pid instead of doc.document._id for better clarity and debugging.

Enums Update (fields.enum.ts):

  • Renamed QueryFields to MustFields for clarity.

Query Builder Service (query-builder.service.ts):

  • Refactored buildFilterFields & buildTermsFilter to support fields that are not included in the FilterFields enum.

Tests included/Docs Updated?

  • Included for each change/fix?
  • Passing? (Merge will not be approved unless this is checked)
  • Docs updated?
  • New packages used/requires npm install?
  • Toggle added for new features?

@Junjiequan Junjiequan self-assigned this Apr 23, 2024
@Junjiequan Junjiequan changed the title fix: improved elasticsearch query fix: behavior of Elasticsearch query differs from mongoDB query Apr 24, 2024
@Junjiequan Junjiequan force-pushed the SWAP-3975-scicat-be-elastic-search-query-should-behave-the branch from 1662f5a to f7ba01f Compare May 14, 2024 14:10
@Junjiequan Junjiequan marked this pull request as ready for review May 16, 2024 11:31
@Junjiequan Junjiequan force-pushed the SWAP-3975-scicat-be-elastic-search-query-should-behave-the branch from e5671a6 to 20844f1 Compare May 21, 2024 10:41
@Junjiequan Junjiequan merged commit 5f237c2 into master May 21, 2024
7 checks passed
@Junjiequan Junjiequan deleted the SWAP-3975-scicat-be-elastic-search-query-should-behave-the branch May 21, 2024 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants