Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Point-in-time API #1141

Closed
danielmitterdorfer opened this issue Dec 21, 2020 · 1 comment · Fixed by #1190
Closed

Support Point-in-time API #1141

danielmitterdorfer opened this issue Dec 21, 2020 · 1 comment · Fixed by #1190
Assignees
@DJRickyB
Labels
enhancement Improves the status quo :Load Driver Changes that affect the core of the load driver such as scheduling, the measurement approach etc.
Milestone
2.x

Comments

@danielmitterdorfer
Copy link
Member

Currently Rally only supports the scroll API for paginating search results. Elasticsearch has introduced a point-in-time API that we should also support similar to how we do it with async search since #1129 (i.e. also assume the same restrictions such as it is only allowed in the context of a composite task). We should specifically make sure to add support for search_after using the point-in-time API.
@danielmitterdorfer danielmitterdorfer added enhancement Improves the status quo :Load Driver Changes that affect the core of the load driver such as scheduling, the measurement approach etc. labels Dec 21, 2020
@danielmitterdorfer danielmitterdorfer added this to the 2.x milestone Dec 21, 2020
@DJRickyB DJRickyB self-assigned this Jan 4, 2021
@DJRickyB
Copy link
Contributor

DJRickyB commented Jan 4, 2021

I've started on this. there are a couple of structural considerations, ultimately I've started down the path of each of the following being a supported operation:

  • Open Point In Time (requires composite context)
  • Close Point In Time (requires composite context)
  • Query with Search After Pagination (P-I-T, and hence composite, optional)
    The meat of this approach is here -> 80b4390

@DJRickyB DJRickyB mentioned this issue Jan 5, 2021
Closed
@DJRickyB DJRickyB mentioned this issue Feb 18, 2021
Merged
DJRickyB pushed a commit that referenced this issue Mar 25, 2021
Add support for search_after and point-in-time (#1190)
62595bd 
This change adds support for `search_after` usage and `point-in-time` interactions.  Specifically:
* Defines `open-point-in-time` and `close-point-in-time` operations, for use in composite contexts
* Adds support to the query/search runner for `search_after` pagination, using a new operation type `paginated-search`
* Defines `scroll-search` operation type for usability
* Moves runner parsing logic to its own module for isolation
* Adds test and documentation for the above

Bonus:
* Re-factors existing query runner to reduce duplicated code
* Adds a benchmark file for demonstrating and experimenting with parsing things in the vein of `detailed-results` and the properties required for `search_after` support
* A minor documentation fix (corrected `get-async-search` to `delete-async-search`)
* Style fixes in docs with extra whitespace

Closes #1141
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DJRickyB DJRickyB

Labels
enhancement Improves the status quo :Load Driver Changes that affect the core of the load driver such as scheduling, the measurement approach etc.
Projects
None yet
Milestone
2.x
2 participants
@danielmitterdorfer @DJRickyB