Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make EQL REST Querying API async #49638

Closed
colings86 opened this issue Nov 27, 2019 · 1 comment · Fixed by #57992
Closed

Make EQL REST Querying API async #49638

colings86 opened this issue Nov 27, 2019 · 1 comment · Fixed by #57992
Assignees
@imotov
Labels
:Analytics/EQL EQL querying Team:QL (Deprecated) Meta label for query languages team

Comments

@colings86
Copy link
Contributor

Once we have the EQL REST Endpoint created from #49634 we should explore making the API asynchronous.

Since EQL queries will require multiple underlying search requests and may need to search through lots of data the queries could take a long time to execute. To give a better user experience it would be good if the query can be executed asynchronously with the user able to request progress and retrieve the final results when they are available.

We already have development underway on adding an asynchronous search API (#49091) so the user experience we add here should be as similar to that as practical so we have consistency in the approaches. The reason we can't just piggyback on the async search API is that the EQL plugin will need to run extra logic on top of the search API, which will require it to have its own async mechanism.
@colings86 colings86 added the :Analytics/EQL EQL querying label Nov 27, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (:Search/EQL)

@colings86 colings86 mentioned this issue Nov 27, 2019
Closed
26 tasks
imotov added a commit to imotov/elasticsearch that referenced this issue Apr 1, 2020
@imotov
EQL: Make EQL search task cancellable
60b6d1c 
First step towards async search execution. At the moment we don't try to cancel
the underlying search requests, and just check if the task is canceled before
performing network operation (such as field caps and search)

Relates to elastic#49638
@imotov imotov mentioned this issue Apr 1, 2020
Merged
imotov added a commit that referenced this issue Apr 6, 2020
@imotov
EQL: Make EQL search task cancellable (#54598)
364c0fe 
First step towards async search execution. At the moment we don't try to cancel
the underlying search requests, and just check if the task is canceled before
performing network operation (such as field caps and search)

Relates to #49638
imotov added a commit to imotov/elasticsearch that referenced this issue Apr 6, 2020
@imotov
EQL: Make EQL search task cancellable (elastic#54598)
ab7c429 
First step towards async search execution. At the moment we don't try to cancel
the underlying search requests, and just check if the task is canceled before
performing network operation (such as field caps and search)

Relates to elastic#49638
@imotov imotov mentioned this issue Apr 6, 2020
Merged
imotov added a commit that referenced this issue Apr 6, 2020
@imotov
EQL: Make EQL search task cancellable (#54598)
1aa87cd 
First step towards async search execution. At the moment we don't try to cancel
the underlying search requests, and just check if the task is canceled before
performing network operation (such as field caps and search)

Relates to #49638
imotov added a commit to imotov/elasticsearch that referenced this issue Apr 6, 2020
@imotov
EQL: Add cascading search cancellation
2a75e88 
EQL search cancellation now propagates cancellation to underlying search
operations.

Relates to elastic#49638
@imotov imotov mentioned this issue Apr 6, 2020
Merged
imotov added a commit that referenced this issue Apr 10, 2020
@imotov
EQL: Add cascading search cancellation (#54843)
9d38dcf 
EQL search cancellation now propagates cancellation to underlying search
operations.

Relates to #49638
imotov added a commit to imotov/elasticsearch that referenced this issue Apr 10, 2020
@imotov
EQL: Add cascading search cancellation (elastic#54843)
55a1cb6 
EQL search cancellation now propagates cancellation to underlying search
operations.

Relates to elastic#49638
@imotov imotov mentioned this issue Apr 10, 2020
Merged
imotov added a commit to imotov/elasticsearch that referenced this issue Apr 13, 2020
@imotov
Make AsyncSearchIndexService reusable
d18c227 
EQL will require very similar functionality to async search. This PR refactors
AsyncSearchIndexService to make it reusable for EQL.

Relates to elastic#49638
@imotov imotov mentioned this issue Apr 13, 2020
Closed
imotov added a commit that referenced this issue Apr 14, 2020
@imotov
EQL: Add cascading search cancellation (#54843)
8a669dc 
EQL search cancellation now propagates cancellation to underlying search
operations.

Relates to #49638
imotov added a commit to imotov/elasticsearch that referenced this issue Apr 22, 2020
@imotov
Make AsyncSearchIndexService reusable
deaeae1 
EQL will require very similar functionality to async search. This PR refactors
AsyncSearchIndexService to make it reusable for EQL.

Supersedes elastic#55119
Relates to elastic#49638
@imotov imotov mentioned this issue Apr 22, 2020
Merged
imotov added a commit that referenced this issue Apr 23, 2020
@imotov
Make AsyncSearchIndexService reusable (#55598)
3bf2df6 
EQL will require very similar functionality to async search. This PR refactors
AsyncSearchIndexService to make it reusable for EQL.

Supersedes #55119
Relates to #49638
imotov added a commit to imotov/elasticsearch that referenced this issue Apr 23, 2020
@imotov
Make AsyncSearchIndexService reusable (elastic#55598)
06b1446 
EQL will require very similar functionality to async search. This PR refactors
AsyncSearchIndexService to make it reusable for EQL.

Supersedes elastic#55119
Relates to elastic#49638
@imotov imotov mentioned this issue Apr 23, 2020
Merged
imotov added a commit that referenced this issue Apr 23, 2020
@imotov
Make AsyncSearchIndexService reusable (#55598)
8c7ef24 
EQL will require very similar functionality to async search. This PR refactors
AsyncSearchIndexService to make it reusable for EQL.

Supersedes #55119
Relates to #49638
@imotov imotov self-assigned this Apr 30, 2020
imotov added a commit to imotov/elasticsearch that referenced this issue May 4, 2020
@imotov
EQL: Adds an ability to start an asynchronous EQL search
d4add59 
Adds support for async searches to eql search API. This commit is limited to
only submitting search API requests and doesn't provide APIs to get results
nor delete the results. These functions will be added in follow up PRs.

Relates to elastic#49638
@imotov imotov mentioned this issue May 4, 2020
Closed
@rjernst rjernst added the Team:QL (Deprecated) Meta label for query languages team label May 4, 2020
imotov added a commit to imotov/elasticsearch that referenced this issue May 12, 2020
@imotov
EQL: Adds an ability to start an asynchronous EQL search
b770022 
Adds support for async searches to eql search API. This commit is limited to
only submitting search API requests and doesn't provide APIs to get results
nor delete the results. These functions will be added in follow up PRs.

Relates to elastic#49638
@imotov imotov mentioned this issue May 12, 2020
Merged
imotov added a commit that referenced this issue May 13, 2020
@imotov
EQL: Adds an ability to start an asynchronous EQL search (#56631)
fcebd4f 
Adds support for async searches to eql search API. This commit is limited to
only submitting search API requests and doesn't provide APIs to get results
nor delete the results. These functions will be added in follow up PRs.

Relates to #49638
imotov added a commit to imotov/elasticsearch that referenced this issue May 13, 2020
@imotov
EQL: Switch to RestCancellableNodeClient in EQL search
e82c1bc 
Switches to RestCancellableNodeClient wrapper for eql search operation in
order to detect clients closing the connection and cancelling the operation.

Relates to elastic#49638
@imotov imotov mentioned this issue May 13, 2020
Merged
imotov added a commit that referenced this issue May 14, 2020
@imotov
EQL: Switch to RestCancellableNodeClient in EQL search (#56692)
c969444 
Switches to RestCancellableNodeClient wrapper for eql search operation in
order to detect clients closing the connection and cancelling the operation.

Relates to #49638
imotov added a commit to imotov/elasticsearch that referenced this issue May 15, 2020
@imotov
EQL: Adds get async EQL search result action
b101b8f 
Adds support for retrieving async EQL search result s to eql search API.

Relates to elastic#49638
@imotov imotov mentioned this issue May 15, 2020
Merged
imotov added a commit that referenced this issue May 27, 2020
@imotov
EQL: Adds get async EQL search result action (#56852)
e999819 
Adds support for retrieving async EQL search result s to eql search API.

Relates to #49638
imotov added a commit to imotov/elasticsearch that referenced this issue May 27, 2020
@imotov
EQL: Adds delete async EQL search result action (elastic#56852)
fce6a02 
Adds support for deleting async EQL search results to eql search API.

Relates to elastic#49638
@imotov imotov mentioned this issue May 27, 2020
Merged
@jrodewig jrodewig mentioned this issue May 29, 2020
Merged
imotov added a commit that referenced this issue Jun 4, 2020
@imotov
EQL: Adds delete async EQL search result action (#57258)
5a95b0f 
Adds support for deleting async EQL search results to EQL search API.

Relates to #49638
imotov added a commit to imotov/elasticsearch that referenced this issue Jun 4, 2020
@imotov
Move async task maintenance service to core plugin
cc05e1b 
The async task task maintenance service is used by both async search plugin
as well as EQL plugin. So it needs to reside in the core.

Relates to elastic#49638
@imotov imotov mentioned this issue Jun 4, 2020
Merged
@jrodewig jrodewig mentioned this issue Jun 5, 2020
Merged
imotov added a commit that referenced this issue Jun 11, 2020
@imotov
Move async task maintenance service to core plugin (#57700)
509c674 
The async task task maintenance service is used by both async search plugin
as well as EQL plugin. So it needs to reside in the core.

Relates to #49638
@imotov imotov mentioned this issue Jun 11, 2020
Merged
imotov added a commit to imotov/elasticsearch that referenced this issue Jun 16, 2020
@imotov @jrodewig
[7.x] EQL: Adds an ability to execute an asynchronous EQL search
a63bb0a 
Adds async support to EQL searches

Closes elastic#49638

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
@imotov imotov mentioned this issue Jun 16, 2020
Merged
imotov added a commit that referenced this issue Jun 25, 2020
@imotov
[7.x] EQL: Adds an ability to execute an asynchronous EQL search (#58192
20af856 
)

Adds async support to EQL searches

Closes #49638

Co-authored-by: James Rodewig james.rodewig@elastic.co
@costin costin mentioned this issue Jul 16, 2020
Closed
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@imotov imotov

Labels
:Analytics/EQL EQL querying Team:QL (Deprecated) Meta label for query languages team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

EQL: Adds an ability to execute an asynchronous EQL search elastic/elasticsearch
4 participants
@colings86 @rjernst @imotov @elasticmachine