zqd: Multipart log ingest #1336
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mattnibs
force-pushed
the
log-post-streaming
branch
from
d20e354
to
c0b2e29
Compare
|
note: This will break brim master. I am working on an update there. |
henridf
reviewed
Sep 22, 2020
henridf
reviewed
Sep 22, 2020
|
General question... with this API it's now possible to post one or more files with |
Yep! Good thinking from @alfred-landrum 😎 |
|
Will hold off on merging until we have accompanying changes on brim. |
nwt
reviewed
Sep 22, 2020
mattnibs
force-pushed
the
log-post-streaming
branch
4 times, most recently
from
607cb1e
to
d599d65
Compare
|
@brimsec/zq this is ready for another review. Now that both multipart and the former post path methods exist, this can go in without any brim dependent changes. |
mattnibs
force-pushed
the
log-post-streaming
branch
2 times, most recently
from
98bc1fb
to
b0b92b3
Compare
nwt
reviewed
Nov 6, 2020
nwt
reviewed
Nov 6, 2020
mattnibs
force-pushed
the
log-post-streaming
branch
2 times, most recently
from
0b59639
to
f0e1a47
Compare
Allow ingest of log data through streaming multi-part http request. The files are submitted as an http multipart request. JSON typing can be submitted as part of the request- it should be given the form value of "json_type_config". For http 1.1 the server cannot send responses back while the requester is sending data. As a result remove status updates and transmit any wires in the server responses. Clients that want to know the amount of data sent while the request is in flight will have to keep track the bytes written over the wire themselves. The zapi post commands have been reformatted. They are as such: - postpcap - post a pcap file to a space - postpath - post paths that will be opened and read by zqd. - post - stream log data to zqd over the network. Closes #1084
mattnibs
force-pushed
the
log-post-streaming
branch
from
f0e1a47
to
8494a9c
Compare
nwt
reviewed
Nov 6, 2020
nwt
reviewed
Nov 6, 2020
Co-authored-by: Noah Treuhaft <noah.treuhaft@gmail.com>
Co-authored-by: Noah Treuhaft <noah.treuhaft@gmail.com>
Co-authored-by: Noah Treuhaft <noah.treuhaft@gmail.com>
nwt
reviewed
Nov 6, 2020
mattnibs
force-pushed
the
log-post-streaming
branch
2 times, most recently
from
45915ae
to
6b90064
Compare
alfred-landrum
approved these changes
Nov 12, 2020
ppl/zqd/handlers_test.go
Outdated
| "#0:record[_path:string,ts:time,uid:bstring]", | ||
| "0:[conn;2;CBrzd94qfowOqJwCHa;]", | ||
| } | ||
| const src1 = `#0:record[_path:string,ts:time,uid:bstring] |
There was a problem hiding this comment.
nit: align these first lines on the left side of the file
mattnibs
force-pushed
the
log-post-streaming
branch
from
6b90064
to
e62a727
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rearrange zqd log post endpoint so log files are transmitted over
the network instead of opened from a locally available file.
The files are submitted as an http multipart request. JSON typing
can be submitted as part of the request- it should be given the
form value of "json_type_config".
For http 1.1 the server cannot send responses back while the
requester is sending data. As a result remove status updates and
transmit any wires in the server responses. Clients that want to
know the amount of data sent while the request is in flight will
have to keep track the bytes written over the wire themselves.
Closes #1084