Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CBOR-friendly bulk format #8481

Open
clintongormley opened this issue Nov 14, 2014 · 4 comments
Open

Add CBOR-friendly bulk format #8481

clintongormley opened this issue Nov 14, 2014 · 4 comments
Assignees
Labels
:Distributed/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. >enhancement help wanted adoptme Team:Distributed Meta label for distributed team

Comments

@clintongormley
Copy link
Contributor

The current bulk format in JSON uses the \n as end-of-line markers, allowing the coordinating node to parse just the metadata line, then skip over the body line.

CBOR doesn't have any disallowed characters or character sequences, so this mechanism will not work. Instead we should have a CBOR specific format which looks like this:

{ metadata line }
int_length_of_body_line
{ body line }

The int_length_of_body_line would allow us to skip to the next metadata line.

For the delete action, no int_length_of_body_line would be required, as we already know that it can't have a body.

Depends on #7640

@tarass
Copy link

tarass commented Aug 12, 2016

Any updates?

CBOR would be great compliment to improved number handling in 5.0 release considering it has variable size numerics.

@tlrx
Copy link
Member

tlrx commented Aug 16, 2016

Since CBOR allows embedded CBOR data item, maybe we could have a CBOR specific bulk format that is basically an array of data items but use https://tools.ietf.org/html/rfc7049#section-2.4.4.1 for the body item?

@clintongormley
Copy link
Contributor Author

@tlrx that's an interesting idea... not sure how easy it'd be to fit into our bulk parsing code. Please could you investigate when you have some time?

@lcawl lcawl added :Distributed/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. and removed :Bulk labels Feb 13, 2018
@rjernst rjernst added the Team:Distributed Meta label for distributed team label May 4, 2020
@vadimkibana
Copy link

vadimkibana commented Jun 29, 2022

Two more options how bulk requests could be achieved:

  1. CBOR documents can simply follow one after another, until the end of HTTP request is reached—this way CBOR parser can simply parse them one by one sequentially.
  2. CBOR specification has an Indefinite length array type, which is an unknown size array, which is terminated by a break 0xFF octet. This way the _bulk endpoint payload could be an indefinite length CBOR array, where each array element is a separate request payload.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. >enhancement help wanted adoptme Team:Distributed Meta label for distributed team
Projects
None yet
Development

No branches or pull requests

8 participants