[BUG] Cannot Send Bulk Requests

[Nesdom13]

What is the bug?
When attempting to use the .bulk endpoint, the following error is thrown: "The bulk request must be terminated by a newline [\n]"

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Build the json for a bulk request (either adding entries to a JsonObject or building a JsonString)
2. Build a BulkRequest giving your json as the value
3. Pass the BulkRequest as the parameter for the client.bulk() endpoint
4. See error message

What is the expected behavior?
The bulk request should be accepted and actioned.

What is your host/environment?

• OS: macOS Big Sur
• Version 11.6
• oOpenJDK version 1.8.0
• OpenJDK Runtime Environment Corretto-8.302.08.1

Do you have any additional context?
I attempted to use both Jakarta JsonObject and Jakarta JsonString. With the object I used the builder .add method to add entries for the actions and the document information, with the JsonString I built a string that held the json and ended in the new line character and that still didn't work.

[reta]

@Nesdom13 could you please elaborate more on what kind of error message you see? Also, how do you configure JsonpMapper instance for the RestClientTransport?

[Nesdom13]

I'm attempting to use the .bulk method on the client which takes in a BulkRequest. To build the BulkRequest I am using the following:
BulkRequest bulkCreateRecordRequest = new BulkRequest.Builder<Map>() .index(INDEX_NAME) .value(recordListJson) .build();

That recordListJson that is being passed in is being built as follows:

`
List recordListJson = records.stream().flatMap(record -> {
JsonObject recordRequestDetails = Json.createObjectBuilder()
.add("index", Json.createObjectBuilder()
.add("_index", INDEX_NAME)
.add("_id", recordId)
.add("_routing", recordRouting))
.build();

    JsonObjectBuilder recordRequestRecordBuilder = Json.createObjectBuilder();

    for (Map.Entry<String, Object> entry : record.toMap().entrySet()) {
      if (entry.getValue() instanceof Long) {
        Long currentValue = (Long) entry.getValue();
        recordRequestRecordBuilder.add(entry.getKey(), currentValue);
      } else if (entry.getValue() instanceof Boolean) {
        boolean currentValue = (Boolean) entry.getValue();
        recordRequestRecordBuilder.add(entry.getKey(), currentValue);
      } else {
        String currentValue = String.valueOf(entry.getValue());
        recordRequestRecordBuilder.add(entry.getKey(), currentValue);
      }
    }

    JsonObject recordRequestRecord = recordRequestRecordBuilder.build();

    return Stream.of(recordRequestDetails, recordRequestRecord);
  }).collect(Collectors.toList());

`

I also tried building it as follows:

`
String bulkRequestString = "";

  for (Record record : records) {
    bulkRequestString += "{'index':{'_index':'testindex', '_id':'1234', '_routing':'12345678'}}\\n";
    bulkRequestString += "{";
    for (Map.Entry<String, Object> entry : record.toMap().entrySet()) {
      if (entry.getValue() instanceof Long || entry.getValue() instanceof Boolean) {
        bulkRequestString += "'" + entry.getKey() + "':" + entry.getValue() + ",";
      } else {
        bulkRequestString += "'" + entry.getKey() + "':'" + entry.getValue() + "',";
      }
    }
    bulkRequestString += "}\\n";
  }

  JsonString recordListJson = Json.createValue(bulkRequestString);

`

Here is an example of the error:
java.lang.RuntimeException: org.opensearch.client.ResponseException: method [POST], host [https://localhost:9200], URI [/locale/_bulk], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"},"status":400}

[reta]

@Nesdom13 ok, it seems like the Bulk request submission for Rest Client is completely broken right now: the request payload is supposed to send individual JSON objects (streams), separated by new line \n [1] but what is being sent instead is JSON array. To illustrate the issue, this is what is being sent:

[{ "index": { "_index": "movies", "_id": "tt1979320" } }, { "title": "Rush", "year": 2013}] 

This is what is server expects:

{ "index": { "_index": "movies", "_id": "tt1979320" } }
{ "title": "Rush", "year": 2013}

I know, the message you are getting is confusing, perhaps the cause why it fails is clear now, although I don't have a workaround for you, it seems like another autogenerator issue.

[1] https://opensearch.org/docs/latest/opensearch/rest-api/document-apis/bulk/

[cherryramatisdev]

Any updates on this ?

[dblock]

If @Nesdom13 @cherryramatisdev want to help, turn this into a failing test (and maybe a fix)?

[reta]

@cherryramatisdev @dblock the 1.0.0 release of the opensearch-client comes with large chunk of changes, the bulk operations seem to work now BUT the API has changed significantly (snippet is below but you would have to adapt your implementation):

final Map<String, Object> record = new TreeMap<>();
record.put("age", 35);

final Builder builder = new BulkRequest.Builder().operations(
    new BulkOperation.Builder()
       .index(new IndexOperation.Builder<>()
       .id("1")
       .routing("1")
       .index(index)
       .document(record)
       .build()).build(),
     new BulkOperation.Builder()
       .index(new IndexOperation.Builder<>()
       .id("2")
       .routing("2")
       .index(index)
       .document(record)
       .build()).build());
    
BulkResponse response = client.bulk(builder.build());
...

That would result in correct JSON being sent out:

{"index":{"_id":"1","_index":"sample-index","routing":"1"}}
{"age":35}
{"index":{"_id":"2","_index":"sample-index","routing":"2"}}
{"age":35}

Hope it helps, sorry for delayed response.

[dblock]

@reta Much thanks!

@VachaShah We probably need some documentation for this and we can close?

[VachaShah]

@reta Much thanks!

@VachaShah We probably need some documentation for this and we can close?

Sure, let me see where we can document this. WDYT of the Using Java Client section?