[FEATURE] Support for filter_path query parameter

[balogal]

OpenSearch supports filtering response bodies with the filter_path query parameter on all REST operations.

In the previous v1 & v2 versions of this library, that parameter could be set with WithFilterPath (e.g. here). However, it seems like the possibility to set this query param was lost with the move to the more go idiomatic API design, with the release of v3. Or at least, I'm not able to figure out how.

A possible workaround is to implement custom request types and use the Do function, but that's of course quite cumbersome.

[Jakob3xD]

This parameter got removed as the response does not match any Response Struct we define. Therefore using the parameter would result in an parsing error. Even if we would find out what struct we can use to parse it, we would need to return it as type any or try to cast it somehow into the response struct.

What is your use case for this parameter?

Do you have an implementation idea?

[balogal]

I'm using the bulk endpoint to index millions of documents. The bulk request returns a lengthy status report about each upserted document in which I am not interested. Following this advice (second to last point), I want to use the filter_path param in order to significantly reduce the response body size and ultimately improve indexing performance.

You raise some valid concerns about the implementation. My assumption was, that filtering the response body will lead to parts of the result structs being returned with their zero-value (which I as a developer would be prepared to deal with, since I'm actively taking the decision to filter out said content). However, I didn't look into it in detail and of course if we end up with some kind of parse error, that would not work.

[balogal]

In case someone stumbles over this, I implemented the workaround with the custom request type as follows:

type CustomBulkReq struct {
	opensearchapi.BulkReq
}

func (r CustomBulkReq) GetRequest() (*http.Request, error) {
	req, err := r.BulkReq.GetRequest()
	if err != nil {
		return req, err
	}
	q := req.URL.Query()
	q.Add("filter_path", "took,errors,items.*.error")
	req.URL.RawQuery = q.Encode()
	return req, nil
}

func main() {
	baseReq := opensearchapi.BulkReq{/* initialize as normal */}
	req := CustomBulkReq{r}
	var data opensearchapi.BulkResp
	resp, err := c.client.Client.Do(ctx, req, &data)
	// ...
}

So far everything works as expected, obviously not every field of the response will be populated.

[MaximMolchanov]

does not match any Response Struct we define

Can you please explain what does it mean? Not sure I got it

would result in an parsing error

Can you please show a small code example where parsing error is returned?

@Jakob3xD ^

I've tried to make search request with filter_path param, I've added a couple of lines into your package - just straightforward FilterPath []string field into SearchParams struct and the same as for all params code to parse it:

	if r.FilterPath != nil {
		params["filter_path"] = strings.Join(r.FilterPath, ",")
	}

And request like this:

	resp, err := client.Search(ctx, &opensearchapi.SearchReq{
		Indices: []string{index},
		Body:    opensearchutil.NewJSONReader(q),
		Params: opensearchapi.SearchParams{
			FilterPath: filterPath,
		},
	})

works for me with no parser errors and the response is really filtered as expected

Also I see that in @balogal example the response is also parsed into your struct with no parser error.

If no parser errors - is it possible to please add this param into package?

[Jakob3xD]

@MaximMolchanov it could be that I am wrong. I tough that filter_path directly returns the wanted fields without the rest of the struct, which would have caused the parsing error. If I understand you correctly the returned data is still in the correct struct format.

If your tests works, feel free to open an PR with your changes.

I am currently working on other things and have to time to do this on my own.