Filebeat google_workspace module uses same 'startTime' url param forever when response is paginated.

[wendeuber]

Our Google workspace logs ingestion project suffers serous performance issue using Filebeat 8.3.2 google_workspace module to pull Google Drive logs into our ElasticSearch cluster.

Our Drive logs has a run rate of 500-600 events/s and can easily get paginated since Google Reports API has max page size of 1000. Once paginated, the requests for the intervals thereafter all use the same startTime because of the recent pagination bug fix.

Captured all the 'RawQuery' of all requests in Debug mode:
"@timestamp":"2022-07-14T19:20:00.707Z","log.logger":"input.httpjson-cursor","log.origin":{"file.name":"httpjson/response.go","file.line":125},"message":"last received page: &httpjson.response{page:30, url:url.URL{Scheme:\"https\", Opaque:\"\", User:(*url.Userinfo)(nil), Host:\"www.googleapis.com\", Path:\"/admin/reports/v1/activity/users/all/applications/drive\", RawPath:\"\", ForceQuery:false, RawQuery:\"pageToken=A%3A1657826281668000%3A2353685954652725353%3A777491262838%3AC040ao83r&startTime=2022-07-14T19%3A17%3A57Z\"
"@timestamp":"2022-07-14T19:20:01.932Z","log.logger":"input.httpjson-cursor","log.origin":{"file.name":"httpjson/response.go","file.line":125},"message":"last received page: &httpjson.response{page:31, url:url.URL{Scheme:\"https\", Opaque:\"\", User:(*url.Userinfo)(nil), Host:\"www.googleapis.com\", Path:\"/admin/reports/v1/activity/users/all/applications/drive\", RawPath:\"\", ForceQuery:false, RawQuery:\"pageToken=A%3A1657826279219000%3A2727150340132871807%3A777491262838%3AC040ao83r&startTime=2022-07-14T19%3A17%3A57Z\"
"@timestamp":"2022-07-14T19:20:03.322Z","log.logger":"input.httpjson-cursor","log.origin":{"file.name":"httpjson/response.go","file.line":125},"message":"last received page: &httpjson.response{page:1, url:url.URL{Scheme:\"https\", Opaque:\"\", User:(*url.Userinfo)(nil), Host:\"www.googleapis.com\", Path:\"/admin/reports/v1/activity/users/all/applications/drive\", RawPath:\"\", ForceQuery:false, RawQuery:\"startTime=2022-07-14T19%3A17%3A57Z\"
"@timestamp":"2022-07-14T19:20:04.843Z","log.logger":"input.httpjson-cursor","log.origin":{"file.name":"httpjson/response.go","file.line":125},"message":"last received page: &httpjson.response{page:2, url:url.URL{Scheme:\"https\", Opaque:\"\", User:(*url.Userinfo)(nil), Host:\"www.googleapis.com\", Path:\"/admin/reports/v1/activity/users/all/applications/drive\", RawPath:\"\", ForceQuery:false, RawQuery:\"pageToken=A%3A1657826389459000%3A1568710631165131716%3A777491262838%3AC040ao83r&startTime=2022-07-14T19%3A17%3A57Z\"
"@timestamp":"2022-07-14T19:20:06.274Z","log.logger":"input.httpjson-cursor","log.origin":{"file.name":"httpjson/response.go","file.line":125},"message":"last received page: &httpjson.response{page:3, url:url.URL{Scheme:\"https\", Opaque:\"\", User:(*url.Userinfo)(nil), Host:\"www.googleapis.com\", Path:\"/admin/reports/v1/activity/users/all/applications/drive\", RawPath:\"\", ForceQuery:false, RawQuery:\"pageToken=A%3A1657826388178000%3A230205037924502923%3A777491262838%3AC040ao83r&startTime=2022-07-14T19%3A17%3A57Z\"
"@timestamp":"2022-07-14T19:20:07.872Z","log.logger":"input.httpjson-cursor","log.origin":{"file.name":"httpjson/response.go","file.line":125},"message":"last received page: &httpjson.response{page:4, url:url.URL{Scheme:\"https\", Opaque:\"\", User:(*url.Userinfo)(nil), Host:\"www.googleapis.com\", Path:\"/admin/reports/v1/activity/users/all/applications/drive\", RawPath:\"\", ForceQuery:false, RawQuery:\"pageToken=A%3A1657826387074000%3A4365156032944158179%3A777491262838%3AC040ao83r&startTime=2022-07-14T19%3A17%3A57Z\"
....................
"@timestamp":"2022-07-14T19:20:27.079Z","log.logger":"input.httpjson-cursor","log.origin":{"file.name":"httpjson/response.go","file.line":125},"message":"last received page: &httpjson.response{page:16, url:url.URL{Scheme:\"https\", Opaque:\"\", User:(*url.Userinfo)(nil), Host:\"www.googleapis.com\", Path:\"/admin/reports/v1/activity/users/all/applications/drive\", RawPath:\"\", ForceQuery:false, RawQuery:\"pageToken=A%3A1657826369819000%3A-8103844197655827342%3A777491262838%3AC040ao83r&startTime=2022-07-14T19%3A17%3A57Z\"

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[jamiehynds]

@marc-gr is this a result of the recent bug fix here: elastic/integrations#3140 ?

[wendeuber]

Any update on this issue, which is hurting us badly in production.

[marc-gr]

Hello @wendeuber ! Would be useful if you could enable debug logs through the fleet ui and share some for one of the infinite requests intervals, as we will be able to gather much more information about what might be going on. Please mind to redact any sensitive data from them since this is an open repository. (Also remember to turn off debugging once done, since having it enabled might impact performance)