[Salesforce] Add pagination support

[kush-elastic]

What does this PR do?

In the current Salesforce Module in filebeat, we don’t have pagination support for all four file-sets (login-rest, logout-rest, apex-rest, setupaudittrail-rest). So if in any case, we have a response body with more than 2000 records, the next page is never called and the data will not be fetched.

Solution:
You can request the next batch of records using the provided URL with your instance and session information, and repeat until all records have been retrieved. These requests use nextRecordsUrl and don’t include any parameters. The final batch of records doesn’t have a nextRecordsUrl field and return true value for field done

Ref Link for Issue: https://blog.bessereau.eu/assets/pdfs/api_rest.pdf

Why is it important?

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Relates [httpjson][Salesforce] Add Pagination Support #34057

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b salesforce_module_pagination upstream/salesforce_module_pagination
git merge upstream/main
git push upstream salesforce_module_pagination

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kush-elastic? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-01-27T08:23:15.531+0000

• Duration: 72 min 24 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2533
Skipped	172
Total	2705

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b salesforce_module_pagination upstream/salesforce_module_pagination
git merge upstream/main
git push upstream salesforce_module_pagination

[agithomas]

CreatedDate is changed to LogDate in the query. Which one is most appropriate? Why ?
Is this change fixing any bug?

Kindly complete the internal code review.

[kush-elastic]

CreatedDate is changed to LogDate in the query. Which one is most appropriate? Why ? Is this change fixing any bug?

Currently, in the httpjson we are not able to use the pagination + split + chain call's , we had a similar Issue(#33213) previously and we are in a chat with @ShourieG who has worked on this issue previously(#33664), He is currently looking into the issue, we have provided the required configurations to dig deep into this issue.

Issue
We are not getting the response for the second page.

Solution
-> We removed the split functionality which was used to set cursor value of (CreatedDate) from the first response so that we are able to receive the response from the second page.

-> We changed the cursor value to more reliable data that we receive in the second call(chain call) TIMESTAMP_DERIVED which is related to the value between (LogDate and LogDate + 1h) and has no relation with CreatedDate.

-> We have tried data collection without split functionality it's working fine as it was used for just setting the cursor value which is now changed to TIMESTAMP_DERIVED. Also, we have updated the query to LogDate as it is more related to TIMESTAMP_DERIVED.

Also, we have used the similar logic in our salesforce integration as well.
JFYI:

• LogDate tracks usage activity of a log file for a 24-hour period, from 12:00 a.m. to 11:59 p.m.
• CreatedDate tracks when the log file was generated.

Kindly complete the internal code review.

This PR is internally reviewed but as discussed here we don't have any other account as of now to provide approval, so consider this PR approved, Harnish can comment the same.

[sonarqubecloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
30.3% Duplication

[harnish-elastic]

LGTM! Approved.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b salesforce_module_pagination upstream/salesforce_module_pagination
git merge upstream/main
git push upstream salesforce_module_pagination

[agithomas]

Approving.

We need to see the outcome of testcases run to resolve the SDH issue. According to the outcome, the queries may need to be modified.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b salesforce_module_pagination upstream/salesforce_module_pagination
git merge upstream/main
git push upstream salesforce_module_pagination

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b salesforce_module_pagination upstream/salesforce_module_pagination
git merge upstream/main
git push upstream salesforce_module_pagination

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b salesforce_module_pagination upstream/salesforce_module_pagination
git merge upstream/main
git push upstream salesforce_module_pagination