[filebeat][cometd] Facing issue with collection of data when leveraging within an integration

[yug-rajani]

While working on the development of the Salesforce integration, we were able to develop the first four data streams using the httpjson input successfully. However, while working on the last two data streams which are supposed to leverage the cometd input, we are facing some issues with data collection (the streaming data doesn't get collected). Here's the issue and all we have tried so far:

• The agent policy template is being set correctly, and there are no errors in the logs. We also looked at the logs from Filebeat started by Elastic-Agent by setting the level to debug.
• We tried using the cometd input from beats repo independently using standalone filebeat, which seemed to work fine. The configuration of cometd in the integration is exactly the same, we are still facing issues with the integration.
• We tried using the salesforce filebeat module from beats repo which internally leverages cometd, which seemed to work fine.
• We tried running the "elastic-agent diagnostics collect" command from inside the elastic-agent container. We identified that the filebeat.sock was not found when running the cometd data streams only. When we enabled the httpjson data streams, we even saw the acknowledgement in the logs that one input is running. However, that isn't the case for cometd.
• On seeing the contents of the filebeat configuration files from the diagnostics collected, we figured out that there was no entry for the cometd configuration out there. However, for the other data streams (login_rest, logout_rest, setupaudittrail) using httpjson, filebeat.yml is getting updated. We have attached the files (filebeat_default.yml and elastic-agent-policy.yml) from the diagnostics here for your reference, along with the working filebeat.yml with standalone filebeat. Post this, we aren't sure where we can look at for the exact cause of the issue.
• We also tried reaching out to the #beats and #integrations channel in Slack (link).

Relevant files:

• filebeat_default.yml.txt
• filebeat.yml.txt
• elastic-agent-policy.yml.txt

Note: Suffix .txt has been added to the files on purpose so that GitHub allows to upload the file. Hence, kindly ignore the extension of the filename.

CC: @lalit-satapathy

Salesforce Integration Development Issue: elastic/integrations#3940

[agithomas]

@yug-elastic , after your debugging, as per the recommendation in the slack conversation, should the issue be assigned to elastic-agent-control-plane-team ?

[yug-rajani]

@yug-elastic , after your debugging, as per the recommendation in the slack conversation, should the issue be assigned to elastic-agent-control-plane-team ?

I pinged the elastic-agent-control-plane-team here and got a response from them that we should reach out to @lalit-satapathy for the same.

[agithomas]

While working on the development of the Salesforce integration, we were able to develop the first four data streams using the httpjson input successfully. However, while working on the last two data streams which are supposed to leverage the cometd input, we are facing some issues with data collection (the streaming data doesn't get collected). Here's the issue and all we have tried so far:

• The agent policy template is being set correctly, and there are no errors in the logs. We also looked at the logs from Filebeat started by Elastic-Agent by setting the level to debug.
• We tried using the cometd input from beats repo independently using standalone filebeat, which seemed to work fine. The configuration of cometd in the integration is exactly the same, we are still facing issues with the integration.

Can you explain in detail the "issues" you are facing in integration?
Are you using standalone agent or fleet agent in docker container ?
Which version of agent are you trying with?
Are there any errors in the agent logs or container logs?

• We tried using the salesforce filebeat module from beats repo which internally leverages cometd, which seemed to work fine.
• We tried running the "elastic-agent diagnostics collect" command from inside the elastic-agent container. We identified that the filebeat.sock was not found when running the cometd data streams only. When we enabled the httpjson data streams, we even saw the acknowledgement in the logs that one input is running. However, that isn't the case for cometd.
• On seeing the contents of the filebeat configuration files from the diagnostics collected, we figured out that there was no entry for the cometd configuration out there. However, for the other data streams (login_rest, logout_rest, setupaudittrail) using httpjson, filebeat.yml is getting updated. We have attached the files (filebeat_default.yml and elastic-agent-policy.yml) from the diagnostics here for your reference, along with the working filebeat.yml with standalone filebeat. Post this, we aren't sure where we can look at for the exact cause of the issue.
• We also tried reaching out to the #beats and #integrations channel in Slack (link).

Relevant files:

• filebeat_default.yml.txt
• filebeat.yml.txt
• elastic-agent-policy.yml.txt

Note: Suffix .txt has been added to the files on purpose so that GitHub allows to upload the file. Hence, kindly ignore the extension of the filename.

CC: @lalit-satapathy

Salesforce Integration Development Issue: elastic/integrations#3940

[yug-rajani]

While working on the development of the Salesforce integration, we were able to develop the first four data streams using the httpjson input successfully. However, while working on the last two data streams which are supposed to leverage the cometd input, we are facing some issues with data collection (the streaming data doesn't get collected). Here's the issue and all we have tried so far:

• The agent policy template is being set correctly, and there are no errors in the logs. We also looked at the logs from Filebeat started by Elastic-Agent by setting the level to debug.
• We tried using the cometd input from beats repo independently using standalone filebeat, which seemed to work fine. The configuration of cometd in the integration is exactly the same, we are still facing issues with the integration.

Can you explain in detail the "issues" you are facing in integration? Are you using standalone agent or fleet agent in docker container ? Which version of agent are you trying with? Are there any errors in the agent logs or container logs?

The issue is basically that the data is not being collected for data streams using cometd input.

We are using the latest (8.4.1) Elastic Stack (with elastic-package tool) which I believe uses the docker-fleet-agent.

No, we couldn't find any errors in the agent logs and container logs related to cometd.

• We tried using the salesforce filebeat module from beats repo which internally leverages cometd, which seemed to work fine.
• We tried running the "elastic-agent diagnostics collect" command from inside the elastic-agent container. We identified that the filebeat.sock was not found when running the cometd data streams only. When we enabled the httpjson data streams, we even saw the acknowledgement in the logs that one input is running. However, that isn't the case for cometd.
• On seeing the contents of the filebeat configuration files from the diagnostics collected, we figured out that there was no entry for the cometd configuration out there. However, for the other data streams (login_rest, logout_rest, setupaudittrail) using httpjson, filebeat.yml is getting updated. We have attached the files (filebeat_default.yml and elastic-agent-policy.yml) from the diagnostics here for your reference, along with the working filebeat.yml with standalone filebeat. Post this, we aren't sure where we can look at for the exact cause of the issue.
• We also tried reaching out to the #beats and #integrations channel in Slack (link).

Relevant files:

• filebeat_default.yml.txt
• filebeat.yml.txt
• elastic-agent-policy.yml.txt

Note: Suffix .txt has been added to the files on purpose so that GitHub allows to upload the file. Hence, kindly ignore the extension of the filename.
CC: @lalit-satapathy
Salesforce Integration Development Issue: elastic/integrations#3940

[agithomas]

I have not got fully gone all the details of cometd and i see that the concept behind works like a subscription. Does the cometd client need to have a port listening to receive the data emitted from the source? If yes, which port it is?

[yug-rajani]

I have not got fully gone all the details of cometd and i see that the concept behind works like a subscription. Does the cometd client need to have a port listening to receive the data emitted from the source? If yes, which port it is?

No, it doesn't require a port. It only requires the client configuration along with the channel name to be subscribed, for example, /event/LoginEventStream.

[milanparmar-crest]

This issue will be resolved as part of this PR: elastic/elastic-agent#1962. These changes will be available in 8.7 release of the elastic-package.

CC: @lalit-satapathy , @agithomas