[Logs onboarding] Determine if logs are being shipped to elastic

[yngrdyn]

At the moment we are determining whether logs are being shipped or not to elastic by performing the following search

const { hits } = await esClient.search({
    index: `logs-${dataset}-${namespace}`,
    terminate_after: 1,
});

This search could give users false positives in regards to the data, What if this index already exists when users are onboarding their logs?

@grabowskit, @ruflin, @gbamparop
Will this be a common scenario for us? Could we assume this scenario to be an edge case and proceed with the current check?
Is there a more robust way to check this? For example, in addition to the index, check for file paths (the ones the user configured in the previous step) as part of determining logs arrival?

[elasticmachine]

Pinging @elastic/apm-ui (Team:APM)

[gbamparop]

I think it's a good idea to have some filters such as path, @timestamp etc.

[ruflin]

Would it be possible to report the host.name from the machine where the script is running? If yes, this could be used to query only for data from this host.

I initially wanted to comment that we should always only query for the last 15min. But this might not provide the expected result. In case someone starts to tail a file with logs from the past few days and we extract the timestamp, only older data will be shipped first.

[gbamparop]

Fleet uses the agent id to query data for the last 5 minutes here. Is this issue related to all shippers?

[gbamparop]

Fleet uses the agent id to query data for the last 5 minutes here. Is this issue related to all shippers?

We've decided in today's meeting to go with this proposal for now, adding it to the Ready column.