[Heartbeat] Private location policy update breaks scheduling

[emilioalvap]

• Version: >8.6.1
• Operating System: Docker

Summary

When running browser monitors through a private location policy, any updates to the policy prevents further iterations from running.
It also seems synthetic process are being intefered with when an agent update occurs, since running monitors at the time reach the timeout and terminate on error consistently:

{"log.level":"info","@timestamp":"2023-02-21T11:52:47.610Z","message":"Command has completed(-1): /usr/share/elastic-agent/.node/node/bin/elastic-synthetics elastic-synthetics --screenshots on --throttling 5d/3u/20l --inline --rich-events","component":{"binary":"heartbeat","dataset":"elastic_agent.heartbeat","id":"synthetics/browser-default","type":"synthetics/browser"},"log":{"source":"synthetics/browser-default"},"service.name":"heartbeat","ecs.version":"1.6.0","log.origin":{"file.line":276,"file.name":"synthexec/synthexec.go"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-02-21T11:52:47.611Z","message":"Error executing command '/usr/share/elastic-agent/.node/node/bin/elastic-synthetics elastic-synthetics --screenshots on --throttling 5d/3u/20l --inline --rich-events' (-1): signal: killed","component":{"binary":"heartbeat","dataset":"elastic_agent.heartbeat","id":"synthetics/browser-default","type":"synthetics/browser"},"log":{"source":"synthetics/browser-default"},"service.name":"heartbeat","ecs.version":"1.6.0","log.origin":{"file.line":282,"file.name":"synthexec/synthexec.go"},"ecs.version":"1.6.0"}

When collecting elastic-agent diagnostics, it seems it's not able to serialize heartbeat integrations correctly:

root@bce4abe537ff:/usr/share/elastic-agent# elastic-agent diagnostics collect
[WARNING] Could not redact state.yaml due to unmarshalling error: yaml: invalid map key: map[interface {}]interface {}{"unitid":"synthetics/browser-default", "unittype":1}
Created diagnostics archive "elastic-agent-diagnostics-2023-02-21T12-52-40Z-00.zip"

How to repro?

1. Create a private location policy with long running monitors, eg:

step("Test", async () => {
    await page.goto("https://google.es");
    await page.waitForTimeout(600e3);
})

2. Enroll and agent and wait for monitors to start running:

{"log.level":"info","@timestamp":"2023-02-21T11:37:17.603Z","message":"Running command: /usr/share/elastic-
agent/.node/node/bin/elastic-synthetics elastic-synthetics --screenshots on...

3. Update any of the monitors to trigger a policy update.
4. Wait for next monitor iteration to happen.

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[emilioalvap]

Adding more info, it seems that browser jobs are not being cleared correctly from the running context when the policy is updated, preventing further jobs from running.

[emilioalvap]

Additionally, two other potential issues:

• Browser project context is not closing all related synthetics context when cancelled. This can be fixed by declaring a project context and passing it as a parent context for generated jobs.

• config Reload() from libbeat is generating different hashes for all new monitors, so it's constantly stopping and reloading running monitors, even when config didn't change. This is probably the case because policy revision is included in the hash struct:, instead of only using the integration local revision: