Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Heartbeat] Fix reload policy hash consistency, browser monitor cleanup and semaphore release #34697

Merged
merged 8 commits into from
Mar 7, 2023

Conversation

emilioalvap
Copy link
Collaborator

@emilioalvap emilioalvap commented Feb 28, 2023

What does this PR do?

Closes #34629.

Addresses multiple issues with private location updates:

  • Remove global fields from policy to prevent agent from stopping integrations which have not been updated.
  • Detect browser monitor has ben cancelled by having a global (to the monitor) context .
  • Fix an issue where job limit semaphore was not released after being acquired.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  1. Create a private location policy with multiple long-running browser monitors.
step("Test", async () => {
    await page.goto("https://google.es");
    await page.waitForTimeout(600e3);
})
  1. Enroll an agent in the policy.
  2. On a different window, monitor processes executed by agent-heartbeat. This is an example to monitor a docker instance:
$: docker exec -it -u root agent watch ps -aef
  1. Start updating monitors in Kibana (can be by enabling/disabling them) one by one and watch for agent reaction.
    • When a monitor that is running is disabled, heartbeat should kill the synthetics process and schedule the next job.
    • When a monitor that is not running is updated, heartbeat should not terminate any of the running jobs.
    • After multiple updates, heartbeat should still continue scheduling jobs since the job limit semaphore is updated even when a job is cancelled.

@emilioalvap emilioalvap added bug Team:obs-ds-hosted-services Label for the Observability Hosted Services team v8.7.0 labels Feb 28, 2023
@emilioalvap emilioalvap requested a review from a team as a code owner February 28, 2023 17:06
@elasticmachine
Copy link
Collaborator

Pinging @elastic/uptime (Team:Uptime)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Feb 28, 2023
@mergify
Copy link
Contributor

mergify bot commented Feb 28, 2023

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @emilioalvap? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@emilioalvap emilioalvap added the backport-skip Skip notification from the automated backport with mergify label Feb 28, 2023
@elasticmachine
Copy link
Collaborator

elasticmachine commented Feb 28, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-03-07T15:38:38.132+0000

  • Duration: 48 min 12 sec

Test stats 🧪

Test Results
Failed 0
Passed 1921
Skipped 25
Total 1946

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@emilioalvap emilioalvap added backport-v8.7.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Mar 2, 2023
@andrewvc
Copy link
Contributor

andrewvc commented Mar 6, 2023

LGTM, confirmed with local testing

@emilioalvap emilioalvap merged commit cbd1a38 into elastic:main Mar 7, 2023
mergify bot pushed a commit that referenced this pull request Mar 7, 2023
…up and semaphore release (#34697)

* Fix reload policy hash consistency, browser monitor cleanup and semaphore release

* Update x-pack/heartbeat/monitors/browser/project.go

* Add changelog

* Update heartbeat/scheduler/schedjob.go

---------

Co-authored-by: Andrew Cholakian <andrewvc@elastic.co>
(cherry picked from commit cbd1a38)
emilioalvap added a commit that referenced this pull request Mar 7, 2023
…up and semaphore release (#34697) (#34760)

* Fix reload policy hash consistency, browser monitor cleanup and semaphore release

* Update x-pack/heartbeat/monitors/browser/project.go

* Add changelog

* Update heartbeat/scheduler/schedjob.go

---------

Co-authored-by: Andrew Cholakian <andrewvc@elastic.co>
(cherry picked from commit cbd1a38)

Co-authored-by: Emilio Alvarez Piñeiro <95703246+emilioalvap@users.noreply.github.com>
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
…up and semaphore release (#34697)

* Fix reload policy hash consistency, browser monitor cleanup and semaphore release

* Update x-pack/heartbeat/monitors/browser/project.go

* Add changelog

* Update heartbeat/scheduler/schedjob.go

---------

Co-authored-by: Andrew Cholakian <andrewvc@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v8.7.0 Automated backport with mergify bug Team:obs-ds-hosted-services Label for the Observability Hosted Services team v8.7.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Heartbeat] Private location policy update breaks scheduling
3 participants