Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] Restart upgrade should be allowed when upgrade is in progress #184212

Closed
juliaElastic opened this issue May 24, 2024 · 5 comments · Fixed by #184586
Closed

[Fleet] Restart upgrade should be allowed when upgrade is in progress #184212

juliaElastic opened this issue May 24, 2024 · 5 comments · Fixed by #184586
Assignees
@juliaElastic
Labels
bug Fixes for quality problems that affect the customer experience Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@juliaElastic
Copy link
Contributor

Issue was reported on a cluster version 8.13.2.

Agent was stuck in updating and it looks like the Restart upgrade action was disabled.
The Restart upgrade action should be enabled if the agent is being upgraded.

image
@juliaElastic juliaElastic added bug Fixes for quality problems that affect the customer experience Team:Fleet Team label for Observability Data Collection Fleet team labels May 24, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@juliaElastic juliaElastic self-assigned this May 31, 2024
@juliaElastic
Copy link
Contributor Author

juliaElastic commented May 31, 2024 •
edited
Loading

I could reproduce:

  • enroll an agent with an older version e.g. 8.13.1
  • upgrade agent with a fictive version e.g. 8.13.5
  • update the agent doc to set the upgrade_started_at to more than 2 hours ago
  • go to Restart upgrade, the modal window doesn't allow restart to submit
image 
   curl -sk -XPOST --user fleet_superuser:password -H 'content-type:application/json' \
   -H'x-elastic-product-origin:fleet' \
   http://localhost:9200/.fleet-agents/_update_by_query -d '
   {
      "script": {
        "source": "ctx._source.upgrade_details.state = \"UPG_DOWNLOADING\"; ctx._source.upgrade_started_at = \"2024-05-31T10:16:25Z\"; ctx._source.upgraded_at = null;",
        "lang": "painless"
      },
      "query": {
        "term": {
          "agent.id":"ade9faa9-ae8c-40a2-905e-2f4b98154c38"
        }
      }
    }'

@juliaElastic juliaElastic mentioned this issue May 31, 2024
Merged
1 task
@cmacknz
Copy link
Member

cmacknz commented May 31, 2024

This needs to be done with some caution, an upgrade during the watching state is potentially destructive to the agent. The watcher is a separate process and will interpret the new upgrade as a crash.

The agent's upgrade watcher needs to be smarter than this, but today it isn't.

Do we know how this happened yet? I want us to avoid putting in potentially destructive work arounds for rare bugs that should just be fixed instead.

@cmacknz
Copy link
Member

cmacknz commented May 31, 2024 •
edited
Loading

upgrade agent with a fictive version e.g. 8.13.5
update the agent doc to set the upgrade_started_at to more than 2 hours ago

What sequence of user actions would cause this?

In between whatever those two steps are, it would be better if we told the agent we were cancelling the upgrade, or perhaps we need to have the agent interpret a request to upgrade to a new version during an upgrade to a different one an implicit cancellation.

@juliaElastic
Copy link
Contributor Author

This needs to be done with some caution, an upgrade during the watching state is potentially destructive to the agent. The watcher is a separate process and will interpret the new upgrade as a crash.

The agent's upgrade watcher needs to be smarter than this, but today it isn't.

Do we know how this happened yet? I want us to avoid putting in potentially destructive work arounds for rare bugs that should just be fixed instead.

I think this is a real UI bug, we don't' allow restart upgrade if the agent is in watching state, but not even when the agent is stuck updating. The fix includes a check before allowing restart upgrade that the agent is in UPG_FAILED state or started upgrade more than 2 hours ago.

juliaElastic added a commit that referenced this issue Jun 3, 2024
@juliaElastic
[Fleet] fix restart upgrade disabled condition (#184586)
1750ab8 
## Summary

Closes #184212

Fix the Restart upgrade modal submit button disabled condition. It was
not correct for single agent restart, the restart was disabled.

To verify:
- enroll an agent with an older version e.g. 8.13.1
- upgrade agent with a fictive version e.g. 8.13.5
- update the agent doc to set the `upgrade_started_at` to more than 2
hours ago
- go to `Restart upgrade`, the modal window show enable submit


<img width="818" alt="image"
src="https://github.com/elastic/kibana/assets/90178898/0edbbb1a-7a51-4287-8598-0d4437bbc874">


### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Jun 3, 2024
@juliaElastic
[Fleet] fix restart upgrade disabled condition (elastic#184586)
4c3d06a 
## Summary

Closes elastic#184212

Fix the Restart upgrade modal submit button disabled condition. It was
not correct for single agent restart, the restart was disabled.

To verify:
- enroll an agent with an older version e.g. 8.13.1
- upgrade agent with a fictive version e.g. 8.13.5
- update the agent doc to set the `upgrade_started_at` to more than 2
hours ago
- go to `Restart upgrade`, the modal window show enable submit

<img width="818" alt="image"
src="https://github.com/elastic/kibana/assets/90178898/0edbbb1a-7a51-4287-8598-0d4437bbc874">

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

(cherry picked from commit 1750ab8)
rohanxz pushed a commit to honeyn303/kibana that referenced this issue Jun 4, 2024
@juliaElastic @rohanxz
[Fleet] fix restart upgrade disabled condition (elastic#184586)
532d81e 
## Summary

Closes elastic#184212

Fix the Restart upgrade modal submit button disabled condition. It was
not correct for single agent restart, the restart was disabled.

To verify:
- enroll an agent with an older version e.g. 8.13.1
- upgrade agent with a fictive version e.g. 8.13.5
- update the agent doc to set the `upgrade_started_at` to more than 2
hours ago
- go to `Restart upgrade`, the modal window show enable submit


<img width="818" alt="image"
src="https://github.com/elastic/kibana/assets/90178898/0edbbb1a-7a51-4287-8598-0d4437bbc874">


### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
kibanamachine referenced this issue Jun 5, 2024
@kibanamachine @juliaElastic
[8.14] [Fleet] fix restart upgrade disabled condition (#184586) (#184656
6f22d69 
)

# Backport

This will backport the following commits from `main` to `8.14`:
- [[Fleet] fix restart upgrade disabled condition
(#184586)](#184586)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Julia
Bardi","email":"90178898+juliaElastic@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-06-03T13:53:06Z","message":"[Fleet]
fix restart upgrade disabled condition (#184586)\n\n##
Summary\r\n\r\nCloses
https://github.com/elastic/kibana/issues/184212\r\n\r\nFix the Restart
upgrade modal submit button disabled condition. It was\r\nnot correct
for single agent restart, the restart was disabled.\r\n\r\nTo
verify:\r\n- enroll an agent with an older version e.g. 8.13.1\r\n-
upgrade agent with a fictive version e.g. 8.13.5\r\n- update the agent
doc to set the `upgrade_started_at` to more than 2\r\nhours ago\r\n- go
to `Restart upgrade`, the modal window show enable
submit\r\n\r\n\r\n<img width=\"818\"
alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/90178898/0edbbb1a-7a51-4287-8598-0d4437bbc874\">\r\n\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"1750ab8676dadb3d6f2f3dbfe0ee9ea95de66733","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:prev-minor","v8.15.0"],"title":"[Fleet]
fix restart upgrade disabled
condition","number":184586,"url":"https://github.com/elastic/kibana/pull/184586","mergeCommit":{"message":"[Fleet]
fix restart upgrade disabled condition (#184586)\n\n##
Summary\r\n\r\nCloses
https://github.com/elastic/kibana/issues/184212\r\n\r\nFix the Restart
upgrade modal submit button disabled condition. It was\r\nnot correct
for single agent restart, the restart was disabled.\r\n\r\nTo
verify:\r\n- enroll an agent with an older version e.g. 8.13.1\r\n-
upgrade agent with a fictive version e.g. 8.13.5\r\n- update the agent
doc to set the `upgrade_started_at` to more than 2\r\nhours ago\r\n- go
to `Restart upgrade`, the modal window show enable
submit\r\n\r\n\r\n<img width=\"818\"
alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/90178898/0edbbb1a-7a51-4287-8598-0d4437bbc874\">\r\n\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"1750ab8676dadb3d6f2f3dbfe0ee9ea95de66733"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/184586","number":184586,"mergeCommit":{"message":"[Fleet]
fix restart upgrade disabled condition (#184586)\n\n##
Summary\r\n\r\nCloses
https://github.com/elastic/kibana/issues/184212\r\n\r\nFix the Restart
upgrade modal submit button disabled condition. It was\r\nnot correct
for single agent restart, the restart was disabled.\r\n\r\nTo
verify:\r\n- enroll an agent with an older version e.g. 8.13.1\r\n-
upgrade agent with a fictive version e.g. 8.13.5\r\n- update the agent
doc to set the `upgrade_started_at` to more than 2\r\nhours ago\r\n- go
to `Restart upgrade`, the modal window show enable
submit\r\n\r\n\r\n<img width=\"818\"
alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/90178898/0edbbb1a-7a51-4287-8598-0d4437bbc874\">\r\n\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"1750ab8676dadb3d6f2f3dbfe0ee9ea95de66733"}}]}]
BACKPORT-->

Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
viduni94 added a commit to viduni94/kibana that referenced this issue Oct 28, 2024
@viduni94
[Obs AI Assistant] Send function description within the system messag…
8bc5e6f 
…e due to token limitations in function description (elastic#184212)

Fix Conflicts in x-pack/plugins/observability_solution/observability_ai_assistant/server/functions/index.ts
viduni94 added a commit to viduni94/kibana that referenced this issue Oct 30, 2024
@viduni94
[Obs AI Assistant] Send function description within the system messag…
0ce486a 
…e due to token limitations in function description (elastic#184212)

Fix Conflicts in x-pack/plugins/observability_solution/observability_ai_assistant/server/functions/index.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@juliaElastic juliaElastic

Labels
bug Fixes for quality problems that affect the customer experience Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Fleet] fix restart upgrade disabled condition juliaElastic/kibana
3 participants
@cmacknz @elasticmachine @juliaElastic