Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch snapshots automation #53706

Merged
merged 38 commits into from
Jan 7, 2020

Conversation

brianseeders
Copy link
Contributor

Closes #52016

There's a couple of small TODOs still there, but this is finished up for the most part.

Need to make sure proper initial snapshots exist before merging this.

Jenkinsfiles:

  • trigger_build_es triggers ES build jobs for each tracked branch
  • build_es builds ES, and uploads the created artifacts to a GCS daily snapshot bucket along with a manifest file. Snapshots in this bucket are kept for 60 days. This job then triggers the verify job
  • verify_es runs most of the standard pipeline, but uses the ES snapshot created from the last job. If the tests succeed, the snapshot is "promoted" by copying its manifest file to the default location used by kbn-es for that ES version, and also copying all of the artifacts to the permanent snapshot bucket. If it fails, an e-mail will be sent out, and the last verified snapshot will continue to be used by kbn-es.

kbn-es:

  • release-manager snapshot functionality removed
  • by default, check the daily gcs snapshot bucket for the last verified snapshot manifest for the required version of ES
  • if the daily snapshot 404s because it expired, fall back to the permanent snapshot storage bucket (this gets us long-lived snapshot storage for old branches and automatic daily artifact cleanup without having to maintain a cleanup process)
  • once a manifest has been downloaded, find the artifact that matches the requested version/license/platform, and continue as normal

@brianseeders brianseeders added Team:Operations Team label for Operations Team v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:CI Continuous integration v7.6.0 labels Dec 20, 2019
Copy link
Contributor

@tylersmalley tylersmalley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - just need to change the build failure email back before merging.

@mistic
Copy link
Member

mistic commented Jan 6, 2020

@elasticmachine merge upstream

@brianseeders
Copy link
Contributor Author

@elasticmachine merge upstream

@brianseeders
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@brianseeders brianseeders merged commit 4b6c2f8 into elastic:master Jan 7, 2020
brianseeders added a commit to brianseeders/kibana that referenced this pull request Jan 7, 2020
brianseeders added a commit to brianseeders/kibana that referenced this pull request Jan 7, 2020
brianseeders added a commit to brianseeders/kibana that referenced this pull request Jan 7, 2020
brianseeders added a commit to brianseeders/kibana that referenced this pull request Jan 7, 2020
jloleysens added a commit to jloleysens/kibana that referenced this pull request Jan 7, 2020
…le-saved-objects

* 'master' of github.com:elastic/kibana: (86 commits)
  [Uptime] Added date range filter into expanded list query (elastic#52609)
  [SIEM] Add react/display-name eslint rule (elastic#53107)
  [SIEM] Enable eslint prefer-template rule (elastic#53983)
  Elasticsearch snapshots automation (elastic#53706)
  [SIEM] Enable eslint react/no-children-prop (elastic#53985)
  [DOCS][Reporting] Adds info about using a custom reporting index (elastic#54052)
  Changing background color to align with EUI color (elastic#54060)
  Fix linting issues (elastic#54068)
  NP Migration: Move doc views registry and existing doc views into discover plugin (elastic#53465)
  [ML] DF Analytics job creation: Add 'excludes' input field to form (elastic#53856)
  EMT-issue-65: add endpoint list api (elastic#53861)
  [SIEM] Fix doubled drag handles in Timeline (elastic#52679)
  Functional tests: refactor visualize_page (elastic#53845)
  [Dashboard] Redesign empty screen in readonly mode (elastic#54073)
  [ML] Support search for partitions on Single Metric Viewer (elastic#53879)
  update apm index pattern (elastic#54095)
  Add data ui for envoyproxy Metricbeat Module (elastic#53476)
  [ML] persist the brush when expanded to full width (elastic#54020)
  Skip failing test (elastic#54100)
  [APM] Show errors on the timeline instead of under the transaction (elastic#53756)
  ...

# Conflicts:
#	src/legacy/core_plugins/console/public/np_ready/application/contexts/services_context.tsx
#	src/legacy/core_plugins/console/public/np_ready/application/index.tsx
jloleysens added a commit to jloleysens/kibana that referenced this pull request Jan 7, 2020
…/kibana into feature/console-saved-objects

* 'feature/console-saved-objects' of github.com:jloleysens/kibana: (86 commits)
  [Uptime] Added date range filter into expanded list query (elastic#52609)
  [SIEM] Add react/display-name eslint rule (elastic#53107)
  [SIEM] Enable eslint prefer-template rule (elastic#53983)
  Elasticsearch snapshots automation (elastic#53706)
  [SIEM] Enable eslint react/no-children-prop (elastic#53985)
  [DOCS][Reporting] Adds info about using a custom reporting index (elastic#54052)
  Changing background color to align with EUI color (elastic#54060)
  Fix linting issues (elastic#54068)
  NP Migration: Move doc views registry and existing doc views into discover plugin (elastic#53465)
  [ML] DF Analytics job creation: Add 'excludes' input field to form (elastic#53856)
  EMT-issue-65: add endpoint list api (elastic#53861)
  [SIEM] Fix doubled drag handles in Timeline (elastic#52679)
  Functional tests: refactor visualize_page (elastic#53845)
  [Dashboard] Redesign empty screen in readonly mode (elastic#54073)
  [ML] Support search for partitions on Single Metric Viewer (elastic#53879)
  update apm index pattern (elastic#54095)
  Add data ui for envoyproxy Metricbeat Module (elastic#53476)
  [ML] persist the brush when expanded to full width (elastic#54020)
  Skip failing test (elastic#54100)
  [APM] Show errors on the timeline instead of under the transaction (elastic#53756)
  ...
gmmorris added a commit to gmmorris/kibana that referenced this pull request Jan 8, 2020
* master: (55 commits)
  [ui/public/utils] Copy rarely used items to where they are consumed (elastic#53819)
  set AppArch team as an owner of the search endpoints (elastic#54131)
  Don't expose Elasticsearch client as Observable (elastic#53824)
  [SIEM] Cleanup unnecessary use of enzyme-to-json (elastic#53980)
  fix ui exports doc (elastic#54138)
  change markdown element title (elastic#54194)
  [Logs UI] Refactor log position to hooks (elastic#53540)
  [SIEM] Implement NP Plugin Setup (elastic#54030)
  [DOCS] Updates ML links (elastic#53613)
  sort renovate packages in config
  Spaces - fix flakey api tests (elastic#54154)
  Remove dependency that was causing effect to re-execute infinitely. (elastic#54160)
  [dev/run] expose unexpected flags as more than just names (elastic#54080)
  [DOCS] Moves index pattern doc to Discover (elastic#53347)
  [SIEM] Cleanup React imports (elastic#53981)
  Update eslint related packages (elastic#54107)
  [Uptime] Added date range filter into expanded list query (elastic#52609)
  [SIEM] Add react/display-name eslint rule (elastic#53107)
  [SIEM] Enable eslint prefer-template rule (elastic#53983)
  Elasticsearch snapshots automation (elastic#53706)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:CI Continuous integration release_note:skip Skip the PR/issue when compiling release notes Team:Operations Team label for Operations Team v6.8.6 v7.4.3 v7.5.1 v7.6.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[kbn-es] Elasticsearch snapshot automation
6 participants