Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RFC: Improve saved object migrations #66056

Merged
merged 38 commits into from
Sep 29, 2020
Merged

RFC: Improve saved object migrations #66056

merged 38 commits into from
Sep 29, 2020

Commits on May 12, 2020

  1. RFC: Improve saved object migrations

    @rudolf
    rudolf committed May 12, 2020
    Configuration menu
    Copy the full SHA
    089e0b0 View commit details
    Browse the repository at this point in the history

  2. Added rolling upgrades as alternative

    @rudolf
    rudolf committed May 12, 2020
    Configuration menu
    Copy the full SHA
    82280bc View commit details
    Browse the repository at this point in the history

Commits on May 15, 2020

  1. Clarify advantage of re-using index for avoiding max shards open errors

    @rudolf
    rudolf committed May 15, 2020
    Configuration menu
    Copy the full SHA
    39ec98c View commit details
    Browse the repository at this point in the history

  2. Add more detail about how dry run migrations and invalid document exp… 

    …orts will be exposed to users
    @rudolf
    rudolf committed May 15, 2020
    Configuration menu
    Copy the full SHA
    148e80f View commit details
    Browse the repository at this point in the history

  3. Add separate section to highlight assumptions and tradeoffs

    @rudolf
    rudolf committed May 15, 2020
    Configuration menu
    Copy the full SHA
    f88c4ba View commit details
    Browse the repository at this point in the history

Commits on May 19, 2020

  1. Apply nits from code review 

    Co-authored-by: Josh Dover <me@joshdover.com>
    @rudolf @joshdover
    rudolf and joshdover committed May 19, 2020
    Configuration menu
    Copy the full SHA
    9fa0ee3 View commit details
    Browse the repository at this point in the history

  2. Move 'Single node migrations coordinated through a lease/lock' under … 

    …alternatives
    @rudolf
    rudolf committed May 19, 2020
    Configuration menu
    Copy the full SHA
    cfd710b View commit details
    Browse the repository at this point in the history

  3. Fix 5.2.2 document lock algorithm step 3.2

    @rudolf
    rudolf committed May 19, 2020
    Configuration menu
    Copy the full SHA
    db021c4 View commit details
    Browse the repository at this point in the history

  4. Minor clarificiations

    @rudolf
    rudolf committed May 19, 2020
    Configuration menu
    Copy the full SHA
    39b980e View commit details
    Browse the repository at this point in the history

  5. Minimizing data loss with mixed Kibana versions: move 7.x algorithm i… 

    …nto alternatives and clarify 8.x algorithm
    @rudolf
    rudolf committed May 19, 2020
    Configuration menu
    Copy the full SHA
    116db68 View commit details
    Browse the repository at this point in the history

Commits on May 25, 2020

  1. Adds impact idempotent restrictions have on upcoming/requested migrat… 

    …ion features
    @rudolf
    rudolf committed May 25, 2020
    Configuration menu
    Copy the full SHA
    821e0d1 View commit details
    Browse the repository at this point in the history

Commits on May 27, 2020

  1. nit: clarify 4.3.1.2.3 

    Clarify that if documents already exist, they aren't overwritten.

Co-authored-by: Brandon Kobel <brandon.kobel@gmail.com>
    @rudolf @kobelb
    rudolf and kobelb committed May 27, 2020
    Configuration menu
    Copy the full SHA
    60a5949 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2020

  1. Update 4.5.2: we need only one version-specific alias, not separate r… 

    …ead/write aliases
    @rudolf
    rudolf committed May 28, 2020
    Configuration menu
    Copy the full SHA
    f28b066 View commit details
    Browse the repository at this point in the history

  2. Add 4.0.4 the assumption that a saved object type will only ever be r… 

    …egistered by one plugin
    @rudolf
    rudolf committed May 28, 2020
    Configuration menu
    Copy the full SHA
    94d2516 View commit details
    Browse the repository at this point in the history

  3. Updates 4.3.1.2: Adds mappings compatibility check, adds soft deletes… 

    … to prevent lost deletes, add note about data loss with mixed plugins on the same kibana version
    @rudolf
    rudolf committed May 28, 2020
    Configuration menu
    Copy the full SHA
    c8551dd View commit details
    Browse the repository at this point in the history

  4. Merge branch 'rfc-saved-object-migrations' of https://github.com/rudo… 

    …lf/kibana into rfc-saved-object-migrations
    @rudolf
    rudolf committed May 28, 2020
    Configuration menu
    Copy the full SHA
    832e839 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2020

  1. Don't try to minimize data loss for unsupported configurations, updat… 

    …e index mappings during 8.x migration

 - Move "4.5.2 Minimizing data loss with mixed Kibana versions (9.0)" to
 alternatives section. Expand this section with a more detailed algorithm as
 well as enumerating some data loss scenarios that could still occur.
 - Add the supported upgrade procedure which guarantees no undetected data loss.
 - Update "4.5.1 Migration algorithm (8.0)"
   - Added a step to update index mappings
   - If there's update version mismatch errors, complete all batches before
     restarting the migration to improve performance.
    @rudolf
    rudolf committed May 29, 2020
    Configuration menu
    Copy the full SHA
    2480909 View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2020

  1. Flesh out upgrade and rollback procedure with gracefull shutdown

    @rudolf
    rudolf committed Jun 9, 2020
    Configuration menu
    Copy the full SHA
    7cbe768 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2020

  1. Merge branch 'master' into rfc-saved-object-migrations

    @rudolf
    rudolf committed Jun 12, 2020
    Configuration menu
    Copy the full SHA
    8727754 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2020

  1. New migration algorithm, upgrade procedure based on index cloning and… 

    … new ES features
    @rudolf
    rudolf committed Jun 17, 2020
    Configuration menu
    Copy the full SHA
    e36ea17 View commit details
    Browse the repository at this point in the history

  2. Minor: clarify that index cloning algorithm solves (2.6) but (2.7) is… 

    … out of scope
    @rudolf
    rudolf committed Jun 17, 2020
    Configuration menu
    Copy the full SHA
    7239124 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2020

  1. Merge branch 'master' into rfc-saved-object-migrations

    @rudolf
    rudolf committed Jun 23, 2020
    Configuration menu
    Copy the full SHA
    8281e20 View commit details
    Browse the repository at this point in the history

  2. Move 'Tag objects as invalid if their transformation fails' to the al… 

    …ternatives section
    @rudolf
    rudolf committed Jun 23, 2020
    Configuration menu
    Copy the full SHA
    e5992f4 View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2020

  1. Don't migrate when there's orphan documents so that rollback is alway… 

    …s possible. Updates: 4.2.1.2 Migration algorithm: Cloned index per version & 4.2.1.4 Handling documents that belong to a disabled plugin
    @rudolf
    rudolf committed Jun 24, 2020
    Configuration menu
    Copy the full SHA
    823a51b View commit details
    Browse the repository at this point in the history

  2. Link to new Elasticsearch features required for the implementation

    @rudolf
    rudolf committed Jun 24, 2020
    Configuration menu
    Copy the full SHA
    469eab3 View commit details
    Browse the repository at this point in the history

  3. 4.2.1.2 Clarify the purpose of the target index postfix

    @rudolf
    rudolf committed Jun 24, 2020
    Configuration menu
    Copy the full SHA
    13c33f2 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2020

  1. Add open questions around mappings growing over time. Fix rollback in… 

    …dex example
    @rudolf
    rudolf committed Jun 26, 2020
    Configuration menu
    Copy the full SHA
    eb3f3ce View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2020

  1. Merge branch 'master' into rfc-saved-object-migrations

    @rudolf
    rudolf committed Jun 30, 2020
    Configuration menu
    Copy the full SHA
    1ce6759 View commit details
    Browse the repository at this point in the history

  2. minor: fix formatting

    @rudolf
    rudolf committed Jun 30, 2020
    Configuration menu
    Copy the full SHA
    c576b01 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2020

  1. Merge branch 'master' into rfc-saved-object-migrations

    @rudolf
    rudolf committed Jul 19, 2020
    Configuration menu
    Copy the full SHA
    d02cebc View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2020

  1. Update assumptions and tradeoffs to match latest algorithm

    @rudolf
    rudolf committed Jul 24, 2020
    Configuration menu
    Copy the full SHA
    a45ffd4 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2020

  1. Review feedback 

     - Remove 4.0.2 since the latest algorithm resolves the tradeoff
 - 4.2.1.2 should handle v6 `.kibana` indexes
    @rudolf
    rudolf committed Sep 14, 2020
    Configuration menu
    Copy the full SHA
    1863ee1 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into rfc-saved-object-migrations

    @rudolf
    rudolf committed Sep 14, 2020
    Configuration menu
    Copy the full SHA
    f35583a View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2020

  1. More alternatives for 4.2.1.4 Handling documents that belong to a dis… 

    …abled plugin. Don't introduce breaking changes in 7.x
    @rudolf
    rudolf committed Sep 15, 2020
    Configuration menu
    Copy the full SHA
    63db2d0 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2020

  1. Merge branch 'master' into rfc-saved-object-migrations

    @rudolf
    rudolf committed Sep 21, 2020
    Configuration menu
    Copy the full SHA
    7ab70fc View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2020

  1. Merge branch 'master' into rfc-saved-object-migrations

    @rudolf
    rudolf committed Sep 23, 2020
    Configuration menu
    Copy the full SHA
    81b1ead View commit details
    Browse the repository at this point in the history

  2. Handle new ES cluster without any indices. Fix clone API parameters

    @rudolf
    rudolf committed Sep 23, 2020
    Configuration menu
    Copy the full SHA
    c4e6b32 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2020

  1. Update 4.2.1.2: transform outdated documents on every startup, only f… 

    …ail for unknown types in 8.x
    @rudolf
    rudolf committed Sep 24, 2020
    Configuration menu
    Copy the full SHA
    8f10cf2 View commit details
    Browse the repository at this point in the history