Upgrade process when data is removed (v4 upgrade)

[axelboc]

Discussion started in #306. The problem with re-importing a deck with fewer notes/cards is summarised in #306 (comment).

The goal is to find an upgrade process for users who may want to keep the notes/cards that are removed from the deck. The process would have to impact "normal" users as little as possible.

One approach, based on releasing a major version, was explained in #306 (comment). Obviously, this impacts "normal" users significantly, so it's far from ideal.

[aplaice]

I agree with @ohare93 and @ukanuk that in principle, a better deck management system would probably be ideal. (Sorry, I still haven't had time to look at Brain Brew!)

---

That said, I don't think that using the currently available tools are insufficient for a "good enough" outcome.

Categories of notes

To summarise, we have three categories of notes:

I. Those which will be kept, with all their current fields. (The bulk)

II. Those which will be kept, but with the contents of some of their fields removed (the "Capital" and "Flag" fields), hence causing some of their cards to become empty and probably, eventually deleted. The cards:

• II.a. The cards that will remain (Map→Country and Country→Map (extended deck))

• II.b. The cards that will be functionally removed from AUG (Country⇔Capital, Country⇔Flag).

III. Those which will be completely removed.

---

Types of users

To slightly modify/re-phrase @axelboc's categorisation of scenarios:

1: New user or upgrading user who doesn't mind losing their existing progress and any now-removed cards. Wants to follow AUG in choice of cards (they don't want the now-removed cards).

They want to only have the cards from I. and II.a..

This group must be able to just import the new deck (possibly deleting their old copy first), without having to fiddle with anything.

2: Upgrading user who'd prefer not to lose progress if possible, but wants to follow AUG in choice of cards.

They want to only have I. and II.a. and lose as little progress as possible.

3: Upgrading user who'd prefer not to lose progress if possible, and doesn't want to lose any now-removed cards. Wants to stay "inter-operable" with AUG.

They want to have all of I., II.a., II.b., III..

(I'm not sure that group 3 is much smaller than 2, if at all.)

---

Approaches

To partially repeat from the previous thread the possible approaches we can have:

A. (Most basic.) Simply remove all the rows corresponding to III. and remove the Capital and Flag fields from II..

• group 1 will now have I. and II.a., as desired.

• groups 2 and 3 will have I., II.a. and III., with progress. i.e.

  • 2 keeps III. despite not wanting it.

  • 3 loses II.b. despite wanting it.

B. (@axelboc's suggestion.)

1. Create a new, temporary deck with rows for III. (all fields) and II. (all fields?).

2. In the main deck, for II., change the guid and remove the Capital and Flag fields.

3. Tell group 1 to ignore the temporary deck and just import the main deck.

4. Tell group 2 to import the temporary deck (just before upgrading the main deck) and delete it.

5. Tell group 3 to import the temporary deck (just before upgrading the main deck) and keep it.

Results:

• group 1 have I. and II.a. as desired.

• group 2 have I. (with progress) and II.a. (without progress).

• group 3 have I., II.a., II.b. and III. (with progress), but with II.a. duplicated (existing in both the "temporary" and the "main" decks; the main deck version can be suspended, though it can't be deleted without losing "interoperability" with AUG).

C. Just like B. but with the first step slightly changed:

1. Create a new, temporary deck with rows for III. (all fields) and II. (complement of the removed fields — i.e. have the Capital and Flag fields (and obviously Country etc.), but not the Map fields). (The temporary deck contains only III. and II.b., but not II.a..)

2.-5. stay the same.

Results:

• group 1 have I. and II.a. as desired.

• group 2 have I. (with progress) and II.a. (without progress).

• group 3 have I., II.b. and III. (with progress), II.a. (without progress). (No duplicates.)

D. Just like B. but with the both first and the second steps modified:

1. Create a new, temporary deck with rows for III. (all fields) and II. (complement of the removed fields — i.e. have the Capital and Flag fields (and obviously Country etc.), but not the Map fields). (The temporary deck contains only III. and II.b., but not II.a..) For II. change the GUIDs!

2. In the main deck, for II. remove the Capital and Flag fields, but do not change the GUIDs.

3.-5. stay the same.

Results:

• group 1 have I. and II.a. as desired.

• group 2 have I. and II.a. (with progress), as desired.

• group 3 have I., II.a. and III. (with progress), II.b. (without progress). (No duplicates.)

E. @ohare93's solution

Like A. but encourage users in 3 to install an add-on and manually copy the notes in II..

Results:

• group 1 have I. and II.a., as desired.

• group 2 have I., II.a. and III. (with progress), despite not wanting III..

• group 3 have I., II.a., II.b. and III. (with progress) and a duplicate of II.a..

F. Just like E. but with temporary decks.

1. We create a temporary, helper deck containing II.b. — i.e. the notes II. with the complement of the removed fields (i.e. Flag and Capital fields), with the same GUIDs.

2. We instruct users in 3 to

   a. Import the temporary deck.

   b. Install the Copy Notes addon.

   c. Copy all the notes in the temporary deck.

   d. Import the new "main" deck.

   e. Remove empty cards (only now — not earlier!).

   The users then have full progress for all their cards, and are also "interoperable" with AUG.

3. We also create a second temporary deck containing III. and encourage users in 2 to import and delete it, before upgrade.

It's the best — everybody achieves their desired outcome, but it's also the most complicated (two different temporary decks) and involved (installing an extra add-on, manually copying cards etc.).

---

Summary of the different approaches compared to the ideal, in a table (as an image since GFM doesn't allow coloured cells):

FWIW, none of the approaches adds any overhead to new/"fresh" users or negatively affects them in any way (other than possibly confusing them).

Some of the approaches add overhead to upgrading-with-progress-but-wanting-to-stick-to-aug-card-selection users (2), but only for their benefit. (They can ignore the overhead, but then they end up keeping all the tiny dependent territories, that we've fully removed.)

Overall, I think that D is the best solution, since it gives the optimal result for groups 1 and 2 and a good-enough result for 3, while remaining relatively simple.

F can exist for that subset of group 3 who is desperate about preserving all progress, but I think that it shouldn't be actively advertised to avoid confusing normal people (all of 1 and 2 and most of 3), with the addition of even more temporary decks...

[axelboc]

Awesome overview, thank you <3

What would be the impact of future upgrades of the deck for each solution, do you think?

[aplaice]

What types future upgrades should we be wary of? Possible scenarios, that I can think of:

i Adding a new "descriptive" field. (One that doesn't result in an extra card, just enhances an existing one — e.g. pronunciation, "flag info" etc.)

ii Adding a "generative" field. (One that does result in an extra field — e.g. currency.)

iii More extensive editing of templates, to use the fields differently (not really clear???).

iv "Non-local" fields (fields whose contents is determined by the contents of other notes — e.g. "Capital hint").

• This will be an issue immediately for most or all of the solutions — see Jamaica/Norfolk Island or Guyana/Cayman Islands.)

• It might need to be somehow addressed even for people who do a fresh import, since they'll still remember that Kingston is the capital of both Jamaica and Norfolk island, and won't immediately know that Norfolk Island is no longer a valid answer.

v Issues when the user exports the deck for collaboration. (As you pointed out in one of the threads, for AUG this is a relatively rare occurrence.)

vi Re-adding the currently removed fields and notes.

I'll think about how these (and any others you or others might think of) might work for the different approaches.

[axelboc]

Sorry I should have been clearer, I thought in some scenarios, the notes in the temporary deck might move back to the UG deck on subsequent imports of the UG deck. But I missed the part in D. where you mention changing the GUIDs of category-II. cards ... so my concern was unfounded.

I think D. is a great solution! Of course it still means that normal users would have to perform a clean import of the deck. That being said, importing and deleting the temporary deck might actually be easier than performing a clean import, so perhaps most users will choose that option.

[aplaice]

Phew, in that case I won't carry out an analysis of the effects of the different approaches on such future issues. :) (FWIW I don't think that the approaches will differ significantly in how they're affected.)

Regarding "Capital hints", in the cases where one country/territory from a capital hint pair is removed, while the other is not, (e.g. Jamaica and Norfolk island) it might be worth keeping the existing "Capital hint" for one more minor version, for the benefit of those who are upgrading, (especially if they're in group 2 but possibly also 1). It would give them time to get used to the fact that the removed country is no longer a valid answer (simply by the lack of its occurrence in the deck) and hence they wouldn't don't try to use it as the answer to the relevant Capital->Country card. I don't think that such a "dangling" temporary capital hint would be very harmful to new users. OTOH a clean break might be simpler and more comprehensible to users.

---

Of course it still means that normal users would have to perform a clean import of the deck.

Yes, with the current tooling, there's no going around the fact that a simple upgrade import wouldn't suffice to lose the fully removed notes,

[axelboc]

I'm still not set on what to do with the removed notes. The plan as discussed so far is for me to generate a temp deck with the removed notes, to attach the ZIPs of all the variants of the temp deck (standard/extended/translated) to the v4 release notes, and to deleted the removed folder from the repo.

Now, @ohare93, if you're keen, you could create a repo under your GitHub account and move the notes there -- even write some release notes so I can just point people to them from the v4 release notes? Up to you. I just don't want to create a repo myself.

Either way, as I've mentioned before, the notes will never be truly gone thanks to git, so we can always revive them into an extension deck when the time comes (for people not in the loop, see ohare93/brain-brew#4).

[axelboc]

And... done, the pre-release notes are out!! 🎉

Please don't hesitate to suggest changes if I've missed anything or something isn't here. Even without the inclusion rules, it's still quite a significant release, so high five! ✋