Add more rolling upgrade E2E tests

[sebgl]

We should improve E2E tests to cover the following PVC reuse scenarios:

• Add some data in the cluster and make sure it's still there once the upgrade is done (related: e2e tests: add support for data integrity tests #608)

• Test migrations from one node role to another: master nodes should not reuse data volumes and data nodes should not reuse master volumes.

• Test mutations with an update strategy that implies both PVC reuse and data migration. Make sure one does not prevent the other to go through.

• Test hot-warm topologies see ECK and ILM  #2077 for that

• Test minor modifications in the VolumeClaimTemplate spec we don't allow changes to the volume claim template spec since moving to StatefulSets

• Test a back-and-forth scenario: ask for a mutation, make sure it's initiated, then revert it

[pebrc]

Test migrations from one node role to another: master nodes should not reuse data volumes and data nodes should not reuse master volumes.

I think this test scenario is somewhat redundant with the move to stateful sets as we would be testing that stateful sets work correctly at this point.

[pebrc]

Test mutations with an update strategy that implies both PVC reuse and data migration. Make sure one does not prevent the other to go through.

cloud-on-k8s/operators/test/e2e/es/mutation_test.go

Lines 18 to 30 in 71de93f

	func TestMutationMdiToDedicated(t *testing.T) {
	// create a 1 md node cluster
	b := elasticsearch.NewBuilder("test-mutation-mdi-to-dedicated").
	WithESMasterDataNodes(1, elasticsearch.DefaultResources)
	// mutate to 1 m node + 1 d node
	mutated := b.
	WithNoESTopology().
	WithESDataNodes(1, elasticsearch.DefaultResources).
	WithESMasterNodes(1, elasticsearch.DefaultResources)
	test.RunMutation(t, b, mutated)
	}

does just that after #608 : it modifies the original sset to remove the data role which reuses the PV and it creates a new data-only sset to which the test data needs to be migrated.

[pebrc]

Test hot-warm topologies

@sebgl I wonder what you had in mind here. What does testing hot-warm mean specifically? I don't think it makes sense to test that Elasticsearch allocation requirements and ILM works.

Hot/warm/cold from an ECK perspective is just three data node sets with different volume claim templates. We can of course test that you can specify more than one data node set. But for example testing that these actually each get different volumes types assigned is quite hard I think.

What could be interesting is testing that ILM allocation requirements don't interfere with our use of allocation requirements on node removal.

[sebgl]

@sebgl I wonder what you had in mind here. What does testing hot-warm mean specifically? I don't think it makes sense to test that Elasticsearch allocation requirements and ILM works.

You're right. I did not have anything particular in mind, I think this was mostly in the list as something to think about when we work on the issue, in case there might be something worth testing I don't know about.

What could be interesting is testing that ILM allocation requirements don't interfere with our use of allocation requirements on node removal.

Indeed, sounds like a tricky bit.