Configuration Test Framework Prototype

[kevin85421]

Why are these changes needed?

• Design Doc

Currently, existing tests for operators only test for the default configuration. However, from the last 6 weeks (the design doc is created on Sep. 26, 2022), 11 out of 34 commits are used to fix bugs caused by configurations. Hence, this project decided to focus on configuration. This PR is a prototype that can reproduce every YAML-related configuration bug (in the 11 reported config bugs) by adding a simple JsonPatch.

if __name__ == '__main__':
    template_name = 'config/ray-cluster.mini.yaml.template'
    namespace = 'default'
    with open(template_name) as base_yaml:
        baseCR = yaml.load(base_yaml, Loader=yaml.FullLoader)

    patch_list = [
        jsonpatch.JsonPatch([{'op': 'replace', 'path': '/spec/headGroupSpec/template/spec/containers/0/name', 'value': 'ray-head-1'}]),
        jsonpatch.JsonPatch([{'op': 'replace', 'path': '/spec/headGroupSpec/template/spec/containers/0/name', 'value': 'ray-head-2'}]),
        # Reproduce #612
        jsonpatch.JsonPatch([{'op': 'replace', 'path': '/spec/headGroupSpec/replicas', 'value': 2}]),
        # Reproduce #587
        jsonpatch.JsonPatch([
            {'op': 'replace', 'path': '/spec/workerGroupSpecs/0/replicas', 'value': 2},
            {'op': 'add', 'path': '/spec/workerGroupSpecs/0/template/metadata/name', 'value': 'haha'}
            ]),
        # Reproduce #585
        jsonpatch.JsonPatch([{'op': 'add', 'path': '/spec/headGroupSpec/rayStartParams/object-manager-port', 'value': '12345'}]),
        # Reproduce #572 #530
        jsonpatch.JsonPatch([{'op': 'add', 'path': '/spec/headGroupSpec/template/metadata/labels/app.kubernetes.io~1name', 'value': 'ray'}]),
        # Reproduce #529
        jsonpatch.JsonPatch([
            {'op': 'replace', 'path': '/spec/headGroupSpec/template/spec/containers/0/resources/requests/memory', 'value': '256Mi'},
            {'op': 'replace', 'path': '/spec/headGroupSpec/template/spec/containers/0/resources/limits/memory', 'value': '512Mi'}
        ])
    ]

    rs = RuleSet([HeadPodNameRule(), EasyJobRule(), HeadSvcRule()])
    mut = Mutator(baseCR, patch_list)
    images = ['rayproject/ray:2.0.0', 'kuberay/operator:v0.3.0', 'kuberay/apiserver:v0.3.0']

    test_cases = unittest.TestSuite()
    for new_cr in mut.mutate():
        addEvent = RayClusterAddCREvent(new_cr, [rs], 90, namespace)
        test_cases.addTest(GeneralTestCase('runTest', images, addEvent))
    runner=unittest.TextTestRunner()
    runner.run(test_cases)

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[DmitriGekhtman]

Notes --
We should be able to use this framework to test deployment using configs other than CRs
(for example, Helm charts).

At some point, we may also wish to support testing configuration for multiple deployed objects.
For example, we may wish to test operator configuration changes.
Or we may wish to test deploying a RayCluster and a RayJob which carries a reference to that RayCluster.

[kevin85421]

[Discussion]: Which one is better, unittest or pytest?

[DmitriGekhtman]

[Discussion]: Which one is better, unittest or pytest?

Or both.
The Ray CI uses pytest to run tests written using unittest.
For example:
https://github.com/ray-project/ray/blob/3e7c207f02e7368e1245e2cfafd27cb0bf179ff7/python/ray/tests/test_autoscaler_yaml.py#L99
https://github.com/ray-project/ray/blob/3e7c207f02e7368e1245e2cfafd27cb0bf179ff7/python/ray/tests/test_autoscaler_yaml.py#L667

@simon-mo
Do you have thoughts on pytest vs unittest?

[simon-mo]

Pytest fixture helps modularize setup and tear down code. Generally considered more powerful. So most of the Ray codebase uses it except autoscaler and tune. Pytest is compatible with unittest anyway. In the end it's up to your taste.

[kevin85421]

TODO:

1. Refactor

def replace_patch(path, value):
    return jsonpatch(...)

2. Add some simple config tests in KubeRay CI

[DmitriGekhtman]

A few more suggestions/questions to see if we can simplify things a bit.

[DmitriGekhtman]

I think we're at a good place for this prototype!
Let's get this running in the KubeRay CI in a follow-up.
After that, we can write more tests using this framework.
Later, we can think about running the tests in the Ray buildkite CI, if it becomes necessary due to compute limitations.