[BUG] pod_memory_hog spec test failing in github actions when PR merged to main

[agentpoyo]

Describe the bug
pod_memory_hog continues to fail in github actions:

I, [2024-04-15 16:45:39 +00:00 #10118]  INFO -- cnf-testsuite: upsert_task: task: pod_memory_hog has status: failed and is awarded: 0 points. Runtime: 96 seconds
I, [2024-04-15 16:45:39 +00:00 #10118]  INFO -- cnf-testsuite: results yaml: {"name" => "cnf testsuite", "testsuite_version" => "main-2024-04-15-164445-5a8da5dc", "status" => nil, "command" => "/home/runner/work/testsuite/testsuite/cnf-testsuite pod_memory_hog verbose", "points" => nil, "exit_code" => 0, "items" => [{"name" => "pod_memory_hog", "status" => "failed", "type" => "normal", "points" => 0}]}


  'pod_memory_hog' A 'Good' CNF should not crash when pod memory hog occurs

Failures:

  1) Resilience pod memory hog Chaos 'pod_memory_hog' A 'Good' CNF should not crash when pod memory hog occurs
     Failure/Error: (/PASSED: pod_memory_hog chaos test passed/ =~ response_s).should_not be_nil

       Expected: nil not to be nil

Error:      # spec/workload/resilience/pod_memory_hog_spec.cr:22

Finished in 2:21 minutes
1 examples, 1 failures, 0 errors, 0 pending

Failed examples:

crystal spec spec/workload/resilience/pod_memory_hog_spec.cr:15 # Resilience pod memory hog Chaos 'pod_memory_hog' A 'Good' CNF should not crash when pod memory hog occurs
Error: Process completed with exit code 1.

To Reproduce
Merges to main reproduce the errors.

Expected behavior
Should pass and not fail.

[taylor]

There is a Litmus Chaos error happening. Before trying to debug we are upgrading Litmus which requires some updates in code integration to the tools and litmus test cases.

This issue is blocking #1949

[taylor]

Need a new issue to handle constraints in pods, dynamically reading the constraint and telling Litmus to not use more than about 95% of the constraint.

If it uses 100% it will cause kubernetes to kill the pod which causes a failure. That is a different type of test.

[taylor]

Spec bug can be fixed with hardcoding to lower than 80% of coredns. This is not ideal though

[kosstennbl]

Issue seems to be fixed with #1987.

Continuation and further improvement of "pod_memory_hog" test is described in #1989