Flaky test: Test_CLI_Delete - Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources] #6872
Comments
vinayada1
added
bug
|
👋 @vinayada1 Thanks for filing this bug report. A project maintainer will review this report and get back to you soon. If you'd like immediate help troubleshooting, please visit our Discord server. For more information on our triage process please visit our triage overview |
vinayada1
added
the
test-failure
|
In the artifacts, there are no containera/containerb logs. In the passed run, these logs exist. This test uses the same container names and deploys them multiple times. In the failed logs, I do not see two deployments of containerb as expected from the test steps. In the passed run, we see multiple operation IDs. In some cases, I see two instances of application-rp logs with different pod names in the failed logs but there are no errors which indicate a failure that caused a restart We also only see the log: |
|
Failed logs: shared_container_logs.zip |
|
Logs with no test failures (Added for comparison with the failure case): shared_container_logs (1).zip |
vinayada1
changed the title
…ete (#6881) # Description Changes to debug test failure issue: #6872 Used a different app name and namespace for the failing test instead of reusing from a previous one in the file Added more logs ## Type of change <!-- Please select **one** of the following options that describes your change and delete the others. Clearly identifying the type of change you are making will help us review your PR faster, and is used in authoring release notes. If you are making a bug fix or functionality change to Radius and do not have an associated issue link please create one now. --> - This pull request is a minor refactor, code cleanup, test improvement, or other maintenance task and doesn't change the functionality of Radius (issue link optional). <!-- Please update the following to link the associated issue. This is required for some kinds of changes (see above). --> Fixes: #issue_number ## Auto-generated summary <!-- GitHub Copilot for docs will auto-generate a summary of the PR --> <!-- copilot:all --> ### <samp>🤖[[deprecated]](https://githubnext.com/copilot-for-prs-sunset) Generated by Copilot at fb725ff</samp> ### Summary 🗑️🐳🧪 <!-- 1. 🗑️ - This emoji represents the deletion of an application and its associated resources, as well as the improved output message for this action. 2. 🐳 - This emoji represents the addition of a new bicep template file that defines an application with two containers, which are a common resource type for Kubernetes applications. 3. 🧪 - This emoji represents the extension of the test function to cover the new scenario of deleting an application with unassociated resources, as well as the added logging to the validation functions. --> This pull request adds a test case and a test data file for deleting an application with unassociated resources using the CLI. It also improves the output message for deleting an application and adds logging to the validation functions. > _Sing, O Muse, of the valiant code reviewers who scrutinized the pull request_ > _And of the skillful coder who improved the CLI with his diligent quest_ > _To make the output message clearer when an application is deleted by name_ > _And to test the deletion of unassociated resources, those containers of fame_ ### Walkthrough * Change the output message for deleting an application to include the application name ([link](https://github.com/radius-project/radius/pull/6881/files?diff=unified&w=0#diff-87907dd86f66c5e0992fef2ea8d8bc09cf79e4cf4e6d9108cb137a536da03c10L166-R166)) * Add a new bicep template file `corerp-kubernetes-cli-with-unassociated-resources.bicep` that defines an application with two unassociated resources ([link](https://github.com/radius-project/radius/pull/6881/files?diff=unified&w=0#diff-9cf9c8faf8c86483d2b71a394c55ab338f6a1fa2d9cc536f8af502718436cd36R1-R40)) - Declare new variables for the application name and the template file path of the application with unassociated resources ([link](https://github.com/radius-project/radius/pull/6881/files?diff=unified&w=0#diff-9f50fd192c3de42201f2c910d3f4ba78ec369dd7c50ea61ba500fa7518f4e81dR529), [link](https://github.com/radius-project/radius/pull/6881/files?diff=unified&w=0#diff-9f50fd192c3de42201f2c910d3f4ba78ec369dd7c50ea61ba500fa7518f4e81dR538-R540)) - Deploy the application with unassociated resources and validate that the pods are running ([link](https://github.com/radius-project/radius/pull/6881/files?diff=unified&w=0#diff-9f50fd192c3de42201f2c910d3f4ba78ec369dd7c50ea61ba500fa7518f4e81dL579-R590)) - Delete one of the unassociated resources using the management client ([link](https://github.com/radius-project/radius/pull/6881/files?diff=unified&w=0#diff-9f50fd192c3de42201f2c910d3f4ba78ec369dd7c50ea61ba500fa7518f4e81dL592-R602)) - Delete the application without deleting the remaining resources using the `DeleteAppWithoutDeletingResources` helper function ([link](https://github.com/radius-project/radius/pull/6881/files?diff=unified&w=0#diff-9f50fd192c3de42201f2c910d3f4ba78ec369dd7c50ea61ba500fa7518f4e81dL592-R602)) - Redeploy the application using an empty template file ([link](https://github.com/radius-project/radius/pull/6881/files?diff=unified&w=0#diff-9f50fd192c3de42201f2c910d3f4ba78ec369dd7c50ea61ba500fa7518f4e81dL592-R602)) - Delete the other unassociated resource using the management client ([link](https://github.com/radius-project/radius/pull/6881/files?diff=unified&w=0#diff-9f50fd192c3de42201f2c910d3f4ba78ec369dd7c50ea61ba500fa7518f4e81dL605-R610)) * Modify the validation functions `ValidateObjectsRunning` and `matchesActualLabels` to log the number and labels of the deployed resources for debugging purposes ([link](https://github.com/radius-project/radius/pull/6881/files?diff=unified&w=0#diff-a52153bfeb452a3dd497767a98db0dc1a0621e012f6e6d8dd9d4b5ec758b4dc2L343-R345), [link](https://github.com/radius-project/radius/pull/6881/files?diff=unified&w=0#diff-a52153bfeb452a3dd497767a98db0dc1a0621e012f6e6d8dd9d4b5ec758b4dc2L535-R535), [link](https://github.com/radius-project/radius/pull/6881/files?diff=unified&w=0#diff-a52153bfeb452a3dd497767a98db0dc1a0621e012f6e6d8dd9d4b5ec758b4dc2R555-R556)) Signed-off-by: vinayada1 <28875764+vinayada1@users.noreply.github.com> Co-authored-by: Nithya Subramanian <98416062+nithyatsu@users.noreply.github.com>
|
Checked in #6881 to see better info if the test fails again |
|
👍 We've reviewed this issue and have agreed to add it to our backlog. Please subscribe to this issue for notifications, we'll provide updates when we pick it up. We also welcome community contributions! If you would like to pick this item up sooner and submit a pull request, please visit our contribution guidelines and assign this to yourself by commenting "/assign" on this issue. For more information on our triage process please visit our triage overview |
|
@vinayada1 , please close the issue once the failure is resolved. cc @lakshmimsft |
|
Haven't seen this failure since #6881. |
Bug information
This is from a failed scheduled test run:-
2023-11-28T17:17:32.9227568Z === �[31mFAIL�[0m: test/functional/shared/cli Test_CLI_Delete (unknown)
2023-11-28T17:17:32.9228777Z test.go:43: Using container registry: ghcr.io/radius-project/dev - set DOCKER_REGISTRY to override
2023-11-28T17:17:32.9229852Z test.go:44: Using container tag: pr-13bf2a8783 - set REL_VERSION to override
2023-11-28T17:17:32.9230857Z test.go:45: Using magpie image: ghcr.io/radius-project/dev/magpiego:pr-13bf2a8783
2023-11-28T17:17:32.9232113Z test.go:49: Using recipe registry: ghcr.io/radius-project/dev - set BICEP_RECIPE_REGISTRY to override
2023-11-28T17:17:32.9233239Z test.go:50: Using recipe tag: pr-13bf2a8783 - set BICEP_RECIPE_TAG_VERSION to override
2023-11-28T17:17:32.9234976Z test.go:53: Using terraform recipe module server URL: http://tf-module-server.radius-test-tf-module-server.svc.cluster.local - set TF_RECIPE_MODULE_SERVER_URL to override
2023-11-28T17:17:32.9236662Z test.go:64: Loaded workspace: kind-radius (Kubernetes (context=kind-radius))
2023-11-28T17:17:32.9240478Z cli.go:471: [heartbeat] command rad deploy /home/runner/work/radius/radius/test/functional/shared/cli/testdata/corerp-kubernetes-cli-with-resources.bicep --application kubernetes-cli-with-resources --parameters magpieimage=ghcr.io/radius-project/dev/magpiego:pr-13bf2a8783 is still running after 10.003208787s
2023-11-28T17:17:32.9243433Z cli.go:418: [rad] Building /home/runner/work/radius/radius/test/functional/shared/cli/testdata/corerp-kubernetes-cli-with-resources.bicep...
2023-11-28T17:17:32.9245897Z cli.go:418: [rad] Deploying template '/home/runner/work/radius/radius/test/functional/shared/cli/testdata/corerp-kubernetes-cli-with-resources.bicep' for application 'kubernetes-cli-with-resources' and environment 'kind-radius' from workspace 'kind-radius'...
2023-11-28T17:17:32.9247399Z cli.go:418: [rad]
2023-11-28T17:17:32.9247811Z cli.go:418: [rad] Deployment In Progress...
2023-11-28T17:17:32.9248249Z cli.go:418: [rad]
2023-11-28T17:17:32.9248572Z cli.go:418: [rad]
2023-11-28T17:17:32.9248935Z cli.go:418: [rad] Deployment Complete
2023-11-28T17:17:32.9249335Z cli.go:418: [rad]
2023-11-28T17:17:32.9249672Z cli.go:418: [rad] Resources:
2023-11-28T17:17:32.9250360Z cli.go:418: [rad] kubernetes-cli-with-resources Applications.Core/applications
2023-11-28T17:17:32.9251254Z cli.go:418: [rad] containera-app-with-resources Applications.Core/containers
2023-11-28T17:17:32.9252128Z cli.go:418: [rad] containerb-app-with-resources Applications.Core/containers
2023-11-28T17:17:32.9252703Z cli.go:418: [rad]
2023-11-28T17:17:32.9253318Z 2023/11/28 16:19:05 validating objects in namespace default-kubernetes-cli-with-resources
2023-11-28T17:17:32.9253996Z cli.go:418: [rad] Application deleted
2023-11-28T17:17:32.9254394Z cli.go:418: [rad]
2023-11-28T17:17:32.9256110Z cli.go:471: [heartbeat] command rad deploy /home/runner/work/radius/radius/test/functional/shared/cli/testdata/corerp-kubernetes-cli-app-empty-resources.bicep --application kubernetes-cli-empty-resources is still running after 10.000189126s
2023-11-28T17:17:32.9258356Z cli.go:418: [rad] Building /home/runner/work/radius/radius/test/functional/shared/cli/testdata/corerp-kubernetes-cli-app-empty-resources.bicep...
2023-11-28T17:17:32.9261040Z cli.go:418: [rad] Deploying template '/home/runner/work/radius/radius/test/functional/shared/cli/testdata/corerp-kubernetes-cli-app-empty-resources.bicep' for application 'kubernetes-cli-empty-resources' and environment 'kind-radius' from workspace 'kind-radius'...
2023-11-28T17:17:32.9262703Z cli.go:418: [rad]
2023-11-28T17:17:32.9263116Z cli.go:418: [rad] Deployment In Progress...
2023-11-28T17:17:32.9263555Z cli.go:418: [rad]
2023-11-28T17:17:32.9263877Z cli.go:418: [rad]
2023-11-28T17:17:32.9264241Z cli.go:418: [rad] Deployment Complete
2023-11-28T17:17:32.9264644Z cli.go:418: [rad]
2023-11-28T17:17:32.9264979Z cli.go:418: [rad] Resources:
2023-11-28T17:17:32.9265678Z cli.go:418: [rad] kubernetes-cli-empty-resources Applications.Core/applications
2023-11-28T17:17:32.9266262Z cli.go:418: [rad]
2023-11-28T17:17:32.9266633Z cli.go:418: [rad] Application deleted
2023-11-28T17:17:32.9267030Z cli.go:418: [rad]
2023-11-28T17:17:32.9267812Z cli.go:418: [rad] Application 'kubernetes-cli-with-resources' does not exist or has already been deleted.
2023-11-28T17:17:32.9268501Z cli.go:418: [rad]
2023-11-28T17:17:32.9270592Z cli.go:471: [heartbeat] command rad deploy /home/runner/work/radius/radius/test/functional/shared/cli/testdata/corerp-kubernetes-cli-with-resources.bicep --application kubernetes-cli-with-resources --parameters magpieimage=ghcr.io/radius-project/dev/magpiego:pr-13bf2a8783 is still running after 10.001016097s
2023-11-28T17:17:32.9273056Z cli.go:418: [rad] Building /home/runner/work/radius/radius/test/functional/shared/cli/testdata/corerp-kubernetes-cli-with-resources.bicep...
2023-11-28T17:17:32.9275454Z cli.go:418: [rad] Deploying template '/home/runner/work/radius/radius/test/functional/shared/cli/testdata/corerp-kubernetes-cli-with-resources.bicep' for application 'kubernetes-cli-with-resources' and environment 'kind-radius' from workspace 'kind-radius'...
2023-11-28T17:17:32.9276930Z cli.go:418: [rad]
2023-11-28T17:17:32.9277342Z cli.go:418: [rad] Deployment In Progress...
2023-11-28T17:17:32.9277773Z cli.go:418: [rad]
2023-11-28T17:17:32.9278100Z cli.go:418: [rad]
2023-11-28T17:17:32.9278467Z cli.go:418: [rad] Deployment Complete
2023-11-28T17:17:32.9278872Z cli.go:418: [rad]
2023-11-28T17:17:32.9279215Z cli.go:418: [rad] Resources:
2023-11-28T17:17:32.9279905Z cli.go:418: [rad] kubernetes-cli-with-resources Applications.Core/applications
2023-11-28T17:17:32.9280796Z cli.go:418: [rad] containera-app-with-resources Applications.Core/containers
2023-11-28T17:17:32.9282444Z cli.go:418: [rad] containerb-app-with-resources Applications.Core/containers
2023-11-28T17:17:32.9283212Z cli.go:418: [rad]
2023-11-28T17:17:32.9283849Z 2023/11/28 16:19:46 validating objects in namespace default-kubernetes-cli-with-resources
2023-11-28T17:17:32.9285235Z 2023/11/28 16:19:49 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9287003Z 2023/11/28 16:19:52 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9288802Z 2023/11/28 16:19:55 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9290538Z 2023/11/28 16:19:58 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9292257Z 2023/11/28 16:20:01 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9293981Z 2023/11/28 16:20:04 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9295878Z 2023/11/28 16:20:07 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9297744Z 2023/11/28 16:20:10 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9299462Z 2023/11/28 16:20:13 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9301388Z 2023/11/28 16:20:16 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9303128Z 2023/11/28 16:20:19 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9304890Z 2023/11/28 16:20:22 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9306637Z 2023/11/28 16:20:25 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9308635Z 2023/11/28 16:20:28 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9310608Z 2023/11/28 16:20:31 Failed to validate resource of type pods with labels map[radapp.io/application:kubernetes-cli-with-resources radapp.io/resource:containerb-app-with-resources]
2023-11-28T17:17:32.9312895Z 2023/11/28 16:20:34 Failed to validate resource of type pods with labels map[radapp.io/applicatio
AB#10578
The text was updated successfully, but these errors were encountered: