Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestFunctional/parallel/ServiceCmd flake: GET http://172.17.0.3:30602 giving up after 5 attempts #8013

Closed
priyawadhwa opened this issue May 5, 2020 · 11 comments
Closed

TestFunctional/parallel/ServiceCmd flake: GET http://172.17.0.3:30602 giving up after 5 attempts #8013

priyawadhwa opened this issue May 5, 2020 · 11 comments
Labels
kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. kind/flake Categorizes issue or PR as related to a flaky test. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@priyawadhwa
Copy link

priyawadhwa commented May 5, 2020 •
edited
Loading 

--- FAIL: TestFunctional/parallel/ServiceCmd (53.31s)
functional_test.go:642: (dbg) Run:  kubectl --context functional-20200505192422-29695 create deployment hello-node --image=k8s.gcr.io/echoserver:1.4
functional_test.go:646: (dbg) Run:  kubectl --context functional-20200505192422-29695 expose deployment hello-node --type=NodePort --port=8080
functional_test.go:651: (dbg) TestFunctional/parallel/ServiceCmd: waiting 10m0s for pods matching "app=hello-node" in namespace "default" ...
helpers.go:320: "hello-node-7bf657c596-77s6z" [7dd5ae6a-d5a5-402d-8551-90cafd9d2db6] Pending / Ready:ContainersNotReady (containers with unready status: [echoserver]) / ContainersReady:ContainersNotReady (containers with unready status: [echoserver])
helpers.go:320: "hello-node-7bf657c596-77s6z" [7dd5ae6a-d5a5-402d-8551-90cafd9d2db6] Running
functional_test.go:651: (dbg) TestFunctional/parallel/ServiceCmd: app=hello-node healthy within 30.143194458s
functional_test.go:655: (dbg) Run:  out/minikube-linux-amd64 -p functional-20200505192422-29695 service list
functional_test.go:668: (dbg) Run:  out/minikube-linux-amd64 -p functional-20200505192422-29695 service --namespace=default --https --url hello-node
functional_test.go:686: (dbg) Run:  out/minikube-linux-amd64 -p functional-20200505192422-29695 service hello-node --url --format={{.IP}}
functional_test.go:695: (dbg) Run:  out/minikube-linux-amd64 -p functional-20200505192422-29695 service hello-node --url
functional_test.go:709: url: http://172.17.0.3:30602
functional_test.go:712: get failed: GET http://172.17.0.3:30602 giving up after 5 attempts
resp: <nil>
@priyawadhwa priyawadhwa added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. labels May 5, 2020
@priyawadhwa priyawadhwa mentioned this issue May 5, 2020
Merged
@medyagh medyagh added this to the v1.10.0 milestone May 5, 2020
@tstromberg tstromberg changed the title TestFunctional/parallel/ServiceCmd flake: get failed: GET http://172.17.0.3:30602 giving up after 5 attempts TestFunctional/parallel/ServiceCmd flake on docker: GET http://172.17.0.3:30602 giving up after 5 attempts May 6, 2020
@tstromberg
Copy link
Contributor

This is apparently specific to docker as a flake:

docker_Linux      | 2020-05-02 |  83% | 10 of 12
docker_Linux      | 2020-05-03 |  92% | 11 of 12
docker_Linux      | 2020-05-04 |  83% | 10 of 12
docker_Linux      | 2020-05-05 |  67% | 8 of 12
docker_Linux      | 2020-05-06 |  60% | 3 of 5

@tstromberg tstromberg changed the title TestFunctional/parallel/ServiceCmd flake on docker: GET http://172.17.0.3:30602 giving up after 5 attempts TestFunctional/parallel/ServiceCmd flake: GET http://172.17.0.3:30602 giving up after 5 attempts May 6, 2020
@priyawadhwa
Copy link
Author

Logs here

@tstromberg tstromberg self-assigned this May 7, 2020
@tstromberg
Copy link
Contributor

I'm now looping this test locally from head to gather more insight into it.

No theories yet as logs don't show much.

@tstromberg
Copy link
Contributor

So far I've run this test in a dual-simultaneous test loop about 50 times and haven't hit the error. It may be squashed. Will add debugging though.

@tstromberg tstromberg mentioned this issue May 7, 2020
Merged
@medyagh
Copy link
Member

medyagh commented May 9, 2020 •
edited
Loading

@tstromberg
here is a test result with the logs you liked to see:
https://storage.googleapis.com/minikube-builds/logs/8041/03059f1/Docker_Linux.html#fail_TestFunctional%2fparallel%2fServiceCmd

@medyagh medyagh added the kind/flake Categorizes issue or PR as related to a flaky test. label May 11, 2020
@medyagh
Copy link
Member

medyagh commented May 11, 2020

also as seen here https://storage.googleapis.com/minikube-builds/logs/8075/b78f66e/Docker_Linux.html#fail_TestFunctional%2fparallel%2fServiceCmd

@medyagh medyagh mentioned this issue May 11, 2020
Merged
@medyagh medyagh modified the milestones: v1.10.0, v1.11.0 May 11, 2020
@medyagh
Copy link
Member

medyagh commented May 13, 2020

still happening on KVM too

@tstromberg tstromberg removed their assignment May 13, 2020
@priyawadhwa priyawadhwa modified the milestones: v1.11.0 May28, v.1.12.0 May 18, 2020
@medyagh medyagh removed this from the v.1.12.0-previous candidate (dumpster fire) milestone Jun 1, 2020
@priyawadhwa
Copy link
Author 

-------------------------------------------------
docker_Linux      | 2020-07-18 | 100% | 12 of 12
docker_Linux      | 2020-07-19 | 100% | 12 of 12
docker_Linux      | 2020-07-20 | 100% | 12 of 12
docker_Linux      | 2020-07-21 | 100% | 12 of 12
docker_Linux      | 2020-07-22 | 100% | 5 of 5
-------------------------------------------------
KVM_Linux         | 2020-07-18 |  50% | 6 of 12
KVM_Linux         | 2020-07-19 |  33% | 4 of 12
KVM_Linux         | 2020-07-20 |  42% | 5 of 12
KVM_Linux         | 2020-07-21 |  40% | 4 of 10
KVM_Linux         | 2020-07-22 |  40% | 2 of 5
-------------------------------------------------
none_Linux        | 2020-07-18 | 100% | 12 of 12
none_Linux        | 2020-07-19 | 100% | 12 of 12
none_Linux        | 2020-07-20 | 100% | 12 of 12
none_Linux        | 2020-07-21 | 100% | 12 of 12
none_Linux        | 2020-07-22 | 100% | 6 of 6
-------------------------------------------------
podman_Linux      | 2020-07-18 |   0% | 0 of 12
podman_Linux      | 2020-07-19 |   0% | 0 of 12
podman_Linux      | 2020-07-20 |   0% | 0 of 12
podman_Linux      | 2020-07-21 |   0% | 0 of 9
podman_Linux      | 2020-07-22 |   0% | 0 of 6
-------------------------------------------------
VirtualBox_Linux  | 2020-07-18 | 100% | 11 of 11
VirtualBox_Linux  | 2020-07-19 | 100% | 12 of 12
VirtualBox_Linux  | 2020-07-20 | 100% | 8 of 8
VirtualBox_Linux  | 2020-07-21 | 100% | 9 of 9
VirtualBox_Linux  | 2020-07-22 | 100% | 3 of 3
-------------------------------------------------
z__ALL__          | 2020-07-18 |  69% | 41 of 59
z__ALL__          | 2020-07-19 |  67% | 40 of 60
z__ALL__          | 2020-07-20 |  66% | 37 of 56
z__ALL__          | 2020-07-21 |  71% | 37 of 52
z__ALL__          | 2020-07-22 |  64% | 16 of 25
-------------------------------------------------

Looks like this test is still failing on KVM

@priyawadhwa priyawadhwa added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Dec 9, 2020
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 9, 2021
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 8, 2021
@k8s-ci-robot k8s-ci-robot added the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Apr 8, 2021
@sharifelgamal
Copy link
Collaborator

This doesn't seem to be an issue anymore, closing for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. kind/flake Categorizes issue or PR as related to a flaky test. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@tstromberg @medyagh @priyawadhwa @k8s-ci-robot @sharifelgamal @fejta-bot