Fix log line container not found in logging api

[tejal29]

relates to GoogleCloudPlatform/cloud-code-intellij-internal#4323
follow from testing of #6370

When testing #6370, @etanshaul mentioned, he saw ApplicationLogEvent

applicationLogEvent {
  containerName: "frontend"
  podName: "java-guestbook-frontend-8cd49999c-tlbr9"
  prefix: "\033[92m[frontend]\033[0m "
  message: "rpc error: code = Unknown desc = Error: No such container: fa7802b2206f84f4f1e166a7a640523a281031c4c95d1709d38d62680391b97c"
  richFormattedMessage: "\033[92m[frontend]\033[0m rpc error: code = Unknown desc = Error: No such container: fa7802b2206f84f4f1e166a7a640523a281031c4c95d1709d38d62680391b97c"
}

Steps to reproduce

1) Run the guestbook app (I used the Java one) 2) Introduce a build error (I made a compilation error in frontend) - let skaffold run and error out 3) Fix the error - let skaffold run 4) Make another change in frontend (one that doesn't break the build) and let skaffold run -> I see multiple frontend containers in app log events

Description of change:

After digging in the code, we stream container logs in pkg/skaffold/kubernetes/logger here https://github.com/GoogleContainerTools/skaffold/blob/main/pkg/skaffold/kubernetes/logger/log.go#L150
in Logger.Start()
However, we never stop this process when a new dev iteration is starts. We only mute logs before a new dev iteration starts. Muting will make sure that no container logs are read and printed since that point of time.
Skaffold relies on the kubectl logs behaviour which exits when an error encounters. kubectl logs pod/xxx -c xxx -f print line rpc error: code = Unknown desc = Error: No such container: before exiting.

We have a race condition here between

1. When a logger is muted and
2. When kubectl log gets killed.

If 1) happens before 2) then you don't see the rpc error: code = Unknown desc = Error: No such container: fa7802b2206f84f4f1e166a7a640523a281031c4c95d1709d38d62680391b97c line.
If 2) happens before 1) then unfortunately this line will be read from log stream and relayed to event API

CLI Reproduction steps.

On master before this change,

1. run skaffold dev on any example
2. run `kubectl logs -f pod/frontend-XXXXX
3. Make a change to frontend and you will see the

kubectl logs -f java-guestbook-frontend-56c884765d-527wm
Picked up JAVA_TOOL_OPTIONS: -agentlib:jdwp=transport=dt_socket,server=y,address=5005,suspend=n,quiet=y
ping....
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.5.1)
....

rpc error: code = Unknown desc = Error: No such container: 51dcf01b9b4c70f785dbe77f1b576baaca406d3f728ca144cedfc9d2c3e7e5cb% 

In this PR,

1. if log for a container are not found, we will ignore them.

2. Don't add containers to track if a pod is deleted so we don't stream logs for deleted pod, containers.

Follow up

1. Should we stop logger instead of mute, unmute between dev loops?

[codecov]

Codecov Report

Merging #6399 (6765171) into main (005dc31) will decrease coverage by 0.00%.
The diff coverage is 60.00%.

@@            Coverage Diff             @@
##             main    #6399      +/-   ##
==========================================
- Coverage   70.38%   70.38%   -0.01%     
==========================================
  Files         499      499              
  Lines       22722    22738      +16     
==========================================
+ Hits        15994    16005      +11     
- Misses       5685     5692       +7     
+ Partials     1043     1041       -2     

Impacted Files	Coverage Δ
pkg/skaffold/kubernetes/logger/log.go	34.06% <0.00%> (-0.77%)	⬇️
pkg/diag/validator/pod.go	1.36% <25.00%> (-0.04%)	⬇️
pkg/skaffold/kubernetes/status/status_check.go	53.40% <100.00%> (+0.74%)	⬆️
pkg/skaffold/log/stream/stream.go	85.71% <100.00%> (+4.46%)	⬆️
pkg/skaffold/util/tar.go	63.21% <0.00%> (-2.30%)	⬇️
pkg/skaffold/docker/parse.go	88.23% <0.00%> (+0.84%)	⬆️
pkg/skaffold/docker/image.go	66.44% <0.00%> (+0.99%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 005dc31...6765171. Read the comment docs.