Inaccurate testing of service account tokens

[atheiman]

We ran popeye with cluster-admin permissions in our 1.13 EKS cluster and it claims all service accounts in our cluster reference a secret that does not exist. Snippet of the output:

  · kube-system/external-dns.......................................................................💥
    💥 [POP-304] References a secret "external-dns-token-tb78n" which does not exists.
  · kube-system/fluentd-elasticsearch..............................................................💥
    💥 [POP-304] References a secret "fluentd-elasticsearch-token-hq72l" which does not exists.
  · kube-system/generic-garbage-collector..........................................................💥
    💥 [POP-304] References a secret "generic-garbage-collector-token-cnlpn" which does not exists.

But looking in the kube-system namespace, the secrets are there:

$ kubectl get serviceaccount -n kube-system \
>   external-dns fluentd-elasticsearch generic-garbage-collector \
>   -o=custom-columns=NAME:.metadata.name,Secrets:.secrets[*].name
NAME                        Secrets
external-dns                external-dns-token-tb78n
fluentd-elasticsearch       fluentd-elasticsearch-token-hq72l
generic-garbage-collector   generic-garbage-collector-token-cnlpn
$ kubectl get secret -n kube-system |
>   grep -e external-dns -e fluentd-elasticsearch -e generic-garbage-collector
external-dns-token-tb78n                         kubernetes.io/service-account-token   3      25d
fluentd-elasticsearch-token-hq72l                kubernetes.io/service-account-token   3      20d
generic-garbage-collector-token-cnlpn            kubernetes.io/service-account-token   3      34d

Not sure if its useful, but we invoke popeye (usingcluster-admin ClusterRole) like this:

popeye --all-namespaces --over-allocs

Popeye version:

Version:   0.4.2
Commit:    ca409ed9c1a9da98986990242f7033563abe2a1c

Kubernetes cluster version:

Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.8-eks-a977ba", GitCommit:"a977bab148535ec195f12edc8720913c7b943f9c", GitTreeState:"clean", BuildDate:"2019-07-29T20:47:04Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

I think the issue is somewhere in here, but I don't know enough Go to figure it out :(

popeye/internal/sanitize/sa.go

Lines 86 to 93 in 15fd321

	func (s *ServiceAccount) checkSecretRefs(fqn string, refs []v1.ObjectReference) {
	for _, ref := range refs {
	sfqn := cache.FQN(ref.Namespace, ref.Name)
	if _, ok := s.ListSecrets()[sfqn]; !ok {
	s.AddCode(304, fqn, sfqn)
	}
	}
	}

[derailed]

@atheiman Thank you for this excellent report for both the details and the quality of this find!!
I think I've resolved this issue and will push a Popeye update. Great find!!

[djablonski-moia]

@derailed I had the same issue, but can confirm it is fixed with 0.4.3

[derailed]

@djablonski-moia Awesome. Thank you the circling back!!