Refactor `kubernetes-secret` rule #1462

rgmz · 2024-08-07T19:44:59Z

Description:

This is a follow-up to #1454.

Changes:

Leverages Use first non-empty group if secretGroup isn't set #1459 to collapse kubernetes-secret-with-data-before and kubernetes-secret-with-data-after into a single rule
Tightens regex to reduce the likelihood of false-positives
Handles multi-line secret definitions (e.g., https://kubernetes.io/docs/concepts/configuration/secret/#docker-config-secrets, https://kubernetes.io/docs/concepts/configuration/secret/#ssh-authentication-secrets)
Partially handles cases with comments at the end of lines, or comments between lines.

TODO

Checklist:

Does your PR pass tests?
Have you written new tests for your changes?
Have you lint your code locally prior to submission?

rgmz · 2024-09-17T20:58:12Z

@marcm-ml @brampat @fabio-sv This seems fairly complete to me. If you have a moment, give the new rule a try and let me know if it fixes the false-positives you're seeing.

Update: it seems to pass the Kubernetes repo with a solid true/false positive ratio.

Scan output (click to expand)

    ○
    │╲
    │ ○
    ○ ░
    ░    gitleaks

7:39PM INF Overriding enabled rules: kubernetes-secret-yaml
Finding:     kind: Secret               
metadata:                  
  name: test-set-env-secret
type: Opaque               
data:                      
                           password: dmFsdWUtMg0K
Secret:      password: dmFsdWUtMg0K
RuleID:      kubernetes-secret-yaml
Entropy:     4.027169
File:        hack/testdata/secret.yaml
Line:        2
Commit:      d7965e9331ed3be777ca4d550ad974ddbf8f6bf2
Author:      Kubernetes Submit Queue
Email:       k8s-merge-robot@users.noreply.github.com
Date:        2017-08-25T13:22:20Z
Fingerprint: d7965e9331ed3be777ca4d550ad974ddbf8f6bf2:hack/testdata/secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret            
metadata:               
  name: environment     
  namespace: kube-system
type: Opaque            
data:                   
                        elasticsearch-password: Y2hhbmdlbWU=
Secret:      elasticsearch-password: Y2hhbmdlbWU=
RuleID:      kubernetes-secret-yaml
Entropy:     4.350209
File:        cluster/addons/fluentd-elasticsearch/env-secret.yaml
Line:        2
Commit:      84e0326eb1f108f0d7aa2e9e48fb0c4a8edb4bd5
Author:      Kubernetes Submit Queue
Email:       k8s-merge-robot@users.noreply.github.com
Date:        2017-08-02T19:46:57Z
Fingerprint: 84e0326eb1f108f0d7aa2e9e48fb0c4a8edb4bd5:cluster/addons/fluentd-elasticsearch/env-secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret                   
metadata:                      
  name: storageos-secret       
type: "kubernetes.io/storageos"
data:                          
                               apiAddress: dGNwOi8vMTI3LjAuMC4xOjU3MDU=
Secret:      apiAddress: dGNwOi8vMTI3LjAuMC4xOjU3MDU=
RuleID:      kubernetes-secret-yaml
Entropy:     4.734184
File:        examples/volumes/storageos/storageos-secret.yaml
Line:        2
Commit:      5e2503e71fa51ae08fbbc90cc94e7d293709528e
Author:      Simon Croome
Email:       simon.croome@storageos.com
Date:        2017-02-24T15:47:40Z
Fingerprint: 5e2503e71fa51ae08fbbc90cc94e7d293709528e:examples/volumes/storageos/storageos-secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret               
metadata:                  
  name: sio-secret         
type: kubernetes.io/scaleio
data:                      
  username: YWRtaW4=       
                           password: c0NhbGVpbzEyMw==
Secret:      password: c0NhbGVpbzEyMw==
RuleID:      kubernetes-secret-yaml
Entropy:     4.315825
File:        examples/volumes/scaleio/secret.yaml
Line:        2
Commit:      98eae9b2222fe2eaacd6e34f2aab18902216fce7
Author:      Kubernetes Submit Queue
Email:       k8s-merge-robot@users.noreply.github.com
Date:        2017-03-03T22:34:37Z
Fingerprint: 98eae9b2222fe2eaacd6e34f2aab18902216fce7:examples/volumes/scaleio/secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret                                                    
metadata:                                                       
  name: heketi-secret                                           
  namespace: default                                            
data:                                                           
  # base64 encoded password. E.g.: echo -n "mypassword" | base64
                                                                key: bXlwYXNzd29yZA==
Secret:      key: bXlwYXNzd29yZA==
RuleID:      kubernetes-secret-yaml
Entropy:     4.106603
File:        examples/experimental/persistent-volume-provisioning/glusterfs-provisioning-secret.yaml
Line:        2
Commit:      1adf8567350caefa4bb5771907f8b5ed9174fa10
Author:      Jan Safranek
Email:       jsafrane@redhat.com
Date:        2016-09-20T14:24:30Z
Fingerprint: 1adf8567350caefa4bb5771907f8b5ed9174fa10:examples/experimental/persistent-volume-provisioning/glusterfs-provisioning-secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret                
metadata:                   
  name: quobyte-admin-secret
data:                       
                            password: cXVvYnl0ZQ==
Secret:      password: cXVvYnl0ZQ==
RuleID:      kubernetes-secret-yaml
Entropy:     4.277614
File:        examples/experimental/persistent-volume-provisioning/quobyte/quobyte-admin-secret.yaml
Line:        2
Commit:      0b7cb5f2ae970624c7ff571b89387887e06f50a4
Author:      Johannes Scheuermann
Email:       johannes.scheuermann@inovex.de
Date:        2016-08-20T13:39:30Z
Fingerprint: 0b7cb5f2ae970624c7ff571b89387887e06f50a4:examples/experimental/persistent-volume-provisioning/quobyte/quobyte-admin-secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret     
metadata:        
  name: ca-secret
type: Opaque     
data:            
                 ca.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR4RENDQXF5Z0F3SUJBZ0lVV3pqUDl5RUk0eHlRSnBzVHVERU4y...
Secret:      ca.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR4RENDQXF5Z0F3SUJBZ0lVV3pqUDl5RUk0eHlRSnBzVHVERU4y...
RuleID:      kubernetes-secret-yaml
Entropy:     5.555561
File:        discovery/ca-secret.yaml
Line:        2
Commit:      d17a236af3314572a092f430520fa35885a2a963
Author:      Devan Goodwin
Email:       dgoodwin@redhat.com
Date:        2016-08-23T18:29:40Z
Fingerprint: d17a236af3314572a092f430520fa35885a2a963:discovery/ca-secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret             
metadata:                
  name: ceph-secret-admin
data:                    
                         key: QVFEQ1pMdFhPUnQrSmhBQUFYaERWNHJsZ3BsMmNjcDR6RFZST0E9PQ==
Secret:      key: QVFEQ1pMdFhPUnQrSmhBQUFYaERWNHJsZ3BsMmNjcDR6RFZST0E9PQ==
RuleID:      kubernetes-secret-yaml
Entropy:     5.092714
File:        examples/experimental/persistent-volume-provisioning/rbd/ceph-secret-admin.yaml
Line:        2
Commit:      f297ea966e0916c18dc008af566975bd26dfb29c
Author:      Kubernetes Submit Queue
Email:       k8s-merge-robot@users.noreply.github.com
Date:        2016-08-23T15:46:32Z
Fingerprint: f297ea966e0916c18dc008af566975bd26dfb29c:examples/experimental/persistent-volume-provisioning/rbd/ceph-secret-admin.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret            
metadata:               
  name: ceph-secret-user
data:                   
                        key: QVFBTWdYaFZ3QkNlRGhBQTlubFBhRnlmVVNhdEdENGRyRldEdlE9PQ==
Secret:      key: QVFBTWdYaFZ3QkNlRGhBQTlubFBhRnlmVVNhdEdENGRyRldEdlE9PQ==
RuleID:      kubernetes-secret-yaml
Entropy:     4.602972
File:        examples/experimental/persistent-volume-provisioning/rbd/ceph-secret-user.yaml
Line:        2
Commit:      f297ea966e0916c18dc008af566975bd26dfb29c
Author:      Kubernetes Submit Queue
Email:       k8s-merge-robot@users.noreply.github.com
Date:        2016-08-23T15:46:32Z
Fingerprint: f297ea966e0916c18dc008af566975bd26dfb29c:examples/experimental/persistent-volume-provisioning/rbd/ceph-secret-user.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret        
metadata:           
  name: azure-secret
type: Opaque        
data:               
                    azurestorageaccountname: azhzdGVzdA==
Secret:      azurestorageaccountname: azhzdGVzdA==
RuleID:      kubernetes-secret-yaml
Entropy:     4.118520
File:        examples/azure_file/secret/azure-secret.yaml
Line:        2
Commit:      d7e4b826b91111445bd2de0f71c39d35baa60a06
Author:      Huamin Chen
Email:       hchen@redhat.com
Date:        2015-11-13T16:47:04Z
Fingerprint: d7e4b826b91111445bd2de0f71c39d35baa60a06:examples/azure_file/secret/azure-secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret       
metadata:          
  name: ceph-secret
data:              
                   key: QVFCMTZWMVZvRjVtRXhBQTVrQ1FzN2JCajhWVUxSdzI2Qzg0SEE9PQ==
Secret:      key: QVFCMTZWMVZvRjVtRXhBQTVrQ1FzN2JCajhWVUxSdzI2Qzg0SEE9PQ==
RuleID:      kubernetes-secret-yaml
Entropy:     5.045915
File:        examples/cephfs/secret/ceph-secret.yaml
Line:        2
Commit:      fe559f27264f424116008307fb49250ad0446afe
Author:      Huamin Chen
Email:       hchen@redhat.com
Date:        2015-04-09T18:05:24Z
Fingerprint: fe559f27264f424116008307fb49250ad0446afe:examples/cephfs/secret/ceph-secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret       
metadata:          
  name: ceph-secret
data:              
                   key: QVFCMTZWMVZvRjVtRXhBQTVrQ1FzN2JCajhWVUxSdzI2Qzg0SEE9PQ==
Secret:      key: QVFCMTZWMVZvRjVtRXhBQTVrQ1FzN2JCajhWVUxSdzI2Qzg0SEE9PQ==
RuleID:      kubernetes-secret-yaml
Entropy:     5.045915
File:        release-0.20.0/examples/rbd/secret/ceph-secret.yaml
Line:        2
Commit:      82f7303a008eaa20aafad071d4e14814beb3559a
Author:      Brendan Burns
Email:       bburns@google.com
Date:        2015-06-26T03:07:34Z
Fingerprint: 82f7303a008eaa20aafad071d4e14814beb3559a:release-0.20.0/examples/rbd/secret/ceph-secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret       
metadata:          
  name: test-secret
data:              
                   data-1: dmFsdWUtMQ0K
Secret:      data-1: dmFsdWUtMQ0K
RuleID:      kubernetes-secret-yaml
Entropy:     3.884184
File:        release-0.20.0/examples/secrets/secret.yaml
Line:        2
Commit:      82f7303a008eaa20aafad071d4e14814beb3559a
Author:      Brendan Burns
Email:       bburns@google.com
Date:        2015-06-26T03:07:34Z
Fingerprint: 82f7303a008eaa20aafad071d4e14814beb3559a:release-0.20.0/examples/secrets/secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret       
metadata:          
  name: ceph-secret
data:              
                   key: QVFCMTZWMVZvRjVtRXhBQTVrQ1FzN2JCajhWVUxSdzI2Qzg0SEE9PQ==
Secret:      key: QVFCMTZWMVZvRjVtRXhBQTVrQ1FzN2JCajhWVUxSdzI2Qzg0SEE9PQ==
RuleID:      kubernetes-secret-yaml
Entropy:     5.045915
File:        release-0.19.0/examples/rbd/secret/ceph-secret.yaml
Line:        2
Commit:      f3208ad4c056a490e8cd55a61098128b2b8f48c3
Author:      Brendan Burns
Email:       bburns@google.com
Date:        2015-06-10T16:23:42Z
Fingerprint: f3208ad4c056a490e8cd55a61098128b2b8f48c3:release-0.19.0/examples/rbd/secret/ceph-secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret       
metadata:          
  name: test-secret
data:              
                   data-1: dmFsdWUtMQ0K
Secret:      data-1: dmFsdWUtMQ0K
RuleID:      kubernetes-secret-yaml
Entropy:     3.884184
File:        release-0.19.0/examples/secrets/secret.yaml
Line:        2
Commit:      f3208ad4c056a490e8cd55a61098128b2b8f48c3
Author:      Brendan Burns
Email:       bburns@google.com
Date:        2015-06-10T16:23:42Z
Fingerprint: f3208ad4c056a490e8cd55a61098128b2b8f48c3:release-0.19.0/examples/secrets/secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret       
metadata:          
  name: ceph-secret
data:              
                   key: QVFCMTZWMVZvRjVtRXhBQTVrQ1FzN2JCajhWVUxSdzI2Qzg0SEE9PQ==
Secret:      key: QVFCMTZWMVZvRjVtRXhBQTVrQ1FzN2JCajhWVUxSdzI2Qzg0SEE9PQ==
RuleID:      kubernetes-secret-yaml
Entropy:     5.045915
File:        examples/rbd/secret/ceph-secret.yaml
Line:        2
Commit:      4a800fd10ea2eb1bd2f424994c6d33b5716940d0
Author:      Huamin Chen
Email:       hchen@redhat.com
Date:        2015-04-07T17:22:23Z
Fingerprint: 4a800fd10ea2eb1bd2f424994c6d33b5716940d0:examples/rbd/secret/ceph-secret.yaml:kubernetes-secret-yaml:2

Finding:     kind: Secret       
metadata:          
  name: test-secret
data:              
                   data-1: dmFsdWUtMQ0K
Secret:      data-1: dmFsdWUtMQ0K
RuleID:      kubernetes-secret-yaml
Entropy:     3.884184
File:        examples/secrets/secret.yaml
Line:        2
Commit:      e1885ba05fbfc8bd4a59ae2c04e1920b39254684
Author:      Paul Morie
Email:       pmorie@gmail.com
Date:        2015-04-29T21:20:14Z
Fingerprint: e1885ba05fbfc8bd4a59ae2c04e1920b39254684:examples/secrets/secret.yaml:kubernetes-secret-yaml:2

7:43PM INF 83725 commits scanned.
7:43PM INF scan completed in 4m10s
7:43PM WRN leaks found: 17

brampat · 2024-09-18T08:06:12Z

@rgmz Thank you for the quick response. I've tested it as so:

downloaded the raw toml file from here to my project folder: https://github.com/gitleaks/gitleaks/blob/b9f875c83a732d00c4ce1d796ac092b68eb91c4d/config/gitleaks.toml
ran gitleaks with -c gitleaks.toml, which gave no leaks found
ran gitleaks without the -c gitleaks.toml (default config), which again gave the 3 false positives.

So, unless my testing method is off, this seems like it solves the false positives for me.

marcm-ml · 2024-09-18T08:39:49Z

My false-positives are fully resolved by this. Thanks for the quick fix 👍

Outside the scope of this PR:
It might be interesting to add a rule for special resources such as ExternalSecret or SopsSecrets and work on an absence of special characters. For example, one could flag an ExternalSecret resource if for each key within the spec.target.template.data field "{{" and "}}" are missing. Similarly, for SopsSecret it would be sufficient to check for the "sops:" field. Of course, you could also add a custom .gitleaks.toml for these personal usecases.

Would like to hear your thoughts about this.

brampat · 2024-09-18T09:30:05Z

If I understand correctly @marcm-ml : See if rules can detect replacement-tags / template usage and in those cases, don't mark those as scanned secrets, since it'll contain a key that references a secret stored somewhere else (eg. in a vault).

rgmz · 2024-09-18T13:06:38Z

It might be interesting to add a rule for special resources such as ExternalSecret or SopsSecrets and work on an absence of special characters. For example, one could flag an ExternalSecret resource if for each key within the spec.target.template.data field "{{" and "}}" are missing. Similarly, for SopsSecret it would be sufficient to check for the "sops:" field. Of course, you could also add a custom .gitleaks.toml for these personal usecases.

Would like to hear your thoughts about this.

I created #1513 to track some follow-up ideas. I don't have personal experience with either of these; if you can provide examples of 'true' positives, that would be useful for future reference.

brampat · 2024-09-23T12:18:24Z

I've upgraded to gitleaks 8.9.2 and scanned the repo again. Unfortunately it now triggers the new rule again.

File:

apiVersion: v1
kind: Secret
metadata:
name: nexus-pull-secret
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: {{ template "imagePullSecret" . }}

Violates (pruned the committer info):

Finding: kind: Secret
metadata:
name: nexus-pull-secret
type: kubernetes.io/dockerconfigjson
data:
.docke...
Secret: kind: Secret
metadata:
name: nexus-pull-secret
type: kubernetes.io/dockerconfigjson
data:
.docke...
RuleID: kubernetes-secret-with-data-after
Entropy: 4.615095
File: helm/petclinic-jboss/templates/secret.yaml
Line: 2

rgmz · 2024-09-23T12:23:39Z

I think this is still pending release.

@zricethezav

brampat · 2024-09-23T13:04:58Z

Are you sure? I've checked the commit log for that release:
v8.19.2...master

And I see the refactoring stuff is in there.

rgmz · 2024-09-23T13:09:13Z

Are you sure? I've checked the commit log for that release: v8.19.2...master

And I see the refactoring stuff is in there.

It's a bit confusing. Those are the changes that have happened since that release.

Regardless, the example you shared had " inside templates {{ }} which may have still caused an issue. I opened #1520 to address that.

rgmz mentioned this pull request Sep 16, 2024

Rule introduced in #1454 matches against other kubernetes resources containing secret in their kind field. #1508

Open

rgmz force-pushed the refactor/k8s-secrets branch from ba6922d to c1f7039 Compare September 16, 2024 12:27

rgmz mentioned this pull request Sep 17, 2024

(feat): Adding secret detection rule for Kubernetes secrets #1454

Merged

3 tasks

rgmz marked this pull request as ready for review September 17, 2024 13:45

rgmz force-pushed the refactor/k8s-secrets branch 5 times, most recently from be527f8 to cc316b0 Compare September 17, 2024 20:56

rgmz force-pushed the refactor/k8s-secrets branch from cc316b0 to 2297d5e Compare September 17, 2024 22:25

refactor(kubernetes-secret): collapse rules and update regex

b9f875c

rgmz force-pushed the refactor/k8s-secrets branch from 2297d5e to b9f875c Compare September 17, 2024 23:42

rgmz mentioned this pull request Sep 18, 2024

Additional Kubernetes Secret Detections #1513

Open

3 tasks

rgmz mentioned this pull request Sep 18, 2024

Kubernetes test cases leaktk/fake-leaks#3

Open

zricethezav merged commit 3698060 into gitleaks:master Sep 18, 2024
1 check passed

rgmz deleted the refactor/k8s-secrets branch September 18, 2024 15:25

rgmz mentioned this pull request Sep 23, 2024

Tweak Kubernetes pattern to handle Helm variables #1520

Merged

3 tasks

rgmz mentioned this pull request Sep 27, 2024

Add kubernetes secrets leaktk/fake-leaks#5

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor `kubernetes-secret` rule #1462

Refactor `kubernetes-secret` rule #1462

rgmz commented Aug 7, 2024 •

edited

Loading

rgmz commented Sep 17, 2024 •

edited

Loading

brampat commented Sep 18, 2024

marcm-ml commented Sep 18, 2024 •

edited

Loading

brampat commented Sep 18, 2024

rgmz commented Sep 18, 2024

brampat commented Sep 23, 2024 •

edited

Loading

rgmz commented Sep 23, 2024

brampat commented Sep 23, 2024

rgmz commented Sep 23, 2024

Refactor kubernetes-secret rule #1462

Refactor kubernetes-secret rule #1462

Conversation

rgmz commented Aug 7, 2024 • edited Loading

Description:

Checklist:

rgmz commented Sep 17, 2024 • edited Loading

brampat commented Sep 18, 2024

marcm-ml commented Sep 18, 2024 • edited Loading

brampat commented Sep 18, 2024

rgmz commented Sep 18, 2024

brampat commented Sep 23, 2024 • edited Loading

rgmz commented Sep 23, 2024

brampat commented Sep 23, 2024

rgmz commented Sep 23, 2024

Refactor `kubernetes-secret` rule #1462

Refactor `kubernetes-secret` rule #1462

rgmz commented Aug 7, 2024 •

edited

Loading

rgmz commented Sep 17, 2024 •

edited

Loading

marcm-ml commented Sep 18, 2024 •

edited

Loading

brampat commented Sep 23, 2024 •

edited

Loading