S3 Target does not work with IRSA (AWS role for svc acct) but works with static IAM User Creds #349

XDanny322 · 2023-10-18T18:39:06Z

Hi -

I have an instance of kyverno, and policy reporter deployed in AWS EKS.

Version:

kyverno: chart version 3.0.5
policy reporter: chart version 2.20.1

Its deployed via ArgoCD, with its valuesObject values (values.yaml) as shown below.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kyverno-policy-reporter
  namespace: argocd
spec:
  destination:
    name: ""
    namespace: policy-reporter
    server: "https://kubernetes.default.svc"
  source:
    path: ""
    repoURL: "https://kyverno.github.io/policy-reporter"
    targetRevision: 2.20.1
    chart: policy-reporter
    helm:
      valuesObject:
        logging:
          encoding: console
          logLevel: -2
          development: true
        kyvernoPlugin:
          enabled: true
          blockReports:
            enabled: true
            source: "Kyverno Event"
          logging:
            encoding: console
            logLevel: -2
            development: true
        ui:
          enabled: true
        serviceAccount:
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNTID_PLACEHOLDER>:role/example-irsa-iam-role
        target:
          s3:
            endpoint: https://s3.amazonaws.com
            region: us-east-1
            bucket: <SOMEBUCKETNAME>
            # secretAccessKey: R1s...
            # accessKeyID: AK...
            minimumPriority: debug
            skipExistingOnStartup: true
            channels:
            - prefix: policy-reporter-20231018/blocked
              sources:
                - Kyverno Event
              customFields:
                foo: blocked
            - prefix: policy-reporter-20231018/audit
              sources:
                - kyverno
              customFields:
                foo: audit

It basically sends events to S3. (i am separating audit and blocked events into diff path, based on sources, in diff channels, TMI). If i pass it hardcoded AWS IAMUser's creds secretAccessKey and accessKeyID all works This is great, everything is working.

However, i would like to use IRSA which is basically an AWS IAM Role, linked with a k8s service account. For this to work, in short, i create (a) AWS OIDC provider (b) AWS role (c) k8s service account, and 'annotation' the service account to use said role. That generates some environment variables (AWS_DEFAULT_REGION, AWS_REGION, AWS_ROLE_ARN, AWS_STS_REGIONAL_ENDPOINTS, AWS_WEB_IDENTITY_TOKEN_FILE). I then turned on debug logging as well. I then started policy-report deployment/pod, all good still.

I then triggered an event, but get some violation error 'Privileged mode is disallowed'. It looks like policy reporting is getting "WebIdentityErr: failed to retrieve credentials. Putting the error together, it looks like the messages is trying to say MethodNotAllowed - The specified method is not allowed against this resource POST. Seems to be getting errors around here:

policy-reporter/pkg/target/s3/s3.go

Line 56 in 9036018

err := c.s3.Upload(body, key)

.

Complete log below.

2023-10-18 17:22:27	INFO	S3 configured
2023-10-18 17:22:27	INFO	 configured
2023-10-18 17:22:27	INFO	 configured
2023-10-18 17:22:27	INFO	UI configured
2023-10-18 17:22:27	INFO	sqlite connection created
2023-10-18 17:22:27	DEBUG	preparing database
2023-10-18 17:22:27	DEBUG	failed to load config	{"error": "no such table: policy_report_config"}
2023-10-18 17:22:27	DEBUG	database schema upgrade started
2023-10-18 17:22:27	INFO	REST api enabled
2023-10-18 17:22:27	DEBUG	register new result listener
2023-10-18 17:22:27	INFO	start client	{"worker": 5}
2023-10-18 17:22:27	INFO	informer sync completed
2023-10-18 17:23:08	DEBUG	Sending request: POST /api/push HTTP/1.1
Host: kyverno-policy-reporter-ui:8080
Content-Type: application/json; charset=utf-8
User-Agent: Policy-Reporter

{"message":"validation error: Privileged mode is disallowed. The fields spec.containers[*].securityContext.privileged and spec.initContainers[*].securityContext.privileged must be unset or set to `false`. rule privileged-containers failed at path /spec/containers/0/securityContext/privileged/","policy":"disallow-privileged-containers","rule":"privileged-containers","priority":"warning","status":"fail","severity":"medium","category":"Pod Security Standards (Baseline)","scored":true,"resource":{"apiVersion":"v1","kind":"Pod","name":"test-pod-1","namespace":"argocd","uid":"7d45730f-ce68-4006-825f-385aaa8de84b"},"creationTimestamp":"2023-10-18T17:22:37Z","source":"kyverno"}

2023-10-18 17:23:08	DEBUG	Received response: HTTP/1.1 200 OK
Content-Length: 0
Content-Type: application/json; charset=UTF-8
Date: Wed, 18 Oct 2023 17:23:08 GMT


2023-10-18 17:23:08	INFO	UI: PUSH OK
2023-10-18 17:23:08	ERROR	S3: S3 Upload error	{"error": "WebIdentityErr: failed to retrieve credentials\ncaused by: SerializationError: failed to unmarshal error message\n\tstatus code: 405, request id: \ncaused by: UnmarshalError: failed to unmarshal error message\n\t00000000  3c 3f 78 6d 6c 20 76 65  72 73 69 6f 6e 3d 22 31  |<?xml version=\"1|\n00000010  2e 30 22 20 65 6e 63 6f  64 69 6e 67 3d 22 55 54  |.0\" encoding=\"UT|\n00000020  46 2d 38 22 3f 3e 0a 3c  45 72 72 6f 72 3e 3c 43  |F-8\"?>.<Error><C|\n00000030  6f 64 65 3e 4d 65 74 68  6f 64 4e 6f 74 41 6c 6c  |ode>MethodNotAll|\n00000040  6f 77 65 64 3c 2f 43 6f  64 65 3e 3c 4d 65 73 73  |owed</Code><Mess|\n00000050  61 67 65 3e 54 68 65 20  73 70 65 63 69 66 69 65  |age>The specifie|\n00000060  64 20 6d 65 74 68 6f 64  20 69 73 20 6e 6f 74 20  |d method is not |\n00000070  61 6c 6c 6f 77 65 64 20  61 67 61 69 6e 73 74 20  |allowed against |\n00000080  74 68 69 73 20 72 65 73  6f 75 72 63 65 2e 3c 2f  |this resource.</|\n00000090  4d 65 73 73 61 67 65 3e  3c 4d 65 74 68 6f 64 3e  |Message><Method>|\n000000a0  50 4f 53 54 3c 2f 4d 65  74 68 6f 64 3e 3c 52 65  |POST</Method><Re|\n000000b0  73 6f 75 72 63 65 54 79  70 65 3e 53 45 52 56 49  |sourceType>SERVI|\n000000c0  43 45 3c 2f 52 65 73 6f  75 72 63 65 54 79 70 65  |CE</ResourceType|\n000000d0  3e 3c 52 65 71 75 65 73  74 49 64 3e 48 58 32 41  |><RequestId>HX2A|\n000000e0  32 57 56 32 50 39 36 50  57 38 30 46 3c 2f 52 65  |2WV2P96PW80F</Re|\n000000f0  71 75 65 73 74 49 64 3e  3c 48 6f 73 74 49 64 3e  |questId><HostId>|\n00000100  76 5a 53 4e 4b 76 64 4a  38 73 2b 6f 65 64 39 61  |vZSNKvdJ8s+oed9a|\n00000110  34 31 41 6e 49 6b 6d 30  32 42 4b 64 75 65 43 41  |41AnIkm02BKdueCA|\n00000120  4f 4a 76 72 61 7a 74 41  4b 56 6d 6a 51 59 45 72  |OJvraztAKVmjQYEr|\n00000130  35 57 6e 36 73 52 56 64  2f 4c 50 30 4e 31 54 61  |5Wn6sRVd/LP0N1Ta|\n00000140  70 74 66 67 36 4c 4f 47  57 68 63 3d 3c 2f 48 6f  |ptfg6LOGWhc=</Ho|\n00000150  73 74 49 64 3e 3c 2f 45  72 72 6f 72 3e           |stId></Error>|\n\ncaused by: unknown error response tag, {{ Error} []}"}
github.com/kyverno/policy-reporter/pkg/target/s3.(*client).Send
	/app/pkg/target/s3/s3.go:58
github.com/kyverno/policy-reporter/pkg/config.(*Resolver).RegisterSendResultListener.NewSendResultListener.func1.1
	/app/pkg/listener/send_result.go:36
2023-10-18 17:23:08	ERROR	: S3 Upload error	{"error": "WebIdentityErr: failed to retrieve credentials\ncaused by: SerializationError: failed to unmarshal error message\n\tstatus code: 405, request id: \ncaused by: UnmarshalError: failed to unmarshal error message\n\t00000000  3c 3f 78 6d 6c 20 76 65  72 73 69 6f 6e 3d 22 31  |<?xml version=\"1|\n00000010  2e 30 22 20 65 6e 63 6f  64 69 6e 67 3d 22 55 54  |.0\" encoding=\"UT|\n00000020  46 2d 38 22 3f 3e 0a 3c  45 72 72 6f 72 3e 3c 43  |F-8\"?>.<Error><C|\n00000030  6f 64 65 3e 4d 65 74 68  6f 64 4e 6f 74 41 6c 6c  |ode>MethodNotAll|\n00000040  6f 77 65 64 3c 2f 43 6f  64 65 3e 3c 4d 65 73 73  |owed</Code><Mess|\n00000050  61 67 65 3e 54 68 65 20  73 70 65 63 69 66 69 65  |age>The specifie|\n00000060  64 20 6d 65 74 68 6f 64  20 69 73 20 6e 6f 74 20  |d method is not |\n00000070  61 6c 6c 6f 77 65 64 20  61 67 61 69 6e 73 74 20  |allowed against |\n00000080  74 68 69 73 20 72 65 73  6f 75 72 63 65 2e 3c 2f  |this resource.</|\n00000090  4d 65 73 73 61 67 65 3e  3c 4d 65 74 68 6f 64 3e  |Message><Method>|\n000000a0  50 4f 53 54 3c 2f 4d 65  74 68 6f 64 3e 3c 52 65  |POST</Method><Re|\n000000b0  73 6f 75 72 63 65 54 79  70 65 3e 53 45 52 56 49  |sourceType>SERVI|\n000000c0  43 45 3c 2f 52 65 73 6f  75 72 63 65 54 79 70 65  |CE</ResourceType|\n000000d0  3e 3c 52 65 71 75 65 73  74 49 64 3e 48 58 32 44  |><RequestId>HX2D|\n000000e0  50 33 38 31 43 34 56 35  48 51 41 35 3c 2f 52 65  |P381C4V5HQA5</Re|\n000000f0  71 75 65 73 74 49 64 3e  3c 48 6f 73 74 49 64 3e  |questId><HostId>|\n00000100  71 6c 68 57 33 48 75 72  39 32 4e 50 43 39 79 39  |qlhW3Hur92NPC9y9|\n00000110  4c 6d 34 37 65 44 57 59  54 74 54 4c 66 47 50 6f  |Lm47eDWYTtTLfGPo|\n00000120  6c 34 79 6e 6d 59 41 44  4e 55 75 49 6c 62 46 42  |l4ynmYADNUuIlbFB|\n00000130  63 79 66 2b 61 36 34 39  5a 70 4d 53 56 51 6a 76  |cyf+a649ZpMSVQjv|\n00000140  73 58 2b 77 51 64 44 72  52 51 63 3d 3c 2f 48 6f  |sX+wQdDrRQc=</Ho|\n00000150  73 74 49 64 3e 3c 2f 45  72 72 6f 72 3e           |stId></Error>|\n\ncaused by: unknown error response tag, {{ Error} []}"}
github.com/kyverno/policy-reporter/pkg/target/s3.(*client).Send
	/app/pkg/target/s3/s3.go:58
github.com/kyverno/policy-reporter/pkg/config.(*Resolver).RegisterSendResultListener.NewSendResultListener.func1.1
	/app/pkg/listener/send_result.go:36

What i tried to debug

use a diff pod with same role - the other pod worked

For completeness sake, i then created another deployment / another service account, that uses the same role, and that works. I diffed the environment variable provided to both deployment, and they were the same.

  - name: AWS_STS_REGIONAL_ENDPOINTS
    value: regional
  - name: AWS_DEFAULT_REGION
    value: us-east-1
  - name: AWS_REGION
    value: us-east-1
  - name: AWS_ROLE_ARN
    value: arn:aws:iam::<ACCOUNTID_PLACEHOLDER>:role/example-irsa-iam-role
  - name: AWS_WEB_IDENTITY_TOKEN_FILE
    value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
--
  - name: AWS_STS_REGIONAL_ENDPOINTS
    value: regional
  - name: AWS_DEFAULT_REGION
    value: us-east-1
  - name: AWS_REGION
    value: us-east-1
  - name: AWS_ROLE_ARN
    value: arn:aws:iam::<ACCOUNTID_PLACEHOLDER>:role/example-irsa-iam-role
  - name: AWS_WEB_IDENTITY_TOKEN_FILE
    value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

shell to check via awscli or soemthing (couldn't)

I also tried get into the pod kubectl exec --stdin --tty kyverno-policy-reporter-c6c864d49-bx658 -- /bin/bash to test, but can't, as it seems there isn't a shell in the scratch base image.

policy-reporter/Dockerfile

Line 18 in 9036018

FROM scratch

Ensure AWS IAM permissions are the same - done

Finallly, worth mentioning, both the IAMuser and the IAMRole has the AWS managed policy AmazonS3FullAccess/

Testing prebuild version 2.16.2

I was follow on this issue / comment: #327 (comment). I think the referenced issue is more "when" do we use which creds... not why we are getting WebIdentityErr. In anycase 2.16.2 image did not help in my case. (Tested by switching just the image key)

The ask

Why does upload to s3 works when using IAMUser status creds, but not IRSA role creds?

Please let me know if i can provide any additional information.

Thanks

The text was updated successfully, but these errors were encountered:

fjogeleit · 2023-10-19T09:33:30Z

Thanks for reporting, I'll will check and let you know when I find something

XDanny322 · 2023-10-19T14:38:13Z

Thankyou @fjogeleit, for getting to this so quickly, and creating image v2.17.0-RC1 for testing. Indeed, updating the aws sdk fixed this issue. I am now able to use IRSA and put files into S3. Complete log below.

This issue has can be closed (or closed when this gets released)

>>   k logs kyverno-policy-reporter-57979f78fb-bwgbt -n policy-reporter
2023-10-19 14:31:52     DEBUG   configure AWS credentals provider       {"provider": "WebIdentityRoleProvider", "WebIdentidyFile": "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"}
2023-10-19 14:31:52     DEBUG   S3 Client created       {"Region": "us-east-1", "Endpoint": "https://s3.amazonaws.com", "PathStyle": false}
2023-10-19 14:31:52     INFO    S3 configured
2023-10-19 14:31:52     DEBUG   configure AWS credentals provider       {"provider": "WebIdentityRoleProvider", "WebIdentidyFile": "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"}
2023-10-19 14:31:52     DEBUG   S3 Client created       {"Region": "us-east-1", "Endpoint": "https://s3.amazonaws.com", "PathStyle": false}
2023-10-19 14:31:52     INFO     configured
2023-10-19 14:31:52     DEBUG   configure AWS credentals provider       {"provider": "WebIdentityRoleProvider", "WebIdentidyFile": "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"}
2023-10-19 14:31:52     DEBUG   S3 Client created       {"Region": "us-east-1", "Endpoint": "https://s3.amazonaws.com", "PathStyle": false}
2023-10-19 14:31:52     INFO     configured
2023-10-19 14:31:52     INFO    UI configured
2023-10-19 14:31:52     INFO    sqlite connection created
2023-10-19 14:31:52     DEBUG   preparing database
2023-10-19 14:31:52     DEBUG   failed to load config   {"error": "no such table: policy_report_config"}
2023-10-19 14:31:52     DEBUG   database schema upgrade started
2023-10-19 14:31:52     INFO    REST api enabled
2023-10-19 14:31:52     DEBUG   register new result listener
2023-10-19 14:31:52     INFO    start client    {"worker": 5}
2023-10-19 14:31:52     INFO    informer sync completed
2023-10-19 14:33:24     DEBUG   Sending request: POST /api/push HTTP/1.1
Host: kyverno-policy-reporter-ui:8080
Content-Type: application/json; charset=utf-8
User-Agent: Policy-Reporter

{"message":"validation error: Privileged mode is disallowed. The fields spec.containers[*].securityContext.privileged and spec.initContainers[*].securityContext.privileged must be unset or set to `false`. rule privileged-containers failed at path /spec/containers/0/securityContext/privileged/","policy":"disallow-privileged-containers","rule":"privileged-containers","priority":"warning","status":"fail","severity":"medium","category":"Pod Security Standards (Baseline)","scored":true,"resource":{"apiVersion":"v1","kind":"Pod","name":"test-pod-1","namespace":"argocd","uid":"4e32e529-85bf-4a34-8c77-0f96ac06a0e1"},"creationTimestamp":"2023-10-19T14:32:53Z","source":"kyverno"}

2023-10-19 14:33:24     DEBUG   Received response: HTTP/1.1 200 OK
Content-Length: 0
Content-Type: application/json; charset=UTF-8
Date: Thu, 19 Oct 2023 14:33:24 GMT


2023-10-19 14:33:24     INFO    UI: PUSH OK
2023-10-19 14:33:24     INFO    S3: PUSH OK
2023-10-19 14:33:24     INFO    : PUSH OK

fjogeleit · 2023-10-19T14:44:46Z

Great, thanks for your feedback and help with testing.

fjogeleit mentioned this issue Oct 19, 2023

Migrate to AWS SDK v2 #352

Merged

fjogeleit closed this as completed Oct 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3 Target does not work with IRSA (AWS role for svc acct) but works with static IAM User Creds #349

S3 Target does not work with IRSA (AWS role for svc acct) but works with static IAM User Creds #349

XDanny322 commented Oct 18, 2023 •

edited

Loading

fjogeleit commented Oct 19, 2023

XDanny322 commented Oct 19, 2023

fjogeleit commented Oct 19, 2023

S3 Target does not work with IRSA (AWS role for svc acct) but works with static IAM User Creds #349

S3 Target does not work with IRSA (AWS role for svc acct) but works with static IAM User Creds #349

Comments

XDanny322 commented Oct 18, 2023 • edited Loading

What i tried to debug

use a diff pod with same role - the other pod worked

shell to check via awscli or soemthing (couldn't)

Ensure AWS IAM permissions are the same - done

Testing prebuild version 2.16.2

The ask

fjogeleit commented Oct 19, 2023

XDanny322 commented Oct 19, 2023

fjogeleit commented Oct 19, 2023

XDanny322 commented Oct 18, 2023 •

edited

Loading