Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deleting s3 object NotFound errors #4859

Closed
r4f4 opened this issue Mar 12, 2024 · 2 comments · Fixed by #4879
Closed

Deleting s3 object NotFound errors #4859

r4f4 opened this issue Mar 12, 2024 · 2 comments · Fixed by #4879
Labels
kind/bug Categorizes issue or PR as related to a bug. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@r4f4
Copy link
Contributor

r4f4 commented Mar 12, 2024 •
edited
Loading

/kind bug

What steps did you take and what happened:
If the bootstrap data s3 object is deleted, further attempts to delete it will cause "NotFound" errors.

What did you expect to happen:
NotFound errors are ignored since the object was already deleted. I confirmed in the AWS console that the openshift-bootstrap-data-rdossant-installer-03-jjf6b bucket contained no objects.

Anything else you would like to add:
Here are the relevant lines from the log output:

I0312 18:00:13.479037     109 s3.go:220] "Deleting S3 object" controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-1" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-1" reconcileID="575c2ec1-a75a-4451-bbd5-dc80f62df49d" machine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-1" cluster="openshift-cluster-api-guests/rdossant-installer-03-jjf6b" bucket="openshift-bootstrap-data-rdossant-installer-03-jjf6b" key="control-plane/rdossant-installer-03-jjf6b-master-1"
I0312 18:00:13.602972     109 s3.go:220] "Deleting S3 object" controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-2" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-2" reconcileID="9cda22be-5acd-4670-840f-8a6708437385" machine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-2" cluster="openshift-cluster-api-guests/rdossant-installer-03-jjf6b" bucket="openshift-bootstrap-data-rdossant-installer-03-jjf6b" key="control-plane/rdossant-installer-03-jjf6b-master-2"
I0312 18:00:13.608919     109 s3.go:220] "Deleting S3 object" controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-0" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-0" reconcileID="1ed0ad52-ffc1-4b62-97e4-876f8e8c3242" machine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-0" cluster="openshift-cluster-api-guests/rdossant-installer-03-jjf6b" bucket="openshift-bootstrap-data-rdossant-installer-03-jjf6b" key="control-plane/rdossant-installer-03-jjf6b-master-0"

E0312 18:04:25.282967     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYY3QSWKBBDZ7R8, host id: 2f3HawFbPheaptP9E+WRbu3fhEXTMwyZQ1DBPGBG7qlg74ssQR0XISM4OSlxvrn59GeFREtN4hp9C+S5LgQD2g==
 >
E0312 18:04:25.284197     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYY3QSWKBBDZ7R8, host id: 2f3HawFbPheaptP9E+WRbu3fhEXTMwyZQ1DBPGBG7qlg74ssQR0XISM4OSlxvrn59GeFREtN4hp9C+S5LgQD2g==
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-0" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-0" reconcileID="7fac94a1-772a-4c7b-a631-5ef7fc015d5b"
E0312 18:04:25.286152     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYPFY0EQBM42VYH, host id: nJZakAhLrbZ1xrSNX3tyk0IKmMgFjsjMSs/D9nzci90GfRNNfUnvwZTbcaUBQYiuSlY5+aysCuwejWpvi8FmGusbQCK1Qtjr9pjqDQfxzY4=
 >
E0312 18:04:25.287353     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYPFY0EQBM42VYH, host id: nJZakAhLrbZ1xrSNX3tyk0IKmMgFjsjMSs/D9nzci90GfRNNfUnvwZTbcaUBQYiuSlY5+aysCuwejWpvi8FmGusbQCK1Qtjr9pjqDQfxzY4=
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-2" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-2" reconcileID="b6c792ad-5519-48d5-a994-18dda76d8a93"
E0312 18:04:25.291383     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYGWSJDR35Q4GWX, host id: Qnltg++ia3VapXjtENZOQIwfAxbxfwVLPlC0DwcRBx+L60h52ENiNqMOkvuNwJyYnPxbo/CaawzMT11oIKGO9g==
 >
E0312 18:04:25.292132     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYGWSJDR35Q4GWX, host id: Qnltg++ia3VapXjtENZOQIwfAxbxfwVLPlC0DwcRBx+L60h52ENiNqMOkvuNwJyYnPxbo/CaawzMT11oIKGO9g==
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-1" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-1" reconcileID="92e1f8ed-b31f-4f75-9083-59aad15efe79"
E0312 18:04:25.679859     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYSBZGYPC7SNJEX, host id: EplmtNQ+RxmbU88z+4App6YEVvniJpyCeMiMZuUegJIMqZgbkA1lmCjHntSLDm4eA857OdhtHsn+zD6AX7uelGIsogzN2ZziiAZXZrbIIEg=
 >
E0312 18:04:25.680663     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYSBZGYPC7SNJEX, host id: EplmtNQ+RxmbU88z+4App6YEVvniJpyCeMiMZuUegJIMqZgbkA1lmCjHntSLDm4eA857OdhtHsn+zD6AX7uelGIsogzN2ZziiAZXZrbIIEg=
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-0" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-0" reconcileID="9e436c67-aca0-409c-9179-0ce4cccce9ad"
E0312 18:04:25.766805     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYVX5DZCE3WREHG, host id: ttRZOKC/DPaAN55tH4SHv5NwPaOtKFxgYtB5w6wC/xWGz5xWtHmrb+QMqestuHJvJDjpxvz1dkOY3IUaDLYPdA==
 >
E0312 18:04:25.767862     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYVX5DZCE3WREHG, host id: ttRZOKC/DPaAN55tH4SHv5NwPaOtKFxgYtB5w6wC/xWGz5xWtHmrb+QMqestuHJvJDjpxvz1dkOY3IUaDLYPdA==
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-2" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-2" reconcileID="ba3546c1-410d-4074-b915-508cebcf6462"
E0312 18:04:25.900297     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYR8FXVXJFPZEBF, host id: /IDCA+L+h9/vU7m777mKwyzP1evlKKxB2l3YUe0ZZNS1MBKdVGxDFKrDIHxkYcm9OX0cyRpbUD5+kaGr/Oit3rLFlBdyPdNrG1UsqqO2/X8=
 >
E0312 18:04:25.901660     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYR8FXVXJFPZEBF, host id: /IDCA+L+h9/vU7m777mKwyzP1evlKKxB2l3YUe0ZZNS1MBKdVGxDFKrDIHxkYcm9OX0cyRpbUD5+kaGr/Oit3rLFlBdyPdNrG1UsqqO2/X8=
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-1" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-1" reconcileID="cabe571a-b320-4ecd-8da4-d3335aee54db"
E0312 18:04:26.081225     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYN8R43NCKYZ7J9, host id: vQw1gpiTnvUzv50VLpKRa8njFma68Ie0LzHB5AXGGlGq/X7kymyA9M17dGgi+hcrvAc4dZ04J9lZrAFDPHbyj+SzzsUOhpneJW8Sr0CafBc=
 >
E0312 18:04:26.082195     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: 9QYN8R43NCKYZ7J9, host id: vQw1gpiTnvUzv50VLpKRa8njFma68Ie0LzHB5AXGGlGq/X7kymyA9M17dGgi+hcrvAc4dZ04J9lZrAFDPHbyj+SzzsUOhpneJW8Sr0CafBc=
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-master-0" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-master-0" reconcileID="c45efec6-6528-4e11-9c81-da27718d1686"
E0312 18:04:26.175397     109 awsmachine_controller.go:576] "controllers/AWSMachine: unable to delete secrets" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: JZAE8SEJZQM2RY2T, host id: EHq4CKwSzqwiQpcOTvRahwsbX+hVY1UinAAQ7vsvDrlpDKEEtxcjtq4F4qa8hzeZECG3OGp+n6FQjLkSw2PXAA==
 >
[...]
E0312 18:17:23.150499     109 awsmachine_controller.go:307] "unable to delete machine" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: EVM8Q64ZV5T5K19G, host id: N6AOedamPQIzaVRnZDz+vg2ueYleJ0e6A0TytrjreNIEj6YUzzzBEEPw2f4WowD+RReANFI0Vac=
 >
E0312 18:17:23.152811     109 controller.go:329] "Reconciler error" err=<
	deleting bootstrap data object: deleting S3 object: NotFound: Not Found
		status code: 404, request id: EVM8Q64ZV5T5K19G, host id: N6AOedamPQIzaVRnZDz+vg2ueYleJ0e6A0TytrjreNIEj6YUzzzBEEPw2f4WowD+RReANFI0Vac=
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="openshift-cluster-api-guests/rdossant-installer-03-jjf6b-bootstrap" namespace="openshift-cluster-api-guests" name="rdossant-installer-03-jjf6b-bootstrap" reconcileID="ddaee7ad-fe36-4aaa-b3d4-8399065e1ee8"

S3 config used: https://github.com/openshift/installer/blob/master/pkg/asset/manifests/aws/cluster.go#L149-L152

Environment:

  • Cluster-api-provider-aws version: main @ 5aa9d22
  • Kubernetes version: (use kubectl version): v1.29.1+5a4819c
  • OS (e.g. from /etc/os-release): RHCOS
@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 12, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@r4f4
Copy link
Contributor Author

r4f4 commented Mar 18, 2024

The problem here is the HeadObject s3 API call:

  1. Object does not exist:
error from HeadObject: NotFound: Not Found
	status code: 404, request id: Q27GZ2A0369275WA, host id: Xy1KhlJvndIjHXHFQepx0y7TH7dvcQ9OhwDRTEQP7SQ1bfv5bgxgqbdKEhHU9VW6ygir4Izllqw=
  1. Bucket does not exist:
error from HeadObject: NotFound: Not Found
	status code: 404, request id: 70TSC48GXFSAW3HQ, host id: XDw0B5SW+79idg59ChYUCRi9+l2GxgKDA7VJH0KxUMa4kHZQiRfsa5MXAq3GN3NQ/q4XxldD9/XQGoEixE5hFg==

@r4f4 r4f4 mentioned this issue Mar 18, 2024
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

🐛 fix: s3: fix bucket object not found r4f4/cluster-api-provider-aws
2 participants
@r4f4 @k8s-ci-robot