Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix s3fs dying after 6h with Unable to locate credentials #137

Closed
rom1504 opened this issue Feb 10, 2022 · 8 comments
Closed

fix s3fs dying after 6h with Unable to locate credentials #137

rom1504 opened this issue Feb 10, 2022 · 8 comments

Comments

@rom1504
Copy link
Owner

rom1504 commented Feb 10, 2022

No description provided.

@rom1504
Copy link
Owner Author

rom1504 commented Feb 10, 2022

fsspec/s3fs#558

@rom1504 rom1504 mentioned this issue Feb 10, 2022
Merged
rom1504 added a commit that referenced this issue Feb 10, 2022
@rom1504
retry whole sharding if it fails
ee357ec 
trying to work around #137
rom1504 added a commit that referenced this issue Feb 10, 2022
@rom1504
retry whole sharding if it fails (#138)
6f5ed17 
trying to work around #137
@rom1504
Copy link
Owner Author

rom1504 commented Feb 10, 2022

boto/botocore#2117 (comment)

@rom1504
Copy link
Owner Author

rom1504 commented Feb 10, 2022

increasing AWS_METADATA_SERVICE_NUM_ATTEMPTS env variable could work

another way is to put credentials in instances rather than using IAM role auth through the metadata service

@rom1504
Copy link
Owner Author

rom1504 commented Feb 10, 2022

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environments-cfg-ec2-imds.html

@rom1504
Copy link
Owner Author

rom1504 commented Feb 10, 2022

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html

@rom1504
Copy link
Owner Author

rom1504 commented Feb 10, 2022

most likely the problem is getting throttled by the instance metadata service because opening thousands of connections
solutions:

  • open less connections
  • somehow share the credential info between connection
  • increase the timeout

opening only 32 connections and keeping them open for a sharding would probably avoid this problem

still doesn't explain why this happens only after 6h

@rom1504
Copy link
Owner Author

rom1504 commented Feb 10, 2022

it seems that 6f5ed17 worked
not a great fix though

@rom1504
Copy link
Owner Author

rom1504 commented Feb 12, 2022

mostly fixed, next steps will happen in #85

@rom1504 rom1504 closed this as completed Feb 12, 2022
@PranshuBansalDev PranshuBansalDev mentioned this issue Mar 30, 2022
Closed
@MarkRoss-Eviden MarkRoss-Eviden mentioned this issue May 11, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rom1504