Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to disable SSL verification for S3Store #82

Merged
merged 5 commits into from
Aug 15, 2023
Merged

Add option to disable SSL verification for S3Store #82

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
06d8901
Add option to disable SSL verification.
janjagusch Jul 11, 2023
27ab8af
Update test to include verify=False.
janjagusch Jul 11, 2023
69457a6
Add entry to changelog.
janjagusch Jul 11, 2023
e0d3639
Merge remote-tracking branch 'upstream/main' into s3-disable-ssl-veri…
janjagusch Aug 8, 2023
7b40a87
Lowercase verify.
janjagusch Aug 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/changes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Changelog
1.8.0
=====
* Fixed the behaviour of ``S3FSStore`` when providing a custom endpoint.
* Added ``verify`` constructor argument to ``S3FSStore`` that disables SSL verification. Use it in an URI as ``?verify=False``.

1.7.0
=====
Expand Down
8 changes: 6 additions & 2 deletions minimalkv/net/s3fsstore.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ def __init__(
reduced_redundancy=False,
public=False,
metadata=None,
verify=True,
):
if isinstance(bucket, str):
import boto3
Expand All @@ -47,6 +48,7 @@ def __init__(
self.reduced_redundancy = reduced_redundancy
self.public = public
self.metadata = metadata or {}
self.verify = verify

# Get endpoint URL
self.endpoint_url = self.bucket.meta.client.meta.endpoint_url
Expand All @@ -66,7 +68,7 @@ def _create_filesystem(self) -> "S3FileSystem":
if not has_s3fs:
raise ImportError("Cannot find optional dependency s3fs.")

client_kwargs = {}
client_kwargs = {"verify": self.verify}
if self.endpoint_url:
client_kwargs["endpoint_url"] = self.endpoint_url

Expand Down Expand Up @@ -187,4 +189,6 @@ def _from_parsed_url(
# The bucket will be created in the `create_filesystem` method if it doesn't exist.
bucket = resource.Bucket(bucket_name)

return cls(bucket)
verify = query.get("verify", "true").lower() == "true"

return cls(bucket, verify=verify)
3 changes: 2 additions & 1 deletion tests/store_creation/test_creation_boto3store.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

storage = pytest.importorskip("google.cloud.storage")

S3_URL = "s3://minio:miniostorage@127.0.0.1:9000/bucketname?create_if_missing=true&is_secure=false"
S3_URL = "s3://minio:miniostorage@127.0.0.1:9000/bucketname?create_if_missing=true&is_secure=false&verify=False"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the F uppecase here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No particular reason. It makes no difference, but I've lowercawed it for consistency.


"""
When using the `s3` scheme in a URL, the new store creation returns an `S3FSStore`.
Expand All @@ -26,6 +26,7 @@ def test_new_s3fs_creation():
bucket_name="bucketname-minio",
is_secure=False,
),
verify=False,
)

actual = get_store_from_url(S3_URL)
Expand Down