Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added x-amz-checksum-sha256 header for Object Lock support #1393

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Added x-amz-checksum-sha256 header for Object Lock support #1393

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8677fc5
Added x-amz-checksum-sha256 header for Object Lock support
AlexisPPLIN Aug 22, 2024
71b8ab3
Fixed import of b64encode from base64
AlexisPPLIN Sep 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions S3/Crypto.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
# Python 2 support
from base64 import encodestring

from base64 import b64encode

from . import Config
from logging import debug
from .BaseUtils import encode_to_s3, decode_from_s3, s3_quote, md5, unicode
Expand Down Expand Up @@ -346,3 +348,11 @@ def calculateChecksum(buffer, mfile, offset, chunk_size, send_chunk):

return md5_hash.hexdigest()
__all__.append("calculateChecksum")

def sha256_hash_to_base64(sha256_hash):
# Extract digest from sha256 hash
sha256_hash_digest = sha256_hash.digest()
# Then convert it to base64
sha256_hash_digest_b64 = b64encode(sha256_hash_digest).decode()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this module encodestring (which imported above as an alias of encodebytes when available) is the function used for base64 encoding.
I think you can use:

sha256_hash_digest_b64 = encodestring(sha256_hash_digest).strip().decode()

to align with other usecases.

return sha256_hash_digest_b64
__all__.append("sha256_hash_to_base64")
7 changes: 6 additions & 1 deletion S3/S3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@
from .ConnMan import ConnMan
from .Crypto import (sign_request_v2, sign_request_v4, checksum_sha256_file,
checksum_sha256_buffer, generate_content_md5,
hash_file_md5, calculateChecksum, format_param_str)
hash_file_md5, calculateChecksum, format_param_str, sha256_hash_to_base64)

try:
from ctypes import ArgumentError
Expand Down Expand Up @@ -1848,6 +1848,11 @@ def send_file(self, request, stream, labels, buffer = '', throttle = 0,
sha256_hash = checksum_sha256_file(stream, offset, size_total)
request.body = sha256_hash

# Provide the checksum with the request. This is important for buckets that have
# Object Lock enabled.

headers['x-amz-checksum-sha256'] = sha256_hash_to_base64(sha256_hash)
Copy link
Contributor

@snosratiershad snosratiershad Jan 19, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As someone could use s3cmd as library this line can override the header potentially comes with request argument's headers attribute. I think it's better to check if it currently has values:

if not headers.get("x-amz-checksum-sha256"):
    headers['x-amz-checksum-sha256'] = sha256_hash_to_base64(sha256_hash)


if use_expect_continue:
if not size_total:
use_expect_continue = False
Expand Down