Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crc64nvme not supported on object uploads #2998

Closed
2 of 3 tasks
niksis02 opened this issue Feb 3, 2025 · 3 comments
Closed
2 of 3 tasks

Crc64nvme not supported on object uploads #2998

niksis02 opened this issue Feb 3, 2025 · 3 comments
Labels
bug This issue is a bug. needs-triage This issue or PR still needs to be triaged.

Comments

@niksis02
Copy link

niksis02 commented Feb 3, 2025

Acknowledgements

Describe the bug

The SDK doesn't support CRC64-NVMe checksum calculations for object uploads.

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

The SDK should properly validate types.ChecksumAlgorithmCrc64NVMe as it is in the supported list of algorithms and should add the required headers to the request:

  • x-amz-sdk-checksum-algorithm: CRC64NVME
  • x-amz-checksum-crc64nvme: <calculated checksum>

The supported algorithms list from the source:

// Algorithm represents the checksum algorithms supported
type Algorithm string

// Enumeration values for supported checksum Algorithms.
const (
	// AlgorithmCRC32C represents CRC32C hash algorithm
	AlgorithmCRC32C Algorithm = "CRC32C"

	// AlgorithmCRC32 represents CRC32 hash algorithm
	AlgorithmCRC32 Algorithm = "CRC32"

	// AlgorithmSHA1 represents SHA1 hash algorithm
	AlgorithmSHA1 Algorithm = "SHA1"

	// AlgorithmSHA256 represents SHA256 hash algorithm
	AlgorithmSHA256 Algorithm = "SHA256"

	// AlgorithmCRC64NVME represents CRC64NVME hash algorithm
	AlgorithmCRC64NVME Algorithm = "CRC64NVME"
)

Current Behavior

Currently the checksum type validation fails with the error message: operation error S3: PutObject, failed to parse algorithm, unknown checksum algorithm, CRC64NVME

Reproduction Steps

package main

import (
	"bytes"
	"context"
	"crypto/rand"
	"log"
	"time"

	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/aws/aws-sdk-go-v2/service/s3/types"
)

func main() {
	cfg, err := config.LoadDefaultConfig(context.TODO())
	if err != nil {
		log.Fatal(err)
	}
	client := s3.NewFromConfig(cfg)
	bucket := "my-bucket"
	key := "my-key"

	data := make([]byte, 100)
	rand.Read(data)

	rdr := bytes.NewReader(data)

	ctx, cancel := context.WithTimeout(context.Background(), time.Second*100)
	_, err = client.PutObject(ctx, &s3.PutObjectInput{
		Bucket:            &bucket,
		Key:               &key,
		Body:              rdr,
		ChecksumAlgorithm: types.ChecksumAlgorithmCrc64nvme,
	})
	cancel()
	if err != nil {
		log.Fatalln(err)
	}
}

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2@v1.36.0 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2@v1.36.0 go@1.21
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.6.8 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.6.8 go@1.21
github.com/aws/aws-sdk-go-v2/config@v1.29.4 github.com/aws/aws-sdk-go-v2@v1.36.0
github.com/aws/aws-sdk-go-v2/config@v1.29.4 github.com/aws/aws-sdk-go-v2/credentials@v1.17.57
github.com/aws/aws-sdk-go-v2/config@v1.29.4 github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.27
github.com/aws/aws-sdk-go-v2/config@v1.29.4 github.com/aws/aws-sdk-go-v2/internal/ini@v1.8.2
github.com/aws/aws-sdk-go-v2/config@v1.29.4 github.com/aws/aws-sdk-go-v2/service/sso@v1.24.14
github.com/aws/aws-sdk-go-v2/config@v1.29.4 github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.28.13
github.com/aws/aws-sdk-go-v2/config@v1.29.4 github.com/aws/aws-sdk-go-v2/service/sts@v1.33.12
github.com/aws/aws-sdk-go-v2/config@v1.29.4 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/config@v1.29.4 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.31
github.com/aws/aws-sdk-go-v2/config@v1.29.4 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.31
github.com/aws/aws-sdk-go-v2/config@v1.29.4 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.12.2
github.com/aws/aws-sdk-go-v2/config@v1.29.4 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.12.12
github.com/aws/aws-sdk-go-v2/config@v1.29.4 go@1.21
github.com/aws/aws-sdk-go-v2/credentials@v1.17.57 github.com/aws/aws-sdk-go-v2@v1.36.0
github.com/aws/aws-sdk-go-v2/credentials@v1.17.57 github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.27
github.com/aws/aws-sdk-go-v2/credentials@v1.17.57 github.com/aws/aws-sdk-go-v2/service/sso@v1.24.14
github.com/aws/aws-sdk-go-v2/credentials@v1.17.57 github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.28.13
github.com/aws/aws-sdk-go-v2/credentials@v1.17.57 github.com/aws/aws-sdk-go-v2/service/sts@v1.33.12
github.com/aws/aws-sdk-go-v2/credentials@v1.17.57 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/credentials@v1.17.57 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.31
github.com/aws/aws-sdk-go-v2/credentials@v1.17.57 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.31
github.com/aws/aws-sdk-go-v2/credentials@v1.17.57 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.12.2
github.com/aws/aws-sdk-go-v2/credentials@v1.17.57 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.12.12
github.com/aws/aws-sdk-go-v2/credentials@v1.17.57 go@1.21
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.27 github.com/aws/aws-sdk-go-v2@v1.36.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.27 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.27 go@1.21
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.31 github.com/aws/aws-sdk-go-v2@v1.36.0
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.31 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.31 go@1.21
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.31 github.com/aws/aws-sdk-go-v2@v1.36.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.31 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.31 go@1.21
github.com/aws/aws-sdk-go-v2/internal/ini@v1.8.2 go@1.21
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.31 github.com/aws/aws-sdk-go-v2@v1.36.0
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.31 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.31 go@1.21
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.12.2 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.12.2 go@1.21
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.5.5 github.com/aws/aws-sdk-go-v2@v1.36.0
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.5.5 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.12.12
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.5.5 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.5.5 go@1.21
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.12.12 github.com/aws/aws-sdk-go-v2@v1.36.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.12.12 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.12.12 go@1.21
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.18.12 github.com/aws/aws-sdk-go-v2@v1.36.0
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.18.12 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.18.12 go@1.21
github.com/aws/aws-sdk-go-v2/service/s3@v1.75.2 github.com/aws/aws-sdk-go-v2@v1.36.0
github.com/aws/aws-sdk-go-v2/service/s3@v1.75.2 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.6.8
github.com/aws/aws-sdk-go-v2/service/s3@v1.75.2 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.31
github.com/aws/aws-sdk-go-v2/service/s3@v1.75.2 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.31
github.com/aws/aws-sdk-go-v2/service/s3@v1.75.2 github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.31
github.com/aws/aws-sdk-go-v2/service/s3@v1.75.2 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.12.2
github.com/aws/aws-sdk-go-v2/service/s3@v1.75.2 github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.5.5
github.com/aws/aws-sdk-go-v2/service/s3@v1.75.2 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.12.12
github.com/aws/aws-sdk-go-v2/service/s3@v1.75.2 github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.18.12
github.com/aws/aws-sdk-go-v2/service/s3@v1.75.2 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/service/s3@v1.75.2 go@1.21
github.com/aws/aws-sdk-go-v2/service/sso@v1.24.14 github.com/aws/aws-sdk-go-v2@v1.36.0
github.com/aws/aws-sdk-go-v2/service/sso@v1.24.14 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.31
github.com/aws/aws-sdk-go-v2/service/sso@v1.24.14 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.31
github.com/aws/aws-sdk-go-v2/service/sso@v1.24.14 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/service/sso@v1.24.14 go@1.21
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.28.13 github.com/aws/aws-sdk-go-v2@v1.36.0
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.28.13 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.31
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.28.13 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.31
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.28.13 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.28.13 go@1.21
github.com/aws/aws-sdk-go-v2/service/sts@v1.33.12 github.com/aws/aws-sdk-go-v2@v1.36.0
github.com/aws/aws-sdk-go-v2/service/sts@v1.33.12 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.31
github.com/aws/aws-sdk-go-v2/service/sts@v1.33.12 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.31
github.com/aws/aws-sdk-go-v2/service/sts@v1.33.12 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.12.2
github.com/aws/aws-sdk-go-v2/service/sts@v1.33.12 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.12.12
github.com/aws/aws-sdk-go-v2/service/sts@v1.33.12 github.com/aws/smithy-go@v1.22.2
github.com/aws/aws-sdk-go-v2/service/sts@v1.33.12 go@1.21

Compiler and Version used

go version go1.23.1 darwin/arm64

Operating System and version

MacOS Sonoma Version 14.5
@niksis02 niksis02 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 3, 2025
@lucix-aws
Copy link
Contributor

#2985

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 3, 2025

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

1 similar comment
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 3, 2025

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

@niksis02 niksis02 changed the title (short issue description) Crc64nvme not supported on object uploads Feb 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue is a bug. needs-triage This issue or PR still needs to be triaged.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@niksis02 @lucix-aws