Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

After chaning S3_PATH backups fail with header mismatch #833

Closed
tman5 opened this issue Feb 15, 2024 · 16 comments
Closed

After chaning S3_PATH backups fail with header mismatch #833

tman5 opened this issue Feb 15, 2024 · 16 comments
Assignees
@Slach

Comments

@tman5
Copy link

tman5 commented Feb 15, 2024

After changing the S3_PATH variable from backups/shard-{shard} to backups/ourapp/environment/shard-{shard} backups are now failing:

error	one of upload table go-routine return error: can\'t upload: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 17B416D892BADB25, HostID: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, api error XAmzContentSHA256Mismatch: The provided \'x-amz-content-sha256\' header does not match what was computed.
@Slach
Copy link
Collaborator

Slach commented Feb 15, 2024

is rollback S3_PATH fix situation?

S3_PATH is just a prefix, nothing else and not affects

Could you share the following commands result, Without sensitive data?

clickhouse-backup version
clickhouse-backup print-config

@tman5
Copy link
Author

tman5 commented Feb 15, 2024

Putting it back doesn't fix it. I even blew away all the contents of that S3 bucket.

Version:	 2.4.30
Git Commit:	 11feb12d60c8c8c7f52478f0550c65630227f252
Build Date:	 2024-02-12

general:
    remote_storage: s3
    max_file_size: 0
    disable_progress_bar: true
    backups_to_keep_local: 0
    backups_to_keep_remote: 3
    log_level: debug
    allow_empty_backups: true
    download_concurrency: 4
    upload_concurrency: 2
    use_resumable_state: true
    restore_schema_on_cluster: ""
    upload_by_part: true
    download_by_part: true
    restore_database_mapping: {}
    retries_on_failure: 3
    retries_pause: 30s
    watch_interval: 1h
    full_interval: 24h
    watch_backup_name_template: shard{shard}-{type}-{time:20060102150405}
    sharded_operation_mode: ""
    cpu_nice_priority: 15
    io_nice_priority: idle
    retriesduration: 30s
    watchduration: 1h0m0s
    fullduration: 24h0m0s
clickhouse:
    username: default
    password: featbit
    host: localhost
    port: 9000
    disk_mapping: {}
    skip_tables:
        - system.*
        - INFORMATION_SCHEMA.*
        - information_schema.*
        - _temporary_and_external_tables.*
    skip_table_engines: []
    timeout: 5m
    freeze_by_part: false
    freeze_by_part_where: ""
    use_embedded_backup_restore: false
    embedded_backup_disk: ""
    backup_mutations: true
    restore_as_attach: false
    check_parts_columns: true
    secure: false
    skip_verify: false
    sync_replicated_tables: false
    log_sql_queries: true
    config_dir: /etc/clickhouse-server/
    restart_command: exec:systemctl restart clickhouse-server
    ignore_not_exists_error_during_freeze: true
    check_replicas_before_attach: true
    tls_key: ""
    tls_cert: ""
    tls_ca: ""
    max_connections: 4
    debug: false
s3:
    access_key: clickhouse_backups_rw
    secret_key: secret-key
    bucket: clickhouse-backups
    endpoint: https://minio.dev.site.com
    region: us-east-1
    acl: public
    assume_role_arn: ""
    force_path_style: true
    path: backup/shard-{shard}
    object_disk_path: ""
    disable_ssl: false
    compression_level: 1
    compression_format: tar
    sse: ""
    sse_kms_key_id: ""
    sse_customer_algorithm: ""
    sse_customer_key: ""
    sse_customer_key_md5: ""
    sse_kms_encryption_context: ""
    disable_cert_verification: true
    use_custom_storage_class: false
    storage_class: STANDARD
    custom_storage_class_map: {}
    concurrency: 5
    part_size: 0
    max_parts_count: 2000
    allow_multipart_download: false
    object_labels: {}
    request_payer: ""
    debug: false
gcs:
    credentials_file: ""
    credentials_json: ""
    credentials_json_encoded: ""
    bucket: ""
    path: ""
    object_disk_path: ""
    compression_level: 1
    compression_format: tar
    debug: false
    force_http: false
    endpoint: ""
    storage_class: STANDARD
    object_labels: {}
    custom_storage_class_map: {}
    client_pool_size: 12
cos:
    url: ""
    timeout: 2m
    secret_id: ""
    secret_key: ""
    path: ""
    compression_format: tar
    compression_level: 1
    debug: false
api:
    listen: 0.0.0.0:7171
    enable_metrics: true
    enable_pprof: false
    username: ""
    password: ""
    secure: false
    certificate_file: ""
    private_key_file: ""
    ca_cert_file: ""
    ca_key_file: ""
    create_integration_tables: true
    integration_tables_host: ""
    allow_parallel: false
    complete_resumable_after_restart: true
ftp:
    address: ""
    timeout: 2m
    username: ""
    password: ""
    tls: false
    skip_tls_verify: false
    path: ""
    object_disk_path: ""
    compression_format: tar
    compression_level: 1
    concurrency: 12
    debug: false
sftp:
    address: ""
    port: 22
    username: ""
    password: ""
    key: ""
    path: ""
    object_disk_path: ""
    compression_format: tar
    compression_level: 1
    concurrency: 12
    debug: false
azblob:
    endpoint_schema: https
    endpoint_suffix: core.windows.net
    account_name: ""
    account_key: ""
    sas: ""
    use_managed_identity: false
    container: ""
    path: ""
    object_disk_path: ""
    compression_level: 1
    compression_format: tar
    sse_key: ""
    buffer_size: 0
    buffer_count: 3
    max_parts_count: 256
    timeout: 4h
    debug: false
custom:
    upload_command: ""
    download_command: ""
    list_command: ""
    delete_command: ""
    command_timeout: 4h
    commandtimeoutduration: 4h0m0s

@warhaus-kc
Copy link

I'm experiencing the same error during upload to MinIO. I updated from clickhouse-backup version 2.3.0 to 2.4.30 and the upload stopped working. If I revert back to 2.3.0, everything works again.

Here's my config and version:

Version:	 2.4.30
Git Commit:	 11feb12d60c8c8c7f52478f0550c65630227f252
Build Date:	 2024-02-12

general:
    remote_storage: s3
    max_file_size: 0
    disable_progress_bar: true
    backups_to_keep_local: 1
    backups_to_keep_remote: 14
    log_level: debug
    allow_empty_backups: true
    download_concurrency: 1
    upload_concurrency: 1
    use_resumable_state: true
    restore_schema_on_cluster: ""
    upload_by_part: true
    download_by_part: true
    restore_database_mapping: {}
    retries_on_failure: 3
    retries_pause: 30s
    watch_interval: 1h
    full_interval: 24h
    watch_backup_name_template: shard{shard}-{type}-{time:20060102150405}
    sharded_operation_mode: ""
    cpu_nice_priority: 15
    io_nice_priority: idle
    retriesduration: 30s
    watchduration: 1h0m0s
    fullduration: 24h0m0s
clickhouse:
    username: default
    password: ""
    host: localhost
    port: 9000
    disk_mapping: {}
    skip_tables:
        - system.*
        - INFORMATION_SCHEMA.*
        - information_schema.*
        - _temporary_and_external_tables.*
    skip_table_engines: []
    timeout: 5m
    freeze_by_part: false
    freeze_by_part_where: ""
    use_embedded_backup_restore: false
    embedded_backup_disk: ""
    backup_mutations: true
    restore_as_attach: false
    check_parts_columns: true
    secure: false
    skip_verify: false
    sync_replicated_tables: false
    log_sql_queries: true
    config_dir: /etc/clickhouse-server/
    restart_command: exec:systemctl restart clickhouse-server
    ignore_not_exists_error_during_freeze: true
    check_replicas_before_attach: true
    tls_key: ""
    tls_cert: ""
    tls_ca: ""
    max_connections: 1
    debug: false
s3:
    access_key: hidden_key
    secret_key: hidden_key
    bucket: tpl-ch-backups
    endpoint: https://s3.example.com
    region: us-east-1
    acl: private
    assume_role_arn: ""
    force_path_style: true
    path: clickhouse
    object_disk_path: ""
    disable_ssl: true
    compression_level: 1
    compression_format: tar
    sse: ""
    sse_kms_key_id: ""
    sse_customer_algorithm: ""
    sse_customer_key: ""
    sse_customer_key_md5: ""
    sse_kms_encryption_context: ""
    disable_cert_verification: false
    use_custom_storage_class: false
    storage_class: STANDARD
    custom_storage_class_map: {}
    concurrency: 1
    part_size: 104857600
    max_parts_count: 2000
    allow_multipart_download: false
    object_labels: {}
    request_payer: ""
    debug: false
gcs:
    credentials_file: ""
    credentials_json: ""
    credentials_json_encoded: ""
    bucket: ""
    path: ""
    object_disk_path: ""
    compression_level: 1
    compression_format: tar
    debug: false
    force_http: false
    endpoint: ""
    storage_class: STANDARD
    object_labels: {}
    custom_storage_class_map: {}
    client_pool_size: 3
cos:
    url: ""
    timeout: 2m
    secret_id: ""
    secret_key: ""
    path: ""
    compression_format: tar
    compression_level: 1
    debug: false
api:
    listen: 0.0.0.0:7171
    enable_metrics: true
    enable_pprof: false
    username: ""
    password: ""
    secure: false
    certificate_file: ""
    private_key_file: ""
    ca_cert_file: ""
    ca_key_file: ""
    create_integration_tables: true
    integration_tables_host: ""
    allow_parallel: false
    complete_resumable_after_restart: true
ftp:
    address: ""
    timeout: 2m
    username: ""
    password: ""
    tls: false
    skip_tls_verify: false
    path: ""
    object_disk_path: ""
    compression_format: tar
    compression_level: 1
    concurrency: 3
    debug: false
sftp:
    address: ""
    port: 22
    username: ""
    password: ""
    key: ""
    path: ""
    object_disk_path: ""
    compression_format: tar
    compression_level: 1
    concurrency: 3
    debug: false
azblob:
    endpoint_schema: https
    endpoint_suffix: core.windows.net
    account_name: ""
    account_key: ""
    sas: ""
    use_managed_identity: false
    container: ""
    path: ""
    object_disk_path: ""
    compression_level: 1
    compression_format: tar
    sse_key: ""
    buffer_size: 0
    buffer_count: 3
    max_parts_count: 256
    timeout: 4h
    debug: false
custom:
    upload_command: ""
    download_command: ""
    list_command: ""
    delete_command: ""
    command_timeout: 4h
    commandtimeoutduration: 4h0m0s

@Slach
Copy link
Collaborator

Slach commented Feb 16, 2024

Is your provider minio or `s3? or both? Do you have the same error on minio and s3?
Could you share:

S3_DEBUG=1 clickhouse-backup upload <backup_name> result?

@Slach
Copy link
Collaborator

Slach commented Feb 16, 2024

something wrong with your s3 section of config

lets remove anything from s3 and stay only

s3:
    access_key: hidden_key
    secret_key: hidden_key
    bucket: tpl-ch-backups
    endpoint: https://s3.example.com
    force_path_style: true
    path: clickhouse

if error the same, please share logs with S3_DEBUG=1

@Slach Slach added this to the 2.4.31 milestone Feb 16, 2024
@Slach Slach self-assigned this Feb 16, 2024
@warhaus-kc
Copy link

with this config error the same

s3:
    access_key: hidden_key
    secret_key: hidden_key
    bucket: tpl-ch-backups
    endpoint: https://s3.example.com
    force_path_style: true
    path: clickhouse

I only use MinIO.

Debug Log

2024/02/16 15:27:34.684213  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/02/16 15:27:34.704998  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/02/16 15:27:34.705087  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/02/16 15:27:34.721175  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/02/16 15:27:34.732301  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/02/16 15:27:34.788715  info SELECT max(toInt64(bytes_on_disk * 1.02)) AS max_file_size FROM system.parts logger=clickhouse
2024/02/16 15:27:34.795203  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/02/16 15:27:34.800347  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/02/16 15:27:34.803364  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/02/16 15:27:34.807464  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/02/16 15:27:34.814220  info [s3:DEBUG] Request
GET /mwl-stage-ch-backups?versioning= HTTP/1.1
Host: minio.example.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.21.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: ab461089-794c-4e06-bb98-2d5f2cc1a3b2
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=hrwRGtmwFAw1sFVG/20240216/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=f346fd37ceeae167f37a65246a374d1a7388ec7bca2c6f2b0b6ff70dcf4dd9a1
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240216T152734Z


2024/02/16 15:27:35.001738  info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 99
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Fri, 16 Feb 2024 15:27:34 GMT
Server: nginx/1.20.1
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17B461674D975783
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/02/16 15:27:35.062032 debug /tmp/.clickhouse-backup-metadata.cache.S3 load 15 elements logger=s3
2024/02/16 15:27:35.067112  info [s3:DEBUG] Request
GET /mwl-stage-ch-backups?delimiter=%2F&list-type=2&max-keys=1000&prefix=clickhouse%2F HTTP/1.1
Host: minio.example.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.21.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: d9e654fa-301a-434b-87ce-f5303c3e504d
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=hrwRGtmwFAw1sFVG/20240216/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=df9308e3abca2de5641072fa77929728e673033b3727f97af23726547a6f7292
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240216T152735Z


2024/02/16 15:27:35.083588  info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 2222
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Fri, 16 Feb 2024 15:27:35 GMT
Server: nginx/1.20.1
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17B4616752864387
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/02/16 15:27:35.120310 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 15 elements logger=s3
2024/02/16 15:27:35.238688 debug prepare table concurrent semaphore with concurrency=1 len(tablesForUpload)=576 backup=chi-stage-ch-mwl-cluster-stage-mwl-0-0-full-2024-02-16-14-46-47 operation=upload
2024/02/16 15:27:35.239118 debug start project_example1_test1.mailgun_event with concurrency=1 len(table.Parts[...])=0 logger=uploadTableData
2024/02/16 15:27:35.239147 debug finish project_example1_test1.mailgun_event with concurrency=1 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/02/16 15:27:35.297167  info [s3:DEBUG] Request
PUT /mwl-stage-ch-backups/clickhouse/chi-stage-ch-mwl-cluster-stage-mwl-0-0-full-2024-02-16-14-46-47/metadata/project_example1_test1/mailgun_event.json?x-id=PutObject HTTP/1.1
Host: minio.example.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.21.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer
Content-Length: 511
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: bc1fa1f9-d962-4934-bbbf-ceff239bad3b
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=hrwRGtmwFAw1sFVG/20240216/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-storage-class;x-amz-trailer, Signature=82648e68b37f5ed1caff86393809a4e8290dd7eb2929a2069c64fa39d01c47d3
Content-Encoding: aws-chunked
Content-Type: application/octet-stream
X-Amz-Acl: private
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20240216T152735Z
X-Amz-Decoded-Content-Length: 468
X-Amz-Sdk-Checksum-Algorithm: CRC32
X-Amz-Storage-Class: STANDARD
X-Amz-Trailer: x-amz-checksum-crc32


2024/02/16 15:27:35.317375  info [s3:DEBUG] Response
HTTP/1.1 400 Bad Request
Content-Length: 620
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Fri, 16 Feb 2024 15:27:35 GMT
Server: nginx/1.20.1
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17B46167603E2D28
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/02/16 15:27:35.317582  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 400, RequestID: 17B46167603E2D28, HostID: c53f04d1-670f-4241-9f5c-2d6227cef7a5, api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.
2024/02/16 15:28:05.323694  info [s3:DEBUG] Request
PUT /mwl-stage-ch-backups/clickhouse/chi-stage-ch-mwl-cluster-stage-mwl-0-0-full-2024-02-16-14-46-47/metadata/project_example1_test1/mailgun_event.json?x-id=PutObject HTTP/1.1
Host: minio.example.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.21.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer
Content-Length: 511
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: d3bc9f6c-6f25-41de-8ab9-4c4b55b4592f
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=hrwRGtmwFAw1sFVG/20240216/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-storage-class;x-amz-trailer, Signature=3ef6693e5b877c26a5c88fe25b837bfd528117c90b9beaefcbf345e75afec349
Content-Encoding: aws-chunked
Content-Type: application/octet-stream
X-Amz-Acl: private
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20240216T152805Z
X-Amz-Decoded-Content-Length: 468
X-Amz-Sdk-Checksum-Algorithm: CRC32
X-Amz-Storage-Class: STANDARD
X-Amz-Trailer: x-amz-checksum-crc32


2024/02/16 15:28:05.348188  info [s3:DEBUG] Response
HTTP/1.1 400 Bad Request
Content-Length: 620
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Fri, 16 Feb 2024 15:28:05 GMT
Server: nginx/1.20.1
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17B4616E5DFFEEAC
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/02/16 15:28:05.348369  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 400, RequestID: 17B4616E5DFFEEAC, HostID: c53f04d1-670f-4241-9f5c-2d6227cef7a5, api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.
2024/02/16 15:28:35.383558  info [s3:DEBUG] Request
PUT /mwl-stage-ch-backups/clickhouse/chi-stage-ch-mwl-cluster-stage-mwl-0-0-full-2024-02-16-14-46-47/metadata/project_example1_test1/mailgun_event.json?x-id=PutObject HTTP/1.1
Host: minio.example.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.21.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer
Content-Length: 511
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 6304662e-1837-4d64-86ef-6bbbe6761d95
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=hrwRGtmwFAw1sFVG/20240216/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-storage-class;x-amz-trailer, Signature=e35d5de8b33cc8eadf3137a0eaa5706b435e390ce6715a2d21fc0e0059a8724e
Content-Encoding: aws-chunked
Content-Type: application/octet-stream
X-Amz-Acl: private
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20240216T152835Z
X-Amz-Decoded-Content-Length: 468
X-Amz-Sdk-Checksum-Algorithm: CRC32
X-Amz-Storage-Class: STANDARD
X-Amz-Trailer: x-amz-checksum-crc32


2024/02/16 15:28:35.401208  info [s3:DEBUG] Response
HTTP/1.1 400 Bad Request
Content-Length: 620
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Fri, 16 Feb 2024 15:28:35 GMT
Server: nginx/1.20.1
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17B461755DB6FE2C
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/02/16 15:28:35.401444  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 400, RequestID: 17B461755DB6FE2C, HostID: c53f04d1-670f-4241-9f5c-2d6227cef7a5, api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.
2024/02/16 15:29:05.496149  info [s3:DEBUG] Request
PUT /mwl-stage-ch-backups/clickhouse/chi-stage-ch-mwl-cluster-stage-mwl-0-0-full-2024-02-16-14-46-47/metadata/project_example1_test1/mailgun_event.json?x-id=PutObject HTTP/1.1
Host: minio.example.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.21.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer
Content-Length: 511
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: dc2187ce-d4eb-4f6a-bb1f-59bc98418205
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=hrwRGtmwFAw1sFVG/20240216/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-storage-class;x-amz-trailer, Signature=5970482314d87d2950e46c9fc92df465815e6335a20343c97083dd04114f21b4
Content-Encoding: aws-chunked
Content-Type: application/octet-stream
X-Amz-Acl: private
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20240216T152905Z
X-Amz-Decoded-Content-Length: 468
X-Amz-Sdk-Checksum-Algorithm: CRC32
X-Amz-Storage-Class: STANDARD
X-Amz-Trailer: x-amz-checksum-crc32


2024/02/16 15:29:05.511843  info [s3:DEBUG] Response
HTTP/1.1 400 Bad Request
Content-Length: 620
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Fri, 16 Feb 2024 15:29:05 GMT
Server: nginx/1.20.1
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17B4617C6090E882
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/02/16 15:29:05.512001  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 400, RequestID: 17B4617C6090E882, HostID: c53f04d1-670f-4241-9f5c-2d6227cef7a5, api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.
2024/02/16 15:29:05.512087 debug start project_example2_test2.zxcf_ef_event with concurrency=1 len(table.Parts[...])=0 logger=uploadTableData
2024/02/16 15:29:05.512111 debug finish project_example2_test2.zxcf_ef_reel_event with concurrency=1 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/02/16 15:29:05.512217 debug start project_example2_test2.migrations_versions with concurrency=1 len(table.Parts[...])=2 logger=uploadTableData
2024/02/16 15:29:05.512436 debug start upload 7 files to chi-stage-ch-mwl-cluster-stage-mwl-0-0-full-2024-02-16-14-46-47/shadow/project_example2_test2/migrations_versions/default_all_0_14_3.tar logger=uploadTableData
2024/02/16 15:29:05.512665 error UploadCompressedStream return error: context canceled logger=uploadTableData
2024/02/16 15:29:05.512703 debug start upload 7 files to chi-stage-ch-mwl-cluster-stage-mwl-0-0-full-2024-02-16-14-46-47/shadow/project_example2_test2/migrations_versions/default_all_15_15_0.tar logger=uploadTableData
2024/02/16 15:29:05.512838 error UploadCompressedStream return error: context canceled logger=uploadTableData
2024/02/16 15:29:05.512883 debug start project_example3_stage_stage.cpa_log with concurrency=1 len(table.Parts[...])=2 logger=uploadTableData
2024/02/16 15:29:05.513084 debug start upload 7 files to chi-stage-ch-mwl-cluster-stage-mwl-0-0-full-2024-02-16-14-46-47/shadow/project_example3_stage/cpa_log/default_all_0_0_0.tar logger=uploadTableData
2024/02/16 15:29:05.513235 error UploadCompressedStream return error: context canceled logger=uploadTableData

@Slach
Copy link
Collaborator

Slach commented Feb 16, 2024

I see

Server: nginx/1.20.1

in logs

could you try to connect to minio directly without nginx in the middle?

@Slach
Copy link
Collaborator

Slach commented Feb 16, 2024

which minio version do you use?

@tman5
Copy link
Author

tman5 commented Feb 16, 2024

I'm using it in Kubernetes and this is the ENV variables set for the backup pod

- name: LOG_LEVEL
                  value: "debug"
                - name: ALLOW_EMPTY_BACKUPS
                  value: "true"
                - name: API_LISTEN
                  value: "0.0.0.0:7171"
                # INSERT INTO system.backup_actions to execute backup
                - name: API_CREATE_INTEGRATION_TABLES
                  value: "true"
                - name: BACKUPS_TO_KEEP_REMOTE
                  value: "3"
                # change it for production S3
                - name: REMOTE_STORAGE
                  value: "s3"
                - name: S3_ACL
                  value: "private"
                - name: S3_ENDPOINT
                  value: https://minio.dev.site.local
                - name: S3_BUCKET
                  value: clickhouse-backups
                # {shard} macro defined by clickhouse-operator
                - name: S3_PATH
                  value: backup/shard-{shard}
                - name: S3_ACCESS_KEY
                  value: access-key
                - name: S3_DISABLE_CERT_VERIFICATION
                  value: "true"
                - name: S3_SECRET_KEY
                   value: secret_key
                - name: S3_FORCE_PATH_STYLE
                  value: "true"

Debug output:

2024/02/16 15:36:18.924501  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/02/16 15:36:18.925786  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/02/16 15:36:18.925837  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/02/16 15:36:18.927464  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/02/16 15:36:18.930731  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/02/16 15:36:18.938078  info SELECT max(toInt64(bytes_on_disk * 1.02)) AS max_file_size FROM system.parts logger=clickhouse
2024/02/16 15:36:18.941548  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/02/16 15:36:18.945921  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/02/16 15:36:18.950712  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/02/16 15:36:18.953304  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/02/16 15:36:18.955462  info [s3:DEBUG] Request
GET /clickhouse-backups?versioning= HTTP/1.1
Host: minio.dev.clarkinc.biz
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.21.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 00f0acb5-2468-4a50-b758-837957201413
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=clickhouse_backups_rw/20240216/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=93153c5879f3788296de0981bcc3b8678681fed0af6220ad3cadab47af0f0fce
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240216T153618Z


2024/02/16 15:36:18.981285  info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 99
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Fri, 16 Feb 2024 15:36:18 GMT
Strict-Transport-Security: max-age=300; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Request-Id: 17B461E14D606B03
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/02/16 15:36:18.981547 debug /tmp/.clickhouse-backup-metadata.cache.S3 load 0 elements logger=s3
2024/02/16 15:36:18.982158  info [s3:DEBUG] Request
GET /clickhouse-backups?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Ffeatbit%2Fdev2-generic%2Fshard-0%2F HTTP/1.1
Host: minio.dev.clarkinc.biz
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.21.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: afabb12f-93ba-4e6b-9439-953b556bb8fa
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=clickhouse_backups_rw/20240216/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=c3774bba372b2d88c1739963255919f95c363d021bc4d6675dc095de42b90b09
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240216T153618Z


2024/02/16 15:36:18.988122  info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 309
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Fri, 16 Feb 2024 15:36:18 GMT
Strict-Transport-Security: max-age=300; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Request-Id: 17B461E14DEB9EA4
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/02/16 15:36:18.988579 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/02/16 15:36:18.988921 debug prepare table concurrent semaphore with concurrency=2 len(tablesForUpload)=0 backup=chi-clickhouse-replicated-0-0-full-2024-02-16-03-01-40 operation=upload
2024/02/16 15:36:18.991216  info [s3:DEBUG] Request
PUT /clickhouse-backups/backup/featbit/dev2-generic/shard-0/chi-clickhouse-replicated-0-0-full-2024-02-16-03-01-40/metadata.json?x-id=PutObject HTTP/1.1
Host: minio.dev.clarkinc.biz
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.21.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer
Content-Length: 558
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 9d816620-36cb-44a9-af03-777a9ede4b3c
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=clickhouse_backups_rw/20240216/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-storage-class;x-amz-trailer, Signature=8e13bfae454160c92de51d743ae28661e4356e66f4b25102b7a8696c168557c4
Content-Encoding: aws-chunked
Content-Type: application/octet-stream
X-Amz-Acl: public
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20240216T153618Z
X-Amz-Decoded-Content-Length: 515
X-Amz-Sdk-Checksum-Algorithm: CRC32
X-Amz-Storage-Class: STANDARD
X-Amz-Trailer: x-amz-checksum-crc32


2024/02/16 15:36:18.998166  info [s3:DEBUG] Response
HTTP/1.1 400 Bad Request
Content-Length: 644
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Fri, 16 Feb 2024 15:36:18 GMT
Strict-Transport-Security: max-age=300; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Request-Id: 17B461E14EC54A33
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/02/16 15:36:18.998366  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 400, RequestID: 17B461E14EC54A33, HostID: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.
2024/02/16 15:36:49.000270  info [s3:DEBUG] Request
PUT /clickhouse-backups/backup/featbit/dev2-generic/shard-0/chi-clickhouse-replicated-0-0-full-2024-02-16-03-01-40/metadata.json?x-id=PutObject HTTP/1.1
Host: minio.dev.clarkinc.biz
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.21.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer
Content-Length: 558
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 2853f743-8b34-4232-b982-14ff7d1e59ee
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=clickhouse_backups_rw/20240216/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-storage-class;x-amz-trailer, Signature=8f9db36eec4358ee1900a91441ef45adac273154228e5e9ccb35476eee43099b
Content-Encoding: aws-chunked
Content-Type: application/octet-stream
X-Amz-Acl: public
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20240216T153649Z
X-Amz-Decoded-Content-Length: 515
X-Amz-Sdk-Checksum-Algorithm: CRC32
X-Amz-Storage-Class: STANDARD
X-Amz-Trailer: x-amz-checksum-crc32


2024/02/16 15:36:49.002913  info [s3:DEBUG] Response
HTTP/1.1 400 Bad Request
Content-Length: 644
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Fri, 16 Feb 2024 15:36:49 GMT
Strict-Transport-Security: max-age=300; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Request-Id: 17B461E84B336D06
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/02/16 15:36:49.003217  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 400, RequestID: 17B461E84B336D06, HostID: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.
2024/02/16 15:37:19.005449  info [s3:DEBUG] Request
PUT /clickhouse-backups/backup/featbit/dev2-generic/shard-0/chi-clickhouse-replicated-0-0-full-2024-02-16-03-01-40/metadata.json?x-id=PutObject HTTP/1.1
Host: minio.dev.clarkinc.biz
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.21.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer
Content-Length: 558
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 75c9d344-56ba-4b7c-9125-070e06463fe2
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=clickhouse_backups_rw/20240216/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-storage-class;x-amz-trailer, Signature=0dde68786fbd16fe8beed0b7357271fd685bd70368ed45e7ce9afac18e4a50bc
Content-Encoding: aws-chunked
Content-Type: application/octet-stream
X-Amz-Acl: public
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20240216T153719Z
X-Amz-Decoded-Content-Length: 515
X-Amz-Sdk-Checksum-Algorithm: CRC32
X-Amz-Storage-Class: STANDARD
X-Amz-Trailer: x-amz-checksum-crc32


2024/02/16 15:37:19.008604  info [s3:DEBUG] Response
HTTP/1.1 400 Bad Request
Content-Length: 644
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Fri, 16 Feb 2024 15:37:19 GMT
Strict-Transport-Security: max-age=300; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Request-Id: 17B461EF47AEF7DB
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/02/16 15:37:19.008841  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 400, RequestID: 17B461EF47AEF7DB, HostID: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.
2024/02/16 15:37:49.011143  info [s3:DEBUG] Request
PUT /clickhouse-backups/backup/featbit/dev2-generic/shard-0/chi-clickhouse-replicated-0-0-full-2024-02-16-03-01-40/metadata.json?x-id=PutObject HTTP/1.1
Host: minio.dev.clarkinc.biz
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.21.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer
Content-Length: 558
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 98061033-b72f-417e-b682-7c31533dfa59
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=clickhouse_backups_rw/20240216/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-storage-class;x-amz-trailer, Signature=aa3f2d1d1cef4d270c977459c6d83d6a86bc9687ea8983a0c79d45a94d3522c6
Content-Encoding: aws-chunked
Content-Type: application/octet-stream
X-Amz-Acl: public
X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
X-Amz-Date: 20240216T153749Z
X-Amz-Decoded-Content-Length: 515
X-Amz-Sdk-Checksum-Algorithm: CRC32
X-Amz-Storage-Class: STANDARD
X-Amz-Trailer: x-amz-checksum-crc32


2024/02/16 15:37:49.014884  info [s3:DEBUG] Response
HTTP/1.1 400 Bad Request
Content-Length: 644
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Fri, 16 Feb 2024 15:37:49 GMT
Strict-Transport-Security: max-age=300; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Request-Id: 17B461F6442AA53B
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/02/16 15:37:49.015108  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 400, RequestID: 17B461F6442AA53B, HostID: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.
2024/02/16 15:37:49.015245  info clickhouse connection closed logger=clickhouse
2024/02/16 15:37:49.015270 error can't upload chi-clickhouse-replicated-0-0-full-2024-02-16-03-01-40/metadata.json: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 17B461F6442AA53B, HostID: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.

minio version: 2023-04-28T18:11:17Z

@Slach
Copy link
Collaborator

Slach commented Feb 16, 2024

could you check altinity/clickhouse-backup:2.4.29 or altinity/clickhouse-backup:master, looks like after we add crc32 in #829
and rollbac this options for #829 should help

@Slach Slach modified the milestones: 2.4.31, 2.5.0 Feb 16, 2024
@warhaus-kc
Copy link

Yes, everything works perfectly on the altinity/clickhouse-backup:2.4.29 version.
Thank you for your help!

@tman5
Copy link
Author

tman5 commented Feb 16, 2024

Confirmed 2.4.29 works for us too

@Slach
Copy link
Collaborator

Slach commented Feb 17, 2024

@tman5 @warhaus-kc please confirm, is 2.4.31 works good for you?

@Slach Slach closed this as completed Feb 17, 2024
@warhaus-kc
Copy link

2.4.31 works for me

@tman5
Copy link
Author

tman5 commented Feb 21, 2024

Confirmed works for me too. Thanks!

@UnamedRus
Copy link

Looks like related minio/minio#17662

@Slach Slach removed this from the 2.5.0 milestone Feb 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Slach Slach

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Slach @UnamedRus @tman5 @warhaus-kc