Uploading release files fails in v2.32.2

[egziko]

CLI Version

2.32.2

Operating System and Architecture

• macOS (arm64)
• macOS (x86_64)
• Linux (i686)
• Linux (x86_64)
• Linux (armv7)
• Linux (aarch64)
• Windows (i686)
• Windows (x86_64)

Operating System Version

AlmaLinux 9.3

Link to reproduction repository

No response

CLI Command

sentry-cli releases files upload ./dist --release ${env.shortCommitId} --project ${env.SENTRY_PROJECT} --log-level=debug

Exact Reproduction Steps

After building Angular app

npm i -g @sentry/cli
sentry-cli releases new ${env.shortCommitId} --project ${env.SENTRY_PROJECT}
sentry-cli releases set-commits ${env.shortCommitId} --auto
sentry-cli releases files upload ./dist --release ${env.shortCommitId} --project ${env.SENTRY_PROJECT} --log-level=debug

Works fine after downgrading to v2.32.1

Expected Results

Expected to upload release files successfully

Actual Results

Command fails with message

error: file could not be read as UTF-8
    caused by: stream did not contain valid UTF-8

Logs

11:28:14  + sentry-cli releases files upload ./dist --release 1dc8a34e --project redacted --log-level=debug
11:28:14    DEBUG   2024-07-22 10:28:14.509294659 +00:00 sentry-cli version: 2.32.2, platform: "linux", architecture: "x86_64"
11:28:14    INFO    2024-07-22 10:28:14.509416932 +00:00 sentry-cli was invoked with the following command line: "/var/jenkins_home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Nodejs_20/lib/node_modules/@sentry/cli/node_modules/@sentry/cli-linux-x64/bin/sentry-cli" "releases" "files" "upload" "./dist" "--release" "1dc8a34e" "--project" "redacted" "--log-level=debug"
11:28:14    DEBUG   2024-07-22 10:28:14.512979105 +00:00 request GET https://sentry.io/api/0/organizations/redacted/chunk-upload/
11:28:14    DEBUG   2024-07-22 10:28:14.513036471 +00:00 using token authentication
11:28:14    DEBUG   2024-07-22 10:28:14.513068439 +00:00 retry number 0, max retries: 0
11:28:14    DEBUG   2024-07-22 10:28:14.578379263 +00:00 > GET /api/0/organizations/redacted/chunk-upload/ HTTP/1.1
11:28:14    DEBUG   2024-07-22 10:28:14.578433014 +00:00 > Host: sentry.io
11:28:14    DEBUG   2024-07-22 10:28:14.578442097 +00:00 > Accept: */*
11:28:14    DEBUG   2024-07-22 10:28:14.578451320 +00:00 > Connection: TE
11:28:14    DEBUG   2024-07-22 10:28:14.578459753 +00:00 > TE: gzip
11:28:14    DEBUG   2024-07-22 10:28:14.578468326 +00:00 > User-Agent: sentry-cli/2.32.2
11:28:14    DEBUG   2024-07-22 10:28:14.579219661 +00:00 > Authorization: Bearer sntrys_e***
11:28:14    DEBUG   2024-07-22 10:28:14.755149625 +00:00 < HTTP/1.1 200 OK
11:28:14    DEBUG   2024-07-22 10:28:14.755212029 +00:00 < server: nginx
11:28:14    DEBUG   2024-07-22 10:28:14.755231909 +00:00 < date: Mon, 22 Jul 2024 10:28:14 GMT
11:28:14    DEBUG   2024-07-22 10:28:14.755250567 +00:00 < content-type: application/json
11:28:14    DEBUG   2024-07-22 10:28:14.755261323 +00:00 < vary: Accept-Encoding,Accept-Language, Cookie
11:28:14    DEBUG   2024-07-22 10:28:14.755271368 +00:00 < allow: GET, POST, HEAD, OPTIONS
11:28:14    DEBUG   2024-07-22 10:28:14.755279901 +00:00 < access-control-allow-methods: GET, POST, HEAD, OPTIONS
11:28:14    DEBUG   2024-07-22 10:28:14.755308654 +00:00 < access-control-allow-headers: X-Sentry-Auth, X-Requested-With, Origin, Accept, Content-Type, Authentication, Authorization, Content-Encoding, sentry-trace, baggage, X-CSRFToken
11:28:14    DEBUG   2024-07-22 10:28:14.755326691 +00:00 < access-control-expose-headers: X-Sentry-Error, X-Sentry-Direct-Hit, X-Hits, X-Max-Hits, Endpoint, Retry-After, Link
11:28:14    DEBUG   2024-07-22 10:28:14.755337898 +00:00 < access-control-allow-origin: *
11:28:14    DEBUG   2024-07-22 10:28:14.755347703 +00:00 < x-sentry-rate-limit-remaining: 39
11:28:14    DEBUG   2024-07-22 10:28:14.755358089 +00:00 < x-sentry-rate-limit-limit: 40
11:28:14    DEBUG   2024-07-22 10:28:14.755400432 +00:00 < x-sentry-rate-limit-reset: 1721644095
11:28:14    DEBUG   2024-07-22 10:28:14.755488242 +00:00 < x-sentry-rate-limit-concurrentremaining: 24
11:28:14    DEBUG   2024-07-22 10:28:14.755500922 +00:00 < x-sentry-rate-limit-concurrentlimit: 25
11:28:14    DEBUG   2024-07-22 10:28:14.755515173 +00:00 < content-language: en
11:28:14    DEBUG   2024-07-22 10:28:14.755523766 +00:00 < x-frame-options: deny
11:28:14    DEBUG   2024-07-22 10:28:14.755541032 +00:00 < x-content-type-options: nosniff
11:28:14    DEBUG   2024-07-22 10:28:14.755549705 +00:00 < x-xss-protection: 1; mode=block
11:28:14    DEBUG   2024-07-22 10:28:14.755561252 +00:00 < content-security-policy: object-src 'none'; script-src 'self' 'unsafe-inline' 'report-sample' s1.sentry-cdn.com js.sentry-cdn.com browser.sentry-cdn.com statuspage-production.s3.amazonaws.com static.zdassets.com aui-cdn.atlassian.com connect-cdn.atl-paas.net js.stripe.com 'strict-dynamic' cdn.pendo.io data.pendo.io pendo-io-static.storage.googleapis.com pendo-static-5634074999128064.storage.googleapis.com; img-src * blob: data:; media-src *; frame-src app.pendo.io demo.arcade.software js.stripe.com sentry.io; frame-ancestors 'self' *.sentry.io; connect-src 'self' *.algolia.net *.algolianet.com *.algolia.io sentry.io *.sentry.io s1.sentry-cdn.com o1.ingest.sentry.io api2.amplitude.com app.pendo.io data.pendo.io reload.getsentry.net t687h3m0nh65.statuspage.io sentry.zendesk.com ekr.zdassets.com maps.googleapis.com; worker-src blob:; default-src 'none'; base-uri 'none'; style-src * 'unsafe-inline'; font-src * data:; report-uri https://o1.ingest.sentry.io/api/54785/security/?sentry_key=--&sentry_release=--
11:28:14    DEBUG   2024-07-22 10:28:14.755590126 +00:00 < x-envoy-attempt-count: 1
11:28:14    DEBUG   2024-07-22 10:28:14.755598678 +00:00 < x-envoy-upstream-service-time: 62
11:28:14    DEBUG   2024-07-22 10:28:14.755607662 +00:00 < x-served-by: getsentry-web-rpc-production---
11:28:14    DEBUG   2024-07-22 10:28:14.755616104 +00:00 < x-sentry-proxy-url: http://10.2.0.67:8999/api/0/organizations/redacted/chunk-upload/
11:28:14    DEBUG   2024-07-22 10:28:14.755625048 +00:00 < x-served-by: getsentry-control-web-default-common-production-6679757c5-6lqtk
11:28:14    DEBUG   2024-07-22 10:28:14.755633972 +00:00 < x-served-by: frontend-default-7db46fd9f5-p29g5
11:28:14    DEBUG   2024-07-22 10:28:14.755642644 +00:00 < strict-transport-security: max-age=31536000; includeSubDomains; preload
11:28:14    DEBUG   2024-07-22 10:28:14.755651258 +00:00 < via: 1.1 google
11:28:14    DEBUG   2024-07-22 10:28:14.755660010 +00:00 < Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
11:28:14    DEBUG   2024-07-22 10:28:14.755670886 +00:00 < Transfer-Encoding: chunked
11:28:14    DEBUG   2024-07-22 10:28:14.757924890 +00:00 response status: 200
11:28:14    DEBUG   2024-07-22 10:28:14.757991240 +00:00 body: {"url":"https://us.sentry.io/api/0/organizations/redacted/chunk-upload/","chunkSize":8388608,"chunksPerRequest":64,"maxFileSize":2147483648,"maxRequestSize":33554432,"concurrency":8,"hashAlgorithm":"sha1","compression":["gzip"],"accept":["debug_files","release_files","pdbs","sources","bcsymbolmaps","il2cpp","portablepdbs","artifact_bundles","artifact_bundles_v2"]}
11:28:14  > Found 178 files
11:28:16  error: file could not be read as UTF-8
11:28:16    caused by: stream did not contain valid UTF-8
11:28:16    DEBUG   2024-07-22 10:28:16.666527603 +00:00 skipping update nagger because session is not attended

[szokeasaurusrex]

@egziko Interesting, my guess is that this changed behavior was introduced by some dependency version bump, since neither of the error messages exist in the Sentry CLI codebase.

Can you isolate the file which is causing the upload to fail? Maybe it is indeed invalid UTF-8, and this was simply not getting checked before?

[egziko]

@szokeasaurusrex
I had assets in my dist folder (images and font files), after removing them it works fine
This used to work with no problem, and I can see them in my Sentry dashboard in previous releases

[szokeasaurusrex]

@egziko Thanks for that information, I will try to see if I can reproduce this problem and determine why it started happening in 2.32.2.

In the meantime, you can work around by using 2.32.1

[szokeasaurusrex]

@egziko I was able to reproduce your bug and identify 9cf9afb as the cause. Looks like we bumped a lot of dependencies in that revision. Will see how I can fix this

[szokeasaurusrex]

@egziko I have prepared a PR (#2108) which should fix this issue. I will aim to have it merged and released sometime this week.

Thank you for reporting this bug, and sorry for any inconvenience it may have caused!

[egziko]

Thank you for quickly taking care of this

[szokeasaurusrex]

You're welcome!

[Swatinem]

I had assets in my dist folder (images and font files), after removing them it works fine

error: file could not be read as UTF-8
    caused by: stream did not contain valid UTF-8

I believe this error comes from getsentry/symbolic#816 (which is already 6 months old)

We should avoid putting non-js assets into the bundles, but we should obviously do that without failing the upload altogether.

[szokeasaurusrex]

@egziko the bug should be fixed in the latest Sentry CLI release, thanks again for reporting this!

[p0358]

I had it happen, then it stopped, and now it seems to be happening again. I use this in the CI script to upload:

          $files_to_upload = Get-ChildItem -Path build\bin -Recurse -Include ("*.pdb", "*.dll", "*.exe")
          echo "Files to upload:" $files_to_upload
          sentry-cli upload-dif --include-sources $files_to_upload

https://github.com/p0358/black_market_edition/actions/runs/10231364720/job/28307610591

Run $files_to_upload = Get-ChildItem -Path build\bin -Recurse -Include ("*.pdb", "*.dll", "*.exe")
Files to upload:
    Directory: D:\a\black_market_edition\black_market_edition\build\bin\x64-Staging
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---            8/3/2024 11:08 PM        1876480 bme.dll
-a---            8/3/2024 11:08 PM       20779008 bme.pdb
-a---            8/3/2024 11:07 PM         389120 curl.pdb
-a---            8/3/2024 11:07 PM         946176 discord-rpc.pdb
-a---            8/3/2024 11:07 PM         921600 imgui.pdb
-a---            8/3/2024 11:07 PM         183808 launcher.dll
-a---            8/3/2024 11:07 PM        4837376 launcher.pdb
-a---            8/3/2024 11:07 PM         110592 minhook.pdb
-a---            8/3/2024 11:07 PM         176128 nghttp2.pdb
-a---            8/3/2024 11:07 PM        3534848 spdlog.pdb
-a---            8/3/2024 11:07 PM         547328 Titanfall_alt.exe
-a---            8/3/2024 11:07 PM        4419584 Titanfall_alt.pdb
> Found 6 debug information files
error: file could not be read as UTF-8
  caused by: stream did not contain valid UTF-8
Add --log-level=[info|debug] or export SENTRY_LOG_LEVEL=[info|debug] to see more output.
Please attach the full debug log to all bug reports.
Error: Process completed with exit code 1.

I am severely confused as to the entire reason of this change about "valid UTF-8". In principle the native apps on Windows require upload of debug symbols .PDB, accompanied with the programs themselves (.EXE, .DLL, maybe .SYS and .SCR too if I was to list it all). These programs obviously aren't UTF-8, they're binary files. upload-dif subcommand is for uploading debug information files, and all always worked as expected for Sentry web UI to display symbolified stack traces.

And what do I see: https://github.com/getsentry/sentry-cli/blob/2a4ae9fc9a56797ddd5dee2230aa0ce344bf4277/Cargo.lock#L2811C1-L2813C19

Looks like in #2109 by @loewenheim the dep was upgraded again and the bug is thus present again... (I don't have permissions to reopen)

[szokeasaurusrex]

@p0358, I replied on getsentry/symbolic#860. It seems to me like you might be experiencing a separate issue, and I would need some more information from you in order to be able to help