proposal: Exit with error message if storage is misconfigured

[Xuanwo]

Background

Sccache will ignore storage errors by default:

• For reading, all non-ok responses will be treated as cache-miss.
• For writing, all non-ok responses will be ignored.

Thus, users don't know storage is misconfigured until they find the cache is always missed.

This problem has been proposed by the community many times:

• Fix Fail compiles on misconfigured S3 bucket #45
• Fix s3 backend silently stopped working after 0.3.1 when AWS region isn't defined #1508
• Fix Silently fails to write GCS cache #1384
• Fix With S3 storage backend, it would be very helpful to print a descriptive error instead of silently timing out when AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY are missing/incorrect #1173
• Fix Don't silently swallow forbidden AWS GET requests (sscache S3 always cache miss) #225

Proposal

So I propose adding a storage check before starting the server by performing write and read operations with file path .sccache_check.

• If both of them succeed, sccache will run in read-write mode.
• If write fails with PermissionDenied and read is Ok or NotFound, sccache will run in read-only mode.
• All other cases will lead sccache exit with corresponding error messages.
  • For example, read returns PermissionDenied or OtherError.

The status will also be shown in sccache --show-stat.

With the super power of OpenDAL, this proposal should be easy (

Error happened during compilation will still be ignored.

Drawbacks

Misconfigured sccache will break users' workflow, which may lead to more severe problems.

Tasks

• feat: Exit while cache is not configured correctly #1515

[sylvestre]

That sounds like a great idea, i am sure that weeks of work have been lost while trying to configure sccache.

I did similar work with logging: #1361

[Xuanwo]

Quick demo show:

Start s3 without ak/sk

:( SCCACHE_START_SERVER=1 SCCACHE_NO_DAEMON=1 SCCACHE_BUCKET=test ./target/debug/sccache                  
sccache: error: cache storage failed to read: ObjectPermissionDenied (permanent) at read => S3Error { code: "AccessDenied", message: "Access Denied", resource: "", request_id: "46RYHDRAN8WD64XW" }

Context:
    response: Parts { status: 403, version: HTTP/1.1, headers: {"x-amz-request-id": "46RYHDRAN8WD64XW", "x-amz-id-2": "R9ldSWAQSbbTSYOowzLoJD0dRgRWbcS0wYV0ibbb+4yVok+oYEmvB+F8+2g05KWvXnGYK8poafw=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Mon, 02 Jan 2023 04:48:47 GMT", "server": "AmazonS3"} }
    service: s3
    path: .sccache_check
    range: 0-

Start gcs without credentials:

:( SCCACHE_START_SERVER=1 SCCACHE_NO_DAEMON=1 SCCACHE_GCS_BUCKET=test ./target/debug/sccache
sccache: error: cache storage failed to write: Unexpected (permanent) at write => signing request

Context:
    service: gcs
    path: .sccache_check

Source: token not found

Start azblob with an invalid config:

:( SCCACHE_START_SERVER=1 SCCACHE_NO_DAEMON=1 SCCACHE_AZURE_BLOB_CONTAINER=test SCCACHE_AZURE_CONNECTION_STRING="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;" ./target/debug/sccache 
sccache: error: cache storage failed to write: Unexpected (temporary) at write => send async request

Context:
    called: http_util::Client::send_async
    service: azblob
    path: .sccache_check

Source: error sending request for url (http://127.0.0.1:10000/devstoreaccount1/test/.sccache_check): error trying to connect: tcp connect error: Connection refused (os error 111)

Caused by:
    0: error trying to connect: tcp connect error: Connection refused (os error 111)
    1: tcp connect error: Connection refused (os error 111)
    2: Connection refused (os error 111)

With RUSTC_WRAPPER:

:( RUSTC_WRAPPER=/home/xuanwo/Code/mozilla/sccache/target/debug/sccache SCCACHE_BUCKET=test CARGO_INCREMENTAL=false SCCACHE_LOG=debug SCCACHE_ERROR_LOG=/tmp/sccache_log.txt cargo build                 
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `/home/xuanwo/Code/mozilla/sccache/target/debug/sccache rustc - --crate-name ___ --print=file-names -C link-arg=-fuse-ld=/usr/bin/mold --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit status: 2)
  --- stderr
  [2023-01-02T12:05:01Z DEBUG sccache::config] Attempting to read config file at "/home/xuanwo/.config/sccache/config"
  [2023-01-02T12:05:01Z DEBUG sccache::config] Couldn't open config file: No such file or directory (os error 2)
  [2023-01-02T12:05:01Z DEBUG sccache::config] Attempting to read config file at "/home/xuanwo/.config/sccache/config"
  [2023-01-02T12:05:01Z DEBUG sccache::config] Couldn't open config file: No such file or directory (os error 2)
  sccache: error: Server startup failed: cache storage failed to read: ObjectPermissionDenied (permanent) at read => S3Error { code: "AccessDenied", message: "Access Denied", resource: "", request_id: "1V54QP0EPZQ5J3A7" }

  Context:
      response: Parts { status: 403, version: HTTP/1.1, headers: {"x-amz-request-id": "1V54QP0EPZQ5J3A7", "x-amz-id-2": "YRQ+E3KrYscCNKSrajVyQU0JT8hxEvdgzBhbHB4OK4Y+m15W6HVFko3NDBaMxIFQRFm/QVl/C+I=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Mon, 02 Jan 2023 12:05:03 GMT", "server": "AmazonS3"} }
      service: s3
      path: .sccache_check
      range: 0-