Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Client panics when initializing on a read only filesystem #2387

Open
irwinp-p opened this issue Oct 3, 2024 · 4 comments · May be fixed by #2391
Open

Client panics when initializing on a read only filesystem #2387

irwinp-p opened this issue Oct 3, 2024 · 4 comments · May be fixed by #2391
Labels
bug Something isn't working

Comments

@irwinp-p
Copy link

irwinp-p commented Oct 3, 2024

Describe the bug

Client initialisation panics on read only filesystems. The failure originates from the rolling file appender in logger_core.
A RollingFilleAppender is created even when file based logging is disabled by setting Logger.setLoggerConfig("warn")

Expected Behavior

When file based logging is disabled, the RollingFileAppender should not be used.

Current Behavior

A panic occurs when calling GlideClient.createClient. Detailed error information below in the Logs section.

Reproduction Steps

Use docker in read-only mode to reproduce

  1. Dockerfile contents
FROM node:20
WORKDIR /usr/app
RUN npm init -y
RUN npm i "@valkey/valkey-glide"
COPY index.js .
CMD ["node", "index.js"]
  1. index.js contents
const { GlideClient, Logger } = require("@valkey/valkey-glide");

const run = async () => {
    Logger.setLoggerConfig("warn");
    const addresses = [ { host: "127.0.0.1", port: 6379 } ];
    await GlideClient.createClient({ addresses: addresses });
}

run();

  1. start a redis server on localhost port 6379

4 build the docker image and run

 docker build -t glide-test .  && docker run --network host -p 6379:6379 --read-only glide-test
  1. Output is displayed below in the Logs section

Possible Solution

Don't create a RollingFilleAppender when file based logging is disabled.

Additional Information/Context

We plan to use Glide on AWS Fargate. All our services there run there with the following setting

readonlyRootFilesystem: true

Also, there may possibly be other issues in addition to the RollingFileAppender that prevents running on read only file systems, e.g. https://github.com/valkey-io/valkey-glide/blob/main/glide-core/src/socket_listener.rs#L793

Client version used

1.1.0

Engine type and version

Valkey 7.2

OS

MacOS

Language

TypeScript

Language Version

node 20

Cluster information

No response

Logs

thread '<unnamed>' panicked at /home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-appender-0.2.3/src/rolling.rs:154:14:
initializing rolling file appender failed: InitError { context: "failed to create log directory", source: Os { code: 30, kind: ReadOnlyFilesystem, message: "Read-only file system" } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at library/core/src/panicking.rs:221:5:
panic in a function that cannot unwind
stack backtrace:
   0:     0xffff6a49c878 - <unknown>
   1:     0xffff6a25c9ac - <unknown>
   2:     0xffff6a47691c - <unknown>
   3:     0xffff6a49ffcc - <unknown>
   4:     0xffff6a4a0fd4 - <unknown>
   5:     0xffff6a4a08d4 - <unknown>
   6:     0xffff6a4a0878 - <unknown>
   7:     0xffff6a4a086c - <unknown>
   8:     0xffff6a22709c - <unknown>
   9:     0xffff6a2270f8 - <unknown>
  10:     0xffff6a2270b0 - <unknown>
  11:     0xffff6a339fb4 - <unknown>
  12:           0xc4057c - _ZN6v8impl12_GLOBAL__N_123FunctionCallbackWrapper6InvokeERKN2v820FunctionCallbackInfoINS2_5ValueEEE
  13:           0xf26e68 - _ZN2v88internal25FunctionCallbackArguments4CallENS0_15CallHandlerInfoE
  14:           0xf27628 - _ZN2v88internal12_GLOBAL__N_119HandleApiCallHelperILb0EEENS0_11MaybeHandleINS0_6ObjectEEEPNS0_7IsolateENS0_6HandleINS0_10HeapObjectEEENS8_INS0_20FunctionTemplateInfoEEENS8_IS4_EEPmi
  15:           0xf27a40 - _ZN2v88internal21Builtin_HandleApiCallEiPmPNS0_7IsolateE
  16:          0x18a5964 - Builtins_CEntry_Return1_ArgvOnStack_BuiltinExit
thread caused non-unwinding panic. aborting.

Other information

No response

@irwinp-p irwinp-p added the bug Something isn't working label Oct 3, 2024
@avifenesh
Copy link
Collaborator

avifenesh commented Oct 3, 2024

@irwinp-p Hi, thanks for bringing it up.
A fix was merged a few days ago where you can turn off the logger completely.
It'll be available in the next minor release around a month from now, if you need this fix before it'll be available in rc when we start the release process soon.

The default behavior still will be to log errors to a file, and in any case of different need it can be configured different.

I'd leave the issue open so in the future we will add special handling for read-only envs, but since there will be a way to mitigate it, the specific handling will have a bit lower priority.

Please explain the other issue you see possible.
If its the logs, as said it can be turn off, and anyhow the default don't log info unless set to do so.

@nakedible
Copy link

It's possible the fix isn't actually sufficient. The client seems to initialize a log appender always, and then reconfigure it, causing the issue.

A possible workaround for now is to change current directory to /dev/shm, which is writable even if the rest of the filesystem isn't.

vnuka-n added a commit to vnuka-n/valkey-glide that referenced this issue Oct 4, 2024
Calling RollingFileAppender::new tries to create the directory for the log files even though logging is turned off by Logger.init('off', null). This change simply postpones the initialization of RollingFileAppender until it is actually needed. This way if we do eg. Logger.init("info", null) it works as expected and does not throw an error on readonly filesystems.

Fixes valkey-io#2387
vnuka-n added a commit to vnuka-n/valkey-glide that referenced this issue Oct 4, 2024
Calling RollingFileAppender::new tries to create the directory for the log files even though logging is turned off by Logger.init('off', null). This change simply postpones the initialization of RollingFileAppender until it is actually needed. This way if we do eg. Logger.init("info", null) it works as expected and does not throw an error on readonly filesystems.

Fixes valkey-io#2387
@vnuka-n vnuka-n linked a pull request Oct 4, 2024 that will close this issue
@vnuka-n
Copy link

vnuka-n commented Oct 4, 2024

Setting the log level to OFF does not fix this issue because the log directory is created in RollingFileAppender::new call which is always done no matter the log level.

I tried to come up with a simple non-breaking change to fix this in PR #2391

@irwinp-p
Copy link
Author

irwinp-p commented Oct 4, 2024

@avifenesh the other issue I alluded is the Unix Socket that's created in glide-core/src/socket_listener.rs. We've now verified there's a workaround for this on read-only file systems by setting the XDG_RUNTIME_DIR env variable to /dev/shm, that will ensure the socket gets created under /dev/shm. So this isn't a problem for us now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants