Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERROR [cubestore::http] pid:1 Websocket error: Protocol(ResetWithoutClosingHandshake) #8669

Closed
jlloyd-widen opened this issue Sep 4, 2024 · 4 comments
Labels
cube store Issues relating to Cube Store question The issue is a question. Please use Stack Overflow for questions.

Comments

@jlloyd-widen
Copy link
Contributor

Describe the bug
The cubestore router and any workers will produce the follow error logs under certain conditions:

2023-10-07 09:06:54,513 ERROR [cubestore::cluster] pid:1 Network error: Connection closed unexpectedly. Please check your worker and meta connection environment variables.
2023-10-07 09:07:46,733 ERROR [cubestore::http] pid:1 Websocket error: Protocol(ResetWithoutClosingHandshake)
2023-10-07 09:07:47,433 ERROR [cubestore::http] pid:1 Websocket error: Protocol(ResetWithoutClosingHandshake)
...

To Reproduce
Steps to reproduce the behavior:

  1. Create an opensource deployment. I'm on AWS EKS but I think this happens on other deployment models too. My deployment is based on the gadsme chart. 1 API, 1 worker, 1 cubestore router, cubestore workers are optional, ingress enabled. I'm working with a Snowflake source but I doubt that even needs to be configured.
  2. Minimal config is required. I have nothing in my cube.js file and only 1 cube with a single measure, and 3 dimensions. No preaggs
  3. The trigger seems to be switching devMode from true to false. When devMode is enabled, everything works fine. When it's disabled the above errors begin to appear.

Expected behavior
Everything should work without errors in devMode

Version:
0.35.22

Additional context
Similar issues are reported in the following places:

@igorlukanin
Copy link
Member

Hi @jlloyd-widen 👋

Thanks for a very elaborate report! Let me clarify something.

The trigger seems to be switching devMode from true to false. When devMode is enabled, everything works fine. When it's disabled the above errors begin to appear.

How does the "switching" happen? You update the env vars and restart the whole cluster, correct?

Also, I have to note that running Cube in a dev mode only makes sense for a single-node instance. Cube is not designed to work in a dev mode in a cluster setting. Why would you want to run Cube in a dev mode in your production environment?

@igorlukanin igorlukanin added question The issue is a question. Please use Stack Overflow for questions. cube store Issues relating to Cube Store labels Sep 5, 2024
@jlloyd-widen
Copy link
Contributor Author

jlloyd-widen commented Sep 5, 2024

@igorlukanin

How does the "switching" happen? You update the env vars and restart the whole cluster, correct?

Yes, basically. In my case, it's updating the K8s/helm values.yaml value for devMode which just injects the env var and then update the deployment.

Also, I have to note that running Cube in a dev mode only makes sense for a single-node instance. Cube is not designed to work in a dev mode in a cluster setting. Why would you want to run Cube in a dev mode in your production environment?

I agree. I don't want to run dev mode in my k8s cluster but this error invalidates my deployment and therefore forces me to run in dev mode if I want anything to work.

@igorlukanin
Copy link
Member

When devMode is enabled, everything works fine. When it's disabled the above errors begin to appear.

forces me to run in dev mode if I want anything to work.

I see. I have a hunch that everything work in a dev mode because Cube Store is basically not used (hard to say for sure without checking your env vars). But it sounds pretty pleasing that once you run in prod mode, then Cube Store is used—and because of some (networking?) issues in a Cube Store cluster you now see these errors from the opening message.

I would recommend to double-check the networking configuration/quality and also check the resources that you provisioned. Might anything be just deprived of them?

@jlloyd-widen
Copy link
Contributor Author

@igorlukanin Good call. The resources do appear to be the problem. I upped the resources in my deployment to match the guidance you linked to and the errors appear to have disappeared. Thank you for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cube store Issues relating to Cube Store question The issue is a question. Please use Stack Overflow for questions.
Projects
None yet
Development

No branches or pull requests

2 participants