Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QUESTION] Dashy 1.5.6 consumes more RAM #136

Closed
EVOTk opened this issue Aug 8, 2021 · 23 comments
Closed

[QUESTION] Dashy 1.5.6 consumes more RAM #136

EVOTk opened this issue Aug 8, 2021 · 23 comments
Assignees
Labels
🐛 Bug [ISSUE] Ticket describing something that isn't working ❕ Medium Priority [ISSUE] Needs to be dealt with at some point 🤷‍♂️ Question [ISSUE] Further information is requested

Comments

@EVOTk
Copy link
Contributor

EVOTk commented Aug 8, 2021

Hello,
Since Dashy 1.5.6 it does not start anymore. Indeed, I allocate 756MB of RAM to it. With Dashy 1.5.6 this is no longer sufficient, I have to give it 1024MB.

With my container limited to 1024MB, Dashy 1.5.6 works.

I already find that allocating 756 MB of RAM is already significant. 1024MB, that starts to do a lot, my machine only has 8GB, more now that Dashy is available on ARM, a lot of ARM machine has little RAM.

Where did this high demand for RAM come from when Dashy was launched? Because we can see that once launched, Dashy consumes much less RAM (around 250MB).

Logs with the container limited to 756MB:

yarn run v1.22.5,
$ npm-run-all --parallel build-watch start,
$ vue-cli-service build --watch --mode production,
$ node server,
,
Checking config file against schema...,
No issues found, your configuration is valid :),
,
,
,
 ██████╗  █████╗ ███████╗██╗  ██╗██╗   ██╗,
 ██╔══██╗██╔══██╗██╔════╝██║  ██║╚██╗ ██╔╝,
 ██║  ██║███████║███████╗███████║ ╚████╔╝,
 ██║  ██║██╔══██║╚════██║██╔══██║  ╚██╔╝,
 ██████╔╝██║  ██║███████║██║  ██║   ██║,
 ╚═════╝ ╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝   ╚═╝,
,
*******************************************************************************************,
Welcome to Dashy! 🚀,
Your new dashboard is now up and running in container ID dc91db3e1af1,
After updating your config file, run  'docker exec -it dc91db3e1af1 yarn build' to rebuild,
*******************************************************************************************,
,
,
Using Dashy V-1.5.6. Update Check Complete,
Unable to check for updates,
,
-  Building for production...,
 WARN  A new version of sass-loader is available. Please upgrade for best experience.,
,
<--- Last few GCs --->,
,
[61:0x5627bf910340]    86833 ms: Scavenge (reduce) 373.1 (378.4) -> 372.8 (379.4) MB, 23.6 / 0.0 ms  (average mu = 0.291, current mu = 0.351) allocation failure ,
[61:0x5627bf910340]    86909 ms: Scavenge (reduce) 373.4 (381.4) -> 373.1 (381.4) MB, 25.1 / 0.0 ms  (average mu = 0.291, current mu = 0.351) allocation failure ,
[61:0x5627bf910340]    87005 ms: Scavenge (reduce) 373.8 (378.4) -> 373.4 (379.9) MB, 25.2 / 0.0 ms  (average mu = 0.291, current mu = 0.351) allocation failure ,
,
,
<--- JS stacktrace --->,
,
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory,
error Command failed with signal "SIGABRT".,
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.,
ERROR: "build-watch" exited with 1.,
error Command failed with exit code 1.,
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.,
@EVOTk EVOTk added the 🤷‍♂️ Question [ISSUE] Further information is requested label Aug 8, 2021
@Lissy93
Copy link
Owner

Lissy93 commented Aug 9, 2021

Oh, that doesn't look good.
It's weird, because this PR mostly just included docs updates, some UI stuff and a slight refactor.

This seems to be a Node.js error, and possibly caused my a memory leak in the version used, I've seen something similar in a previous project a while back. I will take a proper look this evening, and try and get it sorted asap. Thanks for raising it :)

Which tag are you using, is it the normal amd :latest image?

@Lissy93 Lissy93 added ‼️ High Priority [ISSUE] An issue or PR that needs to be dealt with urgently 🐛 Bug [ISSUE] Ticket describing something that isn't working labels Aug 9, 2021
@EVOTk
Copy link
Contributor Author

EVOTk commented Aug 9, 2021

Hello,

Which tag are you using, is it the normal amd :latest image?

Yes.

My compose which no works:

version: "3.8"
services:
  dashy:
    image: lissy93/dashy
    container_name: Dashy
    volumes:
      - /home/evo-dashy/docker/dashy/conf.yml:/app/public/conf.yml
      - /home/evo-dashy/docker/dashy/item-icons:/app/public/item-icons
    ports:
      - 4000:80
    environment:
      - NODE_ENV=production
      - UID=1000
      - GID=1000
    restart: unless-stopped
    healthcheck:
      test: ['CMD', 'node', '/app/services/healthcheck']
      interval: 1m30s
      timeout: 10s
      retries: 3
      start_period: 40s
    mem_limit: 756M
    mem_reservation: 256M

If I increase mem_limit, Dashy works:

version: "3.8"
services:
  dashy:
    image: lissy93/dashy
    container_name: Dashy
    volumes:
      - /home/evo-dashy/docker/dashy/conf.yml:/app/public/conf.yml
      - /home/evo-dashy/docker/dashy/item-icons:/app/public/item-icons
    ports:
      - 4000:80
    environment:
      - NODE_ENV=production
      - UID=1000
      - GID=1000
    restart: unless-stopped
    healthcheck:
      test: ['CMD', 'node', '/app/services/healthcheck']
      interval: 1m30s
      timeout: 10s
      retries: 3
      start_period: 40s
    mem_limit: 1024M
    mem_reservation: 256M

@Lissy93
Copy link
Owner

Lissy93 commented Aug 9, 2021

Thanks, so I've had a quick look, and compared the latest image with version 1.5.0 from last week. For me they look pretty similar

Tag CPU Mem Mem % Net / IO Size
lissy93/dashy:release-1.5.0 0.07% 265.8MiB / 31.42GiB 0.83% 25.4kB / 2.19MB 12.8MB (virtual 951MB)
issy93/dashy:latest 0.08% 260.3MiB / 31.42GiB 0.81% 8.32kB / 1.78kB 14.8MB (virtual 950MB)

But I will look into this futher this evening, because if you container is needing a GB of RAM, that is an issue.

@Lissy93 Lissy93 added ❕ Medium Priority [ISSUE] Needs to be dealt with at some point and removed ‼️ High Priority [ISSUE] An issue or PR that needs to be dealt with urgently labels Aug 9, 2021
@EVOTk
Copy link
Contributor Author

EVOTk commented Aug 9, 2021

Once built, the RAM consumption between 1.5.0 and 1.5.6 is similar yes,around 250M
2021-08-09 13_37_51-Window

On the other hand, for build, 1.5.6 requires more RAM than 1.5.0

My test:
Version - Limit of RAM - Works or not

1.5.0 - 512M - No
1.5.0 - 756M - Yes

1.5.5 - 512M - No
1.5.5 - 756M - Yes

1.5.6 - 512M - No
1.5.6 - 756M - No
1.5.6 - 1024M - Yes

Logs Dashy 1.5.6 with limit 1024M RAM :

yarn run v1.22.5
$ npm-run-all --parallel build-watch start
$ vue-cli-service build --watch --mode production
$ node server
Checking config file against schema...
No issues found, your configuration is valid :)

 ██████╗  █████╗ ███████╗██╗  ██╗██╗   ██╗
 ██╔══██╗██╔══██╗██╔════╝██║  ██║╚██╗ ██╔╝
 ██║  ██║███████║███████╗███████║ ╚████╔╝
 ██║  ██║██╔══██║╚════██║██╔══██║  ╚██╔╝
 ██████╔╝██║  ██║███████║██║  ██║   ██║
 ╚═════╝ ╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝   ╚═╝

*******************************************************************************************
Welcome to Dashy! 🚀
Your new dashboard is now up and running in container ID 1a64e464a4a0
After updating your config file, run  'docker exec -it 1a64e464a4a0 yarn build' to rebuild
*******************************************************************************************
Using Dashy V-1.5.6. Update Check Complete
Unable to check for updates

-  Building for production...
 WARN  A new version of sass-loader is available. Please upgrade for best experience.
 DONE  Compiled successfully in 82958ms11:37:12 AM

  File                                      Size             Gzipped

  dist/js/chunk-vendors.18a6705c.js         1919.94 KiB      559.17 KiB
  dist/js/dashy.fdce4f2d.js                 214.77 KiB       63.39 KiB
  dist/precache-manifest.bc3da6e25287741    21.58 KiB        4.08 KiB
  21f502d2bf108b065.js
  dist/js/about.0c77a650.js                 4.48 KiB         1.99 KiB
  dist/service-worker.js                    1.04 KiB         0.61 KiB
  dist/css/dashy.c0819ca9.css               110.10 KiB       16.69 KiB
  dist/css/chunk-vendors.6991ea48.css       6.59 KiB         1.91 KiB
  dist/css/about.a64c098d.css               1.04 KiB         0.34 KiB

  Images and other types of assets omitted.
 DONE  Build complete. Watching for changes...

@Lissy93
Copy link
Owner

Lissy93 commented Aug 9, 2021

Ah thanks for explaining, it makes sense that the build process uses more memory, but I can't think why this has gone up so much in the most recent version, I will have to go through it this evening and see what I can find.
But thankfully that does not look like a memory leak, which was my initial worry

@EVOTk
Copy link
Contributor Author

EVOTk commented Aug 9, 2021

Thank you for your responsiveness as always.

Sorry for the explanations, I'm trying to put as much information as possible (logs, ..), because I don't speak English, so I use google translate to help me build my sentences.

@shad-lp
Copy link

shad-lp commented Aug 10, 2021

On my side, no abnormal memory usage, considering I'm using the healthchecks feature which eventually increases the ram consumption (peaks are happening when watchtower upgrades all my images, at 5am) :
dashy_memory_usage

@EVOTk
Copy link
Contributor Author

EVOTk commented Aug 10, 2021

On my side, no abnormal memory usage, considering I'm using the healthchecks feature which eventually increases the ram consumption (peaks are happening when watchtower upgrades all my images, at 5am) :

Yes, I'm only talking about RAM consumption when the container was created.

@Lissy93
Copy link
Owner

Lissy93 commented Aug 11, 2021

So I've looked into this, and I'm still not 100% sure what could have caused it, and I haven't been able to recreate it.
There haven't been any recent changed to either the build process, node server code or main Docker image,
and the frontend code wouldn't have an impact on the build process. The only thing it could have been, is the Docker base image wasn't set to a fixed version, so any change in the Alpine Node container could have caused this.

I've created a PR (#147) to set the base image to a fixed version of alpine node FROM node:lts-alpine3.14, which should prevent any random changes in performance happening.

I think that the build process would be a lot more performant if I had used something else other than Node.js. In the future, I would like to rewrite the server part in Go Lang, which should be more light-weight, so hopefully consume less memory. That would be a big job though, so won't happen right away.

@Lissy93
Copy link
Owner

Lissy93 commented Aug 11, 2021

I will keep an eye on this, and reopen the issue again if this keeps happening

@EVOTk
Copy link
Contributor Author

EVOTk commented Aug 11, 2021

Thank you for your explanation ! Indeed, an update of lts-alpine is quite possible, I had not thought of this at all!

Thanks again for your involvement! Rewriting in another language must indeed be a long job!

@EVOTk
Copy link
Contributor Author

EVOTk commented Aug 15, 2021

Hello,
Since the last changes, Dashy, even limited to 756M of RAM, is building correctly!

@Lissy93
Copy link
Owner

Lissy93 commented Aug 15, 2021

I'll revert the change back to the original Dockerfile, and push the multi-architecture image under a separate tag.
Sorry you're having issues :(

@Lissy93
Copy link
Owner

Lissy93 commented Aug 15, 2021

So I've reverted to the previous Dockerfile, and will merge soon, and hopefully this will fix your issue :)

I'm still not sure exactly what is causing it, as on GHCR the build never uses more than 512mb of memory. I did notice that Docker copies all assets into memory during the build, so maybe if you are using your own icons, and if they are large this might possibly have an impact?

@Lissy93
Copy link
Owner

Lissy93 commented Aug 19, 2021

I haven't forgotten about this, the latest PR (#168) halves the size of the Docker container, and I am still looking for further ways to improve it further. How is the memory consumption going, still having issues?

@EVOTk
Copy link
Contributor Author

EVOTk commented Aug 19, 2021

Hello,
Thank you for your work ! It is indeed much better.

Before, with 750M of RAM the container could not build this.
Now with only 512M it works !!!

The image also reduced in size from 943.9 MB to 746.5 MB

@immartian
Copy link

Met the same issue on Raspberry Pi with the latest image. Any solutions?


-  Building for production...
 WARN  A new version of sass-loader is available. Please upgrade for best experience.

<--- Last few GCs --->

[67:0x74504040]   418627 ms: Mark-sweep (reduce) 225.9 (236.3) -> 225.4 (236.6) MB, 4753.2 / 0.3 ms  (average mu = 0.501, current mu = 0.234) allocation failure scavenge might not succeed
[67:0x74504040]   423448 ms: Mark-sweep (reduce) 225.9 (236.6) -> 225.9 (237.1) MB, 4809.9 / 0.3 ms  (average mu = 0.315, current mu = 0.002) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
error Command failed with signal "SIGABRT".
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
ERROR: "build-watch" exited with 1.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
pi@raspberrypi:~ $ docker image

@EDIflyer
Copy link

I just had my Linode server crash and I had to do an out-of-band reset via Lish console. I slowly brought up my containers and it looks like it was an issue with dashy during the initial setup build. It's a 1 CPU/1 GB Linode so sounds like it may be insufficient to run dashy? If so is there a way to detect this at start so there's not the cpu lockup?

The process in question was /app/node_modules/.vin/vue-cli-service build --watch --mode production

When I brought the server back up I restarted containers one by one - when I got to Dashy I saw it doing the initial start phase then said it was taking too long then it crashed. Log file and btop screenshot below in case helpful!

today at 19:15:24Container started
today at 19:15:22info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
today at 19:15:23yarn run v1.22.15
today at 19:15:24$ npm-run-all --parallel build-watch start
today at 19:15:24$ vue-cli-service build --watch --mode production
today at 19:15:24$ node server
today at 19:15:25
today at 19:15:25Checking config file against schema...
today at 19:15:25
today at 19:15:25━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
today at 19:15:25  Warning: 1 issue found in config file  
today at 19:15:25━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
today at 19:15:251.  must be object 
today at 19:15:25
today at 19:15:25SSL Not Enabled: Public key not present
today at 19:15:25
today at 19:15:25
today at 19:15:25 ██████╗  █████╗ ███████╗██╗  ██╗██╗   ██╗
today at 19:15:25 ██╔══██╗██╔══██╗██╔════╝██║  ██║╚██╗ ██╔╝
today at 19:15:25 ██║  ██║███████║███████╗███████║ ╚████╔╝
today at 19:15:25 ██║  ██║██╔══██║╚════██║██╔══██║  ╚██╔╝
today at 19:15:25 ██████╔╝██║  ██║███████║██║  ██║   ██║
today at 19:15:25 ╚═════╝ ╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝   ╚═╝
today at 19:15:25
today at 19:15:25*******************************************************************************************
today at 19:15:25Welcome to Dashy! 🚀
today at 19:15:25Your new dashboard is now up and running with Docker
today at 19:15:25*******************************************************************************************
today at 19:15:25
today at 19:15:25
today at 19:15:25Using Dashy V-2.1.1. Update Check Complete
today at 19:15:25✅ Dashy is Up-to-Date
today at 19:15:25
today at 19:15:26
today at 19:15:26-  Building for production...
today at 19:15:26 WARN  A new version of sass-loader is available. Please upgrade for best experience.
today at 19:16:05
today at 19:16:05<--- Last few GCs --->
today at 19:16:05
today at 19:16:05[62:0x7fcac40c9350]    39855 ms: Scavenge 479.0 (494.7) -> 477.9 (495.0) MB, 14.5 / 0.0 ms  (average mu = 0.472, current mu = 0.312) allocation failure 
today at 19:16:05[62:0x7fcac40c9350]    39888 ms: Scavenge 479.4 (495.0) -> 478.4 (499.5) MB, 8.0 / 0.0 ms  (average mu = 0.472, current mu = 0.312) allocation failure 
today at 19:16:05[62:0x7fcac40c9350]    40734 ms: Mark-sweep 482.2 (499.7) -> 479.2 (501.0) MB, 810.9 / 0.0 ms  (average mu = 0.377, current mu = 0.268) allocation failure scavenge might not succeed
today at 19:16:05
today at 19:16:05
today at 19:16:05<--- JS stacktrace --->
today at 19:16:05
today at 19:16:05FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
today at 19:16:06error Command failed with signal "SIGABRT".
today at 19:16:06info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
today at 19:16:06ERROR: "build-watch" exited with 1.
today at 19:16:07error Command failed with exit code 1.
today at 19:16:07info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
today at 19:16:08Container stopped

image

@mortasoft
Copy link

@EDIflyer EDIflyer the same happend to me. I was using an AWS instance, and it broke everything. 1GB and 1vCPU.
I had to restart and stop the Dashy instance.

It's too resource intensive!

@dsirov
Copy link

dsirov commented Nov 16, 2022

My observation: I am experiencing 1GB memory usage during first 4 minutes after dashy docker start.
Then memory usage is reduced to ~300MB.

By using top in the console, the offender seems to be:

 PID   VSZ^VSZRW^  RSS  (SHR) DIRTY (SHR) STACK COMMAND
   63  1175m 977m  1002m 32676  968m     0   136 /usr/local/bin/node /app/node_modules/.bin/vue-cli-service build --watch --mode prod

Then, it's normalized:

 PID   VSZ^VSZRW^  RSS (SHR) DIRTY (SHR) STACK COMMAND
   63  384m  156m  222m 33036  146m     0   136 /usr/local/bin/node /app/node_modules/.bin/vue-cli-service build --watch --mode prod

@marshes
Copy link

marshes commented Nov 22, 2022

Dashy used to work on my docker instance but for some reason seems to lock my system on startup.

Docker is running in an lxc in proxmox. The lxc has 2 gigs of ram allocated but only 900 megs free.

@Lissy93 Can you reopen this issue or provide some guidance? Thanks!

@Milamber07
Copy link

Hi,
My Dashy crashes with 1024MB limit, without limit it is ok. Considering I have only 8GB, it is quite intense on memory during startup.
I'm using latest tag image.

image

image

Without limit the graph looks like this:
image

@Lissy93 Please, have a look at this memory leak.
If you need more info - just ask.

Thanks for Dashy ;)

@Milamber07
Copy link

A little update, memory usage goes up to 1,5G during editing Dashy.

image

asterling8516 pushed a commit to asterling8516/dashy that referenced this issue Nov 23, 2023
Closes Lissy93#136

Signed-off-by: Bjorn Lammers <walkxnl@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐛 Bug [ISSUE] Ticket describing something that isn't working ❕ Medium Priority [ISSUE] Needs to be dealt with at some point 🤷‍♂️ Question [ISSUE] Further information is requested
Projects
None yet
Development

No branches or pull requests

9 participants