Mesh won't start as a service

[getut]

I'm having an issue that started like 6 releases back where my meshcentral crashes if it is started as a service. I kept hoping that it was a problem that would resolve in a couple of releases but it has not. This is a 100% local installation with the default local database. Originally when it stopped loading, I found that some threshold for java memory had been crossed and I edited meshcentral.service config to include --max-old-space-size=4096 since this finally got mesh to start when starting it manually from the command line. Before adding that to the command line, it would not even start from manual CLI once it started failing. But even after updating the meshcentral.service, mesh no longer starts as a service for me. Above question was posted on reddit, and I was asked to repost here with additional information:

Server specs = Ubuntu 22.04 running in a VM assigned 4 cores and 8GB of RAM and 40GB of disk space (vm is dedicated to mesh)

Command that works when starting manually. This is also the same command in the meshcentral.service file, but when starting manually I start a screen session and the sudo the following command:
node --max-old-space-size=4096 /home/myuser/meshcentral/node_modules/meshcentral

meshcentral.service file:

[Unit]
Description=MeshCentral Server
[Service]
Type=simple
LimitNOFILE=1000000
ExecStart=/usr/bin/node --max-old-space-size=4096 /home/myuser/meshcentral/node_modules/meshcentral
WorkingDirectory=/home/myuser
Environment=NODE_ENV=production
User=myuser
Group=myuser
Restart=always

RestartSec=10

AmbientCapabilities=cap_net_bind_service
[Install]
WantedBy=multi-user.target

Config.json:

{
  "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
  "__comment1__": "This is a simple configuration file, all values and sections that start with underscore (_) are ignored. Edit a section and remove the _ in front of the name. Refer to the user's guide for details.",
  "__comment2__": "See node_modules/meshcentral/sample-config-advanced.json for a more advanced example.",
  "settings": {
    "_cert": "myserver.changed.com",
    "_WANonly": true,
    "_LANonly": true,
    "_sessionKey": "changed",
    "_port": 443,
    "_aliasPort": 443,
    "_redirPort": 80,
    "_redirAliasPort": 80,
    "DbExpire": {
      "events": 1728000,
      "powerevents": 864000,
      "statsevents": 2592000
    },
    "plugins": { "enabled": true }
  },
  "domains": {
    "": {
      "title": "Mycompany Connect",
      "_title2": "Servername",
      "titlePicture": "logo.png",
      "_minify": true,
      "_newAccounts": true,
      "_userNameIsEmail": true,
      "passwordRequirements": {
        "min": 8,
        "max": 128,
        "upper": 1,
        "lower": 1,
        "numeric": 1,
        "nonalpha": 1,
        "reset": 90,
        "force2factor": true,
        "email2factor": true,
	"lock2factor": true,
	"_backupcode2factor": false,
	"single2factorWarning": false,
	"twoFactorCookieDurationDays": 0,
        "skip2factor": "127.0.0.1,192.168.51.0/24",
        "oldPasswordBan": 5,
        "banCommonPasswords": false,
        "twoFactorTimeout": 300
      }
    }
  },
  "letsencrypt": {
    "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before trying Let's Encrypt.",
    "email": "myuser@companydomain.com",
    "names": "meshcentral.companydomain.com",
    "production": true
  },
  "smtp": {
    "host": "mx1.companydomain.com",
    "port": 25,
    "from": "myuserconnect@companydomain.com",
    "tls": false
  }
}

[getut]

Ping. Please help with this. I have been stuck with this issue for a very long time now.

[si458]

sorry didnt see your message,
you shouldnt need --max-old-space-size=4096

also your config.json isnt quite right,
for example.

1. _cert should be cert and it should be set to the same dns name you use to access the web ui
2. _sessionkey should be sessionkey and you should set a unique value in here thats secret

also try the following as your service file looks wrong, from the root of your meshcentral folder
(in your case /home/myuser/meshcentral)

1. stopping meshcentral systemctl stop meshcentral
2. uninstall meshcentral service node node_modules/meshcentral --uninstall
3. reinstall meshcentral service node node_modules/meshcentral --install
4. check status systemctl status meshcentral

[getut]

Sorry for the long delay, but I am limited on when I can bring this server down to do this type of work. But here are the results of what you asked me to try.

When running the uninstall command for node, I get this error:

Error: Cannot find module '/home/lnxusr/meshcentral/meshcentral-data/node_modules/meshcentral' at Module._resolveFilename (node:internal/modules/cjs/loader:1225:15) at Module._load (node:internal/modules/cjs/loader:1051:27) at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:174:12) at node:internal/main/run_main_module:28:49 { code: 'MODULE_NOT_FOUND',
It seems that my node_modules folder is in a different location. It is directly underneath meshcentral folder, not underneath meshcentral-data. I do not believe I have moved this at any point.

Also, before attempting the config.json changes I wanted to ask for more info. Specifically why if this is incorrect did my mesh install function normally before the upgrade that seemed to break its ability to run as a service, and why is it running fine now when started manually if these are incorrect? I'm not trying to be argumentative, but I'm reluctant to change something that seems to be working, at least until I've resolved the service issue. I don't want to introduce changes that werent there before it stopped working.

[getut]

Well, oddly enough both the service uninstall and reinstall worked on a later try. I tried the uninstall multiple times and it failed every time, but then I started gathering information to make the post above this one reporting the failure. After I made that post, I started the server the manual way, then shut it back down again, and then just as one last chance I tried the uninstall command again and it worked this time. I didnt even retype it, I used the up arrow and command history, so I know previous attempts were not a typo... and this time it worked. Then I did the install again and it worked also and the service started properly after that !!!!! YAY!!!!

Thanks for the help!

[si458]

For future use, run all your commands from the same folder, which has the meshcentral-data, meshcentral-backups, node_modules, etc...

dont run anything from inside node_modules or inside meshcentral-data as they won't work!

Always move up a folder!

[getut]

I'm writing a reply here because I'm thinking this new issue is a result of the work that we did here even though it's different. Ever since I got mesh working as a service again, it seems to be failing to be able to update itself. I am currently on 1.1.27 and trying to go to 1.1.29. It starts the update, comes back up, but when I sign back in again, it's still on 1.1.27. If you think this may be something unrelated, let me know and I'll start a new thread.