Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nsqd: failed to instantiate nsqd - --data-path xxx possibly by another instance of nsqd #1217

Closed
coanor opened this issue Nov 22, 2019 · 3 comments
Labels

Comments

@coanor
Copy link

coanor commented Nov 22, 2019

  • docker version: Docker version 18.06.3-ce, build d7080c1

  • docker-compose version:

    docker-compose version 1.17.1, build unknown
    docker-py version: 2.5.1
    CPython version: 2.7.15+
    OpenSSL version: OpenSSL 1.1.1 11 Sep 2018

  • host info: Ubuntu 18.04.3 LTS (ubuntu server)


I'm using docker-compose to setup NSQ according to here.

Additionally, I have add --data-path=/data/nsqd to nsqd node, and I got

FATAL: failed to instantiate nsqd - --data-path=/data/nsqd in use (possibly by another instance of nsqd)

According to source code, this may be caused by a dirlock opertion.

I've used auditctl -w /data/nsqd -p war -k whatsgoingon to monitor /data/nsqd, when someone touched (war: write/access/read) it, some log will print to /var/log/audit/audit.log.

I reastart the nsqd, but still got the same error (via docker-compose logs)

FATAL: failed to instantiate nsqd - --data-path=/data/nsqd in use (possibly by another instance of nsqd)

but no message showing in audit.log (via tailf /var/log/audit/audit.log | grep "whatsgoingon"). this means that during starting the nsqd, no one touched /data/nsqd.

I scraped the dir lock source code and build a test program, when os.Open("/data/nsqd"), the monitor shows the touch action as expected.

I wonder why nsq think someone touching the --data-path but acturally no one there, and this maybe a bug.

@ploxiln
Copy link
Member

ploxiln commented Nov 23, 2019

I don't use docker compose, can't help much with that.

But I wonder if the filesystem type underlying the nsqd storage volume could be the problem?

@coanor coanor closed this as completed Nov 24, 2019
@coanor
Copy link
Author

coanor commented Nov 24, 2019

@ploxiln I've tried in raw docker install, but still failed with the same error. BTW, my fs type is ext4 (via df -T).

@coanor coanor reopened this Dec 2, 2019
@mdh67899
Copy link
Contributor

mdh67899 commented Dec 2, 2019

hi, @coanor , please use this docker-conpose file:

version: '3'
services:
  nsqlookupd:
    image: nsqio/nsq
    command: /nsqlookupd
    ports:
      - "4160"
      - "4161"
  nsqd:
    image: nsqio/nsq
    command: /nsqd --lookupd-tcp-address=nsqlookupd:4160 --data-path=/data/nsqd
    depends_on:
      - nsqlookupd
    ports:
      - "4150"
      - "4151"
    volumes:
      - /tmp/test:/data/nsqd

  nsqadmin:
    image: nsqio/nsq
    command: /nsqadmin --lookupd-http-address=nsqlookupd:4161
    depends_on:
      - nsqlookupd  
    ports:
      - "4171"

in nsqd container, this is no such directory: /data/nsqd, so it will produce this fatal error:

FATAL: failed to instantiate nsqd - --data-path=/data/nsqd in use (possibly by another instance of nsqd)

@coanor coanor closed this as completed Dec 4, 2019
@mreiferson mreiferson changed the title failed to instantiate nsqd - --data-path xxx possibly by another instance of nsqd nsqd: failed to instantiate nsqd - --data-path xxx possibly by another instance of nsqd Jun 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants