[0.3.9] ipfs daemon times out on second start

[dignifiedquire]

Trying to debug some issues with station I stumbled on this timeout issue.

$ rm -rf ~/.ipfs
$ ipfs init
initializing ipfs node at /Users/dignifiedquire/.ipfs
generating 2048-bit RSA keypair...done
peer identity: QmQMG6CjXVuxqUDMSQwVeGai8tTdFjKp6YHFtTzjGsr56m
to get started, enter:

    ipfs cat /ipfs/QmVtU7ths96fMgZ8YSZAbKghyieq7AjxNdcqyVzxTt3qVe/readme
$ ipfs daemon
Initializing daemon...
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/192.168.0.11/tcp/4001
Swarm listening on /ip4/85.216.106.69/tcp/4001
Swarm listening on /ip4/85.216.106.69/tcp/56992
Swarm listening on /ip6/::1/tcp/4001
Swarm listening on /ip6/fd67:d185:7678:b25b:f8b7:874d:59ee:ef4b/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
^C
Received interrupt signal, shutting down...
$ ipfs daemon
Error: Post http://127.0.0.1:5001/api/v0/version?enc=json&stream-channels=true: dial tcp 127.0.0.1:5001: getsockopt: operation timed out

And all subsequent trials to start the daemon timeout the same way.

[dignifiedquire]

$ ipfs diag sys
{
  "diskinfo": {
    "free_space": 9794699264,
    "fstype": "hfs",
    "total_space": 120101797888,
    "used_space": 110307098624
  },
  "environment": {
    "GOPATH": "/Users/dignifiedquire/.go",
    "IPFS_PATH": ""
  },
  "ipfs_git_sha": "",
  "ipfs_version": "0.3.9",
  "memory": {
    "swap": {
      "total": 2048000,
      "used": 708500,
      "free": 1339500,
      "used_percent": 34.5947265625,
      "sin": 0,
      "sout": 0
    },
    "virt": {
      "total": 4294967296,
      "available": 933642240,
      "used": 4278861824,
      "used_percent": 78.26194763183594,
      "free": 16105472,
      "active": 900890624,
      "inactive": 897069056,
      "buffers": 0,
      "cached": 917536768,
      "wired": 1291022336,
      "shared": 0
    }
  },
  "net": {
    "interface_addresses": [
      "/ip6/::1",
      "/ip4/127.0.0.1",
      "/ip6/fe80::1",
      "/ip6/fe80::6aa8:6dff:fe0f:465c",
      "/ip4/192.168.0.11",
      "/ip6/fe80::f8b7:874d:59ee:ef4b",
      "/ip6/fd67:d185:7678:b25b:f8b7:874d:59ee:ef4b"
    ]
  },
  "runtime": {
    "arch": "amd64",
    "compiler": "gc",
    "gomaxprocs": 3,
    "numcpu": 4,
    "numgoroutines": 9,
    "os": "darwin",
    "version": "go1.5.1"
  }
}

[whyrusleeping]

does removing $IPFS_PATH/api fix the issue?

[dignifiedquire]

From IRC:

dignifiedquire: daemon is now initializing again
dignifiedquire: after killing with ctrl + c once same story as before

[dignifiedquire]

After git bisecting the issue:

5040fee906305e074777d4e545c5a35fee35f5b0 is the first bad commit
commit 5040fee906305e074777d4e545c5a35fee35f5b0
Author: Juan Batiz-Benet <juan@benet.ai>
Date:   Thu Aug 27 07:28:27 2015 +0200

    implement support for --api option

    This commit adds support for the --api option, which allows users
    to specify an API endpoint to run the cli command against. It enables
    much easier control of remote daemons.

    It also
    - ensures the API server version matches the API client
    - implements support for the $IPFS_PATH/api file

    Still TODO:
    - tests!
    - multiaddr to support /dns/

    License: MIT
    Signed-off-by: Juan Batiz-Benet <juan@benet.ai>

:040000 040000 44355e5c051099987344121350bee1375f76348b 59c7013dc814ec737b8f091921ce4ed5b843ad57 M  cmd
:040000 040000 a2b968ac1607a1530127a4a4ae53e15125e6bc6a cfa300c96463e15419d9e1355eeb7418ddb74f1a M  core
:040000 040000 d762c647380df25c34e3641b7e93f7bf86dd14e3 e49bada01ce580747dde71032bf532d1aa7ead57 M  repo

[jbenet]

Looks like the api file is not cleaned up from the crash. This is tricky because the api file may be there pointing to a remote daemon (what, you say? yes, you can make a dir like:

> ls ~/.ipfs
api
> cat ~/.ipfs/api
/ip4/1.2.3.4/tcp/5001

to always access a remote api and not have to constantly type --api $api. (very useful with docker containers)

---

How about this:

If user tries to run daemon but there is an api file in the repo -- check if the api works, using something like the version mismatch check.

• if the endpoint is live:

  error: cannot start daemon as repo defines api endpoint in /Users/jbenet/.ipfs/api
  The endpoint (/ip4/1.2.3.4/tcp/5001) is live.
  

• if the endpoint IS NOT live:

  error: cannot start daemon as repo defines api endpoint in /Users/jbenet/.ipfs/api
  But the endpoint (/ip4/1.2.3.4/tcp/5001) is not responding.
  Maybe it is a stale `api` file. Remove it and proceed? (y/N): 
  

• if we don't want the prompt, we can just ask the user to remove it themselves:

  error: cannot start daemon as repo defines api endpoint in /Users/jbenet/.ipfs/api
  But the endpoint (/ip4/1.2.3.4/tcp/5001) is not responding.
  Maybe it is a stale `api` file. Remove it and try again.
  To remove, run:  rm /Users/jbenet/.ipfs/api
  

[dignifiedquire]

The suggested solution would not work for anything controlled through ipfsd-ctl as the api file is never cleaned up at the moment, making it impossible to properly start the daemon a second time.

[jbenet]

@dignifiedquire makes sense. maybe ipfsd-ctl could cleanup the file, as it knows the api file should not be there when it's trying to start the daemon.

[dignifiedquire]

ipfsd-ctl could clean the file up, but it seems like a bug to me that on regular shutdown the file is not cleaned up. (The report above is about regular shutdowns not crashes)