vmpooler fails to fetch vm with dummy provider

[briancain]

I've got vmpooler set up in dev mode with the dummy example and I'm getting an HTTP 503 when I try to get a vm from the pool:

brian@biggermac: ~ % curl http://localhost:4567/vm/debian-7-x86_64 -X POST -v                                        
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 4567 failed: Connection refused
*   Trying fe80::1...
* TCP_NODELAY set
* Connection failed
* connect to fe80::1 port 4567 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 4567 (#0)
> POST /vm/debian-7-x86_64 HTTP/1.1
> Host: localhost:4567
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Content-Type: application/json
< X-Content-Type-Options: nosniff
< Content-Length: 17
<
{
  "ok": false
* Connection #0 to host localhost left intact
}%

brian@biggermac:vmpooler % VMPOOLER_DEBUG=0 ./vmpooler                                                                            ±[●][master]
[2017-09-25 22:05:05] starting vmpooler
== Sinatra (v1.4.8) has taken the stage on 4567 for production with backup from Puma
Puma starting in single mode...
* Version 3.10.0 (ruby 2.2.2-p95), codename: Russell's Teapot
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://0.0.0.0:4567
Use Ctrl-C to stop

After looking to see where it was failing...it looks like maybe it's failing to get something out of redis? https://github.com/puppetlabs/vmpooler/blob/master/lib/vmpooler/api/v1.rb#L40

I've got redis running with homebrew on mac and it seems to be running on the right port.

brian@biggermac:~ % brew services start redis
Service `redis` already started, use `brew services restart redis` to restart.

brian@biggermac:~ % telnet localhost 6379
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying fe80::1...
telnet: connect to address fe80::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Any ideas? Thanks!

[glennsarti]

There should be more text after

* Listening on tcp://0.0.0.0:4567
Use Ctrl-C to stop

This usually indicates that the pooler thread isn't running. It tends to swallow exceptions when running on a different thread. I would check the vmpooler log file (or configure one first)

Also did you try hitting the status endpoint? /api/v1/status

[glennsarti]

Oh interesting thought. Is the pooler/api trying to connect to redis on 127.0.0.1, whereas redis is bound to ::1 ?

[briancain]

@glennsarti - good point on the status endpoint, I did not try that! I'll do that later tonight and see if there's something there that could tell me what's wrong. Also I did see that the pooler swallows exceptions from the dev docs...I'll also see if there's anything of note after I ctrl-c.

So the default config that I'm using from the repo has it on localhost, but no port. I tried doing localhost:6379 and I ended up getting a error not being able to reach redis the moment the pooler starts up. Is the port configurable, or does the pooler expect redis on the same port?

[glennsarti]

Looks like it's always the default port (https://github.com/puppetlabs/vmpooler/blob/master/lib/vmpooler.rb#L87)

You could try ::1 in the vmpooler.conf file for the redis host

[briancain]

@glennsarti - interesting....I made a log file/folder and started seeing log output. And magically the vmpooler started working?

brian@biggermac: ~ % curl http://localhost:4567/vm/debian-7-x86_64 -X POST -v                                             
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 4567 failed: Connection refused
*   Trying fe80::1...
* TCP_NODELAY set
* Connection failed
* connect to fe80::1 port 4567 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 4567 (#0)
> POST /vm/debian-7-x86_64 HTTP/1.1
> Host: localhost:4567
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< X-Content-Type-Options: nosniff
< Content-Length: 112
<
{
  "ok": true,
  "debian-7-x86_64": {
    "hostname": "poolvm-au7c8f7tcf1yyit"
  },
  "domain": "company.com"
* Connection #0 to host localhost left intact
}%

[briancain]

I did notice last night that vmpooler printed an exception saying it couldn't access a dir that didn't exist (which was /var/log/vmpooler) but I didn't think that would cause it to not be operational, but perhaps that was it?

[glennsarti]

Ahh...the logging is pretty naive. If it can't log it is possible it crashed :-)

https://github.com/puppetlabs/vmpooler/blob/master/lib/vmpooler/logger.rb#L17-L19

[briancain]

Cool cool. Well I am guessing that's it! :)