POST to /api/v1/token returns WEBrick::HTTPStatus::LengthRequired error

[timidri]

Hi,

This issue has been reported as a comment in #129 but should have been a separate issue from the start.

We are running into trouble when trying to use the vmpooler API from Beaker.

We've configured a local LDAP server for authentication and when we try to generate a token, it errors out:

curl -X POST -u bas --url localhost:4567/api/v1/token
Enter host password for user 'bas':

!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
HTML>
HEAD>Length Required
BODY>
H1>Length Required
WEBrick::HTTPStatus::LengthRequired
HR>
ADDRESS>
WEBrick/1.3.1 (Ruby/2.0.0/2014-11-13) at
localhost:4567
/ADDRESS>

/BODY>
/HTML>

What are we doing wrong?

[sschneid]

Hi @timidri,

We added LDAP support in vmpooler without a wide range of different LDAP environments to test against, so there are any number of things that could be going on. If you're willing to help with some troubleshooting, I'd be happy to help get this working for a larger number of LDAP servers and configurations.

We're using OpenLDAP 2.4.31 as the LDAP service, and vmpooler is connecting over port 389 and initializing a start_tls (TLSv1) connection. You can see that some of these connection specifics are hard-coded here. Knowing some of your configuration details and if your LDAP service supports a similar connection method would be helpful.

WEBrick::HTTPStatus::LengthRequired is somewhat of a quandary -- can you re-issue the curl command with -v and specify what the HTTP error code is?

[bazzie]

Hi Scott,

This is the requested output:

Enter host password for user 'vmpooler':

• About to connect() to vmpooler.wildemann.nl port 4567 (#0)
• Trying 192.168.1.152...
• Connected to vmpooler.wildemann.nl (192.168.1.152) port 4567 (#0)
• Server auth using Basic with user 'vmpooler'

  POST /api/v1/token HTTP/1.1
  Authorization: Basic dm1wb29sZXI6U3VwZXIxMTM=
  User-Agent: curl/7.29.0
  Host: vmpooler.wildemann.nl:4567
  Accept: /

  < HTTP/1.1 411 Length Required
  < Content-Type: text/html; charset=ISO-8859-1
  < Server: WEBrick/1.3.1 (Ruby/2.0.0/2014-11-13)
  < Date: Fri, 02 Oct 2015 12:22:24 GMT
  < Content-Length: 315
  < Connection: close

I've got some additional info. When I add -d ' ' at the end of the curl command, a token is generated:

curl -v -X POST -u vmpooler --url vmpooler.wildemann.nl:4567/api/v1/token -d ''

Enter host password for user 'vmpooler':

• About to connect() to vmpooler.wildemann.nl port 4567 (#0)
• Trying 192.168.1.152...
• Connected to vmpooler.wildemann.nl (192.168.1.152) port 4567 (#0)
• Server auth using Basic with user 'vmpooler'

  POST /api/v1/token HTTP/1.1
  Authorization: Basic dm1wb29sZXI6U3VwZXIxMTM=
  User-Agent: curl/7.29.0
  Host: vmpooler.wildemann.nl:4567
  Accept: /
  Content-Length: 0
  Content-Type: application/x-www-form-urlencoded

  < HTTP/1.1 200 OK
  < Content-Type: application/json
  < Content-Length: 0
  < X-Content-Type-Options: nosniff
  < Server: WEBrick/1.3.1 (Ruby/2.0.0/2014-11-13)
  < Date: Fri, 02 Oct 2015 12:25:28 GMT
  < Connection: Keep-Alive
  <
• Connection #0 to host vmpooler.wildemann.nl left intact

The token is generated:

curl -u vmpooler --url vmpooler.wildemann.nl:4567/api/v1/token
Enter host password for user 'vmpooler':
{
"ok": true,
"vzjaxlnd00ksniitsnabixf4jaalrg89": {
"created": "2015-10-02 11:40:34 +0000",
"last": "never"
},
"ci42won59vjk9p5dtrhf2laov9tuu6sj": {
"created": "2015-10-02 12:25:28 +0000",
"last": "never"
},
"guypl74heig235fo8x0ph7e1b94398cl": {
"created": "2015-09-28 09:13:31 +0000",
"last": "never"
}
}

[bazzie]

LDAP details:
openldap
x86_64
Version 2.4.39
Release 6.el7

ldaps with tlsv1 is supported in our config.

[timidri]

Hi @sschneid,

We tested some more today and we found that the problem is not the communication with the LDAP server - that works fine. We were firstly mislead by the
curl -X POST -u bas --url localhost:4567/api/v1/token
failing; that is because vmpooler apparently requires a Content-Length header in the request. Changing the curl call to
curl -X POST -u bas --url localhost:4567/api/v1/token -d ''
makes it work.
The real problem, though, lies in the fact that beaker receives HTTP error code 503 from vmpooler when it tries to get a VM. Our VM does get created and booted, but op IP nor DNS resolution work yet. Is that the reason vmpooler doesn't mark them as ready and returns 503 when beaker asks for a VM?

[colinPL]

@timidri,

If vmpooler cannot connect to port 22 (ssh) of the new VM, then the VM is not moved to the ready state:
https://github.com/puppetlabs/vmpooler/blob/master/lib/vmpooler/pool_manager.rb#L33-L44

This check can fail for a variety of reasons:

1. DNS issues
2. new VM is still booting up
3. firewalls
4. and more

VMs that fail to move to ready and reach the timeout limit will be marked as failed and an entry will be made to the vmpooler log file:
https://github.com/puppetlabs/vmpooler/blob/master/lib/vmpooler/pool_manager.rb#L55

NOTE: The vmpooler application will not log in to the VM and does not need valid login credentials! It is just checking that the ssh service of the new VM is reachable.

[colinPL]

I forgot to address the 503 issue in my previous response.

If the response you are receiving is JSON that includes "ok" = false, then the 503 is coming from the vmpooler application.

Code within the POST /vm/ endpoint that sets the response code (503) and ok to false:
https://github.com/puppetlabs/vmpooler/blob/master/lib/vmpooler/api/v1.rb#L327-L336

This behavior is also present within the POST /vm/<pool> endpoint:
https://github.com/puppetlabs/vmpooler/blob/master/lib/vmpooler/api/v1.rb#L398-L406

[bazzie]

@colinPL
I understand the mechanism of moving the VM to the ready queue.
But when a new VM is created, It is cloned from a template vm wich has a hostname configured. When this VM requests an address, it is registered in DNS by the hostname that's configured in the template. So you will be unable to resolve the vmpooler-generated hostname thus never gets moved to the ready queue.

Is there a way to work around this?

[timidri]

More in general: can vmpooler set the hostname itself? I think it's possible using the vsphere API? Otherwize, how would one go about assigning the hostname which should be equal to the vm name early enough in the boot process, while also getting an IP address from a DHCP server?

[bazzie]

Beware: I'm not a Ruby-ninja :)

Im trying to set the hostname via VMware Guest Customization where i can define that the GuestOS hostname should be equal to the VM name.
I'm trying to read the Customization spec in vsphere_helper.rb:

def find_customization(name)
  begin
     @connection.serviceInstance.CurrentTime
  rescue
     initialize
  end

  csm = @connection.serviceContent.customizationSpecManager
  csm.GetCustomizationSpec(:name => name).spec

  csm
end

When i call this in pool_manager.rb, it doesn't work:

csi = find_customization("vmpooler")
cust_spec = csi
ident = RbVmomi::VIM.CustomizationLinuxPrep
ident.hostName = RbVmomi::VIM.CustomizationFixedName
ident.hostName.name = vm['hostname']
cust_spec.identity = ident

Create a clone spec

spec = RbVmomi::VIM.VirtualMachineCloneSpec(
location: relocateSpec,
config: configSpec,
customization: cust_spec,
powerOn: true,
template: false
)

Any suggestions?

[sschneid]

Hi @bazzie,

There are a number of possible ways to set the VM's hostname -- I couldn't get the VMware "customization" stuff working for any of our OSes (Linux/UNIX or Windows) when we started writing vmpooler, so instead settled on init scripts to take care of this process: https://gist.github.com/sschneid/bd4b99a45a2f61a9ecc0

[bazzie]

@sschneid We managed to get it working with the init script you supplied. So that's great news! I would like to thank you and everybody who helped us out.