Crash on MacOS, TF 0.9.6, vsphere_virtual_machine, attempting plan on shut down VMs

[glennmcallister]

Summary:

Previous vSphere VMs created fine on May 26, 2017. Executed plan after making minor changes in a "remote-exec" provisioner. Expected no changes, however TF showed unexpected disk changes:

~ module.haproxy.vsphere_virtual_machine.haproxy.0
    disk.#:                          "1" => "1"
    disk.3862106589.bootable:        "true" => "true"
    disk.3862106589.controller_type: "scsi" => "scsi"
    disk.3862106589.datastore:       "BCL-SIO-VOL-INFRA-001" => "BCL-SIO-VOL-INFRA-001"
    disk.3862106589.iops:            "0" => ""
    disk.3862106589.keep_on_remove:  "false" => "false"
    disk.3862106589.key:             "0" => "<computed>"
    disk.3862106589.name:            "" => ""
    disk.3862106589.size:            "0" => ""
    disk.3862106589.template:        "Shared-Templates/centos-template" => "Shared-Templates/centos-template"
    disk.3862106589.type:            "thin" => "thin"
    disk.3862106589.uuid:            "" => "<computed>"
    disk.3862106589.vmdk:            "" => ""

~ module.haproxy.vsphere_virtual_machine.haproxy.1
    disk.#:                          "1" => "1"
    disk.3862106589.bootable:        "true" => "true"
    disk.3862106589.controller_type: "scsi" => "scsi"
    disk.3862106589.datastore:       "BCL-SIO-VOL-INFRA-001" => "BCL-SIO-VOL-INFRA-001"
    disk.3862106589.iops:            "0" => ""
    disk.3862106589.keep_on_remove:  "false" => "false"
    disk.3862106589.key:             "0" => "<computed>"
    disk.3862106589.name:            "" => ""
    disk.3862106589.size:            "0" => ""
    disk.3862106589.template:        "Shared-Templates/centos-template" => "Shared-Templates/centos-template"
    disk.3862106589.type:            "thin" => "thin"
    disk.3862106589.uuid:            "" => "<computed>"
    disk.3862106589.vmdk:            "" => ""

In preparation for recreating the VMs, I had stopped them but not yet deleted them in vSphere. I usually delete them, but I had forgotten.

I ran plan at this point, which triggered the crash:

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform Version

$ terraform --version
Terraform v0.9.6

Affected Resource(s)

• vsphere_virtual_machine

Terraform Configuration Files

resource "vsphere_virtual_machine" "haproxy" {
  count         = "${var.instance_count}"
  name          = "${var.cluster_name}-k8s-hap-${format("%03d", count.index + 1)}.${var.base_domain}"
  folder        = "${var.folder}"
  vcpu          = "${var.vcpu_count}"
  memory        = "${var.memory_size * 1024}"
  domain        = "${var.base_domain}"
  datacenter    = "${var.datacenter}"
  cluster       = "${var.cluster}"
  resource_pool = "${var.resource_pool}"

  lifecycle {
    ignore_changes = [
      # Have to ignore the network_interface because to create the network, we need 
      # the label value to be 'aci-network/shared|shared-anp|bpc-be01'. However once
      # its created, VMWare reports the portgroup to be 'shared|shared-anp|bpc-be01'.
      "network_interface",
      "disk",
    ]
  }

  network_interface {
    label               = "${var.portgroup}"
    ipv4_address        = "${element(var.ips, count.index)}"
    ipv4_prefix_length  = "${var.ip_prefix_length}"
    ipv4_gateway        = "${var.gateway}"
  }

  disk {
    datastore       = "${var.datastore}"
    template        = "${var.template}"
    type            = "thin"
    bootable        = "true"
    keep_on_remove  = false
  }

  provisioner "file" {
    connection {
      type     = "ssh"
      user     = "root"
      password = "${var.root_passwd}"
    }

    content     = "${file("${path.module}/resources/salt-latest.repo")}"
    destination = "/etc/yum.repos.d/salt-latest.repo"
  }

  provisioner "file" {
    connection {
      type     = "ssh"
      user     = "root"
      password = "${var.root_passwd}"
    }

    content     = "${file("${path.module}/resources/saltstack-signing-key")}"
    destination = "/etc/pki/rpm-gpg/saltstack-signing-key"
  }

  provisioner "remote-exec" {
    connection {
      type     = "ssh"
      user     = "root"
      password = "${var.root_passwd}"
    }

    inline = [
      # several command...
    ]
  }
}

Panic Output

crash.log output: https://gist.github.com/glennmcallister/d7eae5bb403044c86c96f8ab17446ebd

Expected Behavior

Should have ignored the disk difference.

Actual Behavior

Crashed.

Steps to Reproduce

1. terraform plan

Important Factoids

There were 8 VMs the plan was being executed against. Two of the VMs were shut off, but not destroyed.

When I turn the VMs back on, I observe the expected behaviour.

References

None at this time.

[bpoland]

@glennmcallister do you know if your virtualcenter was restarted since the VMs were first deployed? This seems similar to my report #13803 which only seems to happen when vcenter is rebooted.

[glennmcallister]

I checked with our cluster administrator, and he got a list of the events with id 6005, which he's pretty sure is accurate enough for our purposes. If not, let me know and I'll go back to him. The last reboot was on April 26th, 2017. It was a graceful shutdown and restart. It was long before this crash.

BTW, that event represent vCenter boot times, not individual service restarts.

[glennmcallister]

I asked our administrator to double check, and he said that he couldn't find any service restarts that entire day of May 26th, 2017.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.