Skip to content
This repository has been archived by the owner on Dec 5, 2020. It is now read-only.

Openshift 4.1 UPI installation failed on VMware (terraform version - v0.11.14) #65

Open
ghost opened this issue Dec 12, 2019 · 3 comments

Comments

@ghost
Copy link

ghost commented Dec 12, 2019

This issue was originally opened by @arunabhabanerjee as hashicorp/terraform#23640. It was migrated here as a result of the provider split. The original body of the issue is below.


Terraform Version

Terraform v0.11.14

  • provider.external v1.2.0
  • provider.ignition v1.2.0
  • provider.null v2.1.2
  • provider.template v2.1.2
  • provider.vsphere v1.13.0

-->

...

Terraform Configuration Files

``// ID identifying the cluster to create. Use your username so that resources created can be tracked back to you.
cluster_id = "test"

// Domain of the cluster. This should be "${cluster_id}.${base_domain}".
cluster_domain = "test.lab.local"

// Base domain from which the cluster domain is a subdomain.
base_domain = "lab.loacal"

// Name of the vSphere server. The dev cluster is on "vcsa.vmware.devcluster.openshift.com".
vsphere_server = "10.55.138.36"

// User on the vSphere server.
vsphere_user = "administrator@vsphere.local"

// Password of the user on the vSphere server.
vsphere_password = "password"

// Name of the vSphere cluster. The dev cluster is "devel".
vsphere_cluster = "dev"

// Name of the vSphere data center. The dev cluster is "dc1".
vsphere_datacenter = "BTM"

// Name of the vSphere data store to use for the VMs. The dev cluster uses "nvme-ds1".
vsphere_datastore = "datastore2"

// Name of the VM template to clone to create VMs for the cluster. The dev cluster has a template named "rhcos-latest".
vm_template = "rhcos-latest"

// The machine_cidr where IP addresses will be assigned for cluster nodes.
// Additionally, IPAM will assign IPs based on the network ID.
machine_cidr = "10.55.24.0/21"

// The number of control plane VMs to create. Default is 3.
control_plane_count = 3

// The number of compute VMs to create. Default is 3.
compute_count = 3

// URL of the bootstrap ignition. This needs to be publicly accessible so that the bootstrap machine can pull the ignition.
bootstrap_ignition_url = "http://10.55.1.5:8080/ignition/bootstrap.ign"

// Ignition config for the control plane machines. You should copy the contents of the master.ign generated by the installer.
control_plane_ignition = <<END_OF_MASTER_IGNITION
{"ignition":{"config":{"append":[{"source":"https://api-int.test.lab.local:22623/config/master","verification":{}}]},"security":{"tls":{"certificateAuthorities":[{"source":"data:text/plain;charset=utf-8;base64,LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFRENDQWZpZ0F3SUJBZ0lJUU5BQXRDbmhUVkl3RFFZSktvWklodmNOQVFFTEJRQXdKakVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1SQXdEZ1lEVlFRREV3ZHliMjkwTFdOaE1CNFhEVEU1TVRJd016RTBNVFF6T1ZvWApEVEk1TVRFek1ERTBNVFF6T1Zvd0pqRVNNQkFHQTFVRUN4TUpiM0JsYm5Ob2FXWjBNUkF3RGdZRFZRUURFd2R5CmIyOTBMV05oTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF1Q3FDSy8xYUtOOTAKQUFTamxQTTZxaFY2SHlPWGtKaFZKYm1YQzErbUthMG9CbnNiUGN4WE14NkFROXhSMSt0djJveFl1Q0lBRlNNUwplb3pIa2cyS2VhU25mQTJYdHVkRGZnaklTNUdsQ3lPRTJ4ZXZrZ1dzU0VVQXV3TmlTTXRRWS9BTWt1YnMyK3VZCmRVQUxUMHhsZWkyUWhoaFN6OXN3allNNitnaUh5K2JZMDZjZmxkbjF0NG5aRGZFeUlENFNaR25WeDBmVzNNZ3QKdVRPU200Q1E0WWpUZDlQd1pKL0RUckpkNG9sVjZHQlNoSXFBSHh0elBiaWZwMjZaTnY5MjRrN2trMFhKUDI5Lwp3MEErWDRPVG1tdnY0RVQ2SVJISDZlMFp6TW9BUkpReVF1ekJMUFhqZ0VGMFBtRDAvS2ZBSFlEc25Rdklza3JICldJa3A3enc5a3dJREFRQUJvMEl3UURBT0JnTlZIUThCQWY4RUJBTUNBcVF3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBZEJnTlZIUTRFRmdRVTI4TEhRTUlMZkNhREVBUGhhcWg0MmpVZjZvUXdEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUFuQ2dCME9saTl4cmszZkVUK2hCOE9hQWtuRncra2NGVHdGMDNaYS9HeDkydW8wcjJZNlI2bDB2Q1NuCml2VHhoRzBMeDhaTlNYMURlSkFRbDY2YjkrTHF2QVhqdDNqUFpnT1VzRnp5NVBYUnl4emRFMVNPNGRGQ2dZdUcKUHZTYTNObkJaU3JlV0N5Wkp2SWt5S3VjajZRZWhiN0ROUmlYSjJ4WkJSajUya2NGQ29LN1JQODlkNGI4ZlRkNQpjV0dhVDJYcDBsbmJyb0dRWUp2dklmVXZ1N21kdzJ4dUVxMGFFUTFkK00xUnB6eWVxbVRJNTBUcHcxMnBjbnVYCkJnNzI2NVZ3SUxhdVhTWWRxbklnTHJjZU1SejUxbnpaSFdMMHRBMjJMdWRJNE9uNkt0NlJLaXNTQlRsT0ZlYTAKTTQyOFBsVWkrTm9ENm0zUnpmVDhreXFTWS9rPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==","verification":{}}]}},"timeouts":{},"version":"2.2.0"},"networkd":{},"passwd":{},"storage":{},"systemd":{}}
END_OF_MASTER_IGNITION

// Ignition config for the compute machines. You should copy the contents of the worker.ign generated by the installer.
compute_ignition = <<END_OF_WORKER_IGNITION
{"ignition":{"config":{"append":[{"source":"https://api-int.test.lab.local:22623/config/worker","verification":{}}]},"security":{"tls":{"certificateAuthorities":[{"source":"data:text/plain;charset=utf-8;base64,LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFRENDQWZpZ0F3SUJBZ0lJUU5BQXRDbmhUVkl3RFFZSktvWklodmNOQVFFTEJRQXdKakVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1SQXdEZ1lEVlFRREV3ZHliMjkwTFdOaE1CNFhEVEU1TVRJd016RTBNVFF6T1ZvWApEVEk1TVRFek1ERTBNVFF6T1Zvd0pqRVNNQkFHQTFVRUN4TUpiM0JsYm5Ob2FXWjBNUkF3RGdZRFZRUURFd2R5CmIyOTBMV05oTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF1Q3FDSy8xYUtOOTAKQUFTamxQTTZxaFY2SHlPWGtKaFZKYm1YQzErbUthMG9CbnNiUGN4WE14NkFROXhSMSt0djJveFl1Q0lBRlNNUwplb3pIa2cyS2VhU25mQTJYdHVkRGZnaklTNUdsQ3lPRTJ4ZXZrZ1dzU0VVQXV3TmlTTXRRWS9BTWt1YnMyK3VZCmRVQUxUMHhsZWkyUWhoaFN6OXN3allNNitnaUh5K2JZMDZjZmxkbjF0NG5aRGZFeUlENFNaR25WeDBmVzNNZ3QKdVRPU200Q1E0WWpUZDlQd1pKL0RUckpkNG9sVjZHQlNoSXFBSHh0elBiaWZwMjZaTnY5MjRrN2trMFhKUDI5Lwp3MEErWDRPVG1tdnY0RVQ2SVJISDZlMFp6TW9BUkpReVF1ekJMUFhqZ0VGMFBtRDAvS2ZBSFlEc25Rdklza3JICldJa3A3enc5a3dJREFRQUJvMEl3UURBT0JnTlZIUThCQWY4RUJBTUNBcVF3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBZEJnTlZIUTRFRmdRVTI4TEhRTUlMZkNhREVBUGhhcWg0MmpVZjZvUXdEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUFuQ2dCME9saTl4cmszZkVUK2hCOE9hQWtuRncra2NGVHdGMDNaYS9HeDkydW8wcjJZNlI2bDB2Q1NuCml2VHhoRzBMeDhaTlNYMURlSkFRbDY2YjkrTHF2QVhqdDNqUFpnT1VzRnp5NVBYUnl4emRFMVNPNGRGQ2dZdUcKUHZTYTNObkJaU3JlV0N5Wkp2SWt5S3VjajZRZWhiN0ROUmlYSjJ4WkJSajUya2NGQ29LN1JQODlkNGI4ZlRkNQpjV0dhVDJYcDBsbmJyb0dRWUp2dklmVXZ1N21kdzJ4dUVxMGFFUTFkK00xUnB6eWVxbVRJNTBUcHcxMnBjbnVYCkJnNzI2NVZ3SUxhdVhTWWRxbklnTHJjZU1SejUxbnpaSFdMMHRBMjJMdWRJNE9uNkt0NlJLaXNTQlRsT0ZlYTAKTTQyOFBsVWkrTm9ENm0zUnpmVDhreXFTWS9rPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==","verification":{}}]}},"timeouts":{},"version":"2.2.0"},"networkd":{},"passwd":{},"storage":{},"systemd":{}}
END_OF_WORKER_IGNITION

// Set ipam and ipam_token if you want to use the IPAM server to reserve IP
// addresses for the VMs.

// Address or hostname of the IPAM server from which to reserve IP addresses for the cluster machines.
// ipam = "139.178.89.254"

// Token to use to authenticate with the IPAM server.
// ipam_token = "TOKEN_FOR_THE_IPAM_SERVER"

// Set bootstrap_ip, control_plane_ip, and compute_ip if you want to use static
// IPs reserved someone else, rather than the IPAM server.

// The IP address to assign to the bootstrap VM.
bootstrap_ip = "10.55.1.10"

// The IP addresses to assign to the control plane VMs. The length of this list
// must match the value of control_plane_count.
control_plane_ips = ["10.55.1.11", "10.55.1.12", "10.55.1.13"]

// The IP addresses to assign to the compute VMs. The length of this list must
// match the value of compute_count.
compute_ips = ["10.55.1.14", "10.55.1.15", "10.55.1.16"]
root:/home/arunabha/openshift/installer/upi/vsphere>
`hcl
...


### Debug Output
<!--
Full debug output can be obtained by running Terraform with the environment variable `TF_LOG=trace`. Please create a GitHub Gist containing the debug output. Please do _not_ paste the debug output in the issue, since debug output is long.

Debug output may contain sensitive information. Please review it before posting publicly, and if you are concerned feel free to encrypt the files using the HashiCorp security public key.
-->

### Crash Output
<!--
# terraform apply -auto-approve
data.ignition_systemd_unit.restart[1]: Refreshing state...
data.ignition_systemd_unit.restart[2]: Refreshing state...
data.ignition_systemd_unit.restart: Refreshing state...
data.ignition_file.hostname: Refreshing state...
data.ignition_systemd_unit.restart[0]: Refreshing state...
data.ignition_file.hostname[2]: Refreshing state...
data.ignition_systemd_unit.restart[0]: Refreshing state...
data.ignition_file.hostname[2]: Refreshing state...
data.ignition_file.hostname[1]: Refreshing state...
data.ignition_file.hostname[1]: Refreshing state...
data.ignition_file.hostname[0]: Refreshing state...
data.ignition_systemd_unit.restart[2]: Refreshing state...
data.ignition_file.hostname[0]: Refreshing state...
data.ignition_systemd_unit.restart[1]: Refreshing state...
data.ignition_file.static_ip[1]: Refreshing state...
data.ignition_file.static_ip[2]: Refreshing state...
data.ignition_file.static_ip[0]: Refreshing state...
data.ignition_file.static_ip[2]: Refreshing state...
data.ignition_file.static_ip: Refreshing state...
data.ignition_file.static_ip[0]: Refreshing state...
data.ignition_config.ign[1]: Refreshing state...
data.ignition_config.ign[2]: Refreshing state...
data.ignition_file.static_ip[1]: Refreshing state...
data.ignition_config.ign[0]: Refreshing state...
data.ignition_config.ign: Refreshing state...
data.ignition_config.ign[2]: Refreshing state...
data.ignition_config.ign[0]: Refreshing state...
data.ignition_config.ign[1]: Refreshing state...
data.vsphere_datacenter.dc: Refreshing state...
data.vsphere_datastore.datastore: Refreshing state...
data.vsphere_virtual_machine.template: Refreshing state...
data.vsphere_network.network: Refreshing state...
data.vsphere_network.network: Refreshing state...
data.vsphere_compute_cluster.compute_cluster: Refreshing state...
data.vsphere_virtual_machine.template: Refreshing state...
data.vsphere_network.network: Refreshing state...
data.vsphere_datastore.datastore: Refreshing state...
data.vsphere_virtual_machine.template: Refreshing state...
data.vsphere_datastore.datastore: Refreshing state...

Error: Error refreshing state: 3 errors occurred:
        * module.control_plane.data.ignition_config.ign: 3 errors occurred:
        * module.control_plane.data.ignition_config.ign[0]: data.ignition_config.ign.0: No valid JSON found, make sure you're using .rendered and not .id: invalid character 'c' after top-level value
        * module.control_plane.data.ignition_config.ign[1]: data.ignition_config.ign.1: No valid JSON found, make sure you're using .rendered and not .id: invalid character 'b' in literal false (expecting 'a')
        * module.control_plane.data.ignition_config.ign[2]: data.ignition_config.ign.2: No valid JSON found, make sure you're using .rendered and not .id: invalid character 'b' after top-level value


        * module.bootstrap.data.ignition_config.ign: 1 error occurred:
        * module.bootstrap.data.ignition_config.ign: data.ignition_config.ign: No valid JSON found, make sure you're using .rendered and not .id: invalid character 'b' after top-level value


        * module.compute.data.ignition_config.ign: 3 errors occurred:
        * module.compute.data.ignition_config.ign[0]: data.ignition_config.ign.0: No valid JSON found, make sure you're using .rendered and not .id: invalid character 'c' after top-level value
        * module.compute.data.ignition_config.ign[1]: data.ignition_config.ign.1: No valid JSON found, make sure you're using .rendered and not .id: invalid character 'e' after top-level value
        * module.compute.data.ignition_config.ign[2]: data.ignition_config.ign.2: No valid JSON found, make sure you're using .rendered and not .id: invalid character 'c' looking for beginning of value

-->

### Expected Behavior
<!--
terraform should able to deploy VM
-->

### Actual Behavior
<!--
terraform is unable to deply VM
-->

### Steps to Reproduce
<!--
Please list the full steps required to reproduce the issue, for example:
1. `terraform init`
2. `terraform apply -auto-approve`
-->

### Additional Context
<!--
Are there anything atypical about your situation that we should know? For example: is Terraform running in a wrapper script or in a CI system? Are you passing any unusual command line options or environment variables to opt-in to non-default behavior?
-->

### References
<!--
Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

- hashicorp/terraform#6017

-->
@aaronpi
Copy link

aaronpi commented Dec 13, 2019

Seeing the same issue here, with ignition provider 1.2.1 and terraform 0.11.4

  • module.jenkins.data.ignition_config.leader: data.ignition_config.leader: No valid JSON found, make sure you're using .rendered and not .id: invalid character 'a' looking for beginning of value

Rolling back/forcing ignition to use 1.1.0, as mentioned in the previous issue for this, solves for now, but that's not a permanent solution

@LorbusChris
Copy link
Contributor

LorbusChris commented Dec 17, 2019

the error message does deliver the fix though:
You need to adapt your tf files to use .rendered instead of .id to reference things in data.ignition...

@frenchtoasters
Copy link

I am hitting this same issue with the 1.2.1 Ignition provider and terraform 0.12.12, however as @LorbusChris mentioned I am already using the .rendered to access the data. My first thought would be that this has something to do with the how the provider handles ignitions that are rendered and then referenced using count.index.

I am currently referenced the rendered templates in the following manner that is causing this error:

"guestinfo.coreos.config.data" = base64encode(data.ignition_config.aio_ign.*.rendered[count.index])

I have also tested referencing it this way and still getting the same error:

"guestinfo.coreos.config.data" = base64encode(data.ignition_config.aio_ign[count.index].rendered)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants