Terraform Cloud

[neilmock]

This seems to assume an open WG tunnel during provisioning, I can't think of a way this could work in TF Cloud but might be missing something? Thanks!

[mochja]

@neilmock you could theoretically proxy it if you really have to and then set https://registry.terraform.io/providers/fly-apps/fly/latest/docs#fly_http_endpoin and maybe even you could proxy it with another fly app?

I think #31 will solve this in the future.

[DAlperin]

@neilmock yeah this is a known problem. Like @mochja said there is some work going on that should make this easier soon.

[DAlperin]

@mochja @neilmock alrighty we finally have a beta internal tunnel that removes the need for any proxy stuff. See example on how to enable it:

provider "fly" {
  useinternaltunnel    = true
  internaltunnelorg    = "personal"
  internaltunnelregion = "ewr"
}

This is available in v0.0.17 which is releasing now

[hb9cwp]

@DAlperin @jsiebens Tested v.0.0.18 with fly_app, _ip, _machine stanzas and Terraform CLI v1.2.9 from ChromeOS (Debian amd64) works great so far!
Now, I will try with remote Terraform Cloud workspace as well.
Thank you very much for this amazing work.

[hb9cwp]

@DAlperin I just noticed that the same .tf always times out while trying to create fly_machines using Terraform CLI v1.3.0 which was just released, whereas it applies fine using Terraform CLI v1.2.9:

fly_app.flyMachines: Creating...
fly_app.flyMachines: Creation complete after 0s [id=fly-machine-hello04]
fly_ip.IPv4: Creating...
fly_ip.IPv6: Creating...
fly_machine.exampleMachine["ams"]: Creating...
fly_machine.exampleMachine["fra"]: Creating...
fly_ip.IPv6: Creation complete after 2s [id=ip_qr702pxnm6v136zd]
fly_ip.IPv4: Creation complete after 3s [id=ip_3jwv94n3kd81p506]
fly_machine.exampleMachine["ams"]: Still creating... [10s elapsed]
fly_machine.exampleMachine["fra"]: Still creating... [10s elapsed]
fly_machine.exampleMachine["ams"]: Still creating... [20s elapsed]
fly_machine.exampleMachine["fra"]: Still creating... [20s elapsed]
^C
Interrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...

compared to

fly_app.flyMachines: Creating...
fly_app.flyMachines: Creation complete after 0s [id=fly-machine-hello04]
fly_ip.IPv6: Creating...
fly_ip.IPv4: Creating...
fly_machine.exampleMachine["ams"]: Creating...
fly_machine.exampleMachine["fra"]: Creating...
fly_ip.IPv6: Creation complete after 1s [id=ip_lm6k9x4v8kz1qp7r]
fly_machine.exampleMachine["ams"]: Creation complete after 5s [id=6e82929b37d987]
fly_ip.IPv4: Creation complete after 5s [id=ip_degn1rkq86d935om]
fly_machine.exampleMachine["fra"]: Creation complete after 6s [id=4d89044b492287]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

IPv4_address = "137.66.14.187"
IPv6_address = "2a09:8280:1::1:7663"

[DAlperin]

I literally can't imagine what the terraform cli could possibly be breaking, sigh. Thanks for finding that out. Do you mind opening a new issue for that?

In the meantime let me look into it and see what I find out.

[hb9cwp]

Today, I tried both versions of Terraform CLi again. Actually, with both of them terraform apply fails maybe in one out of three or four attempts. Creation of the app, IPv4 and IPv6 addresses always succeeds, sometimes creation of one of the two machines fails, sometimes both of them, and then Terraform aborts with an error message.
Sometimes, the apply stalls while trying to create the machines, until both machines are created after retrying for about 2 minutes:

$ terraform version
Terraform v1.3.0
on linux_amd64
+ provider registry.terraform.io/fly-apps/fly v0.0.18
$ terraform apply
...
Plan: 5 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + IPv4_address = (known after apply)
  + IPv4_region  = (known after apply)
  + IPv6_address = (known after apply)
  + IPv6_region  = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

fly_app.flyMachines: Creating...
fly_app.flyMachines: Creation complete after 0s [id=fly-machine-hello02]
fly_ip.IPv6: Creating...
fly_ip.IPv4: Creating...
fly_machine.exampleMachine["fra"]: Creating...
fly_machine.exampleMachine["ams"]: Creating...
fly_ip.IPv6: Creation complete after 1s [id=ip_qr702pxnpjy136zd]
fly_ip.IPv4: Creation complete after 3s [id=ip_3jwv94n38531p506]
fly_machine.exampleMachine["fra"]: Still creating... [10s elapsed]
fly_machine.exampleMachine["ams"]: Still creating... [10s elapsed]
fly_machine.exampleMachine["fra"]: Still creating... [20s elapsed]
fly_machine.exampleMachine["ams"]: Still creating... [20s elapsed]
fly_machine.exampleMachine["fra"]: Still creating... [30s elapsed]
fly_machine.exampleMachine["ams"]: Still creating... [30s elapsed]
fly_machine.exampleMachine["fra"]: Still creating... [40s elapsed]
fly_machine.exampleMachine["ams"]: Still creating... [40s elapsed]
fly_machine.exampleMachine["ams"]: Still creating... [50s elapsed]
fly_machine.exampleMachine["fra"]: Still creating... [50s elapsed]
fly_machine.exampleMachine["fra"]: Still creating... [1m0s elapsed]
fly_machine.exampleMachine["ams"]: Still creating... [1m0s elapsed]
fly_machine.exampleMachine["ams"]: Still creating... [1m10s elapsed]
fly_machine.exampleMachine["fra"]: Still creating... [1m10s elapsed]
fly_machine.exampleMachine["ams"]: Still creating... [1m20s elapsed]
fly_machine.exampleMachine["fra"]: Still creating... [1m20s elapsed]
fly_machine.exampleMachine["ams"]: Still creating... [1m30s elapsed]
fly_machine.exampleMachine["fra"]: Still creating... [1m30s elapsed]
fly_machine.exampleMachine["fra"]: Still creating... [1m40s elapsed]
fly_machine.exampleMachine["ams"]: Still creating... [1m40s elapsed]
fly_machine.exampleMachine["ams"]: Still creating... [1m50s elapsed]
fly_machine.exampleMachine["fra"]: Still creating... [1m50s elapsed]
fly_machine.exampleMachine["ams"]: Still creating... [2m0s elapsed]
fly_machine.exampleMachine["fra"]: Still creating... [2m0s elapsed]
fly_machine.exampleMachine["fra"]: Creation complete after 2m1s [id=9080542a1e0787]
fly_machine.exampleMachine["ams"]: Creation complete after 2m1s [id=1781900c57d489]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

IPv4_address = "168.220.90.171"
IPv6_address = "2a09:8280:1::3:785f"
$

Re-applying a partially failed apply manages to create the missing machine(s), most of the time in a second or third attempt.
Fortunately, terraform destroy always manages to remove the resources cleanly, no matter in which state they were.

Now, I will try with another image in order to exclude that this behavior is actually related to the image I was testing with so far.

[DAlperin]

@hb9cwp can you drop your config so I can try out a modified version?

[hb9cwp]

@DAlperin I have just sent you my build steps with the sources via PM. Thank you.

[hb9cwp]

Terraform Cloud appears to choke once it tries to open a WG tunnel, although I have added the env variable FLY_API_TOKEN via the workspace of TF Cloud:

Here is the detailed output from Download raw log:

�Terraform v1.3.0
on linux_amd64
Initializing plugins and modules...
{"@level":"info","@message":"Terraform 1.3.0","@module":"terraform.ui","@timestamp":"2022-09-25T12:54:21.841161Z","terraform":"1.3.0","type":"version","ui":"1.0"}
{"@level":"error","@message":"Error: failed to open internal tunnel","@module":"terraform.ui","@timestamp":"2022-09-25T12:54:23.111115Z","diagnostic":{"severity":"error","summary":"failed to open internal tunnel","detail":"tunnel error (doConnect): tunnel error (wgDev.Up()): permission denied","address":"provider[\"registry.terraform.io/fly-apps/fly\"]","range":{"filename":"flyGoHTTPS.tf","start":{"line":34,"column":16,"byte":835},"end":{"line":34,"column":17,"byte":836}},"snippet":{"context":"provider \"fly\"","code":"provider \"fly\" {","start_line":34,"highlight_start_offset":15,"highlight_end_offset":16,"values":[]}},"type":"diagnostic"}
Operation failed: failed running terraform plan (exit 1)

[DAlperin]

@hb9cwp thanks for all the materials. I'll take a look. As for the permission denied error in TF Cloud I've reached out to some folks at hashicorp but we have a running theory about why their environment isn't playing along. Stand by, we'll see what we can do.

[jbarnette]

@DAlperin any word from the HashiCorp folks?

[mattste]

I'm also running into the permission denied issue when running remotely in Terraform Cloud.

[DAlperin]

@mattste @jbarnette @hb9cwp I have a present for you. The machines API is now public, meaning it does not require any wireguard. If you set the env val FLY_HTTP_ENDPOINT or the provider setting fly_http_endpoint to https://api.machines.dev it should work :)

[hb9cwp]

@DAlperin It looks like Xmas is early this year! Do you still need to do other changes before cutting a new release?
Latest v0.0.20 complains about : Error: fly wireguard tunnel must be open ... can't connect to the api, is the tunnel open? :)

Actually, it's just resource "fly_machine" that complains, while resource "fly_app" and resource "fly_ip" are fine! :-)

[hb9cwp]

@DAlperin I can confirm that your provider now works with remote Terraform using a workspace on HashiCorp Cloud Platform (HCP) as well (as long as I don't try to use resource fly_machine)! Thank you very much.

[mattste]

I decided to go ahead and publish a fork of this provider that removes the Wireguard tunnel code until @DAlperin can resolve this issue as they see fit. You can access the fork here.

To use the provider, configure it like this (if you're using cdktf):

new FlyProvider(this, "fly", {
    flyApiToken: "your-api-token-value",
    flyHttpEndpoint: "https://api.machines.dev"
})

[DAlperin]

@mattste thanks for this. I'll get a new version released today or tomorrow

[DAlperin]

@mattste @hb9cwp and co I made a silly mistake. The provider automatically prepends http(s) so just set FLY_HTTP_ENDPOINT or the provider setting fly_http_endpoint to api.machines.dev without https. I just tested it and it works.

[mattste]

@mattste @hb9cwp and co I made a silly mistake. The provider automatically prepends http(s) so just set FLY_HTTP_ENDPOINT or the provider setting fly_http_endpoint to api.machines.dev without https. I just tested it and it works.

Thanks for looking into this! I can confirm changing the endpoint to just the hostname fixes it.

I did not look carefully enough at this code to see that it's just a basic connection test not specific to WG.

Also, is there a security concern with using http as the code does now?

[DAlperin]

I'll check to be sure but I'm fairly sure it gets upgraded to https automatically

[mattste]

I created a post on the Hashicorp forums asking that my provider be unpublished to avoid potential confusion with the official provider. If you happen to have a support contact at Hashicorp then feel free to forward the request.

[DAlperin]

I think we can close this :)