Terragrunt Provider Cache Server does not search multiple provider sources

[kevinanderson1]

Describe the bug

If TERRAGRUNT_PROVIDER_CACHE=1 is set and multiple provider installation blocks are present in the terraform configuration file it will not fall back to different sources even if the requested version is not found in the first source but it is present in the second source.

If TERRAGRUNT_PROVIDER_CACHE is unset then it will successfully search all provider_installation paths.

Steps To Reproduce

I could not really come up with a shorter way to reproduce this error but the steps below do reproduce it reliably.

1. Create a scratch directory and relevant subdirectories for the work.
   mkdir -p terragrunt-bug/{module-null-3.1.1,module-null-3.2.3,targets/null-1,targets/null-2}
2. Create the following files. The filenames are comments in the file and are relative to the terragrunt-bug directory

# module-null-3.1.1/main.tf
terraform {
  required_providers {
    null = {
      source = "hashicorp/null"
      version = "3.1.1"
    }
  }
}

provider "null" {
}

# module-null-3.2.3/main.tf 
terraform {
  required_providers {
    null = {
      source = "hashicorp/null"
      version = "3.2.3"
    }
  }
}

provider "null" {
}

# targets/null-1/terragrunt.hcl 
terraform {
  source = "../../module-null-3.2.3"
}

# targets/null-2/terragrunt.hcl 
terraform {
  source = "../../module-null-3.1.1"
}

Full directory listing:

~/terragrunt-bug$ tree
.
├── module-null-3.1.1
│   └── main.tf
├── module-null-3.2.3
│   └── main.tf
└── targets
    ├── null-1
    │   └── terragrunt.hcl
    └── null-2
        └── terragrunt.hcl

6 directories, 4 files

3. Mirror each provider to a different directory

cd targets/null-1
terragrunt run-all providers mirror ~/offline-providers1
cd ../../

cd targets/null-2
terragrunt run-all providers mirror ~/offline-providers2
cd ../../

4. Host the contents of offline-providers2 on a webserver to simulate a network_mirror. (I'm considering those exact steps as out of scope since it requires a domain name and a TLS certificate but any webserver with a valid certificate should do).
5. Create a Terraform configuration file with the relevant locations.

# /home/snowman/terraform.tfrc
provider_installation {
  filesystem_mirror {
    path = "/home/snowman/offline-providers1/"
  }
  network_mirror {
    # The contents of /home/snowman/offline-providers2 are copied to this network_mirror
    url = "https://<REDACTED>/"
  }
  direct {
    # In my environment I can't reach out directly.
    exclude = ["registry.terraform.io/*/*","registry.opentofu.org/*/*"]
  }
}

6. Export it in your shell

export TF_CLI_CONFIG_FILE=/home/snowman/terraform.tfrc

7. export TERRAGRUNT_PROVIDER_CACHE=1
8. cd targets
9. Execute terragrunt run-all init and experience the failure but only for one module

terragrunt run-all init
12:59:55.499 INFO   Terragrunt Cache server is listening on 127.0.0.1:43409
12:59:55.499 INFO   Start Terragrunt Cache server
12:59:55.500 INFO   The stack at . will be processed in the following order for command init:
Group 1
- Module ./null-1
- Module ./null-2


12:59:55.516 INFO   [null-1] Downloading Terraform configurations from ../module-null-3.2.3 into ./null-1/.terragrunt-cache/CEFrMKAbasLKustgG4oTezLUUU4/06nkfu8QWgqgekYqKwEuh4k0qCw
12:59:55.517 INFO   [null-2] Downloading Terraform configurations from ../module-null-3.1.1 into ./null-2/.terragrunt-cache/ov1bNOFqLh0RF47lnz6DzQXeaAc/vr9GJqZAahmzRe_uiJC5KUZfOcU
12:59:55.687 INFO   [null-1] Caching terraform providers for ./null-1/.terragrunt-cache/CEFrMKAbasLKustgG4oTezLUUU4/06nkfu8QWgqgekYqKwEuh4k0qCw
12:59:55.688 INFO   [null-2] Caching terraform providers for ./null-2/.terragrunt-cache/ov1bNOFqLh0RF47lnz6DzQXeaAc/vr9GJqZAahmzRe_uiJC5KUZfOcU
12:59:55.704 STDERR [null-2] tofu: ╷
12:59:55.704 STDERR [null-2] tofu: │ Error: Failed to resolve provider packages
12:59:55.704 STDERR [null-2] tofu: │ 
12:59:55.704 STDERR [null-2] tofu: │ Could not resolve provider hashicorp/null: no available releases match the
12:59:55.704 STDERR [null-2] tofu: │ given constraints 3.1.1
12:59:55.704 STDERR [null-2] tofu: ╵
12:59:55.704 STDERR [null-2] tofu: 
12:59:55.706 ERROR  [null-2] tofu invocation failed in ./null-2/.terragrunt-cache/ov1bNOFqLh0RF47lnz6DzQXeaAc/vr9GJqZAahmzRe_uiJC5KUZfOcU
12:59:55.706 ERROR  [null-2] Module ./null-2 has finished with an error
12:59:55.724 STDOUT [null-1] tofu: Initializing the backend...
12:59:55.725 STDOUT [null-1] tofu: Initializing provider plugins...
12:59:55.725 STDOUT [null-1] tofu: - Reusing previous version of hashicorp/null from the dependency lock file
12:59:55.725 STDOUT [null-1] tofu: - Installing hashicorp/null v3.2.3...
12:59:55.741 STDOUT [null-1] tofu: - Installed hashicorp/null v3.2.3 (unauthenticated)
12:59:55.741 STDOUT [null-1] tofu: OpenTofu has been successfully initialized!
12:59:55.741 STDOUT [null-1] tofu: 
12:59:55.741 STDOUT [null-1] tofu: You may now begin working with OpenTofu. Try running "tofu plan" to see
12:59:55.741 STDOUT [null-1] tofu: any changes that are required for your infrastructure. All OpenTofu commands
12:59:55.741 STDOUT [null-1] tofu: should now work.
12:59:55.741 STDOUT [null-1] tofu: If you ever set or change modules or backend configuration for OpenTofu,
12:59:55.741 STDOUT [null-1] tofu: rerun this command to reinitialize your working directory. If you forget, other
12:59:55.741 STDOUT [null-1] tofu: commands will detect it and remind you to do so if necessary.
12:59:55.743 INFO   Shutting down Terragrunt Cache server...
12:59:55.743 INFO   Terragrunt Cache server stopped
12:59:55.743 ERROR  error occurred:

* Failed to execute "tofu init -input=false" in ./null-2/.terragrunt-cache/ov1bNOFqLh0RF47lnz6DzQXeaAc/vr9GJqZAahmzRe_uiJC5KUZfOcU
  ╷
  │ Error: Failed to resolve provider packages
  │ 
  │ Could not resolve provider hashicorp/null: no available releases match the
  │ given constraints 3.1.1
  ╵
  
  
  exit status 1

Expected behavior

It's expected that the Terragrunt Provider Cache Server would fallback and search all provider_installation blocks similar to how the terragrunt/tofu commands do without the cache server.

If I unset TERRAGRUNT_PROVIDER_CACHE and run the same command I get a successful run (after cleaning the terragrunt cache at ~/.cache/terragrunt).

# Successful run without the cache server
terragrunt run-all init
13:05:03.424 INFO   The stack at . will be processed in the following order for command init:
Group 1
- Module ./null-1
- Module ./null-2


13:05:03.441 INFO   [null-2] Downloading Terraform configurations from ../module-null-3.1.1 into ./null-2/.terragrunt-cache/ov1bNOFqLh0RF47lnz6DzQXeaAc/vr9GJqZAahmzRe_uiJC5KUZfOcU
13:05:03.441 INFO   [null-1] Downloading Terraform configurations from ../module-null-3.2.3 into ./null-1/.terragrunt-cache/CEFrMKAbasLKustgG4oTezLUUU4/06nkfu8QWgqgekYqKwEuh4k0qCw
13:05:03.455 STDOUT [null-1] tofu: Initializing the backend...
13:05:03.455 STDOUT [null-1] tofu: Initializing provider plugins...
13:05:03.455 STDOUT [null-1] tofu: - Finding hashicorp/null versions matching "3.2.3"...
13:05:03.615 STDOUT [null-1] tofu: - Installing hashicorp/null v3.2.3...
13:05:03.681 STDOUT [null-1] tofu: - Installed hashicorp/null v3.2.3 (unauthenticated)
13:05:03.681 STDOUT [null-1] tofu: OpenTofu has created a lock file .terraform.lock.hcl to record the provider
13:05:03.681 STDOUT [null-1] tofu: selections it made above. Include this file in your version control repository
13:05:03.681 STDOUT [null-1] tofu: so that OpenTofu can guarantee to make the same selections by default when
13:05:03.681 STDOUT [null-1] tofu: you run "tofu init" in the future.
13:05:03.681 STDOUT [null-1] tofu: ╷
13:05:03.681 STDOUT [null-1] tofu: │ Warning: Incomplete lock file information for providers
13:05:03.681 STDOUT [null-1] tofu: │ 
13:05:03.681 STDOUT [null-1] tofu: │ Due to your customized provider installation methods, OpenTofu was forced
13:05:03.681 STDOUT [null-1] tofu: │ to calculate lock file checksums locally for the following providers:
13:05:03.681 STDOUT [null-1] tofu: │   - hashicorp/null
13:05:03.681 STDOUT [null-1] tofu: │ 
13:05:03.681 STDOUT [null-1] tofu: │ The current .terraform.lock.hcl file only includes checksums for
13:05:03.681 STDOUT [null-1] tofu: │ linux_amd64, so OpenTofu running on another platform will fail to install
13:05:03.681 STDOUT [null-1] tofu: │ these providers.
13:05:03.681 STDOUT [null-1] tofu: │ 
13:05:03.681 STDOUT [null-1] tofu: │ To calculate additional checksums for another platform, run:
13:05:03.681 STDOUT [null-1] tofu: │   tofu providers lock -platform=linux_amd64
13:05:03.681 STDOUT [null-1] tofu: │ (where linux_amd64 is the platform to generate)
13:05:03.681 STDOUT [null-1] tofu: ╵
13:05:03.681 STDOUT [null-1] tofu: 
13:05:03.681 STDOUT [null-1] tofu: OpenTofu has been successfully initialized!
13:05:03.681 STDOUT [null-1] tofu: 
13:05:03.681 STDOUT [null-1] tofu: You may now begin working with OpenTofu. Try running "tofu plan" to see
13:05:03.681 STDOUT [null-1] tofu: any changes that are required for your infrastructure. All OpenTofu commands
13:05:03.681 STDOUT [null-1] tofu: should now work.
13:05:03.681 STDOUT [null-1] tofu: If you ever set or change modules or backend configuration for OpenTofu,
13:05:03.681 STDOUT [null-1] tofu: rerun this command to reinitialize your working directory. If you forget, other
13:05:03.681 STDOUT [null-1] tofu: commands will detect it and remind you to do so if necessary.
13:05:03.455 STDOUT [null-2] tofu: Initializing the backend...
13:05:03.456 STDOUT [null-2] tofu: Initializing provider plugins...
13:05:03.456 STDOUT [null-2] tofu: - Finding hashicorp/null versions matching "3.1.1"...
13:05:03.614 STDOUT [null-2] tofu: - Installing hashicorp/null v3.1.1...
13:05:03.816 STDOUT [null-2] tofu: - Installed hashicorp/null v3.1.1 (verified checksum)
13:05:03.816 STDOUT [null-2] tofu: OpenTofu has created a lock file .terraform.lock.hcl to record the provider
13:05:03.816 STDOUT [null-2] tofu: selections it made above. Include this file in your version control repository
13:05:03.816 STDOUT [null-2] tofu: so that OpenTofu can guarantee to make the same selections by default when
13:05:03.816 STDOUT [null-2] tofu: you run "tofu init" in the future.
13:05:03.816 STDOUT [null-2] tofu: ╷
13:05:03.816 STDOUT [null-2] tofu: │ Warning: Incomplete lock file information for providers
13:05:03.816 STDOUT [null-2] tofu: │ 
13:05:03.816 STDOUT [null-2] tofu: │ Due to your customized provider installation methods, OpenTofu was forced
13:05:03.816 STDOUT [null-2] tofu: │ to calculate lock file checksums locally for the following providers:
13:05:03.816 STDOUT [null-2] tofu: │   - hashicorp/null
13:05:03.816 STDOUT [null-2] tofu: │ 
13:05:03.816 STDOUT [null-2] tofu: │ The current .terraform.lock.hcl file only includes checksums for
13:05:03.816 STDOUT [null-2] tofu: │ linux_amd64, so OpenTofu running on another platform will fail to install
13:05:03.816 STDOUT [null-2] tofu: │ these providers.
13:05:03.816 STDOUT [null-2] tofu: │ 
13:05:03.816 STDOUT [null-2] tofu: │ To calculate additional checksums for another platform, run:
13:05:03.816 STDOUT [null-2] tofu: │   tofu providers lock -platform=linux_amd64
13:05:03.816 STDOUT [null-2] tofu: │ (where linux_amd64 is the platform to generate)
13:05:03.816 STDOUT [null-2] tofu: ╵
13:05:03.816 STDOUT [null-2] tofu: 
13:05:03.816 STDOUT [null-2] tofu: OpenTofu has been successfully initialized!
13:05:03.816 STDOUT [null-2] tofu: 
13:05:03.816 STDOUT [null-2] tofu: You may now begin working with OpenTofu. Try running "tofu plan" to see
13:05:03.816 STDOUT [null-2] tofu: any changes that are required for your infrastructure. All OpenTofu commands
13:05:03.816 STDOUT [null-2] tofu: should now work.
13:05:03.816 STDOUT [null-2] tofu: If you ever set or change modules or backend configuration for OpenTofu,
13:05:03.816 STDOUT [null-2] tofu: rerun this command to reinitialize your working directory. If you forget, other
13:05:03.816 STDOUT [null-2] tofu: commands will detect it and remind you to do so if necessary.

Nice to haves

• Terminal output
• Screenshots

Versions

• Terragrunt version: v0.71.1
• OpenTofu/Terraform version: v1.8.8
• Environment details (Ubuntu 20.04, Windows 10, etc.): Fedora 41 x86_64

Additional context

In case it's needed this is the contents of the mirror folders referenced above.

tree ~/offline-providers*
/home/snowman/offline-providers1
└── registry.opentofu.org
    └── hashicorp
        └── null
            ├── 3.2.3.json
            ├── index.json
            └── terraform-provider-null_3.2.3_linux_amd64.zip
/home/snowman/offline-providers2
└── registry.opentofu.org
    └── hashicorp
        └── null
            ├── 3.1.1.json
            ├── index.json
            └── terraform-provider-null_3.1.1_linux_amd64.zip

8 directories, 6 files

[levkohimins]

@kevinanderson1, thanks for the steps to reproduce this issue!

Resolved in v0.73.15 release.