Enhance Terraform Support for Simultaneously Managing Shared and Assigned Repositories in Artifactory

[borkke]

Is your feature request related to a problem? Please describe.
It seems like I can't manage "shared" repositories together with “assigned” repositories and this is what I need to do. My TF state gets messed up.

Describe the solution you'd like
I would like to create remote repos for all public repos we use and share with all the projects (I need sharing here).
Then, we plan to have ~10 projects and for each project we need many local repos assigned to a project and virtual also assigned to a project which contains local and the "shared remotes" (here I need assignments). All via Terraform.

# I'm using "repos" as per the documentation
resource "project" "project" {
  key          = var.tla
  display_name = var.name
  .....
  # Not adding assignments yet on purpose
  repos = []

# 
resource "artifactory_remote_maven_repository" "mvn" {
  url = var.url
  key = var.name
  .....
  #ignoring bause I'm managing assignments via "project"
  #I coundn't find anything separate for "sharing" feature
  lifecycle {
    ignore_changes = [
      project_key
    ]
  }
}

Since I couldn't find an option to "share" via terraform, I went to the UI and shared it. Of course it starts to get messy. Even though I didn't use "assign to project" in the UI, I used "share", I'm seeing this change.

Terraform will perform the following actions:

  # module.onsite_project.project.project will be updated in-place
  ~ resource "project" "project" {
      ~ repos                      = [
          - "shared-docker-remote",
            # (1 unchanged element hidden)
        ]
    }

and then I get a message:

Error: failed to update repos for project:
│ 404 DELETE https://jfrog.bestsecrettec.com/access/api/v1/projects/_/attach/repositories/shared-docker-remote
│ {
│ "errors" : [ {
│ "code" : "NOT_FOUND",
│ "message" : "Could not find resource shared-docker-remote; type repo in Projects configuration"
│ } ]
│ }

In the end, I think it would be a good idea to allow people to "share" repos on the repo level as part of the "artifactory_virtual_maven_repository" resource and leave the "repos = []" on the project level only for project assignments.

Describe alternatives you've considered
I was thinking to use the "project_key" in the repo to do the assignments (which is not suggested by the docs) and use lifecycle to ignore the "repos" on the project level. Then I have to share the repos manually via UI.

[borkke]

I tried the alternative I mentioned above and it seems like that also does not work. Even with ignored repos on the project level the whole project gets tainted and it needs to be recreated after I "share" remote repos to "ALL" via web UI

  # module.onsite_project.project.project is tainted, so must be replaced
-/+ resource "project" "project" {
      ~ id                         = "web" -> (known after apply)
      - repos                      = [
          - "shared-docker-remote",
          - "shared-gradle-remote",
          - "shared-mvn-remote",
          - "shared-npm-remote",
          - "shared-pypi-remote",
        ] -> null
        # (6 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

[alexhung]

@borkke Have you tried the recently released project resource project_repository in v1.5.0? This decouples repository assignment (not sharing) from project settings, and therefore will not trigger state drifts in project.repos attribute.

[alexhung]

@borkke Project sharing support is forthcoming. We are waiting for some projects API enhancement needed before we can add it to the project provider.

[borkke]

@alexhung Thank you for a quick response. I will give it a try on Monday.

Regarding sharing option. This is already good information! Do you know roughly when I can expect this feature? Is there also a ticket somewhere that I can “follow” to get updates on the progress?

[alexhung]

@borkke Currently scheduled for Q2.

[borkke]

@alexhung Thanks for the information, I'm looking forward to this feature.

I tried the project_repository, looks much better, and it fits nicely into how I'm building my infra code. However, I came across another issue, which looks like a bug to me. I will explain here, but please tell me if it's better to open a separate ticket.

In few words:

• I implemented project management, virtual|local|remote repository management via TF
• I'm still trying to workaround the missing "share" functionality for remote repos by doing that manually after they are tf applied
• I'm getting this error when I tf apply "When applying changes to module.onsite_project.project_repository.maven_local_repo_snapshot_assignment[0], provider "provider["registry.terraform.io/jfrog/project"]" produced an unexpected new value: Root resource was present, but now absent."

I exported the log file and added some comments and TF plan
terraform-2.log

[alexhung]

@borkke Which version of the provider and Terraform CLI are you using?

[borkke]

@alexhung Seems like I'm a bit behind Terraform version. I will upgrade and try later today.

terraform --version
Terraform v1.5.7
on darwin_arm64
+ provider registry.terraform.io/jfrog/artifactory v10.4.1
+ provider registry.terraform.io/jfrog/project v1.5.1

[borkke]

@alexhung Same problem with the updated terraform version. Here is another log file.
terraform-2.log

[borkke]

I just noticed that it doesn't happen always. Sometimes works fine and sometimes fails with that error message and when it does, I just run it again it's ok.

[alexhung]

@borkke Thanks for the logs. I'll look into this.

Related jfrog/terraform-provider-artifactory#886

[alexhung]

@borkke So I think the issue is explained in this community post.

The project provider uses the Projects API to assign a repository to a project during the "Create" part of the process. Before the end of this process, the provider set the resource ID to <project-key>-<repo key>.

Then the provider uses the Artifactory Repository API to read back the current state from the repository. This read process also set the same resource ID (because "Read" is also used for refresh and import). However, due to the nature of Artifactory internal working, the repository may not immediately has the latest state. This means the repository's projectKey field may be empty at the time of API request. I included this logging statement for this scenario. And you can find it in your logs.

This is an known issue and we have plan in Q2 to add new Projects API to retrieve the repo assignment status reliably.

In the meantime, I'll add retry logic to the "Read" process and have the provider to make a few (5?) attempts before giving up.

[borkke]

@alexhung good news!
All cases I saw so far it succeeds after second tf apply - which means around 2-3 seconds.

I guess if you put 1s timeout between attempts and fail after 5 attempts, I think it should be fine.

[alexhung]

@borkke The Terraform SDKv2 retry mechanism doesn't support number of retry or interval between retries, unfortunately. It will attempt the retry until timeout is reached, which for Creation is 20 min out of the box. I can potentially set it lower but probably unnecessary.