opentelekomcloud_identity_user_v3: error sending a welcome email: Action Forbidden code": 403,"message": "userId is invalid"

[shaderecker]

Terraform provider version

Terraform v1.8.0
on linux_amd64
+ provider registry.terraform.io/opentelekomcloud/opentelekomcloud v1.36.5

Affected Resource(s)

opentelekomcloud_identity_user_v3

Terraform Configuration Files

resource "opentelekomcloud_identity_user_v3" "user" {
  name               = "test_1"
  description        = "test_1"
  email              = "myemail+test1@gmail.com"
  pwd_reset          = false
  send_welcome_email = true
}

resource "opentelekomcloud_identity_project_v3" "project" {
  name        = "eu-nl_test_1"
  description = "Project Test1"
}

resource "opentelekomcloud_identity_group_v3" "group" {
  name        = "test1"
  description = "group test1"
}

resource "opentelekomcloud_identity_group_membership_v3" "membership" {
  group = opentelekomcloud_identity_group_v3.group.id
  users = [opentelekomcloud_identity_user_v3.user.id]
}

resource "opentelekomcloud_identity_role_assignment_v3" "role_assignment_test" {
  group_id   = opentelekomcloud_identity_group_v3.group.id
  project_id = opentelekomcloud_identity_project_v3.project.id
  role_id    = opentelekomcloud_identity_role_v3.role.id
}

data "opentelekomcloud_identity_role_v3" "ddos_adm" {
  name = "ddos_adm"
}

data "opentelekomcloud_identity_role_v3" "as_adm" {
  name = "as_adm"
}

data "opentelekomcloud_identity_role_v3" "tms_adm" {
  name = "tms_adm"
}

data "opentelekomcloud_identity_projects_v3" "all_projects" {
}

resource "opentelekomcloud_identity_role_assignment_v3" "role_assignment_ddos_adm" {
  group_id   = opentelekomcloud_identity_group_v3.group.id
  project_id = opentelekomcloud_identity_project_v3.project.id
  role_id    = data.opentelekomcloud_identity_role_v3.ddos_adm.id
}

resource "opentelekomcloud_identity_role_assignment_v3" "role_assignment_as_adm" {
  group_id   = opentelekomcloud_identity_group_v3.group.id
  project_id = opentelekomcloud_identity_project_v3.project.id
  role_id    = data.opentelekomcloud_identity_role_v3.as_adm.id
}

resource "opentelekomcloud_identity_role_assignment_v3" "role_assignment_tms_adm" {
  group_id  = opentelekomcloud_identity_group_v3.group.id
  domain_id = data.opentelekomcloud_identity_projects_v3.all_projects.id
  role_id   = data.opentelekomcloud_identity_role_v3.tms_adm.id
}

resource "opentelekomcloud_identity_role_v3" "role" {
  display_name  = "Test_Policy"
  description   = "Test policy description"
  display_layer = "project"

  statement {
    effect = "Allow"
    action = [
      "ecs:*:*",
      "evs:*:*",
      "vpc:*:delete",
      "vpc:*:get",
      "vpc:*:list",
      "vpc:networks:*",
      "vpc:subnets:*",
      "vpc:ports:*",
      "vpc:routers:*",
      "vpc:routeTables:*",
      "vpc:routes:*",
      "vpc:securityGroups:*",
      "vpc:securityGroupRules:*",
      "vpc:floatingIps:*",
      "vpc:publicIps:*",
      "vpc:bandwidths:*",
      "vpc:peerings:*",
      "vpc:vpcTags:*",
      "vpc:subnetTags:*",
      "vpc:publicipTags:*",
      "ims:*:*",
      "ces:*:*"
    ]
  }
}

Debug Output/Panic Output

│ Error: error sending a welcome email: Action Forbidden, error message: {"error": {"code": 403,"message": "userId is invalid","title": "Forbidden"}}
│
│   with opentelekomcloud_identity_user_v3.user,
│   on main.tf line 1, in resource "opentelekomcloud_identity_user_v3" "user":
│    1: resource "opentelekomcloud_identity_user_v3" "user" {
│
╵

https://gist.github.com/shaderecker/39a10b0d236c99a55df34fabee948492

Steps to Reproduce

1. terraform apply

Expected Behavior

• User gets created successfully and welcome email is sent

Actual Behavior

error during user creation, fails at "error sending a welcome email:"

Important Factoids

I did not change anything on my side since the last run.
The last time I ran my terraform code was on 04.04.24 and this was working back then.
The user has the same IAM permissions as before (admin), nothing changed there.

References

[artem-lifshits]

Hello @shaderecker I wasn't able to reproduce the issue.
opentelekomcloud_identity_user_v3 wasn't changed from provider side in a while.

[shaderecker]

Hmm, I can reproduce it just now.
With the provided TF configuration files.
Any ideas what we can do / debug further?

[artem-lifshits]

Maybe something was changed from API side regarding newly created projects. Can you try running the same config on existing one?

[shaderecker]

I now set the project_id to the id of the exiting "eu-nl" project.
Then the TF apply worked correctly.
What is different for newly created projects?

[artem-lifshits]

There shouldn't be any and as you mentioned it worked fine before.
That's why my guess is that something was changed on cloud side, not provider.

[shaderecker]

Albeit, it seems I was just lucky.
Now after a few retries, I am facing the issue also with the eu-nl project id:

[anton-sidelnikov]

Hi @shaderecker please ask support if there any restrictions for sending thise emails? This is not provider issue

[shaderecker]

I wrote an email to the OTC support

[shaderecker]

In the meantime I tracked the requests with mitmproxy and analyzed it a bit:
request:

response:

detail:

It doesn't happen every time. but can be reproduced after 4-5 tries.

[Nils-Magnus]

Did we find out what is the root cause of this behaviour? At first glimpse it looks like some IAM issue of POST https://iam.../v3.0/OS-USER/users/{project}/welcome. Could it be the special email receipient address (email = "myemail+test1@gmail.com") that triggers the issue? I have some suspicion that some validation regexp might fire here. This does not explain why it happens not every time, though.

TBH, technically this is not a Terraform issue, but, I'm pretty sure would also happen if we used the API directly here. Effectively that's what the API returns here (an 403 error). TF is just conveying this error to the user.

(Extra question: Is there a reason why I can't click on the screenshots to see it in original size? Those don't load for me?!)