Error occurred unmarshalling JSON #24758

stumueller · 2024-02-02T10:01:24Z

Is there an existing issue for this?

I have searched the existing issues

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.7.0

AzureRM Provider Version

3.90.0

Affected Resource(s)/Data Source(s)

azurerm_data_factory_pipeline

Terraform Configuration Files

resource "azurerm_data_factory_pipeline" "powerbi_workspace_creation" {
  name            = "powerbi_workspace_creation"
  folder          = local.pbi_dp_folder
  data_factory_id = var.data_factory_id
  activities_json = templatefile(
    "${path.module}/../../templates/activities/common/pipelines/powerbi_workspace_pipeline_activities.json",
    {
      key_vault_linked_service = azurerm_data_factory_linked_service_key_vault.linked_service_key_vault.name
    }
  )
  variables = {
    "rapApiClientId"         = var.rap_api_clientid,
    "rapApiMatApplicationId" = var.rap_api_mat_application_id,
    "simsIdUrl"              = var.simsid_url,
    "rapApiUrl"              = var.rap_api_url,
    "reportingApiUrl"        = var.reporting_api_url,
    "azureKeyVaultUrl"       = var.azure_key_vault_url,
    "powerbiCapacityId"      = var.powerbi_capacityid,
    "resourceGroupName"      = var.databricks_resource_group_name,
    "wsSubscriptionId"       = var.azure_databricks_workspace_resource_id,
  }
}

Template json
[
  {
    "name": "get sims id token",
    "type": "WebActivity",
    "dependsOn": [
      {
        "activity": "get rap api client secret",
        "dependencyConditions": ["Succeeded"]
      }
    ],
    "policy": {
      "timeout": "0.00:05:00",
      "retry": 3,
      "retryIntervalInSeconds": 30,
      "secureOutput": false,
      "secureInput": true
    },
    "userProperties": [],
    "typeProperties": {
      "url": {
        "value": "@concat(variables('simsIdUrl'), 'connect/token')",
        "type": "Expression"
      },
      "method": "POST",
      "headers": {
        "Content-Type": "application/x-www-form-urlencoded"
      },
      "body": {
        "value": "@concat('client_id=',\nvariables('rapApiClientId'),'&client_secret=',\nactivity('get rap api client secret').output.value,'&grant_type=client_credentials','&scope=partner-read')",
        "type": "Expression"
      }
    }
  },
  {
    "name": "Get Sites by Application from RAP API",
    "type": "WebActivity",
    "dependsOn": [
      {
        "activity": "get sims id token",
        "dependencyConditions": ["Succeeded"]
      }
    ],
    "policy": {
      "timeout": "0.00:05:00",
      "retry": 3,
      "retryIntervalInSeconds": 30,
      "secureOutput": false,
      "secureInput": false
    },
    "userProperties": [],
    "typeProperties": {
      "url": {
        "value": "@concat(variables('rapApiUrl'),variables('rapApiMatApplicationId'),'/GetSiteByApplication')",
        "type": "Expression"
      },
      "method": "GET",
      "headers": {
        "authorization": {
          "value": "@concat('bearer',' ',activity('get sims id token').output.access_token)",
          "type": "Expression"
        },
        "content_type": "application/x-www-form-urlencoded"
      }
    }
  },
  {
    "name": "Filter Mats",
    "type": "Filter",
    "dependsOn": [
      {
        "activity": "Get Sites by Application from RAP API",
        "dependencyConditions": ["Succeeded"]
      }
    ],
    "userProperties": [],
    "typeProperties": {
      "items": {
        "value": "@json(activity('Get Sites by Application from RAP API').output.response)",
        "type": "Expression"
      },
      "condition": {
        "value": "@equals(coalesce(item().Authorised, 0), 1)",
        "type": "Expression"
      }
    }
  },
  {
    "name": "get rap api client secret",
    "type": "WebActivity",
    "dependsOn": [],
    "policy": {
      "timeout": "0.00:05:00",
      "retry": 3,
      "retryIntervalInSeconds": 30,
      "secureOutput": true,
      "secureInput": false
    },
    "userProperties": [],
    "typeProperties": {
      "url": {
        "value": "@concat(variables('azureKeyVaultUrl'), 'secrets/rap-api-client-secret/?api-version=7.0')",
        "type": "Expression"
      },
      "method": "GET",
      "authentication": {
        "type": "MSI",
        "resource": "https://vault.azure.net"
      }
    }
  },
  {
    "name": "create missing workspaces ForEach Mat",
    "type": "ForEach",
    "dependsOn": [
      {
        "activity": "Filter Mats",
        "dependencyConditions": ["Succeeded"]
      }
    ],
    "userProperties": [],
    "typeProperties": {
      "items": {
        "value": "@activity('Filter Mats').output.Value",
        "type": "Expression"
      },
      "isSequential": false,
      "batchCount": 2,
      "activities": [
        {
          "name": "create missing Workspace",
          "type": "WebActivity",
          "dependsOn": [],
          "policy": {
            "timeout": "0.00:05:00",
            "retry": 3,
            "retryIntervalInSeconds": 30,
            "secureOutput": false,
            "secureInput": false
          },
          "userProperties": [],
          "typeProperties": {
            "url": {
              "value": "@concat(variables('reportingApiUrl'), 'workspaces/', variables('powerbiCapacityId'))",
              "type": "Expression"
            },
            "method": "POST",
            "headers": {
              "Content-Type": "application/json"
            },
            "body": {
              "value": "@item()",
              "type": "Expression"
            },
            "authentication": {
              "type": "Basic",
              "username": "WorkspaceAdmin",
              "password": {
                "type": "AzureKeyVaultSecret",
                "store": {
                  "referenceName": "${key_vault_linked_service}",
                  "type": "LinkedServiceReference"
                },
                "secretName": "mat-frontend-webapi-basic-auth-secret"
              }
            }
          }
        }
      ]
    }
  }
]

Debug Output/Panic Output

reading the state of Data Factory Pipeline "powerbi_workspace_creation": datafactory.PipelinesClient#Get: Failure responding to request: StatusCode=200 -- Original Error: Error occurred unmarshalling JSON - Error = 'json: cannot unmarshal object into Go struct field WebActivityTypeProperties.headers of type string' JSON = '{

Expected Behaviour

Prior to the upgrade to azurerm 3.90.0 all datafactory pipelines where able to be read. Moving our azurerm provider back to 3.89.0 results in success again

Actual Behaviour

reading the state of Data Factory Pipeline "powerbi_workspace_creation": datafactory.PipelinesClient#Get: Failure responding to request: StatusCode=200 -- Original Error: Error occurred unmarshalling JSON - Error = 'json: cannot unmarshal object into Go struct field WebActivityTypeProperties.headers of type string' JSON = '{

Steps to Reproduce

No response

Important Factoids

No response

References

No response

ziyeqf · 2024-02-04T07:05:39Z

Hi @stumueller, thanks for opening this issue.

There is a data struct change on the WebActivity type activities_json, it's defined as a map now, but interface{} on the before versions. Do you mind checking the json file specified on your configuration ${path.module}/../../templates/activities/common/pipelines/powerbi_workspace_pipeline_activities.json, and share a piece of it, so I can help modifying that.

Thanks.

stumueller · 2024-02-05T09:15:21Z

Hi @stumueller, thanks for opening this issue.

There is a data struct change on the WebActivity type activities_json, it's defined as a map now, but interface{} on the before versions. Do you mind checking the json file specified on your configuration ${path.module}/../../templates/activities/common/pipelines/powerbi_workspace_pipeline_activities.json, and share a piece of it, so I can help modifying that.

Thanks.

yeah sure:

[
{
"name": "get sims id token",
"type": "WebActivity",
"dependsOn": [
{
"activity": "get rap api client secret",
"dependencyConditions": ["Succeeded"]
}
],
"policy": {
"timeout": "0.00:05:00",
"retry": 3,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": true
},
"userProperties": [],
"typeProperties": {
"url": {
"value": "@concat(variables('simsIdUrl'), 'connect/token')",
"type": "Expression"
},
"method": "POST",
"headers": {
"Content-Type": "application/x-www-form-urlencoded"
},
"body": {
"value": "@concat('client_id=',\nvariables('rapApiClientId'),'&client_secret=',\nactivity('get rap api client secret').output.value,'&grant_type=client_credentials','&scope=partner-read')",
"type": "Expression"
}
}
},
{
"name": "Get Sites by Application from RAP API",
"type": "WebActivity",
"dependsOn": [
{
"activity": "get sims id token",
"dependencyConditions": ["Succeeded"]
}
],
"policy": {
"timeout": "0.00:05:00",
"retry": 3,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": {
"value": "@concat(variables('rapApiUrl'),variables('rapApiMatApplicationId'),'/GetSiteByApplication')",
"type": "Expression"
},
"method": "GET",
"headers": {
"authorization": {
"value": "@concat('bearer',' ',activity('get sims id token').output.access_token)",
"type": "Expression"
},
"content_type": "application/x-www-form-urlencoded"
}
}
},
{
"name": "Filter Mats",
"type": "Filter",
"dependsOn": [
{
"activity": "Get Sites by Application from RAP API",
"dependencyConditions": ["Succeeded"]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "@JSON(activity('Get Sites by Application from RAP API').output.response)",
"type": "Expression"
},
"condition": {
"value": "@equals(coalesce(item().Authorised, 0), 1)",
"type": "Expression"
}
}
},
{
"name": "get rap api client secret",
"type": "WebActivity",
"dependsOn": [],
"policy": {
"timeout": "0.00:05:00",
"retry": 3,
"retryIntervalInSeconds": 30,
"secureOutput": true,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": {
"value": "@concat(variables('azureKeyVaultUrl'), 'secrets/rap-api-client-secret/?api-version=7.0')",
"type": "Expression"
},
"method": "GET",
"authentication": {
"type": "MSI",
"resource": "https://vault.azure.net"
}
}
},
{
"name": "create missing workspaces ForEach Mat",
"type": "ForEach",
"dependsOn": [
{
"activity": "Filter Mats",
"dependencyConditions": ["Succeeded"]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "@activity('Filter Mats').output.Value",
"type": "Expression"
},
"isSequential": false,
"batchCount": 2,
"activities": [
{
"name": "create missing Workspace",
"type": "WebActivity",
"dependsOn": [],
"policy": {
"timeout": "0.00:05:00",
"retry": 3,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": {
"value": "@concat(variables('reportingApiUrl'), 'workspaces/', variables('powerbiCapacityId'))",
"type": "Expression"
},
"method": "POST",
"headers": {
"Content-Type": "application/json"
},
"body": {
"value": "@item()",
"type": "Expression"
},
"authentication": {
"type": "Basic",
"username": "WorkspaceAdmin",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "${key_vault_linked_service}",
"type": "LinkedServiceReference"
},
"secretName": "mat-frontend-webapi-basic-auth-secret"
}
}
}
}
]
}
}
]

epicman25 · 2024-02-05T12:02:24Z

Hi @ziyeqf,I am also facing the same issue. Can you exactly say what kind of structure change you mean in WebActivity??
I am not sure where to make the changes.I am giving the current structure in my pipelines.

{
"name": "Name",
"type": "WebActivity",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": true,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": {
"value": "@concat(variables("")",
"type": "Expression"
},
"method": "GET",
"authentication": {
"type": "MSI",
"resource": ""
}
}
}

ziyeqf · 2024-02-06T03:05:55Z

Hi @stumueller and @epicman25,

Thanks for your detail, the struct change is the type of typeProperties.headers became map[string]string instead of interface{} in go (original change on the REST api definition: Azure/azure-rest-api-specs@97cefc8#diff-d5aa9217e2132932fec24cba8c2a103430c34ed925739a4529a0acd16483803aR8246).

So it could only contain one nest level for now, adding jsonencode() to other nest levels might work.

            "headers": {
                "authorization": jsonencode({
                    "value": "@concat('bearer',' ',activity('get sims id token').output.access_token)",
                    "type": "Expression"
                }),
                "content_type": "application/x-www-form-urlencoded"
            }

@epicman25 I'm not sure if it's caused by the same reason for your configuration as I noticed there is no headers specified, while I noticed it might need an escape for the : like the following:

{
    "name": "Name",
    "type": "WebActivity",
    "dependsOn": [],
    "policy": {
        "timeout": "7.00:00:00",
        "retry": 0,
        "retryIntervalInSeconds": 30,
        "secureOutput": true,
        "secureInput": false
    },
    "userProperties": [],
    "typeProperties": {
        "url": {
            "value": "@concat(variables(\"\")",
            "type": "Expression"
        },
        "method": "GET",
        "authentication": {
            "type": "MSI",
            "resource": ""
        }
    }
}

If the error continues, please leave comments.
Thanks.

stumueller · 2024-02-06T17:12:58Z

Hi @stumueller and @epicman25,

Thanks for your detail, the struct change is the type of typeProperties.headers became map[string]string instead of interface{} in go (original change on the REST api definition: Azure/azure-rest-api-specs@97cefc8#diff-d5aa9217e2132932fec24cba8c2a103430c34ed925739a4529a0acd16483803aR8246).

So it could only contain one nest level for now, adding jsonencode() to other nest levels might work.
            "headers": {
                "authorization": jsonencode({
                    "value": "@concat('bearer',' ',activity('get sims id token').output.access_token)",
                    "type": "Expression"
                }),
                "content_type": "application/x-www-form-urlencoded"
            }

Hey just to let you know that didn't work. Within the json file VScode now reports "value expected" and marks the file as red saying there is an error in the file.
If I ignore this and continue with the deployment I get the same error message :-(

ziyeqf · 2024-02-08T03:17:29Z

Hi @stumueller and @epicman25,
Thanks for your detail, the struct change is the type of typeProperties.headers became map[string]string instead of interface{} in go (original change on the REST api definition: Azure/azure-rest-api-specs@97cefc8#diff-d5aa9217e2132932fec24cba8c2a103430c34ed925739a4529a0acd16483803aR8246).
So it could only contain one nest level for now, adding jsonencode() to other nest levels might work.
            "headers": {
                "authorization": jsonencode({
                    "value": "@concat('bearer',' ',activity('get sims id token').output.access_token)",
                    "type": "Expression"
                }),
                "content_type": "application/x-www-form-urlencoded"
            }
Hey just to let you know that didn't work. Within the json file VScode now reports "value expected" and marks the file as red saying there is an error in the file. If I ignore this and continue with the deployment I get the same error message :-(

thanks, will continue looking into it.

pserafin · 2024-02-14T11:00:00Z

Any updates on this?
I have same issue with TF creating data factory pipeline using activities_json
Ive tried to adjust the code but I was unable to make it work.
Unfortunately I cannot return to azurerm 3.87 which is working with existing json definiton, because TF state file contains reference to latest azurerm version, which prevents my pipeline to be deployed.

ziyeqf · 2024-02-18T09:19:05Z

Hi @stumueller and @stumueller , sorry for my wrong answer before.
There are two issues opened on kermit and azure-rest-api-specs for this issue.:

I have also opened a PR(#24921) to the provider to fix this, once the above issues get replied we can continue work on this, or once the PR(#24921) merged, it should be fixed.

Thanks.

* fix #24758 * golint * add comment to workaround * update comment

…icorp#24921) * fix hashicorp#24758 * golint * add comment to workaround * update comment

l33tCod-er · 2024-02-23T10:00:46Z

The bug still exists, I will open a new item for it

…icorp#24921) * fix hashicorp#24758 * golint * add comment to workaround * update comment

github-actions · 2024-04-23T02:02:12Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

github-actions bot added service/data-factory v/3.x labels Feb 2, 2024

ziyeqf mentioned this issue Feb 18, 2024

[datafactory@2018-06-01]: webactivities.headers struct changed tombuildsstuff/kermit#381

Open

ziyeqf added a commit to ziyeqf/terraform-provider-azurerm that referenced this issue Feb 18, 2024

fix hashicorp#24758

9e7c8b2

ziyeqf mentioned this issue Feb 18, 2024

azurerm_data_factory_pipeline: fix headers acceptable value #24921

Merged

katbyte closed this as completed in #24921 Feb 20, 2024

katbyte pushed a commit that referenced this issue Feb 20, 2024

azurerm_data_factory_pipeline: fix headers acceptable value (#24921)

f970438

* fix #24758 * golint * add comment to workaround * update comment

github-actions bot added this to the v3.93.0 milestone Feb 20, 2024

rizkybiz pushed a commit to rizkybiz/terraform-provider-azurerm that referenced this issue Feb 21, 2024

azurerm_data_factory_pipeline: fix headers acceptable value (hash…

e6a5c40

…icorp#24921) * fix hashicorp#24758 * golint * add comment to workaround * update comment

l33tCod-er mentioned this issue Feb 23, 2024

Error unmarshalling JSON still exists #24998

Open

1 task

rizkybiz pushed a commit to rizkybiz/terraform-provider-azurerm that referenced this issue Feb 29, 2024

azurerm_data_factory_pipeline: fix headers acceptable value (hash…

4c2de32

…icorp#24921) * fix hashicorp#24758 * golint * add comment to workaround * update comment

github-actions bot locked as resolved and limited conversation to collaborators Apr 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error occurred unmarshalling JSON #24758

Error occurred unmarshalling JSON #24758

stumueller commented Feb 2, 2024

ziyeqf commented Feb 4, 2024 •

edited

Loading

stumueller commented Feb 5, 2024

epicman25 commented Feb 5, 2024 •

edited

Loading

ziyeqf commented Feb 6, 2024

stumueller commented Feb 6, 2024

ziyeqf commented Feb 8, 2024

pserafin commented Feb 14, 2024

ziyeqf commented Feb 18, 2024

l33tCod-er commented Feb 23, 2024

github-actions bot commented Apr 23, 2024

Error occurred unmarshalling JSON #24758

Error occurred unmarshalling JSON #24758

Comments

stumueller commented Feb 2, 2024

Is there an existing issue for this?

Community Note

Terraform Version

AzureRM Provider Version

Affected Resource(s)/Data Source(s)

Terraform Configuration Files

Debug Output/Panic Output

Expected Behaviour

Actual Behaviour

Steps to Reproduce

Important Factoids

References

ziyeqf commented Feb 4, 2024 • edited Loading

stumueller commented Feb 5, 2024

epicman25 commented Feb 5, 2024 • edited Loading

ziyeqf commented Feb 6, 2024

stumueller commented Feb 6, 2024

ziyeqf commented Feb 8, 2024

pserafin commented Feb 14, 2024

ziyeqf commented Feb 18, 2024

l33tCod-er commented Feb 23, 2024

github-actions bot commented Apr 23, 2024

ziyeqf commented Feb 4, 2024 •

edited

Loading

epicman25 commented Feb 5, 2024 •

edited

Loading