IaC thoughts with Bicep / ARM

[ericcsinger]

Real world examples

I love your hello world template examples, but you folks really need to do a better job of documenting some more realistic scenarios with these templates. Maybe doing case studies where you show how a customer (or you folks) deployed 100 virtual networks, each with 1,000 subnets or something like that. And I don't mean showing a simple template where is 1,000 subnets of the same type. I mean showing how someone manages the parameters, dealing with subnets that might have all kinds of differing cidr ranges, properties, NSG, UDR's, etc. There are so many times where you look at an example and think "oh this will be easy" until you actually sit down and figure out the logistics of it at scale.

Better API documentation (properties / values):

In general you have a decent documentation, but you don't always show all the parameter values. For example, a SKU for a VM or a SQL server. I get it would be a nightmare for you folks to document each possiable sku. HOWEVER, documenting a way to get the values, shouldn't be. For example, run this azure cli command or this PowerShell command or this api to get the values for property "x". This property name matches to this property name in the command. Or something like that.

Modules for scope change lead to template sprawl

Requiring modules as a means to change scopes, adds complexity to our deployment processes. This forces us to craft tons of tiny templates to execute tasks that should be executed easily within a single template. To me, the point of module should only be for the purposes of template reuse / sharing. There is a scope property for resources, and so long as the scope is fully defined, that's all Azure should need to execute a given deployment.

For example, if I have 5 SQL servers, each in a different resourceGroups, each with their own databases to deploy, plus needing to set audit settings. I should be able to craft all of that in a single template. As it stands now, I need to create this convoluted method of having a parent template that kicks off a child template, and sometimes yet another child template.

Scoping issue again, but this time related to parent / child resources

Child resources "parent" property should need nothing more than a resourceId to form a well qualified link. For example, if i wanted to reuse a template for assigning roleassignments. The roleassignment resource needs to know what the parent / scope is. Well the only way you folks let us define that is by using an "existing resource" definition. This makes creating what should be a repeatable / shared template, almost impossible as I would have to define every possiable resource type This seems related to not being able to pass an existing resource as a parameter object, as right now, you only have 'object' as a viable option for a datatype and the scope property won't support that. Again, I don't even think the scope should need this. A resourceID should be all that's needed.

We need more programmatic features.

In the azure resource deployment its self, I don't see why we can't use variables to dynamically create the resource definition. For example, in point 2. It might be more workable for me to call an existing resource, if I could programmatically define it. See below

/// Instead of always being forced to do this
resource logAnalyticsWorkSpaces 'Microsoft.OperationalInsights/workspaces@2021-06-01'  existing = {
 name: ResourceName

///It would be so much more flexible if we could do this instead.
param  ResourceDef string = 'Microsoft.OperationalInsights/workspaces@2021-06-01'
param ResourceName string = 'name'

resource logAnalyticsWorkSpaces invokeExpression(ResourceDef)   existing = {
 name: ResourceName

General thoughts

As a sysadmin with a ton of scripting experience, I find it super frustrating working within all these limits of IaC. It's like for all the power we're supposed to gain for making resource deployments easy, it simply shifts the complexity to different areas. I'm 100% on board for automation and scripting. The concept of IaC is also something I'm 100% on board with. How it's currently articulated and created though to me is a nightmare. I don't have an experience with Terraform, so I have nothing else to compare IaC syntax to.

What I can compare it to though is doing deployments via PS. I think we really need a lot of the programatic flexablity that we have in a scripting language to really let IaC have an easier learning / adaptation curve.

Closing

I feel like there is a lot more I could write about on how I feel IaC is lacking. However, right now, these are the things for me that are leading to template and parameter file sprawl.

[brwilkinson]

Hey @ericcsinger

Wow there is a lot to unroll there 😊 Each one of these is a conversation, so I think it might be best to answer them in different threads here.

Real world examples

• This one is admittedly difficult since no one template fits everyone use-cases.
• However, it's a conversation worth having.

Here are 2 example repo's that show 2 very different ways to build out templates:

• Consultant built templates designed for generic building blocks for Enterprise customer deployments
  • https://github.com/Azure/ResourceModules
  • https://github.com/azure/ResourceModules/wiki <-- getting started wiki
• My personal project designed with a very specific implementation for deploying App Infra Stamps (for a single app platform).
  • https://github.com/brwilkinson/AzureDeploymentFramework/tree/main/ADF/bicep

Some examples...

1. This is the ResoureModules project

   • Each of these has the associate param file in the directory associated for the Resource Type.
     • networkSecurityGroups
     • virtualNetworks
       • Sample of param file for virtualnetwork

2. This is my project, which is specifically designed for rolling out Stamp environments for my customers

   • Sample param file with Subnets: https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/tenants/AOA/ACU1.T5.parameters.json#L360
   • Sample network Modules that consume that object
     • NSG.bicep
     • NetworkFlowLogs.bicep
     • VNET.bicep

[brwilkinson]

Adding this thread with some deployment ideas

#4586

We are also interested to extend the patterns + scenarios that are here and would be interested in ideas

https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/

[brwilkinson]

Better API documentation (properties / values):

All of Azure and Azure Resource Manager is API driven. The docs are all generated with REST based tools directly from the API's.

Whatever you see both in Bicep and in the docs links as below is what has been published by the API endpoints.

• https://docs.microsoft.com/en-us/azure/templates/
• https://docs.microsoft.com/en-us/rest/api/?view=Azure

This repo Azure/bicep takes specific API request to follow up with the Resource owners to request that they improve their API specifications. This is a work in progress and 'Quality' will improve over time as we are able to identify and update specific issues.

This is the actual API specification Repo: https://github.com/Azure/azure-rest-api-specs

If you are authoring Bicep files you should see this message if your code does not match the api specifications

That links to this: http://aka.ms/bicep-type-issues --> #784

• That is open issue for following up on these items.

So feel free to drop any specific examples here, if it doesn't fit the description above...

One additional item.. if you are looking for samples, often you can export live resources to assist with this... This is in the daily builds and will be in the next release which should be out within the next week... keep an eye out. you can paste in a resourceId and it will pull down the settings from the resource that has been deployed into your template, which can enhance your authoring experience.

# get an id from a storage account to export
Get-AzStorageAccount | foreach id

you will see the resource inserted

[brwilkinson]

We need more programmatic features.

This is coming...

PR: #4971

Issues:
#2246
#2245
#4158

[brwilkinson]

General thoughts

Some of what you are saying sounds like it's general Azure API frustration mixed with learning a new 'declarative' language. Also the fact that not all API's (definitions) are perfect and meet standards as being Idempotent, as they should be.

Given that you have PowerShell and AZ PowerShell experience, you likely have a good idea/advantage on understanding Azure Resource Types Etc, however there is definitely more road bumps in the declarative approach as compared to the imperative approach. Especially when you are doing continuous deployment i.e. deploying templates over and over.

Understanding ARM is not trivial, when it comes to deployment and deployment limits Etc.

I will say, as someone who has been doing ARM Template deployments for 4+ years that Bicep Provides a Fresh perspective to deployment over the previous iteration of default ARM template/JSON deployments.

I would definitely recommend a few things:

1. How do you eat and elephant? One bite at a time.
   1. We are current at version 0.4 of Bicep, about to go 0.5, aiming to get to 1.0 in the new year.
   2. The team are dedicated to improving and the reason why this is an open source project is to put this in the hands of the community early, accept feedback and PR's from the community contributors.
   3. We need your help. If you are a willing participant, we would love to hear both feedback and suggestions at a high level like this thread ... We ALSO need to define and prioritize the bites.
      • If you come across specific issues or are able to provide other actionable suggestions, searching current issues and contributing to the conversation or even opening a new individual issue, provides a method to give the most visibility to get a particular issue assigned to one of our Devs to work on.
2. Check out all of the community calls up here: Azure Deployments & Governance Youtube videos channel
3. Check out our Docs and these are also open source, let us know what is missing? Bicep Documentation If you check the recent community call, John Downs presented on some example new pages and these and others are currently being developed and enhanced as we head towards Bicep 1.0 and beyond.
4. Check out the MSLearn topics on Bicep, there are some good tidbits in there that will assist in your journey

   1. There are 5 learning paths up there, plus more coming ... https://docs.microsoft.com/en-us/learn/browse/?terms=bicep&resource_type=learning%20path

      

5. Most of all, use Bicep as much as possible. Then feel free to open Discussions on specific issues that you are facing or ideas that you have based on a specific scenario. As an example, the Subnets/VNet issue that you listed in your first topic above is one where others in the community could provide samples and links to projects and sample code etc.

A few other questions

• Do you have much experience in other Declarative languages maybe Desired State Configuration or something like that?
• If you are able to publish any samples in a public repo (in the future) that is the easiest way to compare notes on design pattens Etc.
• It seems like your design model is maybe based around Governance/Compliance to some degree? How do you approach architectures and what is the goal of your Bicep deployments overall?

[brwilkinson]

Scoping issue again, but this time related to parent / child resources

Child resources "parent" property should need nothing more than a resourceId to form a well qualified link. For example, if i wanted to reuse a template for assigning roleassignments. The roleassignment resource needs to know what the parent / scope is. Well the only way you folks let us define that is by using an "existing resource" definition. This makes creating what should be a repeatable / shared template, almost impossible as I would have to define every possiable resource type This seems related to not being able to pass an existing resource as a parameter object, as right now, you only have 'object' as a viable option for a datatype and the scope property won't support that. Again, I don't even think the scope should need this. A resourceID should be all that's needed.

This one is a specific issue, that I believe should be addressed in the thread above
We need more programmatic features.

Once we get PR: #4971 Then you can simply pass in a resource via parameter our out via an output, that will save you from having to use the existing which is a current pattern in the templates today.

I agree if the above wasn't in place then creating an existing resource from a resourceid is also on my list of wanted items. However I believe what we will end up with is a cleaner/better overall solution. If at that time that does not meet our needs, this is something that has alot of eyes on it, so I am confident that we could improve at that time. There are a lot of features on the backlog and the teams continues to grow, so watch this space.

[brwilkinson]

linking here to workaround to create resource specific role assignments in a generic way.

#5805

[brwilkinson]

Modules for scope change lead to template sprawl

I definitely agree on this in some sense.

There is this issue open:

• Allow targeting multiple scopes in a single file with targetScope #788 to allow a single Bicep template to work over multiple scopes.
• scope property in deployment() function #4698 Here is a related issue:

We also have to deal with ARM - Azure Resource Manager

• It's likely easier to do a subscription deployment, then loop through some resource groups to change Resources and most resources will always live in a resource group. So in this part if it, I think a module makes sense, since you want to run the same code against each resource group?
  • Do you have any examples that you are able to share on which part of this is frustrating?
  • Also what enhancement would make it easier for you to scale Etc ?

Modules are not the only way to get to another scope.

here is a vnet peering bicep module that can go across scopes within the same bicep file.

https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/bicep/VNET-Peering.bicep

param subscriptionID string
param resourceGroupName string
param vNetName string
param vNetNameHub string
param peeringName string

resource VNETHUB 'Microsoft.Network/virtualNetworks@2020-11-01' existing = {
  name: vNetNameHub
}

resource VNET 'Microsoft.Network/virtualNetworks@2020-11-01' existing = {
  name: vNetName
  scope: resourceGroup(subscriptionID,resourceGroupName)
}

resource VNETPeering 'Microsoft.Network/virtualNetworks/virtualNetworkPeerings@2017-10-01' = {
  name: peeringName
  parent: VNETHUB
  properties: {
    allowVirtualNetworkAccess: true
    allowForwardedTraffic: true
    allowGatewayTransit: false
    useRemoteGateways: false
    remoteVirtualNetwork: {
      id: VNET.id
    }
  }
}

[ericcsinger]

Hi Ben, thanks for your very thoughtful and thorough replies. It’s a lot to take in :-) and I will respond more formally once I have some time to go through each reply and digest it. Your example repo looks pretty epic, but it’s like drinking through a fire hose lol. Thanks!

…

Sent from my iPhone

On Dec 6, 2021, at 20:43, Ben Wilkinson ***@***.***> wrote:  Hey @ericcsinger Wow there is a lot to unroll there 😊 Each one of these is a conversation, so I think it might be best to answer them in different threads here. Real world examples This one is admittedly difficult since no one template fits everyone use-cases. However, it's a conversation worth having. Here are 2 example repo's that show 2 very different ways to build out templates: Consultant built templates designed for generic building blocks for Enterprise customer deployments https://github.com/Azure/ResourceModules https://github.com/azure/ResourceModules/wiki <-- getting started wiki My personal project designed with a very specific implementation for deploying App Infra Stamps (for a single app platform). https://github.com/brwilkinson/AzureDeploymentFramework/tree/main/ADF/bicep Some examples... This is the ResoureModules project Each of these has the associate param file in the directory associated for the Resource Type. networkSecurityGroups virtualNetworks Sample of param file for virtualnetwork This is my project, which is specifically designed for rolling out Stamp environments for my customers Sample param file with Subnets: https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/tenants/AOA/ACU1.T5.parameters.json#L360 Sample network Modules that consume that object NSG.bicep NetworkFlowLogs.bicep VNET.bicep — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[azMantas]

Scoping issue again, but this time related to parent / child resources
... if i wanted to reuse a template for assigning roleassignments ...
I've found a solution to assign multiples roles to a resource by using modules, loops and items functions. So far it works fine in my dev environment. Firstly, I need to create a bicep file that handles RBAC assignment for a chosen resource. Notice that rbac supports multiple principal id's

param principalID array
param roledefinitionId string
param resourceNameToAssignRbac string

resource resource 'Microsoft.ContainerRegistry/registries@2021-06-01-preview' existing = {
  name: resourceNameToAssignRbac
}
resource rbac 'Microsoft.Authorization/roleAssignments@2020-08-01-preview' = [for principal in principalID: {
  name: guid(principal, roledefinitionId, resource.id)
  scope: resource
  properties:{
    roleDefinitionId: roledefinitionId
    principalId: principal
  }
}]

Next step is to use it as a module, but this time I call one more function - items().

targetScope = 'subscription'
param projectName string
param containerRegistryRoles object
param utc string = utcNow()

var rgName = 'rg-${projectName}'
var location = 'westeurope'

resource resourceGroupResource 'Microsoft.Resources/resourceGroups@2021-04-01' = {
  name: rgName
  location: location
}

module acr '../modules/cr.bicep' = {
  scope: resourceGroupResource
  name: 'acr-${utc}'
  params: {
    projectName: projectName
  }
}

module rbac '../modules/crRbac.bicep' = [for role in items(containerRegistryRoles): {
  name: '${role.key}-${utc}'
  scope: resourceGroupResource
  params: {
    principalID: role.value.principalId
    roledefinitionId: role.value.roleDefinitionId
    resourceNameToAssignRbac: acr.outputs.acrResourceName
  }
}]

and now we just need to provide roles and role definition Id as an object in parameter file and that is it. The funny part is that it supports multiple RBAC and each role can have multiple principal id's.
P.S. 'acrPull' and 'acrPush' can be named as you like.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "projectName": {
      "value": "bbbb"
    },
    "containerRegistryRoles": {
      "value": {
        "acrPull": {
          "roleDefinitionId": "/providers/Microsoft.Authorization/roleDefinitions/7f951dda-4ed3-4680-a7ca-43fe172d538d",
          "principalId": [
            "11111111-1111-1111-1111-111111111111",
            "22222222-2222-2222-2222-222222222222"
          ]
        },
        "acrPush": {
          "roleDefinitionId": "/providers/Microsoft.Authorization/roleDefinitions/8311e382-0749-4cb8-b61a-304f252e45ec",
          "principalId": [
            "11111111-1111-1111-1111-111111111111",
            "44444444-4444-4444-4444-444444444444",
          ]
        }
      }
    }
  }
}

[brwilkinson]

I'll mark this one as answered for now.

• I think people can still search and contribute, even though it is answered.