Azure Load Balancer Backend Pool remains empty (needs GET request)

[vigor-vavan]

What happened?

When creating an Azure Load Balancer using Pulumi with Python, the backend pool is created but remains empty despite providing the necessary data.

We have attempted to populate the backend pool using LoadBalancerBackendAddressArgs, LoadBalancerBackendAddressArgs with a list of dictionaries, but none of these methods have worked.
If we create the load balancer in the first iteration and then hardcode the backend values in the next iteration, the backend data is populated correctly.

Pulumi completes the process successfully without any errors, but the backend pool remains empty.

Example

Our codebase is quite large, making it impractical to share the exact code. However, we have tested the plain example from the documentation and encountered the same behavior.

Output of pulumi about

python version 3.11.1
pulumi version 3.134.0

Additional context

We have been using the same code for the past few months without any issues. However, in the last 10 days, it has stopped working. We tried updating the Pulumi version, but the load balancer backend pool remains empty.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[justinvp]

Sorry for the trouble, @vigor-vavan. This sounds likely specific to Azure Native, so transferring to that repo and someone will take a look soon.

[thomas11]

Hi @vigor-vavan, which load balancer are you using? There's containerservice.LoadBalancer, kuberrnetesruntime.LoadBalancer, and network.LoadBalancer.

Can you also tell us exactly which example from the docs shows the issue? Many resources have several examples.

Lastly, may I ask you to double-check if any dependencies (azure native provider, pulumi) got updated in the past 10 days? Something must have changed for the behavior to change. Thank you!

[vigor-vavan]

We have been using network.loadbalancer.

I forgot to mention that the preview plan shows the backend address pool will be populated with IPs, but when applied, nothing happens. If we manually add IPs, Pulumi removes them.

Regarding dependencies, we have updated the Pulumi version to test if we will gain any benefit.
We upgraded from pulumi version 3.103.0 to version 3.134.0.

We tried using network.LoadBalancerBackendAddressPool, but encountered a circular dependency issue. The load balancer must exist to create the backend pool, and the backend pool must exist to set load balancer rules which are created with load balancer creation. Additionally, it’s not possible to add rules with Azure Native after the load balancer is created.

We have combined example
https://www.pulumi.com/registry/packages/azure-native/api-docs/network/loadbalancer/#create-load-balancer
with addition of backend pool addresses from
https://www.pulumi.com/registry/packages/azure-native/api-docs/network/loadbalancerbackendaddresspool/#update-load-balancer-backend-pool-with-backend-addresses-containing-virtual-network-and--ip-address

We want in one swoop to create load balancer with all resources backend, frontend, health probes, rules.

[thomas11]

Hi @vigor-vavan, let me try to clarify a few things.

We tried using network.LoadBalancerBackendAddressPool, but encountered a circular dependency issue.

Above, you wrote your code was working until about 10 days ago. Is that also true for the circular dependency problem? Or was that a different approach?

Additionally, it’s not possible to add rules with Azure Native after the load balancer is created.

Why is that? To my knowledge, it should be possible.

The circular dependency issue is probably solvable via $self. Here is an example in Python.

[vigor-vavan]

Circular dependency was another approach we tried. We attempted to create a load balancer without a backend pool so we could add the backend pool through the LoadBalancerBackendAddressPool resource. However, this led to a problem where we couldn’t add rules because there are no resources for rules except as part of the load balancer resource.

We are using $self, and with that, everything was working. In one swoop through the load balancer resources, we created everything.

[thomas11]

I see. With $self, everything was working, but then in the last 10 days it stopped working?

Could you try the latest 2.64.2 release since it contains a Pythin-specific fix?

If that doesn't work, could you please determine the latest version that does work?

[vigor-vavan]

We have updated to the latest version, but the issue persists: the backend pool IPs are still empty.

Regarding when it worked, it stopped functioning even though we haven’t made any changes to our code base.

[thomas11]

Regarding when it worked, it stopped functioning even though we haven’t made any changes to our code base.

Could it be that you have a floating dependency on the provider, like pulumi-azure-native>=2.0.0,<3.0.0, and your package manager pulled in a newer version as it was released?

You could try explicitly using v2.62.0 that was released about two weeks ago.

[vigor-vavan]

We attempted the solution but had no success. We even rebuilt one load balancer. However, despite the loadBalancerBackendAddresses field being populated with IP addresses in the preview, the load balancer pools remain empty after applying the changes on the Azure portal.

[thomas11]

after applying the changes on the Azure portal.

Can you clarify what you mean by that? Were you making additional changes in the Azure portal? If so, Pulumi wouldn't be aware of them unless you run refresh.

If you used the portal only for checking the load balancer, I'm not sure what the issue could be. For a deeper investigation, we'd need a full Pulumi program that reproduces the issue.

[vigor-vavan]

After applying the changes, there were no additional modifications. Pulumi was aware, the refresh flag was used. Where should we check this? Pulumi finishes successfully, but the load balancer backend pool is empty.

We cannot share full program.
Have you tried reproducing the issue from the examples, or with basic example that fills backend pool with IPs.

Let’s summarize:

• Our codebase was working fine up until 10 days ago without any changes.
• We haven’t updated our runner, so no new versions of Pulumi, Python libraries, or Pulumi providers were involved.
• Currently, we have updated the Pulumi version, Python libraries, and provider.
• The preview shows everything is fine, load balancer as we intended will be deployed
• The apply command executes successfully.
• However, on the portal, we observe that the load balancer was not fully deployed.

[vigor-vavan]

We have submitted an issue with the backend to Microsoft as we were able to reproduce the problem manually through the portal. If backends are added quickly, the operation appears successful, but the backend pool remains empty.

The workaround provided is to add backends slowly, which results in successful addition.

Since the issue is not on Pulumi’s side and we cannot expect a quick resolution from Microsoft, is there a possibility to slow down the process of adding backends?

[thomas11]

Hi @vigor-vavan, I'm afraid there can be no wait in between the backends when they're defined as below, as a single request to Azure is made for the backend pool.

load_balancer_backend_address_pool = azure_native.network.LoadBalancerBackendAddressPool("loadBalancerBackendAddressPool",
    backend_address_pool_name="backend",
    load_balancer_backend_addresses=[
        {
            # ...
        },
        {
           # ...
        },
    ],

You could run multiple pulumi up, adding backends via configuration, although that's cumbersome.

[vigor-vavan]

We received feedback from Microsoft stating: "If there are no GET operations before each of the PUT operations, it may result in behavior similar to what was observed when using the Azure Portal."

Could we request a modification in the load balancer backend deployment, either as separate resource or as a part of load balancer deployment?

[vigor-vavan]

Latest response form microsoft regarding
API Load Balancers - Create Or Update

...
2. backendAddressPools Property: The backendAddressPools property in the API works as expected for a Global LB with regional LBs in the backend pool. However, I want to emphasize that this property is read-only when you're using the PUT LB operation for regional LBs where the backend addresses are NICs or their IP addresses. That means, while you can view the backend pool addresses, you cannot modify them directly using the PUT LB operation for regional LBs.
3. How to Update Backend Pools: The key thing to understand is that updates to backend pools must be done separately, depending on the type of pool:
o If you’re working with NIC-based backend pools, you should update them via the PUT NIC operation.
o For IP-based backend pools, you need to use the PUT Backend Pool API to make changes.
...

From their response it looks like it is not possible to have one API call that will setup load balancer backend pools fully, each backend pool should have one API call using
API Load Balancer Backend Address Pools - Create Or Update.

[thomas11]

From their response it looks like it is not possible to have one API call that will setup load balancer backend pools fully, each backend pool should have one API call using

That's what Pulumi would do automatically when you use the standalone LoadBalancerBackendAddressPool resource. When you define the LoadBalancer, leave the backend_address_pools property empty.

[vigor-vavan]

In that case, how would you add load balancer (LB) rules that expect backend pool?
We encountered a problem because we couldn't find a separate resource for LB rules and had to add them through a single call.
We would like to use azure native.

[thomas11]

Unfortunately, I'm no domain expert in Azure load balancers. I think you might need to generate the resource ids of the backend pools in your code so you can pass them to lb.backend_address_pools before the pools are created. Since you determine the pools' names, their ids are known in advance, before they are created.

That's assuming that Azure lets you create the LB referring to non-existent pools. But that seems to be how this Azure example does it. When the LB is created on the line that I linked, the pool is passed as a locally generated id.

[vigor-vavan]

We are encountering an issue when trying to create a load balancer without backend pools, using only references. The error message we receive is:

error: Code="InvalidResourceReference" Message="Resource /subscriptions/***/resourceGroups/***/providers/Microsoft.Network/loadBalancers/***/backendAddressPools/*** referenced by resource /subscriptions/***/resourceGroups/***/providers/Microsoft.Network/loadBalancers/***/loadBalancingRules/*** was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.

However, if we attempt to create a load balancer with empty backend pools, we encounter a different error:

azure-native:network:LoadBalancerBackendAddressPool *** creating (37s) error: cannot create already existing resource '/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/loadBalancers/***/backendAddressPools/***'
azure-native:network:LoadBalancerBackendAddressPool *** creating failed error: cannot create already existing resource '/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/loadBalancers/***/backendAddressPools/***'

This puts us in a difficult position. It seems impossible to create a load balancer with backend pools and rules in a single call. If we try to create the load balancer without backend pool rules, the creation fails. On the other hand, if we try to create it with an empty backend pool, we cannot fill in the IP details.

In Azure Classic, there was a separate resource for load balancer rules, which does not exist in Azure Native.

[stfnzl]

@vigor-vavan I managed to work around this by deploying rules separately using azure classic.

new azure.lb.Rule(
  "the-rule",
  {
    loadbalancerId: loadBalancer.id, // created with azure native and no refs to backend pools at all
    frontendIpConfigurationName: "...",
    backendAddressPoolIds: [pool.id], // pool created via LoadBalancerBackendAddressPool from azure native
    frontendPort: ...,
    backendPort: ...
    protocol: "Tcp",
    probeId: pulumi.interpolate`${loadBalancer.id}/probes/name-of-the-probe-defined-in-the-lb`,
  },
  {
    parent: loadBalancer,
    provider: classicProvider,
  }
);

[vigor-vavan]

We reached the same conclusion and used the same WA with one addition: we had to omit rules from the LB resource lifecycle as it attempted to delete the rules during the running update.

 ignore_changes=["load_balancing_rules"]