Skip to content

Latest commit

 

History

History
176 lines (108 loc) · 9.91 KB

CONTRIBUTING.md

File metadata and controls

176 lines (108 loc) · 9.91 KB
Raw

Contributing to the Azure native provider

Building

Dependencies

  • Go 1.21
  • NodeJS 18.X.X or later
  • Python 3.10 or later
  • .NET 6 or later
  • Gradle 8 or later

Please refer to Contributing to Pulumi for installation guidance.

Building locally

Run the following commands to install Go modules, generate all SDKs, and build the provider:

make

Note: When building via make, go workspaces are ignored.

Running a nodejs example

Navigate to one of the examples and run Pulumi:

make install
cd ./examples/simple
yarn link @pulumi/azure-native
pulumi up

Adding Resource Documentation

Documentation is mostly generated automatically from the specifications. Additional documentation can be added for specific resources by adding markdown files in the docs/resources folder. Each filename is the combination of the module and resource name as it appears in the registry: [module]-[Resource].md e.g. sql-Server.md.

Once documentation has been added, run make generate generate_docs to re-generate the SDKs and the registry docs. Commit the changes and open a pull request.

Azure Versions

Key facts about Azure Versions:

  • Azure's REST API has many versions of each service which are published per-service on their own schedule.
  • Service versions are in the form of an ISO date (e.g. 2020-01-01) followed by an optional suffix (e.g. -preview or -privatepreview).
  • Not all versions of the service contain all resources. Some services will only publish the parts of their API which have changed since the previous version.
  • Some services haven't had a stable (non-suffixed) release for a number of years - and the 'preview' versions are generally considered stable for every-day use.
  • New specification versions are sometimes release before the updated service is deployed.

Terminology

Azure's API specifications are organised into "Resource Providers" (or "Services"), Namespaces and API Versions. We use the term "API Version" in this doc to refer to Azure's Resource Provider API Versions.

Version Sources

  1. spec.json: the azure-rest-api-specs are checked out as a sub-module and include all OpenAPI specifications (for every Resource Provider API Version). The spec.json is a simplified report of versions in the spec - each version and the resources within.
  2. active.json lists the 'active' API versions - the versions of each API which are actually deployed into Azure's data centers. It is generated by running az provider list, stripping excess information (which is written to provider_list.json) then gets summarised into active.json.

Removing Redundant Versions (Squeezing)

Due the very large number of versions, we only include the latest version, or versions which have incompatible resource shapes when compared with the next version.

This should only be generated at the point of releasing a new major version.

This is currently generated by running make schema_squeeze.

Default Version

To provide simpler discovery and usability, the Pulumi Azure Native provider adds a single 'default' version for each API which is the combination of one or more API versions. The aim is to include all resources, at the latest available version, where possible.

Updates to the version selection is done conservatively within the same major version to avoid breaking changes.

Configuration Files

Version configuration files are prefixed with the major version they're for (e.g. v2-) and live within the versions/ folder. There's three steps to the versioning process:

  1. Config: (e.g. v2-config.json) is a hand-edited file to control how the Spec is updated.
  2. Spec: (e.g. v2-spec.yaml) is automatically appended to but can be manually edited.
  3. Lock: (e.g. v2-lock.json) is generated from the Spec and is the exact selection of resource versions to use.

The default version selection is calculated as a part of the make schema target.

Config

The config file performs three functions for each service:

  1. Control which versions will be considered for the spec.
  2. Assert expected properties of the spec - raising warnings when not met.
  3. Document information relevant to the selection of versions and resources.

Spec

A spec file consists of service names, each specifying:

  • tracking specifies a single version of the API to fetch all resources from.
  • additions specified a map of specific resource (or invoke) name to API version to include in the default version.

Both tracking and additions can be specified together but if the set of resources overlaps (the same resource exists) in both the tracking version and the additions, then the versioning program will fail with an error.

From the Spec file, we generate the Lock JSON file (e.g. v2-lock.json) which contain the fully expanded set of default resources. These are the files used at the point of generating the schema.

Creating a new major version default selection

  1. Take a copy of previous config.
  2. Create an empty spec.
  3. Update Makefile: PROVIDER_VERSION and dependencies specific to the previous version (e.g. v2-* mentions).
  4. Calculate removable resources: make schema_squeeze.
  5. Generate schema: make schema.

Type Compatibility

Supporting every version of every API causes the SDK size to be very large. One method we've used to reduce the required size is to only have a single set of types for each API which are re-used across all versions. However, not all the types are compatible and therefore a type with the same name might be incompatible in the next version. This incompatibility is currently detected but ignored on a case-by-case basis for now.

Ideally, at the point a new incompatible version of the type is added, it should then be created with a unique, stable name. Alternatively, we could create a union of the two possible types

Default Version Locks

The default version is calculated and written to a 'lock' file which list every resource (or invoke) at a specific API version. This file is read in during the schema and SDK generation. Currently there is a v1-lock.json and v2-lock.json lock file. These lock files should not be edited directly, but should be calculated by the versioner tool using a specific algorithm.

Additional Reports

  • deprecated.json is a list of API versions which are older than the versions included in the default version and at least 2 years old. These will be removed in the next major version of the provider.
  • pending.json is a list of new API versions which aren't yet included in the default version. These should be included in the default version at the next major release of the provider.

New Go SDK

As the size of the Go SDK is close to exceeding the limit of 512Mb, we've created a new SDK which we're publishing in parallel which defined a Go module per Azure namespace rather than a single root Go module. The additional go modules are auto-generated with the required dependencies in provider/cmd/pulumi-gen-azure-native/main.go.

This new SDK is published to its own repository at github.com/pulumi/pulumi-azure-native-sdk. This is separate partly due to the large number of tags which are created in this new repository per release, but also to remove the need to commit the SDK code into this repository.

Sub-resources

Some resources which are also settable on a parent resource. For instance, the subnets of a virtual network can be specified inline with the virtual network resource:

new network.VirtualNetwork("inline", {
    subnets: [
      { name: "default", addressPrefix: "10.4.1.0/24" },
    ]
})

But they can also be specified as stand-alone resources:

new network.Subnet("third", { ... })

In this case, we call VirtualNetwork.subnets a sub-resource property.

The choice between inline and stand-alone sub-resource definitions offers flexibility but needs some special handling in the CRUD lifecycle.

On Create: when the user opts for stand-alone representations of the sub-resource, they omit the sub-resource property on the parent. For instance, new network.VirtualNetwork will be defined without subnets. In this example, however, creating a VirtualNetwork without subnets will fail in Azure because it's a required property. Therefore, on Create, we find sub-resource properties that are not set, and set them to their default value in the request payload. This happens in the azureNativeProvider.setUnsetSubresourcePropertiesToDefaults method.

On Update: consider the naive implementation. When a virtual network v is updated, any stand-alone subnets are not in v.subnets, therefore they would be removed on update. To prevent this, we need to retrieve the existing sub-resources and fill them into the parent's sub-resource property. This is done in the azureNativeProvider.maintainSubResourcePropertiesIfNotSet method. For example:

new network.VirtualNetwork("vnet", { ... })

new network.Subnet("sub1", { ... })

When vnet is updated, the provider first reads vnet from Azure and populates vnets.subnets with the subnets from the response. This way, the subnets are simply round-tripped and not removed on update.

On Read: when reading a parent resource, the Azure response will contain the sub-resources. If the user defined them stand-alone, we need to reset them to the empty value to avoid recording them in state. This is done in the azureNativeProvider.resetUnsetSubResourceProperties method.

A note on the "default value" mentioned above: it's hard-coded as a an empty array currently. We haven't seen any other type, and it's the only one that really makes sense for sub-resources: they must be a variable number of items.

Relevant PRs:

  • #2755
  • #2950
  • #3054