Consider MatchElementStateForUnknown() Plan Modifiers

[bflad]

Module version

v1.2.0

Use-cases

Over in #709, it was discovered that the UseStateForUnknown() plan modifier (amongst other custom ones) can return unexpected values when a parent list or set has elements rearranged or removed. In particular:

• Attribute-based plan modifiers which attempt to read prior state (such as UseStateForUnknown())
• Implementing those attribute-based plan modifiers on attributes underneath a schema.ListNestedAttribute, schema.ListNestedBlock, schema.SetNestedAttribute, or schema.SetNestedBlock
• The resource is planning/applying an in-place update which rearranges and/or removes those elements within the list/set

This is because the framework's list and set data structures are array index based and the planning implementation will align configuration, plan, and prior state values based on that index. The implementation itself was an design choice where other choices introduced their own potential tradeoffs, such as requiring provider developers to implement much more explicit logic into their schemas. It was particularly desirable to avoid the prior terraform-plugin-sdk model of set and plan difference handling, which consistently had bug reports which were difficult or not possible to fix without breaking changes. In any event, the framework's data handling for attribute plan modifiers cannot potentially be changed until a theoretical next major version, so that is not an option in this case.

This leaves the framework maintainers in an awkward position where UseStateForUnknown() can introduce unexpected plans or Terraform data consistency errors, which framework-native functionality should never do. It is possible to detect when the plan modifier is being called while under a list or set, which means that it can raise its own implementation error to tell developers to remove the plan modifier. This does mean however, that previously working cases such as unchanged lists/sets or ones where only new list/set elements were added will lose their ability to display a more known plan.

Attempted Solutions

Provider developers implementing custom attribute plan modifiers with fairly complex data handling logic.

Proposal

Parent Plan Modifier

One possible proposal would be a new list/set attribute plan modifier that enables provider developers to input tracking attributes which the plan modifier logic would realign the prior state values of any unknown computed values. As a design sketch:

schema.ListNestedAttribute{
  NestedObject: schema.NestedAttributeObject{
    Attributes: map[string]schema.Attribute{
      "attr1": schema.StringAttribute{
         Required: true,
       },
       "attr2": schema.StringAttribute{
         Computed: true,
       },
       // potentially other configurable or non-configurable attributes to handle
    },
  },
  PlanModifiers: []planmodifier.List{
    listplanmodifier.MatchElementStateForUnknown(
      path.MatchRelative().AtAnyListIndex().AtName("attr1"),
      // potentially other configurable attributes to handle
    ),
  },
},

While requiring less provider developer implementation details especially in the case of many computed attributes and potentially being a slightly easier framework implementation, this approach does have a few immediate downsides:

• Action at a distance: Provider developers need to know this implementation goes in a different location than the affected attribute.
• Lack of ability to choose which element attributes it effects: Unless separate function parameters are introduced or a separate plan modifier is created, there's no ability to have the value preservation apply to a subset of computed attribute values. That could introduce Terraform data consistency errors between plan and apply depending on the provider implementation details.

Nested Plan Modifiers

Another possible proposal would be new typed attribute plan modifiers, such as in the resource/schema/stringplanmodifer package, where the provider developer can optionally realign the prior state based on identifying attributes. As a design sketch:

schema.ListNestedAttribute{
  NestedObject: schema.NestedAttributeObject{
    Attributes: map[string]schema.Attribute{
      "attr1": schema.StringAttribute{
         Required: true,
       },
       "attr2": schema.StringAttribute{
         Computed: true,
         PlanModifiers: []planmodifier.String{
           stringplanmodifier.MatchElementStateForUnknown(
             path.MatchRelative().AtParent().AtName("attr1"),
             // potentially other configurable attributes to handle
           ),
         },
       },
       // potentially other configurable or non-configurable attributes to handle
    },
  },
  // ...
},

This means that each underlying attribute can opt into the behavior as appropriate (action where its implemented and explicit choice of effect). There is slightly less discoverability concerns since the implementation would be next to UseStateForUnknown().

The plan modifier logic will need to implement some rigorous validation where it raises errors if:

• The plan modifier is not implemented on an attribute below a list/set.
• It receives 0 path expressions.
• A given path expression goes outside the parent list/set.
• A given path expression goes deeper than nesting of the current path. (Provider developers likely need to implement their own custom plan modifiers for this level of planning complexity.)

References

• Unexpected Behavior With Plan Modifier (UseStateForUnknown) Prior State Under Lists and Sets #709
• resource/schema: Prevent UseStateForUnknown data misalignment by raising implementation error when under a list or set #711