Skip to content

Latest commit

 

History

History
1234 lines (955 loc) · 54.6 KB

README.md

File metadata and controls

1234 lines (955 loc) · 54.6 KB

KEP-4358: Custom Resource Field Selectors

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

  • (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
  • (R) KEP approvers have approved the KEP status as implementable
  • (R) Design details are appropriately documented
  • (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
    • e2e Tests for all Beta API Operations (endpoints)
    • (R) Ensure GA e2e tests meet requirements for Conformance Tests
    • (R) Minimum Two Week Window for GA e2e tests to prove flake free
  • (R) Graduation criteria is in place
  • (R) Production readiness review completed
  • (R) Production readiness review approved
  • "Implementation History" section is up-to-date for milestone
  • User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io
  • Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

This KEP proposes adding field selector support to custom resources.

Motivation

All Kubernetes APIs support field selection for metadata.name, metadata.namespace.

Additionally, built-in APIs support field selection on specific fields such as status.phase for Pods, but CustomResourceDefinitions lack equivalent functionality.

Without this enhancement, an extension author that wants to be be able to filter resources might choose to:

  • Organize data into a label even though it should be a spec or status field.
  • Double writing data both to a label and a spec or status field.

Both of these should be avoided. We should enable extension authors structure their APIs according to best practices, not based on data access patterns.

Goals

  • Add an allow list of selectable fields to CustomResourceDefinitions.
  • Performance and resource consumption characteristics are roughtly equivalent to label selectors.

Non-Goals

  • Arbitrary field selection.
    • This proposal is to support field selection for a small allow list of fields. Expanding support to include all fields is complicated by:
      • Fields contained in, or nested under, lists or maps.
      • The costs to apiserver of evaluating filters on objects or maintaining indices.
      • Lack of direct support for filtering in etcd.
  • Support for conversions, casts or other transforms of field values beyond the automatic cast to string required for field selection.
  • More sophisticated field selection logic. Possible approaches include: CEL based field selection, enhancements to field selection such as set based operators like provided for labels ("in", "not in"), OR clauses, pattern matching... See the "CEL as an accelerated predicate language" section "Future Work" for more details on what makes this challenging.

Proposal

Add selectableFields to the versions of CustomResourceDefinition:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: selector.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      selectableFields:
      - jsonPath: .spec.color 
      - jsonPath: .spec.size 
      additionalPrinterColumns:
      - jsonPath: .spec.color
        name: Color
        type: string
      - jsonPath: .spec.size
        name: Size
        type: string
...

(Alternatively, selectableFields could contain fieldPath: spec.color to align more closely with validationRules.fieldPath. This will be discussed API review).

Clients may then use the field selector to filter resources:

$ kubectl get selector.stable.example.com
NAME       COLOR  SIZE
example1   blue   S
example2   blue   M
example3   green  M

$ kubectl get selector.stable.example.com --field-selector spec.color=blue
NAME       COLOR  SIZE
example1   blue   S
example2   blue   M
$ kubectl get selector.stable.example.com --field-selector spec.color=green,spec.size=M

NAME       COLOR  SIZE
example2   blue   M

Background: Field selection as it exists before this enhancement

Field selection is string equality based. Set-based operators are not supported (in, notin, exists). Field selectors may be "chained" (ANDed together).

All resources have two field selectors available for metadata:

Kind Field Field type field selector Conversions/Notes
* metadata.name string metadata.name
* metadata.namespace string metadata.namespace Absent on cluster scoped resources

Some resources have additional field selectors:

Kind Field Field type field selector Conversions/Notes
ReplicaSet / RC status.replicas int status.replicas itoa(int)
Job status.succeeded int status.successful itoa(int), field selector/name mismatch
CertificateSigningRequest spec.signerName string spec.signerName -
Event involvedObject.kind string involvedObject.kind -
Event involvedObject.namespace string involvedObject.namespace -
Event involvedObject.name string involvedObject.name -
Event involvedObject.uid UID involvedObject.uid string(UID)
Event involvedObject.apiVersion string involvedObject.apiVersion -
Event involvedObject.resourceVersion string involvedObject.resourceVersion -
Event involvedObject.fieldPath string involvedObject.fieldPath -
Event reason string reason -
Event reportingComponent string reportingComponent -
Event source.component string source Set to reportingController if field omitted, field selector/name mismatch
Event type string type -
Namespace/PV/PVC metadata.name 1 string name 1 field selector/name mismatch, metadata.name already selectable
Namespace phase string status.phase -
Node spec.unschedulable boolean spec.unschedulable fmt.Sprint(bool)
Pod spec.nodeName string spec.nodeName -
Pod spec.restartPolicy enum spec.restartPolicy string(enum)
Pod spec.schedulerName enum spec.schedulerName string(enum)
Pod spec.serviceAccountName enum spec.serviceAccountName string(enum)
Pod spec.securityContext.hostNetwork optional boolean spec.hostNetwork strconv.FormatBool(bool) Set to "false" if field is omitted
Pod status.phase string phase field selector/name mismatch
Pod status.podIP slice status.podIPs Set to "" if field is omitted
Pod status.nominatedNodeName string status.nominatedNodeName
Secret type enum type string(enum)

Note that:

  • Resources always provide a value for a selectable field. Notice that in the above table, all field selectors for optional fields provide a fallback if the field is omitted.
  • Field selectors only exist for strings, enums, integers and booleans.
  • Field selectors for name and namespace are provided automatically.
  • Field selectors that do not match the path name of the actual field do exist, for historical reasons, but the best practice is for field selectors to match the exact path name of the field.

Also note that we do not currently publish any field selector information into OpenAPIv3, and we do not document in APIs or on the Kubernetes website which fields may be selected. Because of this, finding the field selectors available for built-in types, to construct the above table, required locating GetAttrs/*ToSelectableFields functions in the Kubernetes source code.

Plan

API

type CustomResourceDefinitionVersion struct {
  
  // selectableFields specifies paths to fields that may be used as field selectors.
  // TODO: A max limit will be decided during API review
	// See https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors
	SelectableFields []SelectableField `json:"selectableFields,omitempty" protobuf:"bytes,9,rep,name=selectableFields"`
}

  // SelectableField specifies the JSON path of a field that may be used with field selectors.
type SelectableField struct {
	// jsonPath is a simple JSON path which is evaluated against each custom resource to produce a
  // field selector value.
  // Only JSON paths without the array notation are allowed.
  // Must point to a field of type string, boolean or integer. Types with enum values
  // and strings with formats are allowed.
  // If jsonPath refers to absent field in a resource, the jsonPath evaluates to an empty string.
  // Must not point to metdata fields.
	JSONPath string `json:"jsonPath" protobuf:"bytes,1,opt,name=jsonPath"`
}

Note that we do not offer a way to set a default value. We discussed this in API review and observed that because OpenAPI schema already provides a way to set a default value for a field, that we don't need (or want) to add a second way to set a default for selectable fields. Instead, if the jsonPath refers to an absent field value in a resource, the selectable field value will be an empty string.

Validaiton rules
  • selectableFields[].jsonPath must be a "simple path" (similar to additionalPrinterColumns[].jsonPath except validated properly). I.e. property names separated by '.', e.g. .spec.x.y (The path may not contain map or array indexing)
  • The schema type referenced by selectableFields[].jsonPath must be one of: string, integer, boolean (enums and fields with string formats are allowed).
  • There is a limit on the maximum selectableFields per CRD version that are allowed.
  • selectableFields may not refer to metadata fields.
  • selectableFields may not contain duplicates.

Discovery

We will add discovery details in to OpenAPIv3. For example:

"org.example.v1.CustomResource": {
  "type": "object",
  "x-kubernetes-group-version-kind": [ ... ],
  "x-kubernetes-selectable-fields": [
    { "fieldPath": "metadata.name"},
    { "fieldPath": "metadata.namespace"},
    { "fieldPath": "spec.myField"},
  ]
},

Implementation

Because generic.StoreOptions consumes selectors from built-in and CRDs in the same way, we would extend the GetAttrs of CRDs to offer fields for selection. https://github.com/jpbetz/kubernetes/tree/crd-object-filters contains a working prototype.

We would leverage the fieldPath validation supported added for CRD Validation Rules to validate paths.

We would leverage the jsonpath library in the same way as used by additionalPrinterColumns to retrieve values from custom resources using jsonpaths.

The majority of the remaining work will be to update the API to add the field and to validate it.

User Stories

  • CustomResourceDefinition author wishes to provide filtered access by a field
    • Author adds the field to an allow list in the CustomResourceDefinition
    • If field is not always present, CustomResourceDefinition author declares how absent fields are handled
  • User uses --field-selector in kubectl to filter a list response.
    • User first needs to know what field selectors are available and what values the selectable fields might have.
      • Places user might look: discovery, CRD resource.
    • User filter by:
      • a single field
      • multiple fields
      • both fields and labels
  • Controller watches custom resources while filtering with field selectors.
    • Controller provides a FieldSelector string in ListOptions

Notes/Constraints/Caveats (Optional)

Discussions about limitations of field selectors:

Problems with escaping:

Future work

Declarative field selector definitions on built-in types

There are two aspects of this:

  • built-in selectableFields converted to declarative go tags
  • built-in selectableFields included in OpenAPIv3

To make it easier and safer to add field selector definitions for built-in types in the future. We will support defining field selectors on API types using go struct tags.

type PodSpec struct {
  // NodeName ...
  //
  // +selectableField
  NodeName string `json:...`

  // Host networking ...
  //
  // +k8s:schema:selectableField:fieldPath="spec.hostNetwork",
  // +k8s:schema:selectableField:default="false"
  // +k8s:schema:selectableField:description="Set to the value of the spec.securityContext.hostNetwork field"
  // +optional
  HostNetwork bool `json:...`
}

String, bool and integer typed fields, and typerefs to these types will be supported and automatically cast to string.

default is needed to handle some of the existing builtin cases that provide a value that is not an empty string if the field is unset.

fieldSelector should only be set for legacy cases where the field selector name does not match the field path. When set, a description will be required.

description is provided for the legacy cases that use fieldSelector and require further clarification. For all other cases, it should not be needed.

Pod.status.podIP[0] is a special legacy case. We will handle it in code. We may need to add special go tag part to allow us to include declarative information in the go struct for discovery purposes while handling the implementation imperatively.

The OpenAPI would also include descriptions for built-in types. This is needed to explain some of the legacy field selectors that have surprising behavior.

"io.k8s.api.core.v1.Pod": {
        "description": "Pod is a collection of containers that can run on a host. This resource is created by clients and scheduled onto hosts.",
        "properties": { ... },
        "type": "object",
        "x-kubernetes-group-version-kind": [ ... ],
        "x-kubernetes-selectable-fields": [
          { "fieldPath": "metadata.name"},
          { "fieldPath": "metadata.namespace"},
          { "fieldPath": "spec.nodeName"},
          { "fieldPath": "spec.restartPolicy"},
          { "fieldPath": "spec.schedulerName"},
          { "fieldPath": "spec.serviceAccountName"},
          { "fieldPath": "spec.hostNetwork", "default": "false", "description": "Set to the value of the spec.securityContext.hostNetwork field"},
          { "fieldPath": "status.phase"},
          { "fieldPath": "status.podIP", "default": "", "description": "Set to the value of status.podIPs[0]"},
          { "fieldPath": "status.nominatedNodeName"},
        ]
      },

Index selected fields

Label and field selectors may be accelerated in the future. Possible approaches:

  • etcd adds filter/index/symlink support in the future and Kubernetes leverages it
  • the watch cache is updated to index labels and field selectors

CEL as an accelerated predicate language

I had drafted a CEL based field selection proposal in the past and discussed it in a larger thread about field selection.

One of the biggest problems is that there is a real possibility that label and field selectors will indexed in the future (as described above), and the community would like to avoid introducing changes to label or field selectors that prevent this from happening. The naive use of CEL--evaluate objects one-by-one in a full scan--would be very expensive in comparison to indexed selectors.

But there is a way CEL expressions could be accelerated, so long as the expressions only contain references to indexed fields.

Imagine a CEL expression like spec.x == 'a' && spec.y == 'b'.

This expression is conceptually very similar to a AND WHERE clause in SQL, which can be optimized SQL query engine so long as the ANDed fields are indexed.

With CEL, an expression's AST could be inspected and evaluated using an approach similar to those used by query engines. For example, for spec.x == 'a' &&spec.y == 'b' the field index for spec.x and spec.y could be inner joined to evaluate the expression.

For cases where this acceleration technique works:

Operations that canot be accelerated this could be assigned high costs, restricting the use of the CEL expressions entirely, or limiting their use.

CEL operations that could potentially be accelerated might include:

  • boolean logic: &&, J||, !
  • set operations: in, sets.intersects()
  • some string operations: string.endsWith() (requires more advanced prefix matching indices)

I do realize this implies the introduction of a CEL query engine for Kubernetes. But not all query engines are large or complex. If indexed, label selectors would require a small query engine that supports AND, NOT and IN. A simple CEL query engine might introduce OR and some other operations, but could be kept small and lean and still offer a substantial boost to the query options available today. A major benefit of CEL is that it provides a stable grammar that is already used elsewhere in Kubernetes.

Risks and Mitigations

Increases watch cache resource utilization

At first glance, the resource utilization cost to the watch cache will be roughly equivalent to the cost of using labels, suggesting that this enhancement doesn't introduce any fudamentally new risks.

But there is still a potential problem. CustomResourceDefinition authors might choose to add a large number of fields to selectableFields. This is plausible because it is so easy to add fields to selectableFields. For comparison, double writing data both to labels and to spec or status fields is significantly significantly more complicated and error prone. (Note, however, that once MutatingAdmissionPolicy is available writing data to labels for selector purposes will become easier and safer, so it might be good to get this enhancement in sooner than later).

Mitigations:

  • Limit selectableFields per CustomResourceDefinition to a small number (e.g. 8). We will pick the exact limit emperically after measuring the change in resource utilization.
  • Clearly document that each selectable field has associated resource costs.

Design Details

Test Plan

[x] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates
Unit tests

  • staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/validation/validation_test.go

    • duplicate selectableFields not allowed
    • invalid path not allowed
    • path to invalid type not allowed
    • too many selectableFields not allowed
    • empty string is set if field may be absent

  • staging/src/k8s.io/apiextensions-apiserver/pkg/registry/customresource/strategy_test.go

    • ensure field paths are not dereferenced if not used, or make it fast and benchmark it
    • all possible type cases, including string format cases with special type bindings (duration, intOrString)

  • staging/src/k8s.io/apiextensions-apiserver/pkg/registry/customresourcedefinition/strategy_test.go

    • all drop field cases

  • <package>: <date> - <test coverage>

Integration tests

For Alpha:

  • test/integration/controlplane/crd_test.go
    • selectableFields cannot be written feature gate disabled
    • feature disablement testing
    • Test list selectable field {with, without default}, {with single field, with multiple fields, with combined use of label selectors} for {list, watch}
    • test with multiple versions where the field moves from field path 1 to field path 2 across serializations and the REST request does not match the serialization stored in etcd
    • Test watch selectable field
      • Only field selected resources are observed
      • When a watched item changes values for its selection. Does a delete get issued to the watch?
    • Test DeleteCollection with a field selector
    • Test reading selectableField data from OpenAPI v2 and v3
e2e tests

For Beta:

  • test/e2e/apimachinery/custom_resource_definition.go

    • TODO: Test selectable fields with combined multiple fields and combined with label selectors for {list, watch}

  • :

Graduation Criteria

Alpha

  • CRD selectableFields implemented
  • CRD selectableField data available in OpenAPI discovery
  • Feature implemented behind a feature flag
  • integration tests completed and enabled

Beta

  • Optimize GetAttrs to only incur JSONPath lookup cost for selectableFields when actually used for field selection
  • e2e tests completed and enabled

Upgrade / Downgrade Strategy

Version Skew Strategy

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?
  • Feature gate (also fill in values in kep.yaml)
    • Feature gate name: CustomResourceFieldSelectors
    • Components depending on the feature gate: kube-apiserver
Does enabling the feature change any default behavior?

No

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

Yes

The enhancement's field selection will become unavailable on CRDs when disabled. When this happens, any clients depending on the new field selectors will receive an HTTP 400 Bad Request response with a message like:

"Unable to find \"stable.example.com/v1, Resource=selector\" that match label selector \"\", field selector \"spec.colorx=blue\": field label not supported: spec.colorx"

This is idential to the behavior before the enhancement is enabled.

selectableFields are removed from write requests to CRDs that do not already have selectableFields.

No other system behavior is impacted.

What happens if we reenable the feature if it was previously rolled back?

Any selectableFields already written to CRDs are reactivated and requests to select by the fields will be filtered correctly.

selectableFields can be added to CRDs again.

Are there any tests for feature enablement/disablement?

Yes

Unit tests: staging/src/k8s.io/apiextensions-apiserver/pkg/registry/customresourcedefinition/strategy_test.go:

=== RUN   TestDropDisabledFields/SelectableFields,_For_create,_FG_disabled,_SelectableFields_in_update,_dropped
=== RUN   TestDropDisabledFields/SelectableFields,_For_create,_FG_enabled,_no_SelectableFields_in_update,_no_drop
=== RUN   TestDropDisabledFields/SelectableFields,_For_create,_FG_enabled,_SelectableFields_in_update,_no_drop
=== RUN   TestDropDisabledFields/SelectableFields,_For_update,_FG_disabled,_oldCRD_has_SelectableFields,_SelectableFields_in_update,_no_drop
=== RUN   TestDropDisabledFields/SelectableFields,_For_update,_FG_disabled,_oldCRD_does_not_have_SelectableFields,_no_SelectableFields_in_update,_no_drop
=== RUN   TestDropDisabledFields/SelectableFields,_For_update,_FG_disabled,_oldCRD_does_not_have_SelectableFields,_SelectableFields_in_update,_dropped
=== RUN   TestDropDisabledFields/SelectableFields,_For_update,_FG_enabled,_oldCRD_has_SelectableFields,_SelectableFields_in_update,_no_drop
=== RUN   TestDropDisabledFields/SelectableFields,_For_update,_FG_enabled,_oldCRD_does_not_have_SelectableFields,_SelectableFields_in_update,_no_drop
=== RUN   TestDropDisabledFields/pre-version_SelectableFields,_For_update,_FG_disabled,_oldCRD_does_not_have_SelectableFields,_SelectableFields_in_update,_dropped

Integration tests: staging/src/k8s.io/apiextensions-apiserver/test/integration/fieldselector_test.go

=== RUN   TestFieldSelectorDropFields

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?
What specific metrics should inform a rollback?
Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?
How can someone using this feature know that it is working for their instance?
  • Events
    • Event Reason:
  • API .status
    • Condition name:
    • Other field:
  • Other (treat as last resort)
    • Details:
What are the reasonable SLOs (Service Level Objectives) for the enhancement?
What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

TODO: Should we add a "filtered" label to the below metrics? It would help isolate problems with selectors better.

Elevated HTTP 400 response codes in request_total for list and watch is a SLI for potential problems with requests using label selectors.

High request_duration_seconds for list requests may indicate a performance problem with label selectors.

Are there any missing metrics that would be useful to have to improve observability of this feature?

Dependencies

Does this feature depend on any specific services running in the cluster?

Scalability

Will enabling / using this feature result in any new API calls?
Will enabling / using this feature result in introducing new API types?
Will enabling / using this feature result in any new calls to the cloud provider?
Will enabling / using this feature result in increasing size or count of the existing API objects?
Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?
What are other known failure modes?
What steps should be taken if SLOs are not being met to determine the problem?

Implementation History

Drawbacks

Alternatives

Discovery

Declare selectable fields to the fieldSelector path parameter of OpenAPI

Path parameters for fieldSelector are arguably the correct place in OpenAPI to document selectable fields.

We decided against this option because, pragmatically, it is less convenient for clients to leverage since clients have significant tooling to process and interpret OpenAPI schemas but have very little tooling for parameters.

 "/api/v1/namespaces/{namespace}/pods": {
      ...
      "parameters": [
        ...
        {
          "description": "A selector to restrict the list of returned objects by their fields. Defaults to everything.",
          "in": "query",
          "name": "fieldSelector",
          "schema": {
            "type": "string",
            "uniqueItems": true,
            "x-kubernetes-selectable-fields": [
              { "fieldPath": "metadata.name"},
              { "fieldPath": "metadata.namespace"},
              { "fieldPath": "spec.nodeName"},
              { "fieldPath": "spec.restartPolicy"},
              { "fieldPath": "spec.schedulerName"},
              { "fieldPath": "spec.serviceAccountName"},
              { "fieldPath": "spec.hostNetwork", "default": "false", "description": "Set to the value of the spec.securityContext.hostNetwork field"},
              { "fieldPath": "status.phase"},
              { "fieldPath": "status.podIP", "default": "", "description": "Set to the value of status.podIPs[0]"},
              { "fieldPath": "status.nominatedNodeName"},
              ...
            ]
          }
        },
        ...
      ...
 }

Declare selectable fields by annotating schema fields as being selectable, e.g.:

"hostNetwork": {
            "description": "...",
            "type": "boolean",
            "x-kubernetes-selectable": {
                "default": "false",
            }
          },

The problem with this approach is that, for legacy reasons, not all selectable fields use an identifier that has a corresponding field in the schema. For example, but pods don't have a status.podIP field. Pods have a status.podIPs which is almost the same, but not quite. There are 6+ fields like this, see the table in the proposal section for full field selectors for more detail.

Infrastructure Needed (Optional)

Footnotes

  1. This field selector is commented with: "This is a bug, but we need to support it for backward compatibility.". ↩ ↩2