Beta label support

[ebeahan]

Summary

Introduces a new key, beta, which when set to True designates a field as being in beta support in the ECS field reference documentation.

Motivation

Fields introduced in RFCs which have advanced to stage three are to be committed to the schema as beta. The "beta" label should appear in the ECS docs next to any fields which are in beta.

The ECS field definitions need an option to specify if a field is currently considered beta. Once an RFC advances to stage four, the beta label will be removed by removing the beta option from the field's definition.

Implementation

A field can be marked as beta by specifying the beta option:

    - name: executable
      level: extended
      type: keyword
      beta: True
      description: >
        Absolute path to the process executable.
      example: /usr/bin/ssh
      multi_fields:
      - type: text
        name: text

Field Reference Docs Example

Once the field reference documentation is regenerated using make, the field now appears in the docs with the beta label if we pass text to the beta:[] argument:

The label is using the default tooltip text, but we can optionally customize the contents.

[ebeahan]

Also shared in elastic/docs#1754 (comment)

There's currently a CSS-related issue in how the generated docs will display the beta:[] tooltip inside a table. Depending on where the [beta] label is placed in a fieldset's table, part of the tooltip may get cut off by the bounds of the table.

[webmat]

This does a good job for any beta fields.

However I'm wondering if automating this beta tagging will be a complex rabbit-hole. Here's various situations I see, with regards to beta concepts in ECS:

• We have a whole section in beta, in the categorization fields. Although it's already marked beta in another way, so we can ignore this.
• RFC 0007 [RFC] Multiple users in an event, stage 3 PR #1017 will introduce beta field reuses
• RFC 0001 [RFC] Wildcard - stage 3 proposal #1015 will migrate existing non-beta fields to a different type. But some of these fields have been around since ECS 0.1.0, so they're not beta fields, only their usage of the new type is :-)
• This will work for most other RFCs in flight, although I could see a desire to flag a whole field set as beta as well.

This opens up a few questions.

Field reuses: I think we could potentially add support inside the small reuse object to identify that this is a beta reuse:

  reusable:
    expected:
      - at: user
        as: target
        beta: true

Field type: this one I'm not sure how we should mark a transition like: used to be keyword and is now wildcard.

Field set: Not sure if we'd want to have the beta label appear on every single field in a new beta field set. A plain text notice might be better at the top of the field set.

With all of these edge cases -- and I might be forgetting others -- I wonder if we shouldn't try to rethink this with a more basic approach.

Perhaps something like a beta notice at the field set level, that takes text (just like the description field). I think this would let us most edge cases quite simply.

Field reuses:

- name: user
  title: User
  beta: >
    Note that the new field reuses `user.effective`, `user.target` and `user.changes` are beta.

Type changes:

- name: user
  title: User
  beta: >
    Note that the new field reuses `user.effective`, `user.target` and `user.changes` are beta.
    Note that usage of type `wildcard` for fields `user.name`, ... is considered beta. These fields used to be using type `keyword`.

Note that these "type change" beta notices would have to be applied to each field set modified.

Note also that by ECS 1.8, the user field set will likely need both of these notices. This approach lets us deal with this in a simple manner with text. Although I'm not sure if having this single growing notice actually becomes a problem with my approach.

Field set:

Any time a whole field set is introduced, we can reuse the same boilerplate text.

- name: Foo
  title: Foo
  beta: >
    Note that this field set was recently introduced, and is considered beta.

I'm still on the fence for one off fields being marked beta, however. With my approach we could absolutely have a beta paragraph that calls out beta fields at the top. However when specific fields are beta, I think it's also great to have a mention in context. So perhaps we should keep the current approach, and pair it with a field set notice that calls out to them?

[ebeahan]

I can see the simplicity of containing everything to the field set level. It would also let us workaround the table formatting issue I mentioned above 😄 .

We had a discussion recently around making each field directly linkable. I can also see value in having the individuals fields tagged with a beta marker for that case. Avoids any chance for a user to visit the direct link and fails to realize the field is beta since they didn't happen to scroll up to view the field set level header. However, it does feel unnecessary to include beta on every field in an entirely beta field set. 🤔

If we agree to leave the beta marker for individual fields, we do have the ability to control the text in the tooltip. Perhaps we can use this feature to better explain when beta is referring to a type migration vs. a net new field?

(Again, though, adds some additional complexity of handling the custom text for a particular subset of fields).

I do agree we will eventually need support for a field set level notice regardless, and if we do collectively decide to continue marking specific fields, I think also having the field set level beta notice too is a good add.

Any objections to implementing the field set level similar to this mock-up? The text would be populated based on the field set level beta description.

- name: agent
  title: Agent
  beta: >
    Note that this field set was recently introduced, and is considered beta.

For the field reuse cases, I think having beta called out somewhere underneath the Field Reuse section is helpful. Each field reuse can already be linked to directly in the current docs. I also think it's clearer to call out when a field reuse is considered beta underneath the Field Reuse section vs. the very top of the field set.

I put together a couple of mocks again to give an idea of what the end results could be:

• Beta fields reuse section header. Text specifies which field reuses are considered beta.

• Marking each field nesting beta individually:

[ebeahan]

I've taken an approach that feels a bit "All the Things!!", but I like that it offers maximum flexibility in methods available for beta labeling.

Placement

The [beta] marker now appears on its own line in the description field vs. the field name. I felt this was a cleaner look. It also eliminates many instances of the CSS issue mentioned earlier #1051 (comment).

Attributes

Fieldsets

Added a beta fieldset-level attribute which adds a beta warning for the entire fieldset. The text provided in the beta attribute value becomes the contents of the marker:

- name: user
  ...
  beta: >
    This fieldset is currently beta

Fields

The field level attribute remains. The text provided in the attribute will become the contents of the beta tooltip.

    - name: name
      ...
      beta: >
        This field is currently beta.

Field Reuse

For field reuses, I've added support for a beta key under the expected object. This has the drawback of not supporting the flat-style notation, but this can be worked around by using the longer object notation.

The text from beta again is used for the tooltip.

  reusable:
    top_level: true
    expected:
      ...
      - at: user
        as: target
        beta: >
          This is a beta reuse field

[webmat]

I love where this is going!

I like the option in field reuses. Don't worry about the flat list, it's now just a shorthand for the most straightforward reuses. The "real" reuse notation now is the object one, so whenever we add features there, we should only worry about the object notation.

Also, between the two renderings of the beta notice for field reuse, I much prefer the one in context ("Marking each field nesting beta individually"). The other one will be too easy to miss, IMO.

Do all 3 new displays of the beta text deal well with multiple paragraphs? If not, perhaps we should watch out for that in the script, and break (in --strict mode).

[webmat]

LGTM

[ebeahan]

Thanks @webmat!

To align with the other changes, I made some final adjustments in schemas/README.md.

[webmat]

Alright, thanks for adding these mentions. Perhaps we should say "Beta notices should not..."

[ebeahan]

Ok, I made that adjustment.

[webmat]

LGTM