[Discuss] Usage of yaml field type / multiline text fields

[jen-huang]

One of the valid user configuration variable types is yaml:

package-spec/versions/1/data_stream/manifest.spec.yml

Line 58 in 1cd07c6

- yaml

I do not recall much discussion during the original implementation of this type, but the intention was for Fleet UI to display a code editor with YAML syntax highlighting when this variable type is encountered:
https://github.com/elastic/kibana/blob/77636e73a918b0946ec3fda0b67745663e3498a5/x-pack/plugins/fleet/public/applications/fleet/sections/agent_policy/create_package_policy_page/components/package_policy_input_var_field.tsx#L64

We assumed that authors would use this field to allow users to input yaml configuration, i.e. key-value pairs (agnostic of nesting levels, or types of values).

In the Crowdstrike integration, there is one field that expects multiline text input. Originally this field was set to type: text which causes Fleet to render a simple one-line input field, which is not a great UX for multiline content, and there was an issue with saving the value due to that. A fix (elastic/integrations#2701) was made by changing it to type: yaml. While this works, I'm not sure that yaml should be used in this manner for values that are not key-value pairs. Might we consider a different spec for non-yaml, multiline text?

This issue is intended to continue the discussion outside of where the original issue was found.

[andrewkroh]

One challenge with unmarshaling yaml type variables and then rendering them back out into the policy is that they can take many forms depending on user input - object, array, string, number, bool, null, etc. In order to form a valid YAML document the template author is required to make assumptions about the data type. For example if the value is an object then the template needs to formed as

my_value:
{{some_var}}

but if the YAML value becomes a string then the template is no longer valid because it needs to be formed as

my_value: {{some_var}}

We solved this in the Beat module templates by including a to_json function that encodes the value to JSON. Because JSON is a subset of YAML and immune to indentation issues my_value: {{some_var | to_json}} works for all data types. Here's an similar handlebars example.

[andrewkroh]

Also I think we need documentation to explain how the different data types defined in the specification are supposed to behave. Developers (and implementors of the spec) need to know things like what validation is associated with each type, if there is any special rendering applied in handlebar templates. I could try to explain this in spec based on my observations of the behavior.

[nchaulet]

Should we enforce more validation on the yaml type or create a new type yaml_object that could eventually be validated by a JSON schema and only allow a key: value object? Allowing to add multiple type in this yaml variables will cause problems during the template rendering