[Ingest Manager] Required fields should be auto-expanded in Config UI

[mostlyjason]

In the integration configuration editor UI, the streams with required fields are collapsed by default. This makes it difficult for users to know what is being collected, or if they need to edit any of the fields.

This is particularly true for custom logs, where the path of the log file is blank by default but it cannot operate without a valid path. The streams should be expanded so that the user can easily discover that they need to provide a path. This will also reduce the number of clicks that the user must perform. It also reduces the number of times the user must scroll down, because they need to scroll down to expand the streams and then scroll down again once its expanded.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[jen-huang]

If streams are all expanded by default, the page can get very long due to number of streams.

IIRC the decision was made to collapse streams by default so that the flow for user adding a data source is a simple as possible (most of the complexity is hidden). cc @hbharding for further input.

[paul-tavares]

Something I have notice that applies to this topic. Personally, I also like the collapsed state for its simplicity and user experience, but what I have found is that some of these collapsed streams have required fields and those are not visible to the user until they try to save at which point you get errors. I think that the ones that have the need for required user interaction should be expanded.

[hbharding]

I think what @paul-tavares and @mostlyjason described is a bug. We discussed this awhile ago and decided at the time to collapse all inputs which do not have any require fields. I.e. if an input has required fields, it should be expanded by default. The goal was to simplify the UX so users don't have to scroll through and see every possible field. We do this again at the stream level by hiding 1) fields which we can supply a good default for and 2) fields which are optional, under "advanced options"

I am a little wary that hiding input fields by default could mislead a user or cause them to forget to configure something. And as Jason points out, it's extra clicks for the users who want to dive in and configure more. I think it depends who we want to cater to. First time users who don't want to be exposed to everything, or seasoned users who have known use case in mind and know what to find.

We should definitely address this "collapsed required field" thing as a bug, but perhaps we could change the expand/collapse logic so that inputs are always expanded (revealing all streams), but advanced stream options remain collapsed.

[mostlyjason]

I think it'd be a fine solution to expand streams with required fields. It looks like the logs integration does make this field required https://github.com/elastic/package-storage/blob/master/packages/log/0.1.0/dataset/log/manifest.yml#L13. The issue is that it didn't expand this required field.

I'll update the issue to mark this as a bug.

[jen-huang]

++ on treating this as a bug for required fields