#113 Improved rendering of config arrays and objects in logs.

[jirifilip]

Improved rendering of config trees. Now arrays and objects are flattened and every line of the rendered config corresponds to a primitive value or array of primitive values.

pramen.sources[0].factory.class="za.co.absa.pramen.core.source.SparkSource"
pramen.sources[0].name="csv"
pramen.sources[0].format="csv"
pramen.sources[0].notification.targets=["a"  "b"  "c"]
...
pramen.sources[1].jdbc.password=[ redacted ]
...

Because of the flattened keys like pramen.sources[1].jdbc.password, the getRedactedFlatConfig() method should now work properly on matching the "password" token inside the key and redacting the value. Before, the password was hiding inside pramen.sources array as a value and was not redacted.

I was thinking how to do this in a nice way and discovered the ConfigValue, ConfigList and ConfigObject interfaces which can be pattern matched against to traverse the config tree. ConfigValueType can then be used to check if the ConfigList only consists of primitive types.

Some ideas for discussion

I don't know if getFlatConfigOfPrimitiveValues() is a good name.

I only substituted the getFlatConfig() for the new getFlatConfigOfPrimitiveValues() in logEffectiveConfigProps() function which logs the config when running a pipeline. But I did not substitute it in other uses of getFlatConfig() (for example, creating .properties from the Config for Kafka configuration or rendering the smtp config in Sendable, getExtraOptions()) because in those other places, it may be undesirable.

[yruslan]

This is fantastic! I like your solution a lot.

Also, thanks a lot for such a detailed PR description!

(The function name getFlatConfigOfPrimitiveValues() is fine for now. If we find a better name in the future, we can always rename it later 😄