Transient transformers (Scala only)

[yruslan]

Background

TBD

Feature

A transient transformation is not persistent and its results can only be used inside the same pipeline.

• Data for from this transformer can only be created for the current information date
• It can be saved to a temporary location for performance
• Not cached (!) since we don't know how many such transformers are in the pipeline.
• If temp dir is not specify - have to recompute every time it is used

Example

• Standardization in Enceladus process

Proposed Solution

• A new 'data format' for metastore tables 'transient'.
• No persistent bookkeeping for such tables, but only for the duration of the session
  • Requires changes to the bookkeeper to support this
• If a transformer depends on a transient transformer, the transient transformer should be in the same pipeline, fail otherwise.