Serialization object Broadcasted

[saif-ellafi]

Description

Parquet proved to be too heavy on heap memory. Switching to an optional configuration setting, plus defaulting to RDD Objects returns good results.

Motivation and Context

Large corpus files make some pipelines very heavy on memory consumption on the driver. This helps there.

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Code improvements with no or little impact
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING page.
• I have added tests to cover my changes.
• All new and existing tests passed.