feat(ingestion): Adds support for memory profiling

[pedro93]

Adds memray as opt-in memory profiler for ingestion processes.
Tested locally that it works.

Recipe:

source:
  type: "file"
  config:
    path: "metadata-ingestion/examples/mce_files/bootstrap_mce.json"

flags:
  generate_memory_profiles: "/tmp/datahub/ingest"

Generates a binary flamegraph file such as file-None-file-2023_09_18-21_38_43.bin that can then be previewed as an HTML after running$ memray flamegraph file-None-file-2023_09_18-21_38_43.bin locally.

Sample output:

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[hsheth2]

a few final things, but would be good to get this merged!

[hsheth2]

don't think this works - if generate_memory_profiles is not set, then we don't do anything

btw you can use contextlib.ExitStack or NullContext to conditionally enter a with clause

[pedro93]

This was an oversight on my part. Already fixed.

[hsheth2]

@pedro93 code looks good, something about the markdown file is causing issues though