refactor(operation_config): change logging on is_profiling_enabled

[cccs-cat001]

due to the is_profiling_enabled method getting called for every data set, it spams log output with these info lines.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines	Coverage Δ
...ion/src/datahub/ingestion/source/sql/sql_config.py	98.68% <ø> (-0.07%)	⬇️
...atahub/ingestion/source_config/operation_config.py	70.00% <100.00%> (+1.91%)	⬆️

... and 5 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 457f96e...e3e7d75. Read the comment docs.

[hsheth2]

Honestly it seems strange that we're repeatedly calling is_profiling_enabled

I think the correct solution is to actually cache the result of is_profiling_enabled / only call it once, not reducing the log level.

[cccs-cat001]

yeah that's fair. is it worthwhile for me to try and make this change? Where would be the best spot to start? This function is called from many different sources.

[cccs-cat001]

I'm no python pro or anything, but would what I did to iceberg be what we want?

I don't mind doing it for the others that use this function.

[hsheth2]

This sort of approach - particularly the lru cache one, feels pretty reasonable. If you're up for it, it'd make sense to make similar changes to other sources as well

[cccs-cat001]

I started going through the other sources and found this - https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/sql/sql_config.py#L121-L126

Would it not make sense to just add the caching to the operation_config function? One stop and it should fix them all eh?

[hsheth2]

@cccs-cat001 yeah I think that makes sense - if we just did what that comment suggested, this would work more nicely

We'd need to add a dependency on cachetools for any source that uses profiling, but I think that's acceptable

[cccs-cat001]

alright... this is my first time contributing... so how can I fix the build 🤣🤣

[hsheth2]

You can run the linter and tests locally: https://datahubproject.io/docs/metadata-ingestion/developing/#code-style

[cccs-cat001]

Hey looks like the CI has passed, can we merge in now? :)