Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: DFP mlflow logger should be configured at same level as morpheus #593

Closed
dagardner-nv opened this issue Jan 4, 2023 · 0 comments · Fixed by #594
Closed

[BUG]: DFP mlflow logger should be configured at same level as morpheus #593

dagardner-nv opened this issue Jan 4, 2023 · 0 comments · Fixed by #594
Assignees
@dagardner-nv
Labels
bug Something isn't working

Comments

@dagardner-nv
Copy link
Contributor

Version

23.01

Which installation method(s) does this occur on?

Source

Describe the bug.

Running the DFP pipeline results in several logged messages from the mlflow client like:

2023/01/04 19:35:37 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-jtaylor@domain.com, version 1

This has two problems:

  1. Interferes with Morpheus' own output.
  2. DFP has a --log_level flag which defaults to WARN but only uses it to configure the morpheus logger not the root logger, allowing INFO level logs from mlflow

Minimum reproducible example

`python dfp_azure_pipeline.py --train_users=all --start_time="2022-08-01" --input_file="/workspace/examples/data/dfp/azure-training-data/*.json"`

Relevant log output

2023/01/04 19:35:37 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-jtaylor@domain.com, version 1


### Full env printout

```shell
Input data rate[Complete]: 3239 messages [00:11, 319.32 messages/s]2023/01/04 19:35:20 INFO mlflow.tracking.fluent: Experiment with name 'dfp/azure/training/DFP-azure-generic_user' does not exist. Creating a new experiment.messages/s]
2023/01/04 19:35:21 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 1
Input data rate[Complete]: 3239 messages [00:11, 319.2023/01/04 19:35:22 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 2
Input data rate[Complete]: 3239 messages [00:11, 319.2023/01/04 19:35:23 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 3
Input data rate[Complete]: 3239 messages [00:11, 319.2023/01/04 19:35:24 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 4
Input data rate[Complete]: 3239 messages [00:11, 319.2023/01/04 19:35:26 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 5
                                                     2023/01/04 19:35:26 INFO mlflow.tracking.fluent: Experiment with name 'dfp/azure/training/DFP-azure-tprice@domain.com' does not exist. Creating a new experiment.8, 32.71 messages/s]
Input data rate[Complete]: 3239 messages [00:11, 319.32 messages/s]2023/01/04 19:35:26 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-tprice@domain.com, version 1
Input data rate[Complete]: 3239 messages [00:11, 319.32 messages/s]2023/01/04 19:35:28 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 6
                                                     2023/01/04 19:35:28 INFO mlflow.tracking.fluent: Experiment with name 'dfp/azure/training/DFP-azure-cperry@domain.com' does not exist. Creating a new experiment.1, 35.60 messages/s]
2023/01/04 19:35:29 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-cperry@domain.com, version 1
Input data rate[Complete]: 3239 messages [00:11, 319.2023/01/04 19:35:29 INFO mlflow.tracking.fluent: Experiment with name 'dfp/azure/training/DFP-azure-attacktarget@domain.com' does not exist. Creating a new experiment.40 messages/s]
2023/01/04 19:35:30 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-attacktarget@domain.com, version 1
Input data rate[Complete]: 3239 messages [00:11, 319.2023/01/04 19:35:31 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 7
                                                     2023/01/04 19:35:32 INFO mlflow.tracking.fluent: Experiment with name 'dfp/azure/training/DFP-azure-acole@domain.com' does not exist. Creating a new experiment.24, 38.36 messages/s]
2023/01/04 19:35:32 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-acole@domain.com, version 1
Input data rate[Complete]: 3239 messages [00:11, 319.32 messages/s]2023/01/04 19:35:34 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 8
Input data rate[Complete]: 3239 messages [00:11, 319.32023/01/04 19:35:37 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 9
                                                                2023/01/04 19:35:37 INFO mlflow.tracking.fluent: Experiment with name 'dfp/azure/training/DFP-azure-jtaylor@domain.com' does not exist. Creating a new experiment., 39.48 messages/s]
2023/01/04 19:35:37 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-jtaylor@domain.com, version 1
Input data rate[Complete]: 3239 messages [00:11, 288.50 messages/s]
Training rate[Complete]: 1176 messages [00:29, 39.43 messages/s]
(morpheus) root@ab07b4964257:/workspace/examples/digital_fingerprinting/production/morpheus# python dfp_azure_pipeline.py --train_users=all --start_time="2022-08-01" --input_file="/workspace/examples/data/dfp/azure-training-data/*.json"
Input data rate[Complete]: 3239 messages [00:01, 16992023/01/04 19:38:28 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 10
Input data rate[Complete]: 3239 messages [00:01, 16992023/01/04 19:38:29 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 11
Input data rate[Complete]: 3239 messages [00:01, 1699.58 messages/s]2023/01/04 19:38:30 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 12
Input data rate[Complete]: 3239 messages [00:01, 16992023/01/04 19:38:31 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 13
Input data rate[Complete]: 3239 messages [00:01, 1699.58 messages/s]2023/01/04 19:38:33 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 14
                                                     2023/01/04 19:38:34 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-tprice@domain.com, version 2
Input data rate[Complete]: 3239 messages [00:01, 16992023/01/04 19:38:36 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 15
Input data rate[Complete]: 3239 messages [00:01, 16992023/01/04 19:38:36 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-cperry@domain.com, version 2
Input data rate[Complete]: 3239 messages [00:01, 1699.58 messages/s]2023/01/04 19:38:37 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-attacktarget@domain.com, version 2
Input data rate[Complete]: 3239 messages [00:01, 1699.58 messages/s]2023/01/04 19:38:39 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 16
                                                     2023/01/04 19:38:40 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-acole@domain.com, version 2
Input data rate[Complete]: 3239 messages [00:01, 1699.2023/01/04 19:38:42 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 17
Input data rate[Complete]: 3239 messages [00:01, 1699.58 messages/s]2023/01/04 19:38:45 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-generic_user, version 18
                                                                2023/01/04 19:38:46 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: DFP-azure-jtaylor@domain.com, version 2
Input data rate[Complete]: 3239 messages [00:01, 1774.95 messages/s]
Training rate[Complete]: 1176 messages [00:24, 48.22 messages/s]



### Other/Misc.

_No response_

### Code of Conduct

- [X] I agree to follow Morpheus' Code of Conduct
- [X] I have searched the [open bugs](https://github.com/nv-morpheus/Morpheus/issues?q=is%3Aopen+is%3Aissue+label%3Abug) and have found no duplicates for this bug report
@dagardner-nv dagardner-nv added the bug Something isn't working label Jan 4, 2023
@dagardner-nv dagardner-nv self-assigned this Jan 4, 2023
@dagardner-nv dagardner-nv changed the title [BUG]: DFP mlflow client is chatty [BUG]: DFP mlflow logger should be configured at same level as morpheus Jan 4, 2023
@dagardner-nv dagardner-nv mentioned this issue Jan 4, 2023
Merged
@ghost ghost closed this as completed in #594 Jan 7, 2023
ghost pushed a commit that referenced this issue Jan 7, 2023
@dagardner-nv
Ensure that the mlflow logger is uses at the same level as the morphe…
1c52ae1 
…us logger (#594)

fixes #593

Authors:
  - David Gardner (https://github.com/dagardner-nv)
  - Eli Fajardo (https://github.com/efajardo-nv)

Approvers:
  - Eli Fajardo (https://github.com/efajardo-nv)

URL: #594
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dagardner-nv dagardner-nv

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ensure that the mlflow logger is uses at the same level as the morpheus logger dagardner-nv/Morpheus
1 participant
@dagardner-nv