DFP: Exclude unwanted columns #583

dagardner-nv · 2022-12-21T01:08:36Z

Filter stage now optionally able to use a DataFrame column as the filter criteria.
Choice of a DataFrame column or a Tensor is specified with a FilterSource enum.
Accepted message type of the Filter stage is now driven by the choice of a DF column or Tensor as the criteria.
DType now has a default constructor.
DFP Post Processing stage no longer performs it's own filtering and no longer converts the output message to a UserMessageMeta.
DFP inference pipelines updated to use Filter & Serialize stages, resulting in unwanted columns excluded from output (fixes [FEA]: DFP should use a serialize stage, and exclude internal fields #582).
Update DFP docs to reflect these changes

Breaking Changes:

morpheus._lib.file_types merged into morpheus._lib.common
DevMemInfo expanded to include shape & stride attributes
MatxUtil updated to take advantage of new attrs in DevMemInfo

…t-messages rather than a meta, allowing serialize-stage to be used

…y_probs=True

… get_slice method so that the return type matches the input

…ssage isn't strictly required

… method

morpheus/_lib/include/morpheus/objects/filter_source.hpp

mdemoret-nv

Looks better using the DevMemInfo everywhere.

mdemoret-nv · 2023-01-21T01:05:49Z

/merge

dagardner-nv added 4 commits December 20, 2022 08:20

Update container version to 23.01

53b1ffa

DFP examples require dask

afcabd3

Slice the output rather than performing a copy, and return sliced mul…

0a27eec

…t-messages rather than a meta, allowing serialize-stage to be used

Add a serialize-stage, TODO: update docs & diagrams

81acb8a

dagardner-nv added non-breaking Non-breaking change improvement Improvement to existing functionality 2 - In Progress labels Dec 21, 2022

dagardner-nv requested a review from a team as a code owner December 21, 2022 01:08

dagardner-nv added 22 commits December 21, 2022 12:05

Merge branch 'david-dfp-23.01' into david-dfp-serialize-stage

0f9e791

Add serializestage to notebooks, and comment the stage

0a90ec9

First pass at filtering on a dataframe column or a tensor

4115112

First pass at moving the FilterSource enum to C++

9eff636

Switch to the C++ impl of the enum

a1ef1d2

New constructor args

fda041a

Remove unused imports, add new args for FilterDetectionsStage

c49cdca

Use field_name var

796c2b0

wip operate on a dataframe

d12d28b

Give Dtype a default value allowing it to be default contrstructable

4aa7db8

Move df column & tensor specific code to their own methods

8ec54d4

Pass new constructor arguments to CPP impl

560ee9c

tests

fdf7a15

Parameterize copy vs slice

f08c689

more tests

567b0a1

Fix cli help string, and always declare probs as a 2d array when empt…

7ffa253

…y_probs=True

wip

1903b8d

wip

81ba67e

override get_slice method

e8bbf29

Accepted type changes to MultiMessage when source is a df column, use…

3a10d18

… get_slice method so that the return type matches the input

Filter stage can accept a MultiMessage when the source is a df column

a411642

Use np.zeros when the dataframe is pandas

ccb646f

dagardner-nv and others added 6 commits January 17, 2023 10:19

Merge branch 'branch-23.01' into david-dfp-serialize-stage

8aeda20

Rename operate_on to data_source

054abe3

Change casing of FilterSource::AUTO to match that of the FileTypes enum

d2c6e01

Rename data_source to filter_source matching the enum type

97391a8

Move bindings for FilterSource into the common module

5037640

Merge the file_types python module into common

0202519

dagardner-nv added breaking Breaking change and removed non-breaking Non-breaking change labels Jan 17, 2023

dagardner-nv added 13 commits January 17, 2023 15:46

Cast to MultiTensorMessage

180abca

Set accepted types to MultiResponseMessage since MultiResponseProbsMe…

f7cac94

…ssage isn't strictly required

Auto determines the filter source based on the message type

18c685a

fix

edf7aef

Move the check of m_filter_source outside of the hot path

5dcef3b

Fix type-o

1a60846

First pass at a refactored DevMemInfo

c74339a

replace the memory_resource and stream methods with a make_new_buffer…

4727191

… method

wip

065c69e

wip

f80ab01

Replace usage of BufferInfo with DevMemInfo

0dfbe40

Move the get methods into private class methods

c830c21

Make private attrs const

27a8f6d

dagardner-nv commented Jan 18, 2023

View reviewed changes

morpheus/_lib/include/morpheus/objects/filter_source.hpp Show resolved Hide resolved

dagardner-nv added 4 commits January 18, 2023 14:46

Remove unused variable

ce587d5

Remove default values for filter_source & copy

2af0bd4

Misc cleanups

c3732e0

Fix missed arguments after data_source->filter_source rename

532c6d8

dagardner-nv requested a review from mdemoret-nv January 19, 2023 00:24

mdemoret-nv approved these changes Jan 21, 2023

View reviewed changes

ghost merged commit 7e4e774 into nv-morpheus:branch-23.01 Jan 21, 2023

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DFP: Exclude unwanted columns #583

DFP: Exclude unwanted columns #583

dagardner-nv commented Dec 21, 2022 •

edited

Loading

mdemoret-nv left a comment

mdemoret-nv commented Jan 21, 2023

DFP: Exclude unwanted columns #583

DFP: Exclude unwanted columns #583

Conversation

dagardner-nv commented Dec 21, 2022 • edited Loading

mdemoret-nv left a comment

Choose a reason for hiding this comment

mdemoret-nv commented Jan 21, 2023

dagardner-nv commented Dec 21, 2022 •

edited

Loading