Generating process tree using alternative data source

[blueteam0ps]

Hi there,
I got a quick question regarding creating a process tree based on an alternative data source. I understand that this will require me to copy an existing schema and make one for my source data set.

Following is the dataframe which I would like to create a process tree of.

I referred to this https://github.com/microsoft/msticpy/blob/c66a03fdd376c86b3ce3af266a91c6cf53c0d3fd/docs/source/visualization/ProcessTree.rst#adapting-the-input-schema-of-your-data to create the following.

Do you think this is the correct way to achieve what I need ?

from msticpy.sectools.proc_tree_builder import WIN_EVENT_SCH
from copy import copy
dfprocs_renamed = dfprocs.rename(columns={'PID': 'process_id', 'PPID': 'parent_id', 'ImageFileName': 'process_name', 'CreateTime': 'time_stamp',  'PPIDName': 'parent_name'})
procs_rmcolumns = ['TreeDepth', 'Offset(V)','Threads','Handles','SessionId','Wow64','ExitTime']
dfprocs_renamed.drop(procs_rmcolumns, inplace=True, axis=1)  
cust_win_schema = copy(WIN_EVENT_SCH)
cust_win_schema.time_stamp = "time_stamp"
cust_win_schema.process_name = "process_name"
cust_win_schema.process_id = "process_id"
cust_win_schema.parent_name = "parent_name"
cust_win_schema.parent_id = "parent_id"
cust_win_schema.logon_id = None
cust_win_schema.cmd_line = None
cust_win_schema.user_name = None
cust_win_schema.path_separator = None
cust_win_schema.host_name_column = None
cust_win_schema.event_id_column = None
cust_win_schema.event_id_identifier = None
process_tree.plot_process_tree(dfprocs_renamed, schema=cust_win_schema)

[ianhelle]

Hi @blueteam0ps.
The idea of the ProcessSchema class is that you shouldn't have to do any renaming in your DataFrame.
You can provide either a ProcSchema instance or a dict with the correct source names to
the plot_process_tree function.

E.g.

my_schema = {
    "time_stamp": "CreateTime",
    "process_name": "ImageFileName",
    "process_id": "PID",
    "parent_name": "PPIDName",
    "parent_id": "PPID",
    "logon_id": "SessionId",
}

dfprocs.mp_plot.process_tree(schema=my_schema)

You can also use ProcSchema

from msticpy.nbtools.process_tree import ProcSchema
my_schema = ProcSchema(
    time_stamp="CreateTime",
    ...

Is the source DF from a common data source (e.g. Splunk, elastic?), if so I can add a builtin ProcSchema and have it auto-infer the source schema.

[blueteam0ps]

@ianhelle Thanks fort the response. The source DF was the output from Volatility's pstree plugin and the output was a CSV. So I just read the CSV into a DF.

Below are the columns available from the raw output
"TreeDepth","PID","PPID","ImageFileName","Offset(V)","Threads","Handles","SessionId","Wow64","CreateTime","ExitTime"

I added the cmd_line, path_separator and host_name_column fields as it complained those are mandatory. So I had to set them to none. Now i'm getting a different error as shown below
my_schema = {
"time_stamp": "CreateTime",
"process_name": "ImageFileName",
"process_id": "PID",
"parent_name": "PPIDName",
"parent_id": "PPID",
"logon_id": "SessionId",
"cmd_line" : None,
"user_name" : None,
"path_separator" : None,
"host_name_column" : None,
}

dfprocs.mp_plot.process_tree(schema=my_schema)

---

IntCastingNaNError Traceback (most recent call last)
Input In [9], in <cell line: 14>()
1 my_schema = {
2 "time_stamp": "CreateTime",
3 "process_name": "ImageFileName",
(...)
11 "host_name_column" : None,
12 }
---> 14 dfprocs.mp_plot.process_tree(schema=my_schema)

File /usr/local/lib/python3.8/dist-packages/msticpy/vis/mp_pandas_plot.py:282, in MsticpyPlotAccessor.process_tree(self, **kwargs)
243 def process_tree(self, **kwargs) -> Tuple[figure, LayoutDOM]:
244 """
245 Build and plot a process tree.
246
(...)
280
281 """
--> 282 return build_and_show_process_tree(data=self._df, **kwargs)

File /usr/local/lib/python3.8/dist-packages/msticpy/nbtools/process_tree.py:122, in build_and_show_process_tree(data, schema, output_var, legend_col, **kwargs)
120 missing_cols = _check_proc_tree_schema(data)
121 if missing_cols:
--> 122 data = build_process_tree(procs=data, schema=schema)
124 return plot_process_tree(
125 data, schema, output_var=output_var, legend_col=legend_col, **kwargs
126 )

File /usr/local/lib/python3.8/dist-packages/msticpy/sectools/proc_tree_builder.py:86, in build_process_tree(procs, schema, show_summary, debug)
84 extr_proc_tree = mde.extract_process_tree(procs, debug=debug)
85 else:
---> 86 extr_proc_tree = winlx.extract_process_tree(procs, schema=schema, debug=debug)
87 merged_procs_keys = _add_tree_properties(extr_proc_tree)
89 # Build process paths

File /usr/local/lib/python3.8/dist-packages/msticpy/sectools/proc_tree_build_winlx.py:53, in extract_process_tree(procs, schema, debug)
26 """
27 Build process trees from the process events.
28
(...)
50
51 """
52 # Clean data
---> 53 procs_cln = _clean_proc_data(procs, schema)
55 # Merge parent-child
56 merged_procs = _merge_parent_by_time(procs_cln, schema)

File /usr/local/lib/python3.8/dist-packages/msticpy/sectools/proc_tree_build_winlx.py:98, in _clean_proc_data(procs, schema)
96 procs_cln = procs_cln[event_type_filter]
97 # Convert any numeric schema cols to str types
---> 98 procs_cln = _num_cols_to_str(procs_cln, schema)
100 procs_cln[Col.EffectiveLogonId] = procs_cln[schema.logon_id]
101 # Create effective logon Id for Windows, if the TargetLogonId is not 0x0

File /usr/local/lib/python3.8/dist-packages/msticpy/sectools/proc_tree_build_winlx.py:140, in _num_cols_to_str(procs_cln, schema)
132 schema_cols = [
133 col for col in attr.asdict(schema).values() if col and col in procs_cln.columns
134 ]
135 force_int_cols = {
136 col: "int"
137 for col, col_type in procs_cln[schema_cols].dtypes.to_dict().items()
138 if pd.api.types.is_float_dtype(col_type)
139 }
--> 140 procs_cln = procs_cln.astype(force_int_cols)
141 # then change any int types in to string types
142 # note: we need the prev
143 int_to_str_cols = {
144 col: "str"
145 for col, col_type in procs_cln[schema_cols].dtypes.to_dict().items()
146 if pd.api.types.is_integer_dtype(col_type)
147 }

File /usr/local/lib/python3.8/dist-packages/pandas/core/generic.py:5906, in NDFrame.astype(self, dtype, copy, errors)
5904 res_col = col.copy() if copy else col
5905 else:
-> 5906 res_col = col.astype(dtype=cdt, copy=copy, errors=errors)
5907 results.append(res_col)
5909 elif is_extension_array_dtype(dtype) and self.ndim > 1:
5910 # GH 18099/22869: columnwise conversion to extension dtype
5911 # GH 24704: use iloc to handle duplicate column names
5912 # TODO(EA2D): special case not needed with 2D EAs

File /usr/local/lib/python3.8/dist-packages/pandas/core/generic.py:5920, in NDFrame.astype(self, dtype, copy, errors)
5913 results = [
5914 self.iloc[:, i].astype(dtype, copy=copy)
5915 for i in range(len(self.columns))
5916 ]
5918 else:
5919 # else, only a single dtype is given
-> 5920 new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
5921 return self._constructor(new_data).finalize(self, method="astype")
5923 # GH 33113: handle empty frame or series

File /usr/local/lib/python3.8/dist-packages/pandas/core/internals/managers.py:419, in BaseBlockManager.astype(self, dtype, copy, errors)
418 def astype(self: T, dtype, copy: bool = False, errors: str = "raise") -> T:
--> 419 return self.apply("astype", dtype=dtype, copy=copy, errors=errors)

File /usr/local/lib/python3.8/dist-packages/pandas/core/internals/managers.py:304, in BaseBlockManager.apply(self, f, align_keys, ignore_failures, **kwargs)
302 applied = b.apply(f, **kwargs)
303 else:
--> 304 applied = getattr(b, f)(**kwargs)
305 except (TypeError, NotImplementedError):
306 if not ignore_failures:

File /usr/local/lib/python3.8/dist-packages/pandas/core/internals/blocks.py:582, in Block.astype(self, dtype, copy, errors)
564 """
565 Coerce to the new dtype.
566
(...)
578 Block
579 """
580 values = self.values
--> 582 new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
584 new_values = maybe_coerce_values(new_values)
585 newb = self.make_block(new_values)

File /usr/local/lib/python3.8/dist-packages/pandas/core/dtypes/cast.py:1292, in astype_array_safe(values, dtype, copy, errors)
1289 dtype = dtype.numpy_dtype
1291 try:
-> 1292 new_values = astype_array(values, dtype, copy=copy)
1293 except (ValueError, TypeError):
1294 # e.g. astype_nansafe can fail on object-dtype of strings
1295 # trying to convert to float
1296 if errors == "ignore":

File /usr/local/lib/python3.8/dist-packages/pandas/core/dtypes/cast.py:1237, in astype_array(values, dtype, copy)
1234 values = values.astype(dtype, copy=copy)
1236 else:
-> 1237 values = astype_nansafe(values, dtype, copy=copy)
1239 # in pandas we don't store numpy str dtypes, so convert to object
1240 if isinstance(dtype, np.dtype) and issubclass(values.dtype.type, str):

File /usr/local/lib/python3.8/dist-packages/pandas/core/dtypes/cast.py:1148, in astype_nansafe(arr, dtype, copy, skipna)
1145 raise TypeError(f"cannot astype a timedelta from [{arr.dtype}] to [{dtype}]")
1147 elif np.issubdtype(arr.dtype, np.floating) and np.issubdtype(dtype, np.integer):
-> 1148 return astype_float_to_int_nansafe(arr, dtype, copy)
1150 elif is_object_dtype(arr.dtype):
1151
1152 # work around NumPy brokenness, #1987
1153 if np.issubdtype(dtype.type, np.integer):

File /usr/local/lib/python3.8/dist-packages/pandas/core/dtypes/cast.py:1193, in astype_float_to_int_nansafe(values, dtype, copy)
1189 """
1190 astype with a check preventing converting NaN to an meaningless integer value.
1191 """
1192 if not np.isfinite(values).all():
-> 1193 raise IntCastingNaNError(
1194 "Cannot convert non-finite values (NA or inf) to integer"
1195 )
1196 return values.astype(dtype, copy=copy)

IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer

[blueteam0ps]

Wonder if I need to build a separate process tree builder for that kind of input?

[ianhelle]

@blueteam0ps - this should be fixed in #513. Due to be published in the next week.

[ianhelle]

Fixed in v2.1.3

[blueteam0ps]

Great Thank you!