Deprecate ResponseMemoryProbs & MultiResponseProbsMessage

examples/abp_nvsmi_detection/README.md

@@ -187,13 +187,13 @@ Added stage: <deserialize-1; DeserializeStage()>
 Added stage: <preprocess-fil-2; PreprocessFILStage()>
   └─ morpheus.MultiMessage -> morpheus.MultiInferenceFILMessage
 Added stage: <inference-3; TritonInferenceStage(model_name=abp-nvsmi-xgb, server_url=localhost:8000, force_convert_inputs=False, use_shared_memory=False)>
-  └─ morpheus.MultiInferenceFILMessage -> morpheus.MultiResponseProbsMessage
+  └─ morpheus.MultiInferenceFILMessage -> morpheus.MultiResponseMessage
 Added stage: <monitor-4; MonitorStage(description=Inference Rate, smoothing=0.001, unit=inf, delayed_start=False, determine_count_fn=None)>
-  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
+  └─ morpheus.MultiResponseMessage -> morpheus.MultiResponseMessage
 Added stage: <add-class-5; AddClassificationsStage(threshold=0.5, labels=[], prefix=)>
-  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
+  └─ morpheus.MultiResponseMessage -> morpheus.MultiResponseMessage
 Added stage: <serialize-6; SerializeStage(include=['mining'], exclude=['^ID$', '^_ts_'], fixed_columns=True)>
-  └─ morpheus.MultiResponseProbsMessage -> morpheus.MessageMeta
+  └─ morpheus.MultiResponseMessage -> morpheus.MessageMeta
 Added stage: <to-file-7; WriteToFileStage(filename=detections.jsonlines, overwrite=True, file_type=FileTypes.Auto)>
   └─ morpheus.MessageMeta -> morpheus.MessageMeta
 ====Building Pipeline Complete!====

examples/digital_fingerprinting/starter/README.md

@@ -135,7 +135,7 @@ The `PreprocessAEStage` is responsible for creating a Morpheus message that cont
 
 **Postprocessing stage** - The DFP pipeline uses the `AddScoresStage` for postprocessing to add anomaly scores and zscores from previous inference stage with matching labels.
 
-**Serialize stage** - `SerializeStage` is used to convert `MultiResponseProbsMessage` from previous stage to a `MessageMeta` to make it suitable for output (i.e. write to file or Kafka).
+**Serialize stage** - `SerializeStage` is used to convert `MultiResponseMessage` from previous stage to a `MessageMeta` to make it suitable for output (i.e. write to file or Kafka).
 
 **Write stage** - `WriteToFileStage` writes input data with inference results to an output file path.
 

examples/nlp_si_detection/README.md

@@ -175,13 +175,13 @@ Added stage: <deserialize-1; DeserializeStage()>
 Added stage: <preprocess-nlp-2; PreprocessNLPStage(vocab_hash_file=/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/data/bert-base-uncased-hash.txt, truncation=True, do_lower_case=True, add_special_tokens=False, stride=-1)>
   └─ morpheus.MultiMessage -> morpheus.MultiInferenceNLPMessage
 Added stage: <inference-3; TritonInferenceStage(model_name=sid-minibert-onnx, server_url=localhost:8000, force_convert_inputs=True, use_shared_memory=False)>
-  └─ morpheus.MultiInferenceNLPMessage -> morpheus.MultiResponseProbsMessage
+  └─ morpheus.MultiInferenceNLPMessage -> morpheus.MultiResponseMessage
 Added stage: <monitor-4; MonitorStage(description=Inference Rate, smoothing=0.001, unit=inf, delayed_start=False, determine_count_fn=None)>
-  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
+  └─ morpheus.MultiResponseMessage -> morpheus.MultiResponseMessage
 Added stage: <add-class-5; AddClassificationsStage(threshold=0.5, labels=[], prefix=)>
-  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
+  └─ morpheus.MultiResponseMessage -> morpheus.MultiResponseMessage
 Added stage: <serialize-6; SerializeStage(include=[], exclude=['^_ts_'], fixed_columns=True)>
-  └─ morpheus.MultiResponseProbsMessage -> morpheus.MessageMeta
+  └─ morpheus.MultiResponseMessage -> morpheus.MessageMeta
 Added stage: <to-file-7; WriteToFileStage(filename=detections.jsonlines, overwrite=True, file_type=FileTypes.Auto)>
   └─ morpheus.MessageMeta -> morpheus.MessageMeta
 ====Building Pipeline Complete!====

examples/root_cause_analysis/README.md

@@ -167,13 +167,13 @@ Added stage: <deserialize-1; DeserializeStage()>
 Added stage: <preprocess-nlp-2; PreprocessNLPStage(vocab_hash_file=/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/data/bert-base-uncased-hash.txt, truncation=True, do_lower_case=True, add_special_tokens=False, stride=-1, column=log)>
   └─ morpheus.MultiMessage -> morpheus.MultiInferenceNLPMessage
 Added stage: <inference-3; TritonInferenceStage(model_name=root-cause-binary-onnx, server_url=localhost:8001, force_convert_inputs=True, use_shared_memory=False)>
-  └─ morpheus.MultiInferenceNLPMessage -> morpheus.MultiResponseProbsMessage
+  └─ morpheus.MultiInferenceNLPMessage -> morpheus.MultiResponseMessage
 Added stage: <monitor-4; MonitorStage(description=Inference rate, smoothing=0.001, unit=inf, delayed_start=False, determine_count_fn=None)>
-  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
+  └─ morpheus.MultiResponseMessage -> morpheus.MultiResponseMessage
 Added stage: <add-scores-5; AddScoresStage(labels=('is_root_cause',), prefix=)>
-  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
+  └─ morpheus.MultiResponseMessage -> morpheus.MultiResponseMessage
 Added stage: <serialize-6; SerializeStage(include=(), exclude=('^ts_',), fixed_columns=True)>
-  └─ morpheus.MultiResponseProbsMessage -> morpheus.MessageMeta
+  └─ morpheus.MultiResponseMessage -> morpheus.MessageMeta
 Added stage: <to-file-7; WriteToFileStage(filename=./root-cause-binary-output.jsonlines, overwrite=True, file_type=FileTypes.Auto, include_index_col=True)>
   └─ morpheus.MessageMeta -> morpheus.MessageMeta
 Inference rate[Complete]: 473 inf [00:01, 340.43 inf/s]

morpheus/_lib/cmake/libraries/morpheus.cmake

@@ -18,28 +18,29 @@ add_library(morpheus
     # Keep these sorted!
     ${MORPHEUS_LIB_ROOT}/src/io/deserializers.cpp
     ${MORPHEUS_LIB_ROOT}/src/io/serializers.cpp
-    ${MORPHEUS_LIB_ROOT}/src/messages/memory/inference_memory.cpp
     ${MORPHEUS_LIB_ROOT}/src/messages/memory/inference_memory_fil.cpp
     ${MORPHEUS_LIB_ROOT}/src/messages/memory/inference_memory_nlp.cpp
-    ${MORPHEUS_LIB_ROOT}/src/messages/memory/response_memory.cpp
+    ${MORPHEUS_LIB_ROOT}/src/messages/memory/inference_memory.cpp
     ${MORPHEUS_LIB_ROOT}/src/messages/memory/response_memory_probs.cpp
+    ${MORPHEUS_LIB_ROOT}/src/messages/memory/response_memory.cpp
     ${MORPHEUS_LIB_ROOT}/src/messages/memory/tensor_memory.cpp
     ${MORPHEUS_LIB_ROOT}/src/messages/meta.cpp
-    ${MORPHEUS_LIB_ROOT}/src/messages/multi.cpp
-    ${MORPHEUS_LIB_ROOT}/src/messages/multi_inference.cpp
     ${MORPHEUS_LIB_ROOT}/src/messages/multi_inference_fil.cpp
     ${MORPHEUS_LIB_ROOT}/src/messages/multi_inference_nlp.cpp
-    ${MORPHEUS_LIB_ROOT}/src/messages/multi_response.cpp
+    ${MORPHEUS_LIB_ROOT}/src/messages/multi_inference.cpp
     ${MORPHEUS_LIB_ROOT}/src/messages/multi_response_probs.cpp
+    ${MORPHEUS_LIB_ROOT}/src/messages/multi_response.cpp
     ${MORPHEUS_LIB_ROOT}/src/messages/multi_tensor.cpp
+    ${MORPHEUS_LIB_ROOT}/src/messages/multi.cpp
     ${MORPHEUS_LIB_ROOT}/src/objects/fiber_queue.cpp
     ${MORPHEUS_LIB_ROOT}/src/objects/file_types.cpp
     ${MORPHEUS_LIB_ROOT}/src/objects/mutable_table_ctx_mgr.cpp
-    ${MORPHEUS_LIB_ROOT}/src/objects/wrapped_tensor.cpp
     ${MORPHEUS_LIB_ROOT}/src/objects/python_data_table.cpp
     ${MORPHEUS_LIB_ROOT}/src/objects/rmm_tensor.cpp
     ${MORPHEUS_LIB_ROOT}/src/objects/tensor.cpp
+    ${MORPHEUS_LIB_ROOT}/src/objects/wrapped_tensor.cpp
     ${MORPHEUS_LIB_ROOT}/src/stages/add_classification.cpp
+    ${MORPHEUS_LIB_ROOT}/src/stages/add_scores_stage_base.cpp
     ${MORPHEUS_LIB_ROOT}/src/stages/add_scores.cpp
     ${MORPHEUS_LIB_ROOT}/src/stages/deserialize.cpp
     ${MORPHEUS_LIB_ROOT}/src/stages/file_source.cpp

morpheus/_lib/include/morpheus/messages/memory/tensor_memory.hpp

@@ -63,7 +63,6 @@ class TensorMemory
     virtual ~TensorMemory() = default;
 
     TensorIndex count{0};
-    TensorMap tensors;
 
     /**
      * @brief Verify whether the specified tensor name is present in the tensor memory
@@ -74,15 +73,6 @@ class TensorMemory
      */
     bool has_tensor(const std::string& name) const;
 
-    /**
-     * @brief Copy tensor ranges
-     *
-     * @param ranges
-     * @param num_selected_rows
-     * @return TensorMap
-     */
-    TensorMap copy_tensor_ranges(const std::vector<RangeType>& ranges, TensorIndex num_selected_rows) const;
-
     /**
      * @brief Get the tensor object identified by `name`
      *
@@ -110,6 +100,13 @@ class TensorMemory
      */
     void set_tensor(const std::string& name, TensorObject&& tensor);
 
+    /**
+     * @brief Get a reference to the internal tensors map
+     *
+     * @return const TensorMap&
+     */
+    const TensorMap& get_tensors() const;
+
     /**
      * @brief Set the tensors object
      *
@@ -118,6 +115,15 @@ class TensorMemory
      */
     void set_tensors(TensorMap&& tensors);
 
+    /**
+     * @brief Copy tensor ranges
+     *
+     * @param ranges
+     * @param num_selected_rows
+     * @return TensorMap
+     */
+    TensorMap copy_tensor_ranges(const std::vector<RangeType>& ranges, TensorIndex num_selected_rows) const;
+
   protected:
     /**
      * @brief Checks if the number of rows in `tensor` matches `count`
@@ -142,6 +148,9 @@ class TensorMemory
      * @throws std::runtime_error If no tensor matching `name` exists
      */
     void verify_tensor_exists(const std::string& name) const;
+
+  private:
+    TensorMap m_tensors;
 };
 
 /****** TensorMemoryInterfaceProxy *************************/
@@ -168,6 +177,14 @@ struct TensorMemoryInterfaceProxy
      */
     static TensorIndex get_count(TensorMemory& self);
 
+    /**
+     * @brief Returns a list of the current tensor names
+     *
+     * @param self
+     * @return std::vector<std::string>
+     */
+    static std::vector<std::string> tensor_names_getter(TensorMemory& self);
+
     /**
      * @brief Returns true if a tensor with the requested name exists in the tensors object
      *

morpheus/_lib/include/morpheus/messages/multi_inference.hpp

@@ -17,7 +17,7 @@
 
 #pragma once
 
-#include "morpheus/messages/memory/inference_memory.hpp"
+#include "morpheus/messages/memory/tensor_memory.hpp"
 #include "morpheus/messages/meta.hpp"
 #include "morpheus/messages/multi.hpp"
 #include "morpheus/messages/multi_tensor.hpp"
@@ -60,13 +60,15 @@ class MultiInferenceMessage : public DerivedMultiMessage<MultiInferenceMessage,
      * @param memory Holds the generic tensor data in cupy arrays that will be used for inference stages
      * @param offset Message offset in inference memory instance
      * @param count Message count in inference memory instance
+     * @param id_tensor_name Name of the tensor that correlates tensor rows to message IDs
      */
     MultiInferenceMessage(std::shared_ptr<MessageMeta> meta,
-                          TensorIndex mess_offset                 = 0,
-                          TensorIndex mess_count                  = -1,
-                          std::shared_ptr<InferenceMemory> memory = nullptr,
-                          TensorIndex offset                      = 0,
-                          TensorIndex count                       = -1);
+                          TensorIndex mess_offset              = 0,
+                          TensorIndex mess_count               = -1,
+                          std::shared_ptr<TensorMemory> memory = nullptr,
+                          TensorIndex offset                   = 0,
+                          TensorIndex count                    = -1,
+                          std::string id_tensor_name           = "seq_ids");
 
     /**
      * @brief Returns the input tensor for the given `name`.
@@ -108,14 +110,16 @@ struct MultiInferenceMessageInterfaceProxy : public MultiTensorMessageInterfaceP
      * @param memory Holds the generic tensor data in cupy arrays that will be used for inference stages
      * @param offset Message offset in inference memory instance
      * @param count Message count in inference memory instance
+     * @param id_tensor_name Name of the tensor that correlates tensor rows to message IDs
      * @return std::shared_ptr<MultiInferenceMessage>
      */
     static std::shared_ptr<MultiInferenceMessage> init(std::shared_ptr<MessageMeta> meta,
                                                        TensorIndex mess_offset,
                                                        TensorIndex mess_count,
-                                                       std::shared_ptr<InferenceMemory> memory,
+                                                       std::shared_ptr<TensorMemory> memory,
                                                        TensorIndex offset,
-                                                       TensorIndex count);
+                                                       TensorIndex count,
+                                                       std::string id_tensor_name);
 };
 #pragma GCC visibility pop
 /** @} */  // end of group

morpheus/_lib/include/morpheus/messages/multi_inference_fil.hpp

@@ -17,8 +17,8 @@
 
 #pragma once
 
-#include "morpheus/messages/memory/inference_memory.hpp"  // for InferenceMemory
-#include "morpheus/messages/meta.hpp"                     // for MessageMeta
+#include "morpheus/messages/memory/tensor_memory.hpp"
+#include "morpheus/messages/meta.hpp"  // for MessageMeta
 #include "morpheus/messages/multi.hpp"
 #include "morpheus/messages/multi_inference.hpp"
 #include "morpheus/objects/tensor_object.hpp"
@@ -27,6 +27,7 @@
 #include <pybind11/pytypes.h>  // for object
 
 #include <memory>
+#include <string>
 
 namespace morpheus {
 /****** Component public implementations *******************/
@@ -57,13 +58,15 @@ class MultiInferenceFILMessage : public DerivedMultiMessage<MultiInferenceFILMes
      * @param memory Holds the generic tensor data in cupy arrays that will be used for inference stages
      * @param offset Message offset in inference memory object
      * @param count Message count in inference memory object
+     * @param id_tensor_name Name of the tensor that correlates tensor rows to message IDs
      */
     MultiInferenceFILMessage(std::shared_ptr<MessageMeta> meta,
-                             TensorIndex mess_offset                           = 0,
-                             TensorIndex mess_count                            = -1,
-                             std::shared_ptr<morpheus::InferenceMemory> memory = nullptr,
-                             TensorIndex offset                                = 0,
-                             TensorIndex count                                 = -1);
+                             TensorIndex mess_offset              = 0,
+                             TensorIndex mess_count               = -1,
+                             std::shared_ptr<TensorMemory> memory = nullptr,
+                             TensorIndex offset                   = 0,
+                             TensorIndex count                    = -1,
+                             std::string id_tensor_name           = "seq_ids");
 
     /**
      * @brief Returns the 'input__0' tensor, throws a `std::runtime_error` if it does not exist
@@ -114,14 +117,16 @@ struct MultiInferenceFILMessageInterfaceProxy : public MultiInferenceMessageInte
      * @param memory Holds the generic tensor data in cupy arrays that will be used for inference stages
      * @param offset Message offset in inference memory object
      * @param count Message count in inference memory object
+     * @param id_tensor_name Name of the tensor that correlates tensor rows to message IDs
      * @return std::shared_ptr<MultiInferenceFILMessage>
      */
     static std::shared_ptr<MultiInferenceFILMessage> init(std::shared_ptr<MessageMeta> meta,
                                                           TensorIndex mess_offset,
                                                           TensorIndex mess_count,
-                                                          std::shared_ptr<InferenceMemory> memory,
+                                                          std::shared_ptr<TensorMemory> memory,
                                                           TensorIndex offset,
-                                                          TensorIndex count);
+                                                          TensorIndex count,
+                                                          std::string id_tensor_name);
 
     /**
      * @brief Get  'input__0' tensor as a python object

morpheus/_lib/include/morpheus/messages/multi_inference_nlp.hpp

@@ -17,8 +17,8 @@
 
 #pragma once
 
-#include "morpheus/messages/memory/inference_memory.hpp"  // for InferenceMemory
-#include "morpheus/messages/meta.hpp"                     // for MessageMeta
+#include "morpheus/messages/memory/tensor_memory.hpp"
+#include "morpheus/messages/meta.hpp"  // for MessageMeta
 #include "morpheus/messages/multi.hpp"
 #include "morpheus/messages/multi_inference.hpp"
 #include "morpheus/objects/tensor_object.hpp"
@@ -27,6 +27,7 @@
 #include <pybind11/pytypes.h>  // for object
 
 #include <memory>
+#include <string>
 
 namespace morpheus {
 /****** Component public implementations *******************/
@@ -57,13 +58,15 @@ class MultiInferenceNLPMessage : public DerivedMultiMessage<MultiInferenceNLPMes
      * @param memory Holds the generic tensor data in cupy arrays that will be used for inference stages
      * @param offset Message offset in inference memory object
      * @param count Message count in inference memory object
+     * @param id_tensor_name Name of the tensor that correlates tensor rows to message IDs
      */
     MultiInferenceNLPMessage(std::shared_ptr<MessageMeta> meta,
-                             TensorIndex mess_offset                 = 0,
-                             TensorIndex mess_count                  = -1,
-                             std::shared_ptr<InferenceMemory> memory = nullptr,
-                             TensorIndex offset                      = 0,
-                             TensorIndex count                       = -1);
+                             TensorIndex mess_offset              = 0,
+                             TensorIndex mess_count               = -1,
+                             std::shared_ptr<TensorMemory> memory = nullptr,
+                             TensorIndex offset                   = 0,
+                             TensorIndex count                    = -1,
+                             std::string id_tensor_name           = "seq_ids");
 
     /**
      * @brief Returns the 'input_ids' tensor, throws a `std::runtime_error` if it does not exist.
@@ -130,14 +133,16 @@ struct MultiInferenceNLPMessageInterfaceProxy : public MultiInferenceMessageInte
      * @param memory Holds the generic tensor data in cupy arrays that will be used for inference stages
      * @param offset Message offset in inference memory object
      * @param count Message count in inference memory object
+     * @param id_tensor_name Name of the tensor that correlates tensor rows to message IDs
      * @return std::shared_ptr<MultiInferenceNLPMessage>
      */
     static std::shared_ptr<MultiInferenceNLPMessage> init(std::shared_ptr<MessageMeta> meta,
                                                           TensorIndex mess_offset,
                                                           TensorIndex mess_count,
-                                                          std::shared_ptr<InferenceMemory> memory,
+                                                          std::shared_ptr<TensorMemory> memory,
                                                           TensorIndex offset,
-                                                          TensorIndex count);
+                                                          TensorIndex count,
+                                                          std::string id_tensor_name);
 
     /**
      * @brief Get  'input_ids' tensor as a python object