init version of online serving and rolling

docs/advanced/serial.rst

@@ -14,6 +14,9 @@ Serializable Class
 
 ``Qlib`` provides a base class ``qlib.utils.serial.Serializable``, whose state can be dumped into or loaded from disk in `pickle` format. 
 When users dump the state of a ``Serializable`` instance, the attributes of the instance whose name **does not** start with `_` will be saved on the disk.
+However, users can use ``config`` method or override ``default_dump_all`` attribute to prevent this feature.
+
+Users can also override ``pickle_backend`` attribute to choose a pickle backend. The supported value is "pickle" (default and common) and "dill" (dump more things such as function, more information in `here <https://pypi.org/project/dill/>`_).
 
 Example
 ==========================

docs/advanced/task_management.rst

@@ -19,7 +19,7 @@ An example of the entire process is shown `here <https://github.com/microsoft/ql
 
 Task Generating
 ===============
-A ``task`` consists of `Model`, `Dataset`, `Record` or anything added by users. 
+A ``task`` consists of `Model`, `Dataset`, `Record`, or anything added by users. 
 The specific task template can be viewed in 
 `Task Section <../component/workflow.html#task-section>`_.
 Even though the task template is fixed, users can customize their ``TaskGen`` to generate different ``task`` by task template.
@@ -30,15 +30,15 @@ Here is the base class of ``TaskGen``:
     :members:
 
 ``Qlib`` provides a class `RollingGen <https://github.com/microsoft/qlib/tree/main/qlib/workflow/task/gen.py>`_ to generate a list of ``task`` of the dataset in different date segments.
-This class allows users to verify the effect of data from different periods on the model in one experiment. More information in `here <../reference/api.html#TaskGen>`_.
+This class allows users to verify the effect of data from different periods on the model in one experiment. More information is `here <../reference/api.html#TaskGen>`_.
 
 Task Storing
 ===============
 To achieve higher efficiency and the possibility of cluster operation, ``Task Manager`` will store all tasks in `MongoDB <https://www.mongodb.com/>`_.
 ``TaskManager`` can fetch undone tasks automatically and manage the lifecycle of a set of tasks with error handling.
-Users **MUST** finished the configuration of `MongoDB <https://www.mongodb.com/>`_ when using this module.
+Users **MUST** finish the configuration of `MongoDB <https://www.mongodb.com/>`_ when using this module.
 
-Users need to provide the MongoDB URL and database name for using ``TaskManager`` in `initialization <../start/initialization.html#Parameters>`_ or make statement like this.
+Users need to provide the MongoDB URL and database name for using ``TaskManager`` in `initialization <../start/initialization.html#Parameters>`_ or make a statement like this.
 
     .. code-block:: python
 
@@ -55,32 +55,35 @@ More information of ``Task Manager`` can be found in `here <../reference/api.htm
 
 Task Training
 ===============
-#FIXME: Trainer
-After generating and storing those ``task``, it's time to run the ``task`` which are in the *WAITING* status.
+After generating and storing those ``task``, it's time to run the ``task`` which is in the *WAITING* status.
 ``Qlib`` provides a method called ``run_task`` to run those ``task`` in task pool, however, users can also customize how tasks are executed.
 An easy way to get the ``task_func`` is using ``qlib.model.trainer.task_train`` directly.
 It will run the whole workflow defined by ``task``, which includes *Model*, *Dataset*, *Record*.
 
 .. autofunction:: qlib.workflow.task.manage.run_task
 
 Meanwhile, ``Qlib`` provides a module called ``Trainer``. 
-``Trainer`` will train a list of tasks and return a list of model recorder.
-``Qlib`` offer two kind of Trainer, TrainerR is the simplest way and TrainerRM is based on TaskManager to help manager tasks lifecycle automatically. 
+
+.. autoclass:: qlib.model.trainer.Trainer
+    :members:
+
+``Trainer`` will train a list of tasks and return a list of model recorders.
+``Qlib`` offer two kinds of Trainer, TrainerR is the simplest way and TrainerRM is based on TaskManager to help manager tasks lifecycle automatically. 
 If you do not want to use ``Task Manager`` to manage tasks, then use TrainerR to train a list of tasks generated by ``TaskGen`` is enough.
-More information is in `here <../reference/api.html#Trainer>`_.
+`Here <../reference/api.html#Trainer>`_ are the details about different ``Trainer``.
 
 Task Collecting
 ===============
 To collect the results of ``task`` after training, ``Qlib`` provides `Collector <../reference/api.html#Collector>`_, `Group <../reference/api.html#Group>`_ and `Ensemble <../reference/api.html#Ensemble>`_ to collect the results in a readable, expandable and loosely-coupled way.
 
-`Collector <../reference/api.html#Collector>`_ can collect object from everywhere and process them such as merging, grouping, averaging and so on. It has 2 step action including ``collect`` (collect anything in a dict) and ``process_collect`` (process collected dict).
+`Collector <../reference/api.html#Collector>`_ can collect objects from everywhere and process them such as merging, grouping, averaging and so on. It has 2 step action including ``collect`` (collect anything in a dict) and ``process_collect`` (process collected dict).
 
 `Group <../reference/api.html#Group>`_ also has 2 steps including ``group`` (can group a set of object based on `group_func` and change them to a dict) and ``reduce`` (can make a dict become an ensemble based on some rule).
 For example: {(A,B,C1): object, (A,B,C2): object} ---``group``---> {(A,B): {C1: object, C2: object}} ---``reduce``---> {(A,B): object}
 
 `Ensemble <../reference/api.html#Ensemble>`_ can merge the objects in an ensemble. 
 For example: {C1: object, C2: object} ---``Ensemble``---> object
 
-So the hierarchy is ``Collector``'s second step correspond to ``Group``. And ``Group``'s second step correspond to ``Ensemble``.
+So the hierarchy is ``Collector``'s second step corresponds to ``Group``. And ``Group``'s second step correspond to ``Ensemble``.
 
-For more information, please see `Collector <../reference/api.html#Collector>`_, `Group <../reference/api.html#Group>`_ and `Ensemble <../reference/api.html#Ensemble>`_, or the `example <https://github.com/microsoft/qlib/tree/main/examples/model_rolling/task_manager_rolling.py>`_
+For more information, please see `Collector <../reference/api.html#Collector>`_, `Group <../reference/api.html#Group>`_ and `Ensemble <../reference/api.html#Ensemble>`_, or the `example <https://github.com/microsoft/qlib/tree/main/examples/model_rolling/task_manager_rolling.py>`_.

docs/component/online.rst

@@ -9,12 +9,12 @@ Online Serving
 Introduction
 =============
 In addition to backtesting, one way to test a model is effective is to make predictions in real market conditions or even do real trading based on those predictions.
-``Online Serving`` is a set of module for online models using latest data,
+``Online Serving`` is a set of modules for online models using the latest data,
 which including `Online Manager <#Online Manager>`_, `Online Strategy <#Online Strategy>`_, `Online Tool <#Online Tool>`_, `Updater <#Updater>`_. 
 
 `Here <https://github.com/microsoft/qlib/tree/main/examples/online_srv>`_ are several examples for reference, which demonstrate different features of ``Online Serving``.
-If you have many models or `task` need to be managed, please consider `Task Management <../advanced/task_management.html>`_.
-The `examples <https://github.com/microsoft/qlib/tree/main/examples/online_srv>`_ maybe based on `Task Management <../advanced/task_management.html>`_ such as ``TrainerRM`` or ``Collector``.
+If you have many models or `task` needs to be managed, please consider `Task Management <../advanced/task_management.html>`_.
+The `examples <https://github.com/microsoft/qlib/tree/main/examples/online_srv>`_ are based on some components in `Task Management <../advanced/task_management.html>`_ such as ``TrainerRM`` or ``Collector``.
 
 Online Manager
 =============

docs/reference/api.rst

@@ -226,4 +226,7 @@ Serializable
 --------------------
 
 .. automodule:: qlib.utils.serial.Serializable
-    :members:
+    :members:
+
+
+

examples/model_rolling/task_manager_rolling.py

@@ -2,8 +2,8 @@
 # Licensed under the MIT License.
 
 """
-This example shows how a TrainerRM work based on TaskManager with rolling tasks.
-After training, how to collect the rolling results will be showed in task_collecting.
+This example shows how a TrainerRM works based on TaskManager with rolling tasks.
+After training, how to collect the rolling results will be shown in task_collecting.
 """
 
 from pprint import pprint

examples/online_srv/online_management_simulate.py

@@ -2,15 +2,15 @@
 # Licensed under the MIT License.
 
 """
-This examples is about how can simulate the OnlineManager based on rolling tasks. 
+This example is about how can simulate the OnlineManager based on rolling tasks. 
 """
 
 import fire
 import qlib
-from qlib.model.trainer import DelayTrainerRM
+from qlib.model.trainer import DelayTrainerR, DelayTrainerRM, TrainerR, TrainerRM
 from qlib.workflow import R
 from qlib.workflow.online.manager import OnlineManager
-from qlib.workflow.online.strategy import RollingAverageStrategy
+from qlib.workflow.online.strategy import RollingStrategy
 from qlib.workflow.task.gen import RollingGen
 from qlib.workflow.task.manage import TaskManager
 
@@ -112,10 +112,10 @@ def __init__(
         qlib.init(provider_uri=provider_uri, region=region, mongo=mongo_conf)
         self.rolling_gen = RollingGen(
             step=rolling_step, rtype=RollingGen.ROLL_SD, ds_extra_mod_func=None
-        )  # The rolling tasks generator, ds_extra_mod_func is None because we just need simulate to 2018-10-31 and needn't change handler end time.
-        self.trainer = DelayTrainerRM(self.exp_name, self.task_pool)
+        )  # The rolling tasks generator, ds_extra_mod_func is None because we just need to simulate to 2018-10-31 and needn't change the handler end time.
+        self.trainer = DelayTrainerRM(self.exp_name, self.task_pool)  # Also can be TrainerR, TrainerRM, DelayTrainerR
         self.rolling_online_manager = OnlineManager(
-            RollingAverageStrategy(exp_name, task_template=tasks, rolling_gen=self.rolling_gen),
+            RollingStrategy(exp_name, task_template=tasks, rolling_gen=self.rolling_gen),
             trainer=self.trainer,
             begin_time=self.start_time,
         )
@@ -138,8 +138,6 @@ def main(self):
         print(self.rolling_online_manager.get_collector()())
         print("========== signals ==========")
         print(self.rolling_online_manager.get_signals())
-        print("========== online history ==========")
-        print(self.rolling_online_manager.history)
 
 
 if __name__ == "__main__":

examples/online_srv/rolling_online_management.py

@@ -2,21 +2,20 @@
 # Licensed under the MIT License.
 
 """
-This example show how OnlineManager works with rolling tasks.
-There are two parts including first train and routine.
+This example shows how OnlineManager works with rolling tasks.
+There are four parts including first train, routine 1, add strategy and routine 2.
 Firstly, the OnlineManager will finish the first training and set trained models to `online` models.
-Next, the OnlineManager will finish a routine process, including update online prediction -> prepare signals -> prepare tasks -> prepare new models -> reset online models
+Next, the OnlineManager will finish a routine process, including update online prediction -> prepare tasks -> prepare new models -> prepare signals
+Then, we will add some new strategies to the OnlineManager. This will finish first training of new strategies.
+Finally, the OnlineManager will finish second routine and update all strategies.
 """
 
 import os
-from pathlib import Path
-import pickle
 import fire
 import qlib
 from qlib.workflow import R
-from qlib.workflow.online.strategy import RollingAverageStrategy
+from qlib.workflow.online.strategy import RollingStrategy
 from qlib.workflow.task.gen import RollingGen
-from qlib.workflow.task.manage import TaskManager
 from qlib.workflow.online.manager import OnlineManager
 
 data_handler_config = {
@@ -84,77 +83,98 @@ def __init__(
         task_url="mongodb://10.0.0.4:27017/",
         task_db_name="rolling_db",
         rolling_step=550,
-        tasks=[task_xgboost_config, task_lgb_config],
+        tasks=[task_xgboost_config],
+        add_tasks=[task_lgb_config],
     ):
         mongo_conf = {
             "task_url": task_url,  # your MongoDB url
             "task_db_name": task_db_name,  # database name
         }
         qlib.init(provider_uri=provider_uri, region=region, mongo=mongo_conf)
         self.tasks = tasks
+        self.add_tasks = add_tasks
         self.rolling_step = rolling_step
-        strategy = []
+        strategies = []
         for task in tasks:
             name_id = task["model"]["class"]  # NOTE: Assumption: The model class can specify only one strategy
-            strategy.append(
-                RollingAverageStrategy(
+            strategies.append(
+                RollingStrategy(
                     name_id,
                     task,
                     RollingGen(step=rolling_step, rtype=RollingGen.ROLL_SD),
                 )
             )
 
-        self.rolling_online_manager = OnlineManager(strategy)
-        self.collector = self.rolling_online_manager.get_collector()
+        self.rolling_online_manager = OnlineManager(strategies)
 
     _ROLLING_MANAGER_PATH = (
         ".RollingOnlineExample"  # the OnlineManager will dump to this file, for it can be loaded when calling routine.
     )
 
     # Reset all things to the first status, be careful to save important data
     def reset(self):
-        for task in self.tasks:
+        for task in self.tasks + self.add_tasks:
             name_id = task["model"]["class"]
-            TaskManager(name_id).remove()
             exp = R.get_exp(experiment_name=name_id)
             for rid in exp.list_recorders():
                 exp.delete_recorder(rid)
 
-            if os.path.exists(self._ROLLING_MANAGER_PATH):
-                os.remove(self._ROLLING_MANAGER_PATH)
+        if os.path.exists(self._ROLLING_MANAGER_PATH):
+            os.remove(self._ROLLING_MANAGER_PATH)
 
     def first_run(self):
         print("========== reset ==========")
         self.reset()
         print("========== first_run ==========")
         self.rolling_online_manager.first_train()
+        print("========== collect results ==========")
+        print(self.rolling_online_manager.get_collector()())
         print("========== dump ==========")
         self.rolling_online_manager.to_pickle(self._ROLLING_MANAGER_PATH)
-        print("========== collect results ==========")
-        print(self.collector())
 
     def routine(self):
         print("========== load ==========")
-        with Path(self._ROLLING_MANAGER_PATH).open("rb") as f:
-            self.rolling_online_manager = pickle.load(f)
+        self.rolling_online_manager = OnlineManager.load(self._ROLLING_MANAGER_PATH)
         print("========== routine ==========")
         self.rolling_online_manager.routine()
         print("========== collect results ==========")
-        print(self.collector())
+        print(self.rolling_online_manager.get_collector()())
         print("========== signals ==========")
         print(self.rolling_online_manager.get_signals())
+        print("========== dump ==========")
+        self.rolling_online_manager.to_pickle(self._ROLLING_MANAGER_PATH)
+
+    def add_strategy(self):
+        print("========== load ==========")
+        self.rolling_online_manager = OnlineManager.load(self._ROLLING_MANAGER_PATH)
+        print("========== add strategy ==========")
+        strategies = []
+        for task in self.add_tasks:
+            name_id = task["model"]["class"]  # NOTE: Assumption: The model class can specify only one strategy
+            strategies.append(
+                RollingStrategy(
+                    name_id,
+                    task,
+                    RollingGen(step=self.rolling_step, rtype=RollingGen.ROLL_SD),
+                )
+            )
+        self.rolling_online_manager.add_strategy(strategies=strategies)
+        print("========== dump ==========")
+        self.rolling_online_manager.to_pickle(self._ROLLING_MANAGER_PATH)
 
     def main(self):
         self.first_run()
         self.routine()
+        self.add_strategy()
+        self.routine()
 
 
 if __name__ == "__main__":
     ####### to train the first version's models, use the command below
     # python rolling_online_management.py first_run
 
     ####### to update the models and predictions after the trading time, use the command below
-    # python rolling_online_management.py after_day
+    # python rolling_online_management.py routine
 
     ####### to define your own parameters, use `--`
     # python rolling_online_management.py first_run --exp_name='your_exp_name' --rolling_step=40

examples/online_srv/update_online_pred.py

@@ -2,10 +2,10 @@
 # Licensed under the MIT License.
 
 """
-This example show how OnlineTool works when we need update prediction.
+This example shows how OnlineTool works when we need update prediction.
 There are two parts including first_train and update_online_pred.
-Firstly, we will finish the training and set the trained model to `online` model.
-Next, we will finish updating online prediction.
+Firstly, we will finish the training and set the trained models to the `online` models.
+Next, we will finish updating online predictions.
 """
 import fire
 import qlib

qlib/data/dataset/__init__.py

@@ -172,7 +172,10 @@ def _prepare_seg(self, slc: slice, **kwargs):
         ----------
         slc : slice
         """
-        return self.handler.fetch(slc, **kwargs, **self.fetch_kwargs)
+        if hasattr(self, "fetch_kwargs"):
+            return self.handler.fetch(slc, **kwargs, **self.fetch_kwargs)
+        else:
+            return self.handler.fetch(slc, **kwargs)
 
     def prepare(
         self,