* v0.2.2

* Added predict method to trainer
* Added State.PREDICT enum
* Added Phase.PREDICT_BEGIN and Phase.PREDICT_END enums
* Added CollectOutputs callback
* Added predict example
* Moving some elements from nn.functional to nn
* Added sample count to save/load model

CHANGELOG.txt

@@ -1,6 +1,18 @@
 Change Log
 ==========
 
+
+0.2.2 (18/10/2020)
+-----------------
+* Added predict method to trainer
+* Added State.PREDICT enum 
+* Added Phase.PREDICT_BEGIN and Phase.PREDICT_END enums
+* Added CollectOutputs callback
+* Added predict example
+* Moving some elements from nn.functional to nn
+* Added sample count to save/load model
+
+
 0.2.1 (15/10/2020)
 -----------------
 * Fix in Tensorboard metrics output

README.md

@@ -29,17 +29,18 @@ A Fast, Flexible Trainer with Callbacks and Extensions for PyTorch
     from lpd.trainer import Trainer
     from lpd.enums import Phase, State, MonitorType, MonitorMode, StatsType
     from lpd.callbacks import StatsPrint, ModelCheckPoint, Tensorboard, EarlyStopping, SchedulerStep
+    from lpd.extensions.custom_schedulers import KerasDecay
     from lpd.extensions.custom_metrics import binary_accuracy_with_logits
     from lpd.utils.torch_utils import get_gpu_device_if_available
     from lpd.utils.general_utils import seed_all
 
-    seed_all(seed=42)
+    seed_all(seed=42) # because its the answer to life and the universe
 
     device = get_gpu_device_if_available() # with fallback to CPU if GPU not avilable
-    model = TestModel(config, num_embeddings).to(device) #this is your model class, and its being sent to the relevant device
+    model = TestModel(config, num_embeddings).to(device) # this is your model class, and its being sent to the relevant device
     optimizer = optim.SGD(params=model.parameters())
-    scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', patience=5, verbose=True)
-    loss_func = nn.BCEWithLogitsLoss().to(device) #this is your loss class, already sent to the relevant device
+    scheduler = KerasDecay(optimizer, decay=0.01, last_step=-1) # decay scheduler using keras formula 
+    loss_func = nn.BCEWithLogitsLoss().to(device) # this is your loss class, already sent to the relevant device
     metric_name_to_func = {'acc':binary_accuracy_with_logits} # add as much metrics as you like
 
     # you can use some of the defined callbacks, or you can create your own
@@ -73,6 +74,11 @@ A Fast, Flexible Trainer with Callbacks and Extensions for PyTorch
     trainer.evaluate(test_data_loader, test_steps)
 ```
 
+### Making predictions
+```python
+    predictions = trainer.predict(data_loader, steps)
+```
+
 ## TrainerStats
 ``Trainer`` tracks stats for `train/validate/test` and you can access them in your custom callbacks
 or any other place that has access to your trainer.
@@ -131,25 +137,29 @@ Evaluation phases and states will behave as follow
         State.EXTERNAL
         Phase.TEST_END
 ```
-With phases and states, you have full control over the timing of your callbacks,
 
-### SchedulerStep Callback
-
-Will invoke ``step()`` on your scheduler.
 
-For example, SchedulerStep callback to control your scheduler,
-but only at the end of every batch, and only when in train state (as opposed to validation and test)
-then define your SchedulerStep callback like so:
+Predict phases and states will behave as follow
 ```python
-    from lpd.callbacks import SchedulerStep
-    from lpd.enums import Phase, State
-    SchedulerStep(apply_on_phase=Phase.BATCH_END, apply_on_states=State.TRAIN)
-```
-In case you need it on validation state as well, pass a list for ``apply_on_states`` like so:
-```python
-    SchedulerStep(apply_on_phase=Phase.BATCH_END, apply_on_states=[State.TRAIN, State.VAL])
+        State.EXTERNAL
+        Phase.PREDICT_BEGIN
+        State.PREDICT
+        # batches loop:
+            Phase.BATCH_BEGIN
+            # batch
+            Phase.BATCH_END
+        State.EXTERNAL
+        Phase.PREDICT_END
 ```
 
+With phases and states, you have full control over the timing of your callbacks,
+
+### StatsPrint Callback
+Below is an output example for ``StatsPrint`` callback that will print an epoch summary at the end of every epoch
+
+![EpochSummary](https://raw.githubusercontent.com/RoySadaka/ReposMedia/main/lpd/images/epoch_summary.png)
+
+
 ### ModelCheckPoint Callback
 Saving a checkpoint when a monitored loss/metric has improved.
 The callback will save the model, optimizer, scheduler, and epoch number.
@@ -182,6 +192,23 @@ epochs, and stop the trainer in that case
                   monitor_mode=MonitorMode.MIN)
 ```
 
+### SchedulerStep Callback
+
+Will invoke ``step()`` on your scheduler.
+
+For example, SchedulerStep callback to control your scheduler,
+but only at the end of every batch, and only when in train state (as opposed to validation and test)
+then define your SchedulerStep callback like so:
+```python
+    from lpd.callbacks import SchedulerStep
+    from lpd.enums import Phase, State
+    SchedulerStep(apply_on_phase=Phase.BATCH_END, apply_on_states=State.TRAIN)
+```
+In case you need it on validation state as well, pass a list for ``apply_on_states`` like so:
+```python
+    SchedulerStep(apply_on_phase=Phase.BATCH_END, apply_on_states=[State.TRAIN, State.VAL])
+```
+
 
 ### Tensorboard Callback
 Will export the loss and the metrics at a given phase and state, in a format that can be viewed on Tensorboard 
@@ -192,13 +219,15 @@ Will export the loss and the metrics at a given phase and state, in a format tha
 ```
 
 
+### CollectOutputs Callback
+In case you want to collect the outputs of any given state during training
+```python
+    CollectOutputs(apply_on_phase=Phase.BATCH_END, apply_on_states=State.VAL)
+```
+CollectOutputs is automatically used in ``trainer.predict(...)`` to collect the predictions
 
-### StatsPrint Callback
-Below is an output example for ``StatsPrint`` callback that will print an epoch summary at the end of every epoch
-
-![EpochSummary](https://raw.githubusercontent.com/RoySadaka/ReposMedia/main/lpd/images/epoch_summary.png)
 
-You can also create custom callbacks
+### Create your custom callbacks
 
 ```python
     from lpd.enums import Phase, State

examples/basic/train.py

@@ -17,7 +17,7 @@ def get_parameters():
     # N is batch size; D_in is input dimension;
     # H is hidden dimension; D_out is output dimension.
     N, D_in, H, D_out = 64, 1000, 100, 10
-    num_epochs = 50
+    num_epochs = 10
     data_loader = eu.examples_data_generator(N, D_in, D_out)
     data_loader_steps = 100
     return N, D_in, H, D_out, num_epochs, data_loader, data_loader_steps

examples/data_loader/train.py

@@ -6,7 +6,6 @@
 import torch as T
 import torch.nn as nn
 import torch.optim as optim
-import torch.nn.functional as F
 from torch.utils.data import Dataset, DataLoader
 
 from lpd.trainer import Trainer
@@ -88,7 +87,7 @@ def __init__(self, D_in, H, D_out, num_embeddings, embedding_dim):
                                             embedding_dim=embedding_dim)
         # nn.init.uniform_(self.embedding_layer.weight, a=-0.05, b=0.05) # I PREFER THE INIT THAT TensorFlow DO FOR Embedding
 
-        self.dense = Dense(embedding_dim, H, use_bias=True, activation=F.relu)
+        self.dense = Dense(embedding_dim, H, use_bias=True, activation=nn.ReLU())
         self.dense_out = Dense(H, D_out, use_bias=True, activation=None)
 
     def forward(self, x):               # (batch, D_in)

examples/multiple_inputs/model.py

@@ -1,6 +1,5 @@
 import torch as T
 import torch.nn as nn
-import torch.nn.functional as F
 import torch.optim as optim
 
 from lpd.trainer import Trainer

examples/scheduler_step_on_batch_end/train.py

@@ -13,7 +13,7 @@ def get_parameters():
     # N is batch size; D_in is input dimension;
     # H is hidden dimension; D_out is output dimension.
     N, D_in, H, D_out = 64, 1000, 100, 10
-    num_epochs = 50
+    num_epochs = 10
     data_loader = eu.examples_data_generator(N, D_in, D_out)
     data_loader_steps = 100
     return N, D_in, H, D_out, num_epochs, data_loader, data_loader_steps
@@ -40,7 +40,7 @@ def get_trainer(N, D_in, H, D_out, num_epochs, data_loader, data_loader_steps):
     # NOTICE!!! WE USE verbose=1 TO SEE THE PRINTS FOR THIS EXAMPLE, BUT YOU MIGHT PREFER TO USE verbose=0 or verbose=2
     # BECAUSE ON BATCH LEVEL IT WILL PRINT A LOT 
     callbacks = [   
-                    SchedulerStep(apply_on_phase=Phase.BATCH_END, apply_on_states=State.TRAIN, verbose=1), #CAN ALSO BE apply_on_states=[State.TRAIN]
+                    SchedulerStep(apply_on_phase=Phase.BATCH_END, apply_on_states=State.TRAIN, verbose=1), #CAN ALSO BE IN FORM OF ARRAY - apply_on_states=[State.TRAIN]
                     StatsPrint(apply_on_phase=Phase.EPOCH_END)
                 ]
 

examples/train_evaluate_predict/train.py

@@ -0,0 +1,67 @@
+import torch as T
+import torch.nn as nn
+import torch.optim as optim
+from lpd.trainer import Trainer
+from lpd.callbacks import StatsPrint
+from lpd.extensions.custom_schedulers import DoNothingToLR
+import lpd.utils.torch_utils as tu
+import lpd.utils.general_utils as gu
+import examples.utils as eu
+
+def get_parameters():
+    # N is batch size; D_in is input dimension;
+    # H is hidden dimension; D_out is output dimension.
+    N, D_in, H, D_out = 8, 1000, 100, 10
+    num_epochs = 5
+    data_loader = eu.examples_data_generator(N, D_in, D_out)
+    data_loader_steps = 100
+    return N, D_in, H, D_out, num_epochs, data_loader, data_loader_steps
+
+def get_trainer(N, D_in, H, D_out, num_epochs, data_loader, data_loader_steps):
+
+    device = tu.get_gpu_device_if_available()
+
+    model = eu.get_basic_model(D_in, H, D_out).to(device)
+
+    loss_func = nn.MSELoss(reduction='sum').to(device)
+
+    optimizer = optim.Adam(model.parameters(), lr=1e-4)
+
+    scheduler = DoNothingToLR() #CAN ALSO USE scheduler=None, BUT DoNothingToLR IS MORE EXPLICIT
+
+    metric_name_to_func = None # THIS EXAMPLE DOES NOT USE METRICS, ONLY LOSS
+
+    callbacks = [   
+                    StatsPrint()
+                ]
+
+    trainer = Trainer(model=model, 
+                      device=device, 
+                      loss_func=loss_func, 
+                      optimizer=optimizer,
+                      scheduler=scheduler,
+                      metric_name_to_func=metric_name_to_func, 
+                      train_data_loader=data_loader, 
+                      val_data_loader=data_loader,
+                      train_steps=data_loader_steps,
+                      val_steps=data_loader_steps,
+                      num_epochs=num_epochs,
+                      callbacks=callbacks,
+                      name='Train-Evaluate-Predict-Example')
+    return trainer
+
+def run():
+    gu.seed_all(42)  # BECAUSE ITS THE ANSWER TO LIFE AND THE UNIVERSE
+
+    N, D_in, H, D_out, num_epochs, data_loader, data_loader_steps = get_parameters()
+
+    trainer = get_trainer(N, D_in, H, D_out, num_epochs, data_loader, data_loader_steps)
+
+    trainer.summary()
+
+    trainer.train()
+
+    trainer.evaluate(data_loader, data_loader_steps)
+
+    data_generator_for_predictions = eu.examples_prediction_data_generator(data_loader, data_loader_steps)
+    predictions = trainer.predict(data_generator_for_predictions, data_loader_steps)

examples/utils.py

@@ -1,6 +1,5 @@
 import torch as T
 import torch.nn as nn
-import torch.nn.functional as F
 from lpd.extensions.custom_layers import Dense
 
 def examples_data_generator(N, D_in, D_out, binary_out=False):
@@ -13,9 +12,16 @@ def examples_data_generator(N, D_in, D_out, binary_out=False):
     while True:
         yield x, y #YIELD THE SAME X,y every time
 
+def examples_prediction_data_generator(origin_data_loader, steps):
+    for x,y in origin_data_loader:
+        steps -= 1
+        yield x
+        if steps == 0:
+            break
+
 
 def get_basic_model(D_in, H, D_out):
     return nn.Sequential(
-                            Dense(D_in, H, use_bias=True, activation=F.relu),
+                            Dense(D_in, H, use_bias=True, activation=nn.ReLU()),
                             Dense(H, D_out, use_bias=True, activation=None)
                         )

lpd/callbacks/__init__.py

@@ -1,9 +1,11 @@
+from lpd.callbacks.callback_base import CallbackBase
+from lpd.callbacks.callback_monitor import CallbackMonitor, CallbackMonitorResult
 from lpd.callbacks.stats_print import StatsPrint
 from lpd.callbacks.model_checkpoint import ModelCheckPoint
 from lpd.callbacks.tensorboard import Tensorboard
 from lpd.callbacks.early_stopping import EarlyStopping
 from lpd.callbacks.scheduler_step import SchedulerStep
 from lpd.callbacks.callback_context import CallbackContext
-from lpd.callbacks.callback_monitor import CallbackMonitor, CallbackMonitorResult
+from lpd.callbacks.collect_outputs import CollectOutputs
 
 

lpd/callbacks/callback_base.py

@@ -82,10 +82,12 @@ def _validations(self):
                         Phase.TRAIN_END:{None, State.EXTERNAL}, 
                         Phase.EPOCH_BEGIN:{None, State.EXTERNAL}, 
                         Phase.EPOCH_END:{None, State.EXTERNAL}, 
-                        Phase.BATCH_BEGIN:{None, State.TRAIN, State.VAL, State.TEST}, 
-                        Phase.BATCH_END:{None, State.TRAIN, State.VAL, State.TEST}, 
+                        Phase.BATCH_BEGIN:{None, State.TRAIN, State.VAL, State.TEST, State.PREDICT}, 
+                        Phase.BATCH_END:{None, State.TRAIN, State.VAL, State.TEST, State.PREDICT}, 
                         Phase.TEST_BEGIN:{None, State.EXTERNAL}, 
                         Phase.TEST_END:{None, State.EXTERNAL}, 
+                        Phase.PREDICT_BEGIN:{None, State.EXTERNAL}, 
+                        Phase.PREDICT_END:{None, State.EXTERNAL}, 
                         }
 
         if self.apply_on_states is not None:

lpd/callbacks/collect_outputs.py

@@ -0,0 +1,39 @@
+from lpd.enums import Phase, State, MonitorType, MonitorMode, StatsType
+from lpd.callbacks.callback_base import CallbackBase
+from lpd.callbacks.callback_context import CallbackContext
+from lpd.callbacks.callback_monitor import CallbackMonitor, CallbackMonitorResult
+from typing import Union, List, Optional, Dict
+
+class CollectOutputs(CallbackBase):
+    """
+        This callback will collect outputs per each state, (it is currently used in trainer.predict() method.)
+        It will collect the numpy outputs in the defined states to a dictionary.
+
+        Methods:
+            get_outputs_for_state - for a given state, returns the collected outputs
+
+        Args:
+            apply_on_phase - see in CallbackBase
+            apply_on_states - see in CallbackBase
+    """
+
+    def __init__(self, 
+                    apply_on_phase: Phase=Phase.BATCH_END, 
+                    apply_on_states: Union[State, List[State]]=None):
+        super(CollectOutputs, self).__init__(apply_on_phase, apply_on_states)
+        self.state_to_outputs = {}
+
+    def get_outputs_for_state(self, state: State):
+        return self.state_to_outputs[state]
+
+    def __call__(self, callback_context: CallbackContext):
+        c = callback_context #READABILITY DOWN THE ROAD
+        state = c.trainer_state
+
+        if self.should_apply_on_state(c):
+
+            if state not in self.state_to_outputs:
+                self.state_to_outputs[state] = []
+
+            last_outputs = c.trainer.get_last_outputs()
+            self.state_to_outputs[state].append(last_outputs)

lpd/enums/phase.py

@@ -10,5 +10,8 @@ class Phase(Enum):
     BATCH_END     = auto()
     TEST_BEGIN    = auto()
     TEST_END      = auto()
+    PREDICT_BEGIN = auto()
+    PREDICT_END   = auto()
+
     def __str__(self):
         return self.name

lpd/enums/state.py

@@ -12,5 +12,6 @@ class State(Enum):
     TRAIN    = auto()
     VAL      = auto() 
     TEST     = auto()
+    PREDICT  = auto()
     def __str__(self):
         return self.name