Merge pull request #15 from zezhishao/dev/refactor

update configurations for PEMS04 dataset; update REAMD; update training logs.

README.md

@@ -31,6 +31,9 @@ scripts   --> Data preprocessing scripts.
 step      --> The implementation of STEP, including the architecture, dataloader, loss, and runner for STEP.
 
 tsformer_ckpt --> Pre-trained TSFormer for METR-LA, PEMS-BAY, and PEMS04 dataset.
+
+training_logs --> Training logs of STEP and TSFormer.
+
 ```
 
 ## 💿 Requirements
@@ -81,7 +84,7 @@ datasets
 
 ```bash
 python step/run.py --cfg='step/STEP_$DATASET.py' --gpus='0, 1'
-# python step/run.py --cfg='step/STEP_METR-LA.py' --gpus='0'
+# python step/run.py --cfg='step/STEP_METR-LA.py' --gpus='0, 1'
 # python step/run.py --cfg='step/STEP_PEMS-BAY.py' --gpus='0, 1'
 # python step/run.py --cfg='step/STEP_PEMS04.py' --gpus='0, 1'
 ```
@@ -91,7 +94,7 @@ Configuration file `step/STEP_$DATASET.py` describes the forecasting configurati
 Edit `BATCH_SIZE` and `GPU_NUM` in the configuration file and `--gpu` in the command line to run on your own hardware.
 Note that different GPU number leads to different real batch sizes, affecting the learning rate setting and the forecasting accuracy.
 
-Our training logs are shown in `train_logs/STEP_METR-LA.log`, `train_logs/STEP_PEMS04.log`, and `train_logs/STEP_PEMS-BAY.log`.
+Our training logs are shown in [training_logs/STEP_METR-LA.log](./training_logs/STEP_METR-LA.log), [training_logs/STEP_METR-LA.log](./training_logs/STEP_METR-LA.log) and [training_logs/STEP_PEMS-BAY.log](./training_logs/STEP_PEMS-BAY.log).
 
 ## ⚒ Train STEP from Scratch
 
@@ -107,10 +110,11 @@ python step/run.py --cfg='step/TSFormer_$DATASET.py' --gpus '0'
 Replace `$DATASET_NAME` with one of `METR-LA`, `PEMS-BAY`, `PEMS04` as shown in the code above.
 Configuration file `step/TSFormer_$DATASET.py` describes the pre-training configurations.
 Edit the `BATCH_SIZE` and `GPU_NUM` in the configuration file and `--gpu` in the command line to run on your own hardware.
+
 All the training logs, including the config file, training log, and checkpoints, will be saved in `checkpoints/MODEL_EPOCH/MD5_of_config_file`.
 For example, `checkpoints/TSFormer_100/5afe80b3e7a3dc055158bcfe99afbd7f`.
 
-Our training logs are shown in `train_logs/TSFormer_METR-LA.log`, `train_logs/TSFormer_PEMS04.log`, and `train_logs/TSFormer_PEMS-BAY.log`, and the pre-trained TSFormers for each datasets are placed in `tsformer_ckpt` folder.
+Our training logs are shown in [training_logs/TSFormer_METR-LA.log](./training_logs/TSFormer_METR-LA.log), [training_logs/TSFormer_PEMS04.log](./training_logs/TSFormer_PEMS04.log), and [training_logs/TSFormer_PEMS04.log](./training_logs/TSFormer_PEMS-BAY.log), and the our pre-trained TSFormers for each datasets are placed in `tsformer_ckpt` folder.
 
 ### **Forecasting Stage**
 
@@ -137,6 +141,16 @@ Then train the downstream STGNN (Graph WaveNet) like in section 4.
 
 - [BasicTS: An Open Source Standard Time Series Forecasting Benchmark.](https://github.com/zezhishao/BasicTS)
 
+## QA:
+
+Q1: Why is the performance in the training log slightly different from the performance in the paper?
+
+A1: 
+STEP's code is now refactored based on [BasicTS](https://github.com/zezhishao/BasicTS), to provide fair comparisons with all baselines.
+BasicTS unifies the training pipeline and evaluation metrics, which is slightly different from the original implementation of STEP.
+Therefore, there may be small differences (better or worse) between training logs and papers that do not affect the conclusions of the paper.
+Moreover, you can refer to BasicTS to find the true performance of many baseline models.
+
 ## Citing
 
 If you find this repository useful for your work, please consider citing it as follows:

step/TSFormer_PEMS04.py

@@ -21,7 +21,7 @@
 CFG.DATASET_TYPE = "Traffic flow"
 CFG.DATASET_INPUT_LEN = 288 * 7 * 2
 CFG.DATASET_OUTPUT_LEN = 12
-CFG.GPU_NUM = 1
+CFG.GPU_NUM = 2
 
 # ================= environment ================= #
 CFG.ENV = EasyDict()
@@ -55,7 +55,7 @@
 CFG.TRAIN.OPTIM = EasyDict()
 CFG.TRAIN.OPTIM.TYPE = "Adam"
 CFG.TRAIN.OPTIM.PARAM= {
-    "lr":0.0005,
+    "lr":0.001,
     "weight_decay":0,
     "eps":1.0e-8,
     "betas":(0.9, 0.95)
@@ -82,7 +82,7 @@
 # read data
 CFG.TRAIN.DATA.DIR = "datasets/" + CFG.DATASET_NAME
 # dataloader args, optional
-CFG.TRAIN.DATA.BATCH_SIZE = 4
+CFG.TRAIN.DATA.BATCH_SIZE = 6
 CFG.TRAIN.DATA.PREFETCH = False
 CFG.TRAIN.DATA.SHUFFLE = True
 CFG.TRAIN.DATA.NUM_WORKERS = 2

step/run.py

@@ -14,7 +14,7 @@ def parse_args():
 
     # parser.add_argument("-c", "--cfg", default="step/TSFormer_PEMS04.py", help="training config")
     parser.add_argument("-c", "--cfg", default="step/STEP_PEMS04.py", help="training config")
-    parser.add_argument("--gpus", default="0", help="visible gpus")
+    parser.add_argument("--gpus", default="0, 1", help="visible gpus")
     return parser.parse_args()
 
 if __name__ == "__main__":