[devops] Upgrade to lightning 2.0 #1514

ourownstory · 2024-01-17T00:41:05Z

Upgrade guide
https://lightning.ai/docs/pytorch/latest/upgrade/from_1_9.html

github-actions · 2024-06-20T00:31:17Z

Model Benchmark

Benchmark	Metric	main	current	diff
PeytonManning	MAE_val	0.34955	0.35511	1.59%	✅
PeytonManning	RMSE_val	0.50049	0.50375	0.65%	✅
PeytonManning	Loss_val	0.01771	0.01801	1.69%	✅
PeytonManning	train_loss	nan	0.01463	0.0%	✅
PeytonManning	reg_loss	nan	0	0.0%	✅
PeytonManning	MAE	0.34653	0.34734	0.24%	✅
PeytonManning	RMSE	0.49312	0.49386	0.15%	✅
PeytonManning	Loss	0.01464	0.01463	-0.09%	✅
PeytonManning	time	11.8065	12.7	7.57%	❌
AirPassengers	MAE_val	30.6285	30.8786	0.82%	✅
AirPassengers	RMSE_val	31.5254	31.8365	0.99%	✅
AirPassengers	Loss_val	0.01277	0.01303	1.98%	✅
AirPassengers	train_loss	nan	0.00071	0.0%	✅
AirPassengers	reg_loss	nan	0	0.0%	✅
AirPassengers	MAE	6.16861	6.85845	11.18%	❌
AirPassengers	RMSE	7.85225	8.90745	13.44%	❌
AirPassengers	Loss	0.00065	0.00076	16.92%	❌
AirPassengers	time	7.29709	7.76	6.34%	⚠️
EnergyPriceDaily	MAE_val	5.41157	5.64156	4.25%	⚠️
EnergyPriceDaily	RMSE_val	6.71538	7.19554	7.15%	❌
EnergyPriceDaily	Loss_val	0.02525	0.0289	14.47%	❌
EnergyPriceDaily	train_loss	nan	0.02962	0.0%	✅
EnergyPriceDaily	reg_loss	nan	0	0.0%	✅
EnergyPriceDaily	MAE	5.94936	6.40067	7.59%	❌
EnergyPriceDaily	RMSE	7.9833	8.56962	7.34%	❌
EnergyPriceDaily	Loss	0.02579	0.02941	14.05%	❌
EnergyPriceDaily	time	14.2089	15.82	11.34%	❌
YosemiteTemps	MAE_val	0.56412	0.48186	-14.58%	🎉
YosemiteTemps	RMSE_val	0.83161	0.71525	-13.99%	🎉
YosemiteTemps	Loss_val	0.0004	0.0003	-26.0%	🎉
YosemiteTemps	train_loss	nan	0.00103	0.0%	✅
YosemiteTemps	reg_loss	nan	0	0.0%	✅
YosemiteTemps	MAE	0.98449	0.85851	-12.8%	🎉
YosemiteTemps	RMSE	1.75389	1.54684	-11.81%	🎉
YosemiteTemps	Loss	0.00132	0.00103	-21.4%	🎉
YosemiteTemps	time	30.8231	35.24	14.33%	❌

\n

Model training plots

\n ## Model Training ### PeytonManning ![](https://asset.cml.dev/1d71ab731130716a13ca5609c24d52e4e540dbcd?cml=svg%2Bxml&cache-bypass=3c0b1bd7-0cd7-448a-aa0b-27a9d908b270) ### YosemiteTemps ![](https://asset.cml.dev/25c0d646f1807f399110bf3dd41a6af7b5e4d0bb?cml=svg%2Bxml&cache-bypass=07bc006a-81a4-40cd-a3ea-4902b537b8b9) ### AirPassengers ![](https://asset.cml.dev/85d58e6f0d87f903ddc54d6984086a6e1661dec8?cml=svg%2Bxml&cache-bypass=4a361a46-2823-4af3-8f49-d1143b7be716) ### EnergyPriceDaily ![](https://asset.cml.dev/88a4f18f9122a4b1cfb9ad479dcdb8c1c0883759?cml=svg%2Bxml&cache-bypass=499b3718-3366-4903-a754-8d5a27f5859a) \n

ourownstory

It seems like the tests are yielding differing metrics - Do you have any idea what might be the cause?

ourownstory · 2024-06-20T04:05:15Z

pyproject.toml

@@ -16,28 +16,27 @@ classifiers = [
 Homepage = "https://github.com/ourownstory/neural_prophet"

 [tool.poetry.dependencies]
-python = "^3.9"
-typing-extensions = "^4.5.0"
+python = ">=3.9,<3.12"


Is there a reason that we are no longer including python 3.12?

I haven't changed that requirement. I'll add it shortly.

pyproject.toml

MaiBe-ctrl · 2024-06-20T16:37:51Z

@ourownstory Not sure where the differences in the metrics are coming from, but I had to cast some variables to float32 to fix some tests which might caused these differences.

ourownstory

Looks like two changes to the if-check have unintentional consequences. @MaiBe-ctrl

ourownstory · 2024-06-21T01:12:37Z

neuralprophet/configure.py

@@ -311,12 +311,12 @@ def __post_init__(self):
            self.trend_local_reg = False

        # If trend_local_reg = True
-        if self.trend_local_reg == True:
+        if self.trend_local_reg:


This change looks wrong - see comment in other PR

ourownstory · 2024-06-21T01:13:05Z

neuralprophet/configure.py

@@ -398,12 +398,12 @@ def __post_init__(self):
            self.seasonality_local_reg = False

        # If seasonality_local_reg = True
-        if self.seasonality_local_reg == True:
+        if self.seasonality_local_reg:


This also overwrites existing config, which should not happen

ourownstory · 2024-06-21T01:42:08Z

neuralprophet/configure.py

            log.error("trend_local_reg = True. Default trend_local_reg value set to 1")
            self.trend_local_reg = 1

        # If Trend modelling is global.
-        if self.trend_global_local == "global" and self.trend_local_reg:
+        if self.trend_global_local == "global" and self.trend_local_reg is True:


Here, your change was actually correct before:
if self.trend_global_local == "global" and self.trend_local_reg:

ourownstory · 2024-06-21T01:42:40Z

neuralprophet/configure.py

            log.error("seasonality_local_reg = True. Default seasonality_local_reg value set to 1")
            self.seasonality_local_reg = 1

        # If Season modelling is global.
-        if self.global_local == "global" and self.seasonality_local_reg:
+        if self.global_local == "global" and self.seasonality_local_reg is True:


Here, your change was actually correct before:
if self.global_local == "global" and self.seasonality_local_reg:

codecov · 2024-06-21T05:39:09Z

Codecov Report

Attention: Patch coverage is 93.33333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 87.46%. Comparing base (79ea9cd) to head (cfbe96e).
Report is 3 commits behind head on main.

❗ Current head cfbe96e differs from pull request most recent head c4037b0

Please upload reports for the commit c4037b0 to get more accurate results.

Files	Patch %	Lines
neuralprophet/forecaster.py	66.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1514      +/-   ##
==========================================
- Coverage   88.16%   87.46%   -0.70%     
==========================================
  Files          41       41              
  Lines        5374     5378       +4     
==========================================
- Hits         4738     4704      -34     
- Misses        636      674      +38

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ourownstory · 2024-06-21T05:46:09Z

@ourownstory Not sure where the differences in the metrics are coming from, but I had to cast some variables to float32 to fix some tests which might caused these differences.

I think the difference comes from the potentially changed implementation of the learning rate finder, that now is part of lightning Tuner.

ourownstory added 7 commits January 16, 2024 15:11

relax torch to greater 2

26cb5bd

upgrade lightning to 2.0, open version limits of other packages

6cac2bb

upgrade to python3.11

35133f5

re-relax python versions

32146f7

merge main

4176a75

merge py3.11

2fd30b1

merge main

329931a

ourownstory marked this pull request as draft January 17, 2024 00:41

ourownstory changed the title ~~[devops] Upgrade lightning 2 [WIP]~~ [devops] Upgrade to lightning 2.0 [WIP] Jan 17, 2024

ourownstory and others added 8 commits January 16, 2024 16:45

Merge branch 'main' into upgrade-lightning-2

1222282

Merge branch 'main' into upgrade-lightning-2

b059658

upgrade to lightning 2+

5e0428d

Merge branch 'main' into upgrade-lightning-2

293755e

fix toml file

65caa47

fic poetry

7f62f38

added kaleido

7cc25c9

downgrade numpy

7f71169

fix linting

2b2a060

ourownstory commented Jun 20, 2024

View reviewed changes

added python 3.12

2001d8d

ourownstory marked this pull request as ready for review June 21, 2024 00:23

ourownstory added 3 commits June 20, 2024 17:50

clean up pyproject.toml dependencies

6ebb274

update poetry lock

a0f921a

Merge branch 'main' into upgrade-lightning-2

cadcc7b

ourownstory commented Jun 21, 2024

View reviewed changes

changed is True

64b4e17

ourownstory commented Jun 21, 2024

View reviewed changes

changed bool check

c4773e9

Merge branch 'main' into upgrade-lightning-2

cfbe96e

uncomment test_glocal lines 152 onward

891ba53

ourownstory changed the title ~~[devops] Upgrade to lightning 2.0 [WIP]~~ [devops] Upgrade to lightning 2.0 Jun 21, 2024

ourownstory added 2 commits June 20, 2024 22:50

Merge branch 'main' into upgrade-lightning-2

0be90e7

Merge branch 'main' into upgrade-lightning-2

c4037b0

ourownstory merged commit 79a6907 into main Jun 21, 2024
7 of 11 checks passed

ourownstory deleted the upgrade-lightning-2 branch June 21, 2024 07:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[devops] Upgrade to lightning 2.0 #1514

[devops] Upgrade to lightning 2.0 #1514

ourownstory commented Jan 17, 2024

github-actions bot commented Jun 20, 2024 •

edited

Loading

ourownstory left a comment

ourownstory Jun 20, 2024

MaiBe-ctrl Jun 20, 2024

MaiBe-ctrl commented Jun 20, 2024

ourownstory left a comment

ourownstory Jun 21, 2024

ourownstory Jun 21, 2024

ourownstory Jun 21, 2024

ourownstory Jun 21, 2024

codecov bot commented Jun 21, 2024 •

edited

Loading

ourownstory commented Jun 21, 2024

[devops] Upgrade to lightning 2.0 #1514

[devops] Upgrade to lightning 2.0 #1514

Conversation

ourownstory commented Jan 17, 2024

github-actions bot commented Jun 20, 2024 • edited Loading

Model Benchmark

ourownstory left a comment

Choose a reason for hiding this comment

ourownstory Jun 20, 2024

Choose a reason for hiding this comment

MaiBe-ctrl Jun 20, 2024

Choose a reason for hiding this comment

MaiBe-ctrl commented Jun 20, 2024

ourownstory left a comment

Choose a reason for hiding this comment

ourownstory Jun 21, 2024

Choose a reason for hiding this comment

ourownstory Jun 21, 2024

Choose a reason for hiding this comment

ourownstory Jun 21, 2024

Choose a reason for hiding this comment

ourownstory Jun 21, 2024

Choose a reason for hiding this comment

codecov bot commented Jun 21, 2024 • edited Loading

Codecov Report

ourownstory commented Jun 21, 2024

github-actions bot commented Jun 20, 2024 •

edited

Loading

codecov bot commented Jun 21, 2024 •

edited

Loading