Test fixes #154

ejm714 · 2025-03-03T20:28:28Z

set minimum version for scikit-learn and use new root_mean_squared_error function (API change details)

The squared parameter of metrics.mean_squared_error and metrics.mean_squared_log_error is deprecated and will be removed in 1.6. Use the new functions metrics.root_mean_squared_error and metrics.root_mean_squared_log_error instead

the SCL band has been reprocessed as it's picking up more water pixels and a couple more available images. this re-makes the tests assets in accordance with this (reprocessing details). Retrain base model to account for reprocessed SCL band #155 tracks a full training to ensure consistency in data processing. EDIT: some of these differences are due to underlying variations in openCV implementations

The improvements introduced in processing baselines up to baseline 04.00 (in operations since 25 January 2022) have been generalised to the historical archive:

Improved L2A processing algorithms for scene classification and surface reflectance (PB 04.00 feature)

Bonus fixes

uncap matplotlib
remove duplicate clean make command
address pandas warnings

…st assets

…pected features

…each test run; note: this is a workaround for better test asset management tracked in #61

codecov · 2025-03-03T21:17:53Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.0%. Comparing base (63e490f) to head (d9e91ad).
Report is 2 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #154   +/-   ##
=====================================
  Coverage   75.0%   75.0%           
=====================================
  Files         12      12           
  Lines        863     863           
=====================================
  Hits         648     648           
  Misses       215     215

Files with missing lines	Coverage Δ
cyfi/evaluate.py	`91.0% <100.0%> (ø)`

klwetstone

@ejm714 Generally looks good. I am getting slightly different values for the test assets when I rerun make test -- can we huddle quickly tomorrow?

Based on sentinel_metadata_[train/test].csv, I'm getting all the same tiles, but with different values for the number of water pixels. As a result of the different water masks, but features, r-squared values (results.json), and model are also different.

tests/assets/experiment/config_artifact.yaml

pyproject.toml

tests/test_experiment.py

tests/assets/experiment/config_artifact.yaml

klwetstone · 2025-03-03T22:21:28Z

tests/assets/experiment/sentinel_metadata_test.csv

-a9af79990527a83e31f9a954d3e334d5,S2B_MSIL2A_20180708T155819_R097_T17SPV_20201011T133356,0.004069006543723974,56382.0,3,https://sentinel2l2a01.blob.core.windows.net/sentinel2-l2/17/S/PV/2018/07/08/S2B_MSIL2A_20180708T155819_N0212_R097_T17SPV_20201011T133356.SAFE/GRANULE/L2A_T17SPV_A006987_20180708T161248/IMG_DATA/R10m/T17SPV_20180708T155819_TCI_10m.tif
-d823c957bedf702bb1dd8d562e45a7b8,S2B_MSIL2A_20170713T173909_R098_T13TDE_20210210T083500,0.014554095195663475,66558.0,15,https://sentinel2l2a01.blob.core.windows.net/sentinel2-l2/13/T/DE/2017/07/13/S2B_MSIL2A_20170713T173909_N0212_R098_T13TDE_20210210T083500.SAFE/GRANULE/L2A_T13TDE_A001840_20170713T174714/IMG_DATA/R10m/T13TDE_20170713T173909_TCI_10m.tif
-f79c66af395e5de8fb3c5cc1b1ed6993,S2A_MSIL2A_20180521T154911_R054_T18TUK_20201012T122951,0.0155145746579417,6416.0,0,https://sentinel2l2a01.blob.core.windows.net/sentinel2-l2/18/T/UK/2018/05/21/S2A_MSIL2A_20180521T154911_N0212_R054_T18TUK_20201012T122951.SAFE/GRANULE/L2A_T18TUK_A015209_20180521T155356/IMG_DATA/R10m/T18TUK_20180521T154911_TCI_10m.tif
+494fb91d1fb8697e73b90e5d0c9420ed,S2A_MSIL2A_20190818T170851_R112_T15SUD_20201005T015156,0.010019129603060737,1096.0,8,https://sentinel2l2a01.blob.core.windows.net/sentinel2-l2/15/S/UD/2019/08/18/S2A_MSIL2A_20190818T170851_N0212_R112_T15SUD_20201005T015156.SAFE/GRANULE/L2A_T15SUD_A021702_20190818T171838/IMG_DATA/R10m/T15SUD_20190818T170851_TCI_10m.tif


When I rerun make test on this branch, I get different values for num_water_pixels. Mine generally align with the previous version of this file. I'm not sure why mine is different -- I started with a clean environment and re-installed the requirements.

I'm also getting slightly different values for features_test.csv and features_train.csv

hmmm that's very curious and good to know

after a long investigation, the number of water pixel changes (with cascading effects on band summary stats) appear to be coming from OpenCV (due to variations in underlying hardware, operating systems, and library implementations).

Different processors and architectures may handle floating-point calculations with slight variations, leading to minor differences in the computed pixel values during interpolation.

this is the relevant part of the code

cyfi/cyfi/data/features.py

Lines 165 to 169 in 85c82ca

if config.filter_to_water_area:

if band != "SCL":

scaled_scl = cv2.resize(scl_array[0], (arr.shape[2], arr.shape[1]))

arr = arr[0][scaled_scl == 6]

sample_item_features["num_water_pixels"] = arr.size

tests/test_experiment.py

klwetstone

@ejm714 LGTM! Just one small question.

All tests pass for me, and make tests doesn't change any of the test asset files
make assets runs and does change test asset files (expected because of weird opencv behavior). The same images are selected for both train and test, although the number of water pixels is different

klwetstone · 2025-03-04T21:45:54Z

pyproject.toml

@@ -34,7 +34,7 @@ dependencies = [
  "gradio",
  "lightgbm",
  "loguru",
-  "matplotlib<3.8.0",
+  "matplotlib",


Why do we no longer need to enforce a version of matplotlib?

There was a mpl bug that was the cause of this ceiling (e379dec) but I can't remember the specifics. In general, we only want floors not ceilings for packages. Ceilings are often a temporary workaround and it looked like this one was outdated (indeed as it was given tests pass with later versions)

set minimum scikit-learn version and use root_mean_squared_error

dbe6279

ejm714 deployed to test-fixes - CyFi PR #154 March 3, 2025 20:28 — with Render View deployment

ejm714 added 3 commits March 3, 2025 12:55

sentinel-2 data has been reprocessed leading to changes: part 1 -- te…

56d2e23

…st assets

sentinel-2 data has been reprocessed leading to changes: part 2 -- ex…

d997368

…pected features

avoid writing out git hash to config yaml so it does not change with …

54e9c65

…each test run; note: this is a workaround for better test asset management tracked in #61

ejm714 deployed to test-fixes - CyFi PR #154 March 3, 2025 21:14 — with Render View deployment

ejm714 marked this pull request as ready for review March 3, 2025 21:36

ejm714 requested a review from klwetstone March 3, 2025 21:37

klwetstone reviewed Mar 3, 2025

View reviewed changes

ejm714 added 2 commits March 3, 2025 14:46

be more flexible with scikit-learn

80b9d62

override save path in experiment test to use tmp_path

cc6417b

ejm714 deployed to test-fixes - CyFi PR #154 March 3, 2025 22:58 — with Render View deployment

ejm714 added 2 commits March 3, 2025 15:33

remove ceiling for matplotlib and remove duplicate clean command

c3347fa

update test assets with latest matplotlib

19d9ef4

ejm714 deployed to test-fixes - CyFi PR #154 March 3, 2025 23:33 — with Render View deployment

address pandas warnings

38ffda0

ejm714 deployed to test-fixes - CyFi PR #154 March 3, 2025 23:54 — with Render View deployment

klwetstone reviewed Mar 4, 2025

View reviewed changes

tests/test_experiment.py Outdated Show resolved Hide resolved

fix test from writing out to assets

0cb6efd

ejm714 deployed to test-fixes - CyFi PR #154 March 4, 2025 17:56 — with Render View deployment

latest hash

72576dd

ejm714 deployed to test-fixes - CyFi PR #154 March 4, 2025 21:26 — with Render View deployment

ejm714 force-pushed the test-fixes branch from bcfa2f1 to 72576dd Compare March 4, 2025 21:36

ejm714 deployed to test-fixes - CyFi PR #154 March 4, 2025 21:37 — with Render View deployment

make assets should be runnable again if it failed halfway through

d9e91ad

ejm714 temporarily deployed to test-fixes - CyFi PR #154 March 4, 2025 21:40 — with Render Destroyed

klwetstone approved these changes Mar 4, 2025

View reviewed changes

ejm714 merged commit 62fef8b into main Mar 4, 2025
10 checks passed

ejm714 deleted the test-fixes branch March 4, 2025 22:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test fixes #154

Test fixes #154

ejm714 commented Mar 3, 2025 •

edited

Loading

codecov bot commented Mar 3, 2025 •

edited

Loading

klwetstone left a comment

klwetstone Mar 3, 2025

klwetstone Mar 3, 2025

ejm714 Mar 3, 2025

ejm714 Mar 4, 2025 •

edited

Loading

klwetstone left a comment

klwetstone Mar 4, 2025

ejm714 Mar 4, 2025

	if config.filter_to_water_area:
	if band != "SCL":
	scaled_scl = cv2.resize(scl_array[0], (arr.shape[2], arr.shape[1]))
	arr = arr[0][scaled_scl == 6]
	sample_item_features["num_water_pixels"] = arr.size

Test fixes #154

Test fixes #154

Conversation

ejm714 commented Mar 3, 2025 • edited Loading

Bonus fixes

codecov bot commented Mar 3, 2025 • edited Loading

Codecov Report

klwetstone left a comment

Choose a reason for hiding this comment

klwetstone Mar 3, 2025

Choose a reason for hiding this comment

klwetstone Mar 3, 2025

Choose a reason for hiding this comment

ejm714 Mar 3, 2025

Choose a reason for hiding this comment

ejm714 Mar 4, 2025 • edited Loading

Choose a reason for hiding this comment

klwetstone left a comment

Choose a reason for hiding this comment

klwetstone Mar 4, 2025

Choose a reason for hiding this comment

ejm714 Mar 4, 2025

Choose a reason for hiding this comment

ejm714 commented Mar 3, 2025 •

edited

Loading

codecov bot commented Mar 3, 2025 •

edited

Loading

ejm714 Mar 4, 2025 •

edited

Loading