Skip to content

Commit

Permalink
clean up for release; fixed landsat subset pytests, added release not…
Browse files Browse the repository at this point in the history 
…es for v4.4.0, removed debug print in python client
  • Loading branch information
@jpswinski
jpswinski committed May 3, 2024
1 parent a92b431 commit 6d18ed1
Show file tree
Hide file tree
Showing 5 changed files with 173 additions and 24 deletions.
1 change: 0 additions & 1 deletion clients/python/sliderule/sliderule.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -460,7 +460,6 @@ def __logeventrec(rec):
# __exceptrec
#
def __exceptrec(rec):
print("Received", rec["text"])
if rec["code"] == exceptioncodes["SIMPLIFY"]:
raise RetryRequest("cmr simplification requested")
elif rec["code"] < 0:
Expand Down
19 changes: 0 additions & 19 deletions clients/python/tests/test_landsat.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,16 +36,7 @@ def test_subset1(self, init):
rsps = sliderule.source("subsets", rqst)
assert init
assert len(rsps) > 0
assert len(rsps['subsets'][0][0]['data']) > 0
assert rsps['subsets'][0][0]['rows'] == 1030
assert rsps['subsets'][0][0]['cols'] == 2504
assert rsps['subsets'][0][0]['size'] == 5158240
assert rsps['subsets'][0][0]['datatype'] == 1 # INT16
assert abs(rsps['subsets'][0][0]['ulx'] - 300000) < 1e-9
assert abs(rsps['subsets'][0][0]['uly'] - 5623518.2868507) < 1e-9
assert rsps['subsets'][0][0]['cellsize'] == 30
assert rsps['subsets'][0][0]['wkt'] != ""


def test_subset167(self, init):
time_start = "2021-01-01T00:00:00Z"
Expand All @@ -64,19 +55,9 @@ def test_subset167(self, init):
subsets = rsps['subsets'][0]
assert init
assert len(subsets) == 167

for subset in subsets:
assert subset['rows'] > 0
assert subset['cols'] > 0
assert subset['size'] > 0

# Some rsters have datatype as INT16 others as UINT16
assert subset['datatype'] == 1 or subset['datatype'] == 5
assert subset['ulx'] > 0 # for this test both ulx and uly in map coords are positive
assert subset['uly'] > 0
assert subset['cellsize'] == 30
assert subset['wkt'] != ""


def test_ndvi(self, init):
region = sliderule.toregion(os.path.join(TESTDIR, "data/grandmesa.geojson"))
Expand Down
1 change: 0 additions & 1 deletion clients/python/utils/benchmark.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@
import sliderule
from sliderule import earthdata, h5, icesat2, gedi
import argparse
import logging
import time
import os

Expand Down
170 changes: 170 additions & 0 deletions docs/rtd/source/release_notes/release-v4-4-0.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
# Release v4.4.0

2024-04-04

Version description of the v4.4.0 release of SlideRule Earth.

## New Features

* Resources are queried from servers instead of client. If a processing request does not include a list of `resources` to process, the server processing the request will query CMR and populate the `resources` parameter. In addition, any sampling requests that need a populated `catalog` parameter will also be queried on the server side and have that parameter populated.

* [389](https://github.com/SlideRuleEarth/sliderule/pull/389) and [383](https://github.com/SlideRuleEarth/sliderule/pull/383) - updates to demo plotting and added support for downloading results

* Raster sampling support when the output is an Arrow generated format (Geo/Parquet, CSV, Feather).

* Added Feather output support

* [43d536b](https://github.com/SlideRuleEarth/sliderule/commit/43d536b690ac4fe8337634980092bbf164ece366) - Request parameters and record information added to metadata of generated Parquet files.

* [763e553](https://github.com/SlideRuleEarth/sliderule/commit/763e5537d23483601535f622d9386f93456ba967) - max confidence in the signal_conf variable can be selected when filtering ATL03 photons based on confidence level

* [392](https://github.com/SlideRuleEarth/sliderule/pull/392) - GEBCO raster sampling support added

* [9d71b6e](https://github.com/SlideRuleEarth/sliderule/commit/9d71b6ef8da004c18a2a40f18035ad3eb3d36b66) - Meta global canopy height raster support added

## Issues Resolved

* Logs returned to users are now consistently implemented as exception records through the `alert` function.

## Development Updates

* GDAL headers removed from exported package headers

* GDAL log messages are defaulted off

## Getting This Release

[https://github.com/SlideRuleEarth/sliderule/releases/tag/v4.4.0](https://github.com/SlideRuleEarth/sliderule/releases/tag/v4.4.0)

## Benchmarks

```
atl06_aoi
output: 625445 x 16 elements
total: 26.511650 secs
gdf2poly: 0.000324 secs
toregion: 0.094860 secs
__parse_native: 21.172510 secs
todataframe: 0.391770 secs
merge: 0.000002 secs
flatten: 2.783902 secs
atl06p: 26.266139 secs
atl06_ancillary
output: 1180 x 17 elements
total: 3.049564 secs
gdf2poly: 0.000324 secs
toregion: 0.094860 secs
__parse_native: 0.690858 secs
todataframe: 0.005812 secs
merge: 0.013563 secs
flatten: 0.030884 secs
atl06p: 2.857920 secs
atl03_ancillary
output: 1180 x 17 elements
total: 3.047120 secs
gdf2poly: 0.000324 secs
toregion: 0.094860 secs
__parse_native: 0.158048 secs
todataframe: 0.005754 secs
merge: 0.013669 secs
flatten: 0.031304 secs
atl06p: 3.045915 secs
atl06_parquet
output: 1577 x 17 elements
total: 2.755643 secs
gdf2poly: 0.000324 secs
toregion: 0.094860 secs
__parse_native: 0.209622 secs
todataframe: 0.005754 secs
merge: 0.013669 secs
flatten: 0.044833 secs
atl06p: 2.755176 secs
atl03_parquet
output: 22833 x 22 elements
total: 1.633988 secs
gdf2poly: 0.000324 secs
toregion: 0.094860 secs
__parse_native: 0.011271 secs
todataframe: 0.005754 secs
merge: 0.013669 secs
flatten: 0.044833 secs
atl06p: 2.755176 secs
atl03sp: 1.607882 secs
atl06_sample_landsat
output: 914 x 20 elements
total: 7.073046 secs
gdf2poly: 0.000324 secs
toregion: 0.094860 secs
__parse_native: 0.370676 secs
todataframe: 0.006028 secs
merge: 0.012087 secs
flatten: 0.031790 secs
atl06p: 4.672313 secs
atl03sp: 1.607882 secs
atl06_sample_zonal_arcticdem
output: 1651 x 27 elements
total: 6.384218 secs
gdf2poly: 0.000320 secs
toregion: 0.006032 secs
__parse_native: 0.319357 secs
todataframe: 0.005968 secs
merge: 0.020238 secs
flatten: 0.045042 secs
atl06p: 6.375683 secs
atl03sp: 1.607882 secs
atl06_sample_nn_arcticdem
output: 1651 x 20 elements
total: 4.758471 secs
gdf2poly: 0.000278 secs
toregion: 0.005407 secs
__parse_native: 0.091373 secs
todataframe: 0.005742 secs
merge: 0.018872 secs
flatten: 0.038760 secs
atl06p: 4.751282 secs
atl03sp: 1.607882 secs
atl06_msample_nn_arcticdem
output: 1596050 x 20 elements
total: 143.910649 secs
gdf2poly: 0.000268 secs
toregion: 0.005141 secs
__parse_native: 89.897940 secs
todataframe: 0.975277 secs
merge: 15.259323 secs
flatten: 29.145164 secs
atl06p: 142.751770 secs
atl03sp: 1.607882 secs
atl06_no_sample_arcticdem
output: 1596050 x 16 elements
total: 69.063557 secs
gdf2poly: 0.000306 secs
toregion: 0.006468 secs
__parse_native: 53.275093 secs
todataframe: 0.964930 secs
merge: 0.000003 secs
flatten: 7.419176 secs
atl06p: 67.920378 secs
atl03sp: 1.607882 secs
atl03_rasterized_subset
output: 51968 x 22 elements
total: 4.176219 secs
gdf2poly: 0.000306 secs
toregion: 0.006468 secs
__parse_native: 1.858839 secs
todataframe: 0.042089 secs
merge: 0.000003 secs
flatten: 0.371834 secs
atl06p: 67.920378 secs
atl03sp: 3.623235 secs
atl03_polygon_subset
output: 50799 x 22 elements
total: 3.140495 secs
gdf2poly: 0.000306 secs
toregion: 0.006468 secs
__parse_native: 1.370758 secs
todataframe: 0.041492 secs
merge: 0.000003 secs
flatten: 0.359666 secs
atl06p: 67.920378 secs
atl03sp: 3.114903 secs
```
6 changes: 3 additions & 3 deletions targets/slideruleearth-aws/docker/sliderule/libdep.lock
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
arrow: apache-arrow-15.0.2-0-ge03105efc
gdal: v3.8.4-0-gc2d2a61739
gdal: v3.8.5-0-g1d418c1c1a
geos: 3.12.1-0-g9a5d0587f
libgeotiff: 1.7.1-0-gbdeff84
pistache: 0.0.5-79-g875c973
pistache: 0.0.5-100-g9331499
PROJ: 9.4.0-0-g7cd00d0b
rapidjson: v1.1.0-751-g5ec44fb9
rapidjson: v1.1.0-752-gab1842a2

0 comments on commit 6d18ed1

Please sign in to comment.