Fix for #44, add support for modis collection 6 (and retain support for 5) #68

ra-tolson · 2016-08-02T14:04:11Z

This was made by: git checkout modis-6 && git rebase -i master HEAD --onto dev

The diff doesn't look exactly the same as the old PR:

@bhbraswell could you read through this and confirm that it looks right? If there are any problems we can add commits until it's okay. Likewise I would like to add unit tests. I also want to confirm system tests still pass, but for that I have to wait on other PRs.

… collection

…r one. fixed archive update

(Edited to resolve conflicts by tolson during rebase to branch `dev`.)

(Edited by tolson during rebase onto branch `dev`.)

bhbraswell · 2016-08-02T14:11:50Z

Except for the print statement it looks ok to me.

We can talk about tests. I tested gips_process with:

no asset present, update not set —> you get a v6 asset
no asset present, update set —> you get a v6 asset
v5 asset present, update not set —> nothing happens
v5 asset present, update set —> you get a v6 asset
v6 asset present, update not set —> nothing happens
v6 asset present, update set —> nothing happens*
if the version string evaluates to a larger number in the downloaded data, then it will replace the current asset. I think this is possible.

On Aug 2, 2016, at 10:04 AM, ags-tolson notifications@github.com wrote:

This was made by: git checkout modis-6 && git rebase -i master HEAD --onto dev

The diff doesn't look exactly the same as the old PR:

#53 #53
@bhbraswell https://github.com/bhbraswell could you read through this and confirm that it looks right? If there are any problems we can add commits until it's okay. Likewise I would like to add unit tests. I also want to confirm system tests still pass, but for that I have to wait on other PRs.

You can view, comment on, or merge this pull request online at:

#68 #68
Commit Summary

clean up data init. include version which represents file version and collection
update option added to inventory. update with fetch forces the fetch
specifying fetch and update will replace an older version with a newer one. fixed archive update
modis indices now can handle versions 5 or 6
fixed modis version assignment
File Changes

M gips/data/core.py https://github.com/Applied-GeoSolutions/gips/pull/68/files#diff-0 (10)
M gips/data/modis/modis.py https://github.com/Applied-GeoSolutions/gips/pull/68/files#diff-1 (89)
M gips/inventory.py https://github.com/Applied-GeoSolutions/gips/pull/68/files#diff-2 (12)
M gips/parsers.py https://github.com/Applied-GeoSolutions/gips/pull/68/files#diff-3 (2)
M gips/scripts/inventory.py https://github.com/Applied-GeoSolutions/gips/pull/68/files#diff-4 (2)
Patch Links:

https://github.com/Applied-GeoSolutions/gips/pull/68.patch https://github.com/Applied-GeoSolutions/gips/pull/68.patch
https://github.com/Applied-GeoSolutions/gips/pull/68.diff https://github.com/Applied-GeoSolutions/gips/pull/68.diff
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #68, or mute the thread https://github.com/notifications/unsubscribe-auth/ADD4XMxK-H3Mq9EEZKtAsu0b9geZ0z2_ks5qb05cgaJpZM4JapH1.

ra-tolson · 2016-08-02T19:54:37Z

So the system tests need revision to pass, for obvious reasons: Source data changes result in process & project data changes. I want to double-check what I'm seeing in these test runs so I'm going to post outcomes here and I'd like to get feedback.

To start with, here's what had to change to make the test of gips_inventory --fetch pass again; what it's running that gets these files is effectively gips_inventory modis -s /home/tolson/src/gips/gips/test/NHseacoast.shp -d 2012-12-01,2012-12-01 -v 4 --fetch

+        'modis/tiles/h12v04/2012336/MCD43A2.A2012336.h12v04.006.2016112010833.hdf': 531008224,
+        'modis/tiles/h12v04/2012336/MCD43A4.A2012336.h12v04.006.2016112010833.hdf': -955061246,
         'modis/tiles/h12v04/2012336/MOD10A1.A2012336.h12v04.005.2012339213007.hdf': 1588268768,
         'modis/tiles/h12v04/2012336/MOD11A1.A2012336.h12v04.005.2012339180517.hdf': -868909291,
         'modis/tiles/h12v04/2012336/MYD10A1.A2012336.h12v04.005.2012340031954.hdf': 1810195064,

ra-tolson · 2016-08-02T20:30:51Z

UPDATE: There seems to be a bug of some kind: Products for 2012-338 are not generated if you run gips_process . . . -d 2012-12-01,2012-12-03 but they are generated if you run with -d set to the third day only.

Here're the change that makes the system test for gips_process work. I'm nervous about some of these lines as it looks like no products are produced for 2012-338. Command line: gips_process modis -s /home/tolson/src/gips/gips/test/NHseacoast.shp -d 2012-12-01,2012-12-03 -v 4

Here's the changes to the list of created files:

     'created': {
         # TODO Are these broken or what?  Each None is a broken symlink:
         # See https://github.com/Applied-GeoSolutions/gips/issues/54
+        'modis/tiles/h12v04/2012336/h12v04_2012336_MCD_quality.tif': None,
         'modis/tiles/h12v04/2012337/h12v04_2012337_MCD_quality.tif': None,
         'modis/tiles/h12v04/2012337/h12v04_2012337_MOD_temp8td.tif': None,
         'modis/tiles/h12v04/2012337/h12v04_2012337_MOD_temp8tn.tif': None,
+        'modis/tiles/h12v04/2012336/MCD43A2.A2012336.h12v04.006.2016112010833.hdf.index': 1818315139,
+        'modis/tiles/h12v04/2012336/MCD43A4.A2012336.h12v04.006.2016112010833.hdf.index': 502383905,
         'modis/tiles/h12v04/2012336/MOD10A1.A2012336.h12v04.005.2012339213007.hdf.index': -1075525670,
         'modis/tiles/h12v04/2012336/MOD11A1.A2012336.h12v04.005.2012339180517.hdf.index': -1602319177,
         'modis/tiles/h12v04/2012336/MYD10A1.A2012336.h12v04.005.2012340031954.hdf.index': 1623945316,
         'modis/tiles/h12v04/2012336/MYD11A1.A2012336.h12v04.005.2012341040543.hdf.index': -1720582124,
         'modis/tiles/h12v04/2012336/h12v04_2012336_MCD_fsnow.tif': -843500181,
+        'modis/tiles/h12v04/2012336/h12v04_2012336_MCD_indices.tif': -1977616018,
         'modis/tiles/h12v04/2012336/h12v04_2012336_MCD_snow.tif': 388495321,
         'modis/tiles/h12v04/2012336/h12v04_2012336_MOD-MYD_obstime.tif': 1994827924,
         'modis/tiles/h12v04/2012336/h12v04_2012336_MOD-MYD_temp.tif': 2094570047,
         'modis/tiles/h12v04/2012336/h12v04_2012336_MOD_clouds.tif': 161070470,
-        'modis/tiles/h12v04/2012337/MCD43A2.A2012337.h12v04.005.2012356160504.hdf.index': 1869798455,
-        'modis/tiles/h12v04/2012337/MCD43A4.A2012337.h12v04.005.2012356160504.hdf.index': 1702701995,
+        'modis/tiles/h12v04/2012337/MCD43A2.A2012337.h12v04.006.2016112013509.hdf.index': 335987188,
+        'modis/tiles/h12v04/2012337/MCD43A4.A2012337.h12v04.006.2016112013509.hdf.index': -1583910758,
         'modis/tiles/h12v04/2012337/MOD09Q1.A2012337.h12v04.005.2012346141041.hdf.index': 1528708875,
         'modis/tiles/h12v04/2012337/MOD10A1.A2012337.h12v04.005.2012340033542.hdf.index': 1739917027,
         'modis/tiles/h12v04/2012337/MOD11A1.A2012337.h12v04.005.2012339204007.hdf.index': 640817914,
         'modis/tiles/h12v04/2012337/MOD11A2.A2012337.h12v04.005.2012346152330.hdf.index': 53371709,
         'modis/tiles/h12v04/2012337/MYD10A1.A2012337.h12v04.005.2012340112013.hdf.index': 531935583,
         'modis/tiles/h12v04/2012337/MYD11A1.A2012337.h12v04.005.2012341072847.hdf.index': 1676310978,
         'modis/tiles/h12v04/2012337/h12v04_2012337_MCD_fsnow.tif': 297883486,
-        'modis/tiles/h12v04/2012337/h12v04_2012337_MCD_indices.tif': -2140726827,
+        'modis/tiles/h12v04/2012337/h12v04_2012337_MCD_indices.tif': 1828799894,
         'modis/tiles/h12v04/2012337/h12v04_2012337_MCD_snow.tif': -748640537,
         'modis/tiles/h12v04/2012337/h12v04_2012337_MOD-MYD_obstime.tif': -1729084231,
         'modis/tiles/h12v04/2012337/h12v04_2012337_MOD-MYD_temp.tif': -1718009535,
         'modis/tiles/h12v04/2012337/h12v04_2012337_MOD_clouds.tif': -832284681,
-        'modis/tiles/h12v04/2012337/h12v04_2012337_MOD_ndvi8.tif': -593200294,
         'modis/tiles/h12v04/2012338/MOD10A1.A2012338.h12v04.005.2012341091201.hdf.index': 1725484908,
-        'modis/tiles/h12v04/2012338/MOD11A1.A2012338.h12v04.005.2012341041222.hdf.index': 838676814,
-        'modis/tiles/h12v04/2012338/MYD10A1.A2012338.h12v04.005.2012340142152.hdf.index': -130649785,
-        'modis/tiles/h12v04/2012338/MYD11A1.A2012338.h12v04.005.2012341075802.hdf.index': -642783734,
-        'modis/tiles/h12v04/2012338/h12v04_2012338_MCD_fsnow.tif': -1930181337,
-        'modis/tiles/h12v04/2012338/h12v04_2012338_MCD_snow.tif': 387672365,
-        'modis/tiles/h12v04/2012338/h12v04_2012338_MOD-MYD_obstime.tif': -1693632983,
-        'modis/tiles/h12v04/2012338/h12v04_2012338_MOD-MYD_temp.tif': 1712906003,
-        'modis/tiles/h12v04/2012338/h12v04_2012338_MOD_clouds.tif': 296967275,
     },
 }

ra-tolson · 2016-08-03T21:10:07Z

Update: I spent today trying to figure this out, to no avail. Commenting out ndvi8 seems to fix the problem for the rest of the products, so per advice from @bhbraswell I'm leaving it that way for the time being. Apparently ndvi8 is less important as a product so it's okay.

I'll file an issue for this with details on errors observed etc.

bhbraswell · 2016-08-03T21:14:50Z

Now that you know that remove ndvi8 gets rid of the problem, I can look a little more closely at that processing block.

On Aug 3, 2016, at 5:10 PM, ags-tolson notifications@github.com wrote:

Update: I spent today trying to figure this out, to no avail. Commenting out ndvi8 seems to fix the problem for the rest of the products, so per advice from @bhbraswell https://github.com/bhbraswell I'm leaving it that way for the time being. Apparently ndvi8 is less important as a product so it's okay.

I'll file an issue for this with details on errors observed etc.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #68 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/ADD4XEnOMVXffKE_Jtq6rdgXkoxy3MvXks5qcQOvgaJpZM4JapH1.

ra-tolson · 2016-08-03T21:32:10Z

@bhbraswell Okay cool, see issue #70 for details I found after our mtg today. I'm going to pivot back to catching up on system tests now.

fix busted products by deactivating ndvi8, details: #70. MOD09Q1 is no longer needed so remove it from asset lists.

ra-tolson · 2016-08-04T18:35:40Z

Tests pass now; I merged in dev to this branch since it'd moved and I wanted to make sure some new unit tests passed.

I think this may be ready to merge.

bhbraswell · 2016-08-10T23:43:05Z

I just talked to @justinfisk about this. There were some questions about whether the update functionality could create a problem for a user who might want to maintain continuity with a thread of work that is an old version. I more or less BSed my way to an agreement with him that for RS data, you either always want the newer version, or for some reason possibly you don't care about the version. But I think in any case we should be on a path to get this into dev unless he has further comments.

third-party-tool · 2016-08-11T01:32:21Z

I bought the BS. This PR has the coveted fisk seal of approval.

ircwaves · 2016-08-11T01:51:50Z

I believe you mean the "highly coveted"

Bobby Braswell and others added 5 commits August 2, 2016 09:30

clean up data init. include version which represents file version and…

4c71867

… collection

update option added to inventory. update with fetch forces the fetch

db39189

specifying fetch and update will replace an older version with a newe…

eebbf88

…r one. fixed archive update

modis indices now can handle versions 5 or 6

6aeb89f

(Edited to resolve conflicts by tolson during rebase to branch `dev`.)

fixed modis version assignment

865313f

(Edited by tolson during rebase onto branch `dev`.)

44 revert unnecessary edits

1b7b21a

ra-tolson added 2 commits August 4, 2016 13:41

Merge branch 'dev' into m6-squash-try

c3286c6

44 bugfix & test updates for modis collection 6:

903235c

fix busted products by deactivating ndvi8, details: #70. MOD09Q1 is no longer needed so remove it from asset lists.

ircwaves closed this Aug 11, 2016

ircwaves reopened this Aug 11, 2016

ircwaves merged commit 23c0bad into dev Aug 11, 2016

ra-tolson mentioned this pull request Aug 23, 2016

handle new asset collections (e.g. MODIS collection 6) #44

Closed

ra-tolson deleted the 44-modis-6 branch January 5, 2017 16:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix for #44, add support for modis collection 6 (and retain support for 5) #68

Fix for #44, add support for modis collection 6 (and retain support for 5) #68

ra-tolson commented Aug 2, 2016

bhbraswell commented Aug 2, 2016

ra-tolson commented Aug 2, 2016

ra-tolson commented Aug 2, 2016 •

edited

Loading

ra-tolson commented Aug 3, 2016

bhbraswell commented Aug 3, 2016

ra-tolson commented Aug 3, 2016

ra-tolson commented Aug 4, 2016

bhbraswell commented Aug 10, 2016 •

edited

Loading

third-party-tool commented Aug 11, 2016

ircwaves commented Aug 11, 2016

Fix for #44, add support for modis collection 6 (and retain support for 5) #68

Fix for #44, add support for modis collection 6 (and retain support for 5) #68

Conversation

ra-tolson commented Aug 2, 2016

bhbraswell commented Aug 2, 2016

ra-tolson commented Aug 2, 2016

ra-tolson commented Aug 2, 2016 • edited Loading

ra-tolson commented Aug 3, 2016

bhbraswell commented Aug 3, 2016

ra-tolson commented Aug 3, 2016

ra-tolson commented Aug 4, 2016

bhbraswell commented Aug 10, 2016 • edited Loading

third-party-tool commented Aug 11, 2016

ircwaves commented Aug 11, 2016

ra-tolson commented Aug 2, 2016 •

edited

Loading

bhbraswell commented Aug 10, 2016 •

edited

Loading