Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

processing errors #754

Closed
fvanderbiest opened this issue May 3, 2017 · 4 comments
Closed

processing errors #754

fvanderbiest opened this issue May 3, 2017 · 4 comments
Assignees
@fvanderbiest
Milestone
2.1

Comments

@fvanderbiest
Copy link
Collaborator

On our dev server, make process yields

INFO     EQ-CUB-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 443 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (30, 21)

INFO     EQ-SAR-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 8364 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (464, 458)

INFO     EQ-US-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 3106 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (94, 90)

INFO     EQ-SOA-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 8365 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (486, 480)

INFO     EQ-SSA-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 1220 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (218, 217)

INFO     FL-TLS-WB
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 25 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 811, in rasterio._io.RasterReader.read (rasterio/_io.c:10482)
MemoryError

INFO     EQ-TWN-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 2 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (81, 79)

INFO     EQ-LEA-GEM
INFO       Cleaning previous outputs
INFO       Opening raster files
INFO       Iterating over 187 administrative divisions
INFO       Closing raster files
ERROR    Traceback (most recent call last):
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 78, in do_execute
    self.process_hazardset(id[0])
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 145, in process_hazardset
    outputs = self.create_outputs(hazardset)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 224, in create_outputs
    shape)
  File "/srv/apache/wb-thinkhazard/private/thinkhazard/thinkhazard/processing/processing.py", line 333, in notpreprocessed_hazardlevel
    data = reader.read(1, window=window, masked=True)
  File "rasterio/_io.pyx", line 752, in rasterio._io.RasterReader.read (rasterio/_io.c:9256)
  File "rasterio/_base.pyx", line 719, in rasterio._base.eval_window (rasterio/_base.c:15813)
  File "rasterio/_base.pyx", line 741, in rasterio._base.eval_window (rasterio/_base.c:15407)
ValueError: invalid window: row range (100, 96)
@stufraser1
Copy link
Member

We had a problem with the projection of the first set of GEM rasters delivered. This error looks to be the same issue. The errors were corrected in new layers; the layers on old geonode were replaced with ones that we then successfully processed in TH v1.

The transfer of data from old to new geonode may have been done before the problematic files were replaced.
Please copy the latest set of data from old geonode to new geonode, and try this again - the problem should be fixed

@fvanderbiest
Copy link
Collaborator Author

I'll try that, thanks !

@fvanderbiest fvanderbiest self-assigned this May 12, 2017
@arnaud-morvan arnaud-morvan mentioned this issue May 23, 2017
Merged
@arnaud-morvan
Copy link
Contributor

I've added, in complete task, a test on rasters. If bounds.bottom > bounds.top, hazardset is considered incomplete and this displayed in admin interface.

@fvanderbiest fvanderbiest mentioned this issue Jun 2, 2017
Closed
@stufraser1
Copy link
Member

@arnaud-morvan @fvanderbiest
is this fixed now? the above layers seem to show as imported to TH at http://int.thinkhazard.org/admin/hazardsets

@stufraser1 stufraser1 modified the milestone: 2.1 Aug 4, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fvanderbiest fvanderbiest

Labels
None yet
Projects
None yet
Milestone
2.1
Development

No branches or pull requests
3 participants
@fvanderbiest @arnaud-morvan @stufraser1