Cannot convert float NaN to integer in offwind renewable profiles for NZ

[davide-f]

Checklist

• I am using the current main branch or the latest release. Please indicate.
• I am running on an up-to-date pypsa-earth environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

When running the workflow for NZ the following error message is thrown.
This may be an issue to be notified to atlite PyPSA/atlite#282

Error Message

If applicable, paste any terminal output to help illustrating your problem.
In some cases it may also be useful to share your list of installed packages: conda list.

Traceback (most recent call last):
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/atlite/gis.py", line 537, in _process_func
    return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/atlite/gis.py", line 512, in shape_availability_reprojected
    masked, transform = shape_availability(geometry, excluder)
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/atlite/gis.py", line 452, in shape_availability
    masked, transform = projected_mask(
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/atlite/gis.py", line 366, in projected_mask
    masked, transform_ = mask(raster, geom, crop=True, **kwargs)
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/rasterio/mask.py", line 178, in mask
    shape_mask, transform, window = raster_geometry_mask(
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/rasterio/mask.py", line 80, in raster_geometry_mask
    window = geometry_window(dataset, shapes, pad_x=pad_x, pad_y=pad_y)
  File "/home/davidef/miniconda3/envs/pypsa-earth/lib/python3.8/site-packages/rasterio/features.py", line 460, in geometry_window
    row_start, row_stop = int(math.floor(min(rows))), int(math.ceil(max(rows)))
ValueError: cannot convert float NaN to integer
"""

[euronion]

Can you share the config and snakemake command which trigger the issue?

[davide-f]

Sure! I'm sending it in discord.
I'll soon be repeating the scenario, I've deleted the folder.
In case I may send you the entire folder, however I'm using a world cutout (380Gb...) the config I'm sending has build_cutout enabled.

The snakemake command is snakemake -j all solve_all_networks

[davide-f]

My feeling is that this issue has something to do with these white bands:

[davide-f]

By doing an additional check, it is clear that some data from copernicus are nans, those most likely trigger the issue

A possible fix could be to fill nans with zeros.

What do you think @euronion ? may this feature be interesting for atlite?

[euronion]

Let's please first track down where the problem is coming from, instead of filling with zero values. cdsapi shouldn't deliver nan values if working properly.

Did you check the data from cdsapi directly? Or did you look at a freshly prepared cutout?

[davide-f]

That was the data saved into cutout and it happened twice: it's the second time that I download this type of data with missing values.
They should be missing values in copernicus.

However I am unsure whether this can be generated by atlite, personally I don't think so.

This is also known link

[euronion]

I'm asking because we have another issue currently under investigation with atlite related to xarray creating NaN values where no NaN should be see SO.

If I understand the link you sent correctly, it's about a different dataset and observational data, not ERA5 reanalysis. The whole point of using reanalysis is, that values for each grid are calculated using climate models. NaNs shouldn't occur.

[euronion]

Could you directly use the cdsapi and try to retrieve some data for the affected grid points? See if they are also NaN if you skip atlite?

Adjust the request below as you see fit:

import cdsapi

c = cdsapi.Client()

c.retrieve(
    'reanalysis-era5-single-levels',
    {
        'product_type': 'reanalysis',
        'format': 'netcdf',
        'variable': [
            '100m_u_component_of_wind', '100m_v_component_of_wind',
        ],
        'month': [
            '01', '02', '03',
            '04', '05', '06',
            '07', '08', '09',
            '10', '11', '12',
        ],
        'day': [
            '01', '02', '03',
            '04', '05', '06',
            '07', '08', '09',
            '10', '11', '12',
            '13', '14', '15',
            '16', '17', '18',
            '19', '20', '21',
            '22', '23', '24',
            '25', '26', '27',
            '28', '29', '30',
            '31',
        ],
        'time': [
            '00:00', '01:00', '02:00',
            '03:00', '04:00', '05:00',
            '06:00', '07:00', '08:00',
            '09:00', '10:00', '11:00',
            '12:00', '13:00', '14:00',
            '15:00', '16:00', '17:00',
            '18:00', '19:00', '20:00',
            '21:00', '22:00', '23:00',
        ],
        'year': '2013',
        'area': [
            90, -150, -90,
            140,
        ],
    },
    'download.nc')

[davide-f]

I think this issue is outdated; a simple fix in atlite has been added to account for this.
I'm closing: if this problem appears, let's open it back