Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mask_landsea fails when using 2D fx data on 4D cube data #1349

Closed
valeriupredoi opened this issue Oct 8, 2021 · 6 comments · Fixed by #1350
Closed

mask_landsea fails when using 2D fx data on 4D cube data #1349

valeriupredoi opened this issue Oct 8, 2021 · 6 comments · Fixed by #1350
Assignees
@valeriupredoi @sloosvel
Labels
bug Something isn't working

Comments

@valeriupredoi
Copy link
Contributor

Minimal recipe

documentation:
  description: |
    Various example preprocessors.
  authors:
    - righi_mattia
  title: Test recipe for ESMValCore preprocessors.
  maintainer:
    - righi_mattia
  projects:
    - c3s-magic
datasets:
  - {dataset: bcc-csm1-1, project: CMIP5, exp: historical, ensemble: r1i1p1}
preprocessors:
  preprocessor_1:
    extract_levels:
      levels: bcc-csm1-1
      scheme: linear
    mask_landsea:
      mask_out: land
diagnostics:
  diagnostic_1: &diag
    description: Preprocessor test diagnostic.
    variables:
      ta:
        preprocessor: preprocessor_1
        mip: Amon
        start_year: 2000
        end_year: 2002
      orog:
        mip: fx
      sftlf:
        mip: fx
    scripts: null

Trace (full)

2021-10-08 12:44:56,533 UTC [25166] ERROR   Failed to run mask_landsea(air_temperature / (K)               (time: 36; air_pressure: 17; latitude: 64; longitude: 128)
    Dimension coordinates:
        time                             x                 -             -              -
        air_pressure                     -                 x             -              -
        latitude                         -                 -             x              -
        longitude                        -                 -             -              x
    Ancillary variables:
        land_area_fraction               -                 -             x              x
    Cell methods:
        mean                        time (20 mintues)
    Attributes:
        Conventions                 CF-1.4
        associated_files            baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_bcc-csm1-1_historical_r0i0p0.nc...
        branch_time                 470.0
        cmor_version                2.5.6
        contact                     Dr. Tongwen Wu (twwu@cma.gov.cn)
        experiment                  historical
        experiment_id               historical
        forcing                     Nat Ant GHG SD Oz Sl Vl SS Ds BC OC
        frequency                   mon
        initialization_method       1
        institute_id                BCC
        institution                 Beijing Climate Center(BCC),China Meteorological Administration,China
        model_id                    bcc-csm1-1
        modeling_realm              atmos
        original_name               T
        parent_experiment           pre-industrial control
        parent_experiment_id        piControl
        parent_experiment_rip       r1i1p1
        physics_version             1
        product                     output
        project_id                  CMIP5
        realization                 1
        source                      bcc-csm1-1:atmosphere:  BCC_AGCM2.1 (T42L26); land: BCC_AVIM1.0;ocean:...
        table_id                    Table Amon (11 April 2011) 1cfdc7322cf2f4a32614826fab42c1ab
        title                       bcc-csm1-1 model output prepared for CMIP5 historical, {'mask_out': 'land'})
2021-10-08 12:44:56,682 UTC [25155] INFO    Maximum memory used (estimate): 0.7 GB
2021-10-08 12:44:56,684 UTC [25155] INFO    Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2021-10-08 12:44:56,685 UTC [25155] ERROR   Program terminated abnormally, see stack trace below for more information:
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 780, in _run_task
    output_files = task.run()
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 253, in run
    self.output_files = self._run(input_files)
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 489, in _run
    product.apply(step, self.debug)
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 356, in apply
    self.cubes = preprocess(self.cubes, step, **self.settings[step])
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 302, in preprocess
    result.append(_run_preproc_function(function, item, settings))
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 285, in _run_preproc_function
    return function(items, **kwargs)
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/_mask.py", line 118, in mask_landsea
    fx_cube.data = da.broadcast_to(fx_cube.core_data(), cube.shape)
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/iris/coords.py", line 740, in data
    self._values = data
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/iris/coords.py", line 241, in _values
    self._values_dm.data = values
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/iris/_data_manager.py", line 259, in data
    raise ValueError(emsg.format(self.shape, data.shape))
ValueError: Require data with shape (64, 128), got (36, 17, 64, 128).
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/valeriu/ESMValCore/esmvalcore/_main.py", line 479, in run
    fire.Fire(ESMValTool())
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/valeriu/ESMValCore/esmvalcore/_main.py", line 423, in run
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/home/valeriu/ESMValCore/esmvalcore/_main.py", line 125, in process_recipe
    recipe.run()
  File "/home/valeriu/ESMValCore/esmvalcore/_recipe.py", line 1557, in run
    self.tasks.run(max_parallel_tasks=self._cfg['max_parallel_tasks'])
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 709, in run
    self._run_parallel(max_parallel_tasks)
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 752, in _run_parallel
    _copy_results(task, running[task])
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 775, in _copy_results
    task.output_files, task.products = future.get()
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 780, in _run_task
    output_files = task.run()
  File "/home/valeriu/ESMValCore/esmvalcore/_task.py", line 253, in run
    self.output_files = self._run(input_files)
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 489, in _run
    product.apply(step, self.debug)
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 356, in apply
    self.cubes = preprocess(self.cubes, step, **self.settings[step])
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 302, in preprocess
    result.append(_run_preproc_function(function, item, settings))
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/__init__.py", line 285, in _run_preproc_function
    return function(items, **kwargs)
  File "/home/valeriu/ESMValCore/esmvalcore/preprocessor/_mask.py", line 118, in mask_landsea
    fx_cube.data = da.broadcast_to(fx_cube.core_data(), cube.shape)
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/iris/coords.py", line 740, in data
    self._values = data
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/iris/coords.py", line 241, in _values
    self._values_dm.data = values
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-stock/lib/python3.9/site-packages/iris/_data_manager.py", line 259, in data
    raise ValueError(emsg.format(self.shape, data.shape))
ValueError: Require data with shape (64, 128), got (36, 17, 64, 128).
2021-10-08 12:44:56,690 UTC [25155] INFO    
If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions
If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues
To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory.

Issue in a nutshell

fx data has shape: (64, 128) and data cube has shape: (36, 17, 64, 128); in _mask.py l.117 (not 118 as in the trace, I added a line at pos 117 inspect the two cubes)

fx_cube.data = da.broadcast_to(fx_cube.core_data(), cube.shape)

fails to broadcast fx data onto cube data.
@valeriupredoi valeriupredoi added the bug Something isn't working label Oct 8, 2021
@valeriupredoi
Copy link
Contributor Author

iris version: iris 3.1.0 py39hf3d152e_1 conda-forge

@valeriupredoi
Copy link
Contributor Author

looks like iris is not permitting reshaping: in lib/python3.9/site-packages/iris/_data_manager.py l.259

        if init_done and self.shape != data.shape:
            # The _ONLY_ data reshape permitted is converting a 0-dimensional
            # array i.e. self.shape == () into a 1-dimensional array of length
            # one i.e. data.shape == (1,)
            if self.shape or data.shape != (1,):
                emsg = "Require data with shape {!r}, got {!r}."
                raise ValueError(emsg.format(self.shape, data.shape))

@zklaus
Copy link

zklaus commented Oct 8, 2021

Well, yes, the shape of a cube can not be changed like this.

@valeriupredoi
Copy link
Contributor Author

yip, but broadcasting the fx data onto the shape of the data cube should be allowed no?

@zklaus
Copy link

zklaus commented Oct 8, 2021

Yes. I think a straight-forward fix is as follows:
Since fx_cube.data effectively is only a temporary variable here, just use it as such, i.e. s/fx_cube.data/fx_cube_data/ in L117-118. Beware that the same problem happens a little later in the file, so perhaps best to search for broadcast_to. Also, we should add a test with different size cubes.

@valeriupredoi
Copy link
Contributor Author

right on, Klaus! work started in #1350

@valeriupredoi valeriupredoi mentioned this issue Oct 8, 2021
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@valeriupredoi valeriupredoi

@sloosvel sloosvel

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
3 participants
@zklaus @valeriupredoi @sloosvel