Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

encountered errors applying CMEMS (zos) data to py-eddy-tracker identification #207

Closed
doge-py opened this issue Jul 25, 2023 · 8 comments
Closed

encountered errors applying CMEMS (zos) data to py-eddy-tracker identification #207

doge-py opened this issue Jul 25, 2023 · 8 comments

Comments

@doge-py
Copy link

doge-py commented Jul 25, 2023

Hello,
I'm trying to apply adt data from cmems GLOBAL_MULTIYEAR_PHY_001_030 to py-eddy-tracker identification, but encounter error as following:

error:

WARNING 2023-07-25 09:49:47,698 grid.__init__ :
        We assume pixel position of grid is centered for /home/user/mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc
        DEBUG 2023-07-25 09:49:47,698   grid.   load_general_features :
                Load general feature from /home/user/mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc
        DEBUG 2023-07-25 09:49:47,704   grid.   bessel_high_filter :
                Run filtering with wavelength of 500 km and order of 3 ...
        DEBUG 2023-07-25 09:49:47,704   grid.   grid :
                Load surf_el from /home/user/mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc
Remain  0:00:00.045110 ETA  2023-07-25 09:49:55.191888 current kernel size : (443, 337) Step : 125/126
        DEBUG 2023-07-25 09:49:55,194   grid.   bessel_high_filter :
                Filtering done
INFO 2023-07-25 09:49:55,195 grid.add_uv :
        Add u/v variable with stencil method
        DEBUG 2023-07-25 09:49:55,195   grid.   compute_stencil :
                Stencil half width apply : 4
        DEBUG 2023-07-25 09:49:55,198   grid.   compute_stencil :
                Stencil half width apply : 4
INFO 2023-07-25 09:49:55,924 grid.eddy_identification :
        We will apply on step a factor to be coherent with grid : 1.000000
/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/numpy/lib/function_base.py:4737: UserWarning: Warning: 'partition' will ignore the 'mask' of the MaskedArray.
  arr.partition(
        DEBUG 2023-07-25 09:49:55,927   grid.   eddy_identification :
                Levels from -0.160388 to 0.145720
INFO 2023-07-25 09:49:55,927 eddy_feature.__init__ :
        Start computing iso lines
        DEBUG 2023-07-25 09:49:55,945   eddy_feature.   __init__ :
                X shape : (251,)
        DEBUG 2023-07-25 09:49:55,946   eddy_feature.   __init__ :
                Y shape : (126,)
        DEBUG 2023-07-25 09:49:55,946   eddy_feature.   __init__ :
                Z shape : (251, 126)
INFO 2023-07-25 09:49:55,946 eddy_feature.__init__ :
        Start computing iso lines with 156 levels from -0.162000 to 0.148000 ...
INFO 2023-07-25 09:49:56,153 eddy_feature.__init__ :
        Finish computing iso lines
INFO 2023-07-25 09:49:56,169 eddy_feature.__init__ :
        Repair 79 closed contours and 0 almost closed contours / 1026 contours
        DEBUG 2023-07-25 09:49:56,172   grid.   eddy_identification :
                doing collection 1, contour value -0.1600, 1 paths
        DEBUG 2023-07-25 09:49:56,188   grid.   eddy_identification :
                doing collection 2, contour value -0.1580, 1 paths
        DEBUG 2023-07-25 09:49:56,201   grid.   eddy_identification :
                doing collection 3, contour value -0.1560, 1 paths
        DEBUG 2023-07-25 09:49:56,201   grid.   eddy_identification :
                doing collection 4, contour value -0.1540, 1 paths
        DEBUG 2023-07-25 09:49:56,201   grid.   eddy_identification :
                doing collection 5, contour value -0.1520, 1 paths
        DEBUG 2023-07-25 09:49:56,201   grid.   eddy_identification :
                doing collection 6, contour value -0.1500, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 7, contour value -0.1480, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 8, contour value -0.1460, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 9, contour value -0.1440, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 10, contour value -0.1420, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 11, contour value -0.1400, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 12, contour value -0.1380, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 13, contour value -0.1360, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 14, contour value -0.1340, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 26, contour value -0.1100, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 27, contour value -0.1080, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 28, contour value -0.1060, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 29, contour value -0.1040, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 30, contour value -0.1020, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 31, contour value -0.1000, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 32, contour value -0.0980, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 33, contour value -0.0960, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 34, contour value -0.0940, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 35, contour value -0.0920, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 36, contour value -0.0900, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 37, contour value -0.0880, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 38, contour value -0.0860, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 39, contour value -0.0840, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 40, contour value -0.0820, 2 paths
        DEBUG 2023-07-25 09:49:56,206   grid.   eddy_identification :
                doing collection 41, contour value -0.0800, 3 paths
        DEBUG 2023-07-25 09:49:56,206   grid.   eddy_identification :
                doing collection 42, contour value -0.0780, 2 paths
        DEBUG 2023-07-25 09:49:56,206   grid.   eddy_identification :
                doing collection 43, contour value -0.0760, 2 paths
        DEBUG 2023-07-25 09:49:56,207   grid.   eddy_identification :
                doing collection 44, contour value -0.0740, 2 paths
        DEBUG 2023-07-25 09:49:56,207   grid.   eddy_identification :
                doing collection 45, contour value -0.0720, 3 paths
        DEBUG 2023-07-25 09:49:56,207   grid.   eddy_identification :
                doing collection 46, contour value -0.0700, 3 paths
        DEBUG 2023-07-25 09:49:56,207   grid.   eddy_identification :
                doing collection 47, contour value -0.0680, 6 paths
        DEBUG 2023-07-25 09:49:56,208   grid.   eddy_identification :
                doing collection 48, contour value -0.0660, 9 paths
        DEBUG 2023-07-25 09:49:56,208   grid.   eddy_identification :
                doing collection 49, contour value -0.0640, 9 paths
        DEBUG 2023-07-25 09:49:56,209   grid.   eddy_identification :
                doing collection 50, contour value -0.0620, 10 paths
        DEBUG 2023-07-25 09:49:56,210   grid.   eddy_identification :
                doing collection 51, contour value -0.0600, 9 paths
        DEBUG 2023-07-25 09:49:56,210   grid.   eddy_identification :
                doing collection 52, contour value -0.0580, 12 paths
        DEBUG 2023-07-25 09:49:56,211   grid.   eddy_identification :
                doing collection 53, contour value -0.0560, 11 paths
        DEBUG 2023-07-25 09:49:56,212   grid.   eddy_identification :
                doing collection 54, contour value -0.0540, 11 paths
        DEBUG 2023-07-25 09:49:56,212   grid.   eddy_identification :
                doing collection 55, contour value -0.0520, 9 paths
        DEBUG 2023-07-25 09:49:56,213   grid.   eddy_identification :
                doing collection 56, contour value -0.0500, 11 paths
        DEBUG 2023-07-25 09:49:56,214   grid.   eddy_identification :
                doing collection 57, contour value -0.0480, 11 paths
        DEBUG 2023-07-25 09:49:56,215   grid.   eddy_identification :
                doing collection 58, contour value -0.0460, 15 paths
        DEBUG 2023-07-25 09:49:56,216   grid.   eddy_identification :
                doing collection 59, contour value -0.0440, 16 paths
        DEBUG 2023-07-25 09:49:56,217   grid.   eddy_identification :
                doing collection 60, contour value -0.0420, 21 paths
        DEBUG 2023-07-25 09:49:56,218   grid.   eddy_identification :
                doing collection 61, contour value -0.0400, 21 paths
        DEBUG 2023-07-25 09:49:56,220   grid.   eddy_identification :
                doing collection 62, contour value -0.0380, 19 paths
        DEBUG 2023-07-25 09:49:56,221   grid.   eddy_identification :
                doing collection 63, contour value -0.0360, 22 paths
        DEBUG 2023-07-25 09:49:56,223   grid.   eddy_identification :
                doing collection 64, contour value -0.0340, 18 paths
        DEBUG 2023-07-25 09:49:56,225   grid.   eddy_identification :
                doing collection 65, contour value -0.0320, 17 paths
        DEBUG 2023-07-25 09:49:56,226   grid.   eddy_identification :
                doing collection 66, contour value -0.0300, 18 paths
        DEBUG 2023-07-25 09:49:56,228   grid.   eddy_identification :
                doing collection 67, contour value -0.0280, 14 paths
        DEBUG 2023-07-25 09:49:56,230   grid.   eddy_identification :
                doing collection 68, contour value -0.0260, 13 paths
        DEBUG 2023-07-25 09:49:56,240   grid.   eddy_identification :
                doing collection 69, contour value -0.0240, 11 paths
        DEBUG 2023-07-25 09:49:56,241   grid.   eddy_identification :
                doing collection 70, contour value -0.0220, 12 paths
        DEBUG 2023-07-25 09:49:56,242   grid.   eddy_identification :
                doing collection 71, contour value -0.0200, 10 paths
        DEBUG 2023-07-25 09:49:56,243   grid.   eddy_identification :
                doing collection 72, contour value -0.0180, 8 paths
        DEBUG 2023-07-25 09:49:56,244   grid.   eddy_identification :
                doing collection 73, contour value -0.0160, 5 paths
        DEBUG 2023-07-25 09:49:56,244   grid.   eddy_identification :
                doing collection 74, contour value -0.0140, 5 paths
        DEBUG 2023-07-25 09:49:56,244   grid.   eddy_identification :
                doing collection 75, contour value -0.0120, 3 paths
        DEBUG 2023-07-25 09:49:56,245   grid.   eddy_identification :
                doing collection 76, contour value -0.0100, 5 paths
        DEBUG 2023-07-25 09:49:56,245   grid.   eddy_identification :
                doing collection 77, contour value -0.0080, 6 paths

Traceback (most recent call last):
  File "/home/user/eddy_identification.py", line 19, in <module>
    a, c = h.eddy_identification(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 849, in eddy_identification
    ) = self.get_uavg(
        ^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 996, in get_uavg
    max_average_speed = self.speed_coef_mean(original_contour)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 1911, in speed_coef_mean
    return mean_on_regular_contour(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/numba/core/dispatcher.py", line 468, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/numba/core/dispatcher.py", line 409, in error_rewrite
    raise e.with_traceback(None)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<built-in function getitem>) found for signature:

 >>> getitem(bool, UniTuple(int64 x 2))

There are 22 candidate implementations:
      - Of which 22 did not match due to:
      Overload of function 'getitem': File: <numerous>: Line N/A.
        With argument(s): '(bool, UniTuple(int64 x 2))':
       No match.

During: typing of intrinsic-call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py (280)

File "../../../mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py", line 280:
def interp2d_bilinear(x_g, y_g, z_g, m_g, x, y):
    <source elided>
            if not masked:
                if m_g[i0, j0] or m_g[i0, j1] or m_g[i1, j0] or m_g[i1, j1]:
                ^

During: resolving callee type: type(CPUDispatcher(<function interp2d_bilinear at 0x7efed0ffdc60>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py (201)

During: resolving callee type: type(CPUDispatcher(<function interp2d_bilinear at 0x7efed0ffdc60>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py (201)


File "../../../mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py", line 201:
def interp2d_geo(x_g, y_g, z_g, m_g, x, y, nearest=False):
    <source elided>
    else:
        return interp2d_bilinear(x_g, y_g, z_g, m_g, x, y)
        ^

During: resolving callee type: type(CPUDispatcher(<function interp2d_geo at 0x7efed0ffd620>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py (130)

During: resolving callee type: type(CPUDispatcher(<function interp2d_geo at 0x7efed0ffd620>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py (130)


File "../../../mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 130:
def mean_on_regular_contour(
    <source elided>
    x_new, y_new = uniform_resample(x_val, y_val, num_fac, fixed_size)
    values = interp2d_geo(x_g, y_g, z_g, m_g, x_new[1:], y_new[1:])

my code:

#Activate verbose

from py_eddy_tracker import start_logger

start_logger().setLevel("DEBUG")  # Available options: ERROR, WARNING, INFO, DEBUG
#Run identification

from datetime import datetime
from py_eddy_tracker.dataset.grid import RegularGridDataset

grid_name = "mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc"
lon_name = "longitude"
lat_name = "latitude"

h = RegularGridDataset(grid_name, lon_name, lat_name)
h.bessel_high_filter("zos", 500, order=3)
date = datetime(2011, 1, 1)
h.add_uv("zos","ugos","vgos")
a, c = h.eddy_identification(
    "zos",
    "ugos",
    "vgos",  # Variables used for identification
    date,  # Date of identification
    0.002,  # step between two isolines of detection (m)
    pixel_limit=(5, 2000),  # Min and max pixel count for valid contour
    shape_error=55,  # Error max (%) between ratio of circle fit and contour
)
#Save identification data

from netCDF import Dataset

with Dataset(date.strftime("Anticyclonic_%Y%m%d.nc"), "w") as h:
    a.to_netcdf(h)
with Dataset(date.strftime("Cyclonic_%Y%m%d.nc"), "w") as h:
    c.to_netcdf(h)

other information:

numpy version : 1.24.4
numba version : 0.57.1
python version : 3.11.3

Best regards,

doge
@doge-py doge-py changed the title encounter error in eddy identification using cmems data encountered errors applying CMEMS (zos) data to py-eddy-tracker identification Jul 25, 2023
@doge-py
Copy link
Author

doge-py commented Jul 27, 2023 •
edited
Loading

after changing acess data method from MOTU to ftp, got the same errors from here Unit error block EddyId #2.

July 27, 2023
after some research, problem solved when I add force_speed_unit="m/s" to eddy_identification, but I still want to know how to apply data obtained from MOTU because it seems that ftp can't subset data variables? (not quite sure about it)

new code:

#Activate verbose

from py_eddy_tracker import start_logger

start_logger().setLevel("DEBUG")  # Available options: ERROR, WARNING, INFO, DEBUG
#Run identification

from datetime import datetime
from py_eddy_tracker.dataset.grid import RegularGridDataset

grid_name = "mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc"
lon_name = "longitude"
lat_name = "latitude"

h = RegularGridDataset(grid_name, lon_name, lat_name)
h.bessel_high_filter("zos", 500, order=3)
date = datetime(2011, 1, 1)
h.add_uv("zos","ugos","vgos")
a, c = h.eddy_identification(
    "zos",
    "ugos",
    "vgos",  # Variables used for identification
    date,  # Date of identification
    0.002,  # step between two isolines of detection (m)
    pixel_limit=(5, 2000),  # Min and max pixel count for valid contour
    shape_error=55,  # Error max (%) between ratio of circle fit and contour
    force_speed_unit="m/s"
)
#Save identification data

from netCDF import Dataset

with Dataset(date.strftime("Anticyclonic_%Y%m%d.nc"), "w") as h:
    a.to_netcdf(h)
with Dataset(date.strftime("Cyclonic_%Y%m%d.nc"), "w") as h:
    c.to_netcdf(h)

Best regards,

doge

@AntSimi
Copy link
Owner

AntSimi commented Aug 9, 2023

MOTU do some modifications from original file, last time i check this tools create a mix of nan and masked value.
Maybe you could try to use 'nan_masking' options #111 (comment)

@doge-py
Copy link
Author

doge-py commented Aug 9, 2023

I followed your suggestions and add "nan_masking=True" to the code, but unfortunately, the issue still persists and got exactly same error message.
I'm thankful for your assistance.

@AntSimi
Copy link
Owner

AntSimi commented Aug 10, 2023

could you share your file in order to reproduce bug?

@doge-py
Copy link
Author

doge-py commented Aug 11, 2023

I subsetted it as following:
time: 2020,12,31 ~ 2020,12,31
lon: 67 ~ 105
lat: -37 ~ -29
elevation: -0.49 ~ -0.49
variable: uo, vo, zos

AntSimi added a commit that referenced this issue Sep 27, 2023
@AntSimi
issue #207 change speed compute formula to avoid mask shrinking
73b017c
@AntSimi
Copy link
Owner

AntSimi commented Sep 27, 2023

with latest version you must avoid this exception with& without nan_masking option

@doge-py
Copy link
Author

doge-py commented Dec 3, 2023

It worked; I really appreciate your help!

@doge-py doge-py closed this as completed Dec 3, 2023
@SMLZzzzzzz
Copy link

mark，nice reply!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@AntSimi @doge-py @SMLZzzzzzz