Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cpld_gridgen program does not work on WCOSS2. #663

Closed
GeorgeGayno-NOAA opened this issue Jul 7, 2022 · 12 comments
Closed

cpld_gridgen program does not work on WCOSS2. #663

GeorgeGayno-NOAA opened this issue Jul 7, 2022 · 12 comments
Assignees
Labels
bug Something isn't working

Comments

@GeorgeGayno-NOAA
Copy link
Collaborator

Something is not working. Looks like an ESMF error.

@GeorgeGayno-NOAA GeorgeGayno-NOAA added the bug Something isn't working label Jul 7, 2022
This was referenced Jul 7, 2022
@GeorgeGayno-NOAA
Copy link
Collaborator Author

I just ran the cpld_gridgen regression test on Cactus using 'develop' at ca9bed8. It ran to completion. I don't know what changed on WCOSS2 since I opened this issue. @DeniseWorthen, can you or someone on your team help me evaluate the results?

@DeniseWorthen
Copy link
Contributor

The tools I generally use to inspect the files are not available on wcoss2. Would you be able to put the files on Hera for me to evaluate?

@GeorgeGayno-NOAA
Copy link
Collaborator Author

The baseline data was moved from Hera to Cactus. Here are the file differences from the Hera baseline:

0.25-degree test

Variable  Group Count          Sum      AbsSum          Min         Max       Range         Mean      StdDev
land_frac /     90300 -7.76871e-12 7.71727e-10 -1.55509e-12 2.08733e-12 3.64242e-12 -8.60322e-17 1.74286e-14
Variable  Group Count         Sum     AbsSum          Min         Max       Range        Mean      StdDev
land_frac /     92755 2.12493e-12 6.3463e-10 -1.01841e-12 9.53015e-13 1.97142e-12 2.29091e-17 1.36747e-14
Variable  Group Count         Sum      AbsSum          Min         Max       Range         Mean      StdDev
land_frac /     72576 -3.7161e-12 5.19475e-10 -1.77858e-13 1.16018e-13 2.93876e-13 -5.12029e-17 1.04768e-14
Variable  Group  Count         Sum      AbsSum         Min         Max       Range         Mean      StdDev
land_frac /     132071 -1.3019e-11 1.02577e-09 -2.0195e-12 7.34524e-13 2.75402e-12 -9.85755e-17 1.72286e-14
Variable  Group  Count          Sum      AbsSum          Min         Max       Range         Mean      StdDev
land_frac /     109459 -4.13377e-12 8.34639e-10 -1.65123e-12 1.18383e-12 2.83507e-12 -3.77655e-17 1.49571e-14
Variable  Group  Count          Sum      AbsSum          Min         Max       Range         Mean      StdDev
land_frac /     127224 -4.77261e-13 9.84517e-10 -3.53051e-13 2.21378e-13 5.74429e-13 -3.75134e-18 1.07783e-14
Variable Group   Count          Sum      AbsSum          Min         Max       Range         Mean      StdDev
area_a   /     1524665  3.32988e-17 3.92867e-14 -2.37169e-19 2.40557e-19 4.77727e-19  2.18401e-23  3.5764e-20
area_b   /      861490  1.18491e-16 3.48869e-14 -3.45589e-19 3.50672e-19 6.96261e-19  1.37542e-22 5.16047e-20
frac_a   /      861607 -8.49742e-12 6.97487e-09 -2.20046e-12  2.3832e-12 4.58367e-12  -9.8623e-18 1.48218e-14
frac_b   /      624385 -2.69899e-11 4.77076e-09  -2.0195e-12 2.08733e-12 4.10683e-12 -4.32263e-17 1.44963e-14
S        /     3241434 -2.68928e-11 7.75689e-09 -2.01483e-12 2.08678e-12 4.10161e-12 -8.29657e-18 6.20992e-15
Variable Group Count         Sum      AbsSum          Min         Max       Range        Mean      StdDev
angle    /        13 1.97325e-16 3.57787e-16 -5.55112e-17 1.11022e-16 1.66533e-16 1.51788e-17 4.06852e-17
Variable Group Count        Sum      AbsSum          Min         Max       Range       Mean      StdDev
anglet   /        10 5.5728e-17 2.89916e-16 -1.11022e-16 1.11022e-16 2.22045e-16 5.5728e-18 5.52313e-17

@GeorgeGayno-NOAA
Copy link
Collaborator Author

0.5-degree test

Variable  Group Count         Sum      AbsSum          Min         Max       Range        Mean      StdDev
land_frac /     22623 8.23097e-13 1.03496e-10 -6.65246e-13 1.83853e-13 8.49099e-13 3.63832e-17 8.85007e-15
Variable  Group Count          Sum      AbsSum          Min         Max       Range         Mean      StdDev
land_frac /     23313 -1.42338e-12 8.93204e-11 -4.40981e-13 2.62901e-13 7.03881e-13 -6.10552e-17 7.98463e-15
Variable  Group Count         Sum      AbsSum          Min         Max       Range        Mean      StdDev
land_frac /     18510 5.03627e-13 6.48901e-11 -4.11893e-14 4.25215e-14 8.37108e-14 2.72084e-17 5.12935e-15
Variable  Group Count         Sum      AbsSum          Min         Max       Range        Mean      StdDev
land_frac /     32931 2.76971e-12 1.41934e-10 -3.03757e-13 4.46976e-13 7.50733e-13 8.41065e-17 8.71972e-15
Variable  Group Count         Sum      AbsSum          Min         Max       Range       Mean      StdDev
land_frac /     27326 1.71044e-12 1.15291e-10 -3.26517e-13 2.54019e-13 5.80536e-13 6.2594e-17 7.81463e-15
Variable  Group Count         Sum      AbsSum          Min        Max       Range       Mean      StdDev
land_frac /     31715 7.53296e-13 1.24314e-10 -6.43929e-14 6.9833e-14 1.34226e-13 2.3752e-17 5.39094e-15
Variable Group  Count         Sum      AbsSum          Min         Max       Range        Mean      StdDev
area_a   /     400289 1.35826e-17 2.03978e-14 -4.20128e-19 4.20128e-19 8.40257e-19 3.39319e-23 6.94282e-20
area_b   /     210354 8.12474e-18 1.78667e-14 -6.84403e-19 6.50521e-19 1.33492e-18 3.86241e-23 1.06665e-19
frac_a   /     237535 3.24762e-12 1.01431e-09 -7.28417e-13 5.10481e-13  1.2389e-12 1.36722e-17 8.40645e-15
frac_b   /     156418  5.1368e-12 6.39246e-10 -6.65246e-13 4.46976e-13 1.11222e-12 3.28402e-17 7.51749e-15
S        /     858157 5.07044e-12 1.02366e-09 -6.59806e-13 4.49418e-13 1.10922e-12 5.90852e-18 3.12237e-15
Variable Group Count         Sum      AbsSum         Min         Max Range        Mean StdDev
angle    /         1 2.77556e-17 2.77556e-17 2.77556e-17 2.77556e-17     0 2.77556e-17      0
Variable Group Count Sum AbsSum Min Max Range     Mean   StdDev
col      /        83 -11     83  -1   1     2 -0.13253 0.997204

@GeorgeGayno-NOAA
Copy link
Collaborator Author

1.0-degree test

Variable  Group Count         Sum      AbsSum          Min         Max       Range        Mean      StdDev
land_frac /      5680 5.56754e-14 1.41329e-11 -1.36002e-13 1.08025e-13 2.44027e-13 9.80201e-18 4.76013e-15
Variable  Group Count         Sum     AbsSum          Min        Max       Range      Mean      StdDev
land_frac /      5889 3.55519e-13 1.2329e-11 -7.22755e-14 1.0536e-13 1.77636e-13 6.037e-17 4.22046e-15
Variable  Group Count          Sum      AbsSum          Min         Max       Range        Mean      StdDev
land_frac /      4795 -1.54524e-13 8.46134e-12 -1.53211e-14 1.75415e-14 3.28626e-14 -3.2226e-17 2.56522e-15
Variable  Group Count         Sum      AbsSum          Min         Max       Range       Mean      StdDev
land_frac /      8133 8.80558e-14 1.89385e-11 -9.35918e-14 1.34226e-13 2.27818e-13 1.0827e-17 4.50742e-15
Variable  Group Count         Sum      AbsSum          Min         Max       Range       Mean      StdDev
land_frac /      6829 1.45027e-13 1.47667e-11 -7.39409e-14 8.29337e-14 1.56875e-13 2.1237e-17 3.92277e-15
Variable  Group Count         Sum      AbsSum          Min        Max       Range       Mean      StdDev
land_frac /      7875 5.54109e-13 1.59147e-11 -1.54321e-14 2.4647e-14 4.00791e-14 7.0363e-17 2.75835e-15
Variable Group  Count          Sum      AbsSum          Min         Max       Range         Mean      StdDev
area_a   /     107957 -1.96603e-16 1.13336e-14 -9.21572e-19 9.21572e-19 1.84314e-18 -1.82113e-21 1.44012e-19
area_b   /      50432 -1.67509e-17 9.67612e-15 -1.27394e-18 1.13841e-18 2.41235e-18 -3.32149e-22 2.35753e-19
frac_a   /      63534  3.31224e-12 1.49069e-10 -2.53575e-13 2.86993e-13 5.40568e-13  5.21333e-17 5.18187e-15
frac_b   /      39201  1.04386e-12  8.4543e-11 -1.36002e-13 1.34226e-13 2.70228e-13  2.66285e-17 3.89709e-15
S        /     223246  1.02333e-12 1.34134e-10 -1.32783e-13 1.34559e-13 2.67342e-13  4.58386e-18 1.57841e-15
Variable Group Count Sum AbsSum Min Max Range      Mean StdDev
col      /        15   1     15  -1   1     2 0.0666667 1.0328

@GeorgeGayno-NOAA
Copy link
Collaborator Author

The tools I generally use to inspect the files are not available on wcoss2. Would you be able to put the files on Hera for me to evaluate?

Yes, I can move the files to hera.

@GeorgeGayno-NOAA
Copy link
Collaborator Author

@DeniseWorthen Find the files on Hera: /scratch2/NCEPDEV/stmp1/George.Gayno/CPLD_GRIDGEN.wcoss2.4.denise

@DeniseWorthen
Copy link
Contributor

@GeorgeGayno-NOAA The land_frac differences (these are mapped ocean fractions on the atm tiles) look fine (I spot checked multiple files). The differences are not systematic.

I don't quite understand the report for the variable "col" --it should be a positive integer. It is use in mapping the ocean land mask (src_field) to the ATM tile (dst_field).

    dst_field = 0.0
    do i = 1,n_s
       ii = row(i); jj = col(i)
       dst_field(ii) = dst_field(ii) + S(i)*real(src_field(jj),dbl_kind)
    enddo

I don't understand why the 050 case should report the min as -1? Am I misunderstanding what it is reporting?

@GeorgeGayno-NOAA
Copy link
Collaborator Author

@GeorgeGayno-NOAA The land_frac differences (these are mapped ocean fractions on the atm tiles) look fine (I spot checked multiple files). The differences are not systematic.

I don't quite understand the report for the variable "col" --it should be a positive integer. It is use in mapping the ocean land mask (src_field) to the ATM tile (dst_field).

    dst_field = 0.0
    do i = 1,n_s
       ii = row(i); jj = col(i)
       dst_field(ii) = dst_field(ii) + S(i)*real(src_field(jj),dbl_kind)
    enddo

I don't understand why the 050 case should report the min as -1? Am I misunderstanding what it is reporting?

The report lists the differences between the baseline file (brought over from Hera) and the WCOSS2 computed file. That is why 'col' can be negative. The number of 'col' points that are different is a very small percentage of the total. For the 0.5-degree case, the number of points different is 83 and the total number of points is 867,529.

@GeorgeGayno-NOAA
Copy link
Collaborator Author

@GeorgeGayno-NOAA The land_frac differences (these are mapped ocean fractions on the atm tiles) look fine (I spot checked multiple files). The differences are not systematic.
I don't quite understand the report for the variable "col" --it should be a positive integer. It is use in mapping the ocean land mask (src_field) to the ATM tile (dst_field).

    dst_field = 0.0
    do i = 1,n_s
       ii = row(i); jj = col(i)
       dst_field(ii) = dst_field(ii) + S(i)*real(src_field(jj),dbl_kind)
    enddo

I don't understand why the 050 case should report the min as -1? Am I misunderstanding what it is reporting?

The report lists the differences between the baseline file (brought over from Hera) and the WCOSS2 computed file. That is why 'col' can be negative. The number of 'col' points that are different is a very small percentage of the total. For the 0.5-degree case, the number of points different is 83 and the total number of points is 867,529.

@DeniseWorthen Does my explanation make sense? If so, I would like to close this issue. And turn on the daily regression testing on WCOSS2.

@DeniseWorthen
Copy link
Contributor

Yes, it does. I think you can close the issue. Thanks.

@GeorgeGayno-NOAA GeorgeGayno-NOAA self-assigned this Sep 19, 2022
@GeorgeGayno-NOAA
Copy link
Collaborator Author

Sometime after this issue was opened, the cpld_gridgen program started working on WCOSS2. I can't explain it. Will reopen this issue if the problem returns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants