ASCII2NC error converting to a usable nc file

[YakelynRJ]

Hello, I am trying to convert a txt file to a NetCDF file so I can use the point-stat tool.
I believe I have to create a new use case given that the test cases provided in the tutorial are not working for my purposes.
I have created a txt file in python following the instructions and Table 4.4 in the Model Evaluation Tools Version 9.0.2 User's Guide. This txt file comes from this website (https://data.pmel.noaa.gov/pmel/erddap/tabledap/saildrone_2019_arctic_flux.html) and it doesn't have information on elevation, Level, Height, QC_String so I put NA in the txt file creation. I am attaching a txt file as an example, this data corresponds to a Sensible heat flux which I believe corresponds to a grib_code = 122.

I am trying to modify the Ascii2NcConfig_default and getting it to work but it doesn't.
I am running ascii2nc like this:

ascii2nc
saildrone_sensible_2019.txt
sensible_test.nc
-format python
-config Ascii2NcConfig
-v 2

Some help would be appreciated.
saildrone_sensible_2019.txt

[j-opatz]

Hello, and thank you for your question. It looks like you've already done a good job of getting the data into a proper format for ASCII2NC. In your post, it seems like the error is in relation to a character space that makes the script try and read line 14 as a line of code, which is possibly the fault of line 1 where you have "Ascii2NcConfig/usr/local/share//usr/local/share/////" without a "//" at the beginning of the line to comment it out.

I also saw you're utilizing the MET User's Guide for version 9.0.4. I strongly recommend that you begin using the latest version of the User's Guide, which is online for your convenience.

Luckily, the ASCII2NC tool does not require the configuration file to run. And the -format python flag should only be used if you are planning to run your python script over the data first to get it in MET-acceptable format (which it sounds like you are, but the file you provided was already MET-acceptable). When I ran the command ascii2nc saildrone_sensible_2019.txt sensible_test.nc -v 2 on the file you provided, the tool ran without any issues. Try it without the configuration file (and without the -format flag, if you're not going to use Python during runtime) and see if that helps.

ascii2nc saildrone_sensible_2019.txt saildrone_sensible_2019.nc
DEBUG 1: Config File: /Volumes/d1/projects/MET/MET_development/MET-develop/met/share/met/config/Ascii2NcConfig_default
DEBUG 2: Rejected 0 observations off the masking grid.
DEBUG 2: Rejected 0 observations outside the masking polyline.
DEBUG 2: Rejected 0 observations not matched with station ID's.
DEBUG 2: Processing observations for 4871 headers.
DEBUG 1: Creating NetCDF Observation file: saildrone_sensible_2019.nc
DEBUG 2: Finished processing 4871 observations for 4871 headers.

[YakelynRJ]

Thank you!
You were right, the command ascii2nc saildrone_sensible_2019.txt sensible_test.nc -v 2 worked!
Now, I am trying to run point-stat. So, I obtained data from ECMWF:

• link: (https://apps.ecmwf.int/datasets/data/tigge/levtype=sfc/type=cf/)
• Type of level: Surface
• Type: Controle forecast
• Month: June - October 2019
• Origin and time: ECMWF, 00:00:00
• Step: all
• Parameter: Surface latent heat flux, surface sensible heat flux

I obtained two nc files using ascii2nc, one for latent heat flux and another for sensible heat fluxes. A few station IDs are on these files, and I am trying to
PointStatConfig_tutorial_run1.txt
pair all saildrone data with the ECMWF and obtain statistics.
I have noticed that the configuration file asks for climatology files but I don't want that.

I am attaching my configuration file and the screenshot of my error.

Thanks in advance for any help.

[JohnHalleyGotway]

@YakelynRJ I apologize for the VERY long delay in responding to this issue! @TaraJensen notified us about this orphaned issue. We're still figuring out these sort of details after transitioning from direct email support (met_help@ucar.edu) over to GitHub discussions. Once we answered your initial question about ascii2nc, I marked it as being ANSWERED which caused us to miss your follow up question about point_stat. Of course, it's often the case that one question leads to another!

Our team has decided to start LOCKING conversations once they've come to a natural conclusion and the user has confirmed the answer. That should help us avoid this in the future. Locking a discussion prevents future posts and encourages users to ask new questions in new discussions. Our hope is that keeping each discussion topic as focussed as possible will make them more helpful to other users.

But that's a digression. On to your question about Point-Stat. Basically, you're requesting that Point-Stat read "LHTFL/L0" data from an ECMWF GRIB2 file named forecast_october_2019.grib2 but MET can't find the requested data in that file. Whenever getting started with new datasets in MET, I always start by running MET's plot_data_plane utility. That confirms that MET can read data for the "name" and "level" requested, that it plots the data at the right spot on the earth, and that its oriented correctly.

For GRIB2 data, there are seldom issues with the grid definition or orientation. Those issues are more common for custom NetCDF files. Regardless, plot_data_plane is a quick and easy way of getting the name and level settings right. But I need some sample data for testing. So I created an account with ECMWF and followed your instructions to retrieve data from apps.ecmwf.int/datasets/data/tigge/levtype=sfc/type=cf.

Data for the month of October 2019 contains a whopping 3782 records. Here's the first 5 of them, as reported by wgrib2:

wgrib2 LHTFL_SHTFL_October_2019.grib2 | head -5
1:0:d=2019100100:SHTFL:surface:0-0 day acc fcst:ENS=low-res ctl
2:211:d=2019100100:LHTFL:surface:0-0 day acc fcst:ENS=low-res ctl
3:422:d=2019100100:SHTFL:surface:0-6 hour acc fcst:ENS=low-res ctl
4:520473:d=2019100100:LHTFL:surface:0-6 hour acc fcst:ENS=low-res ctl
5:1040524:d=2019100100:SHTFL:surface:0-12 hour acc fcst:ENS=low-res ctl

Running plot_data_plane...

plot_data_plane LHTFL_SHTFL_October_2019.grib2 LHTFL.ps 'name="LHTFL"; level="L0";'

It find 1891 matching records! One for each date:

WARNING: MetGrib2DataFile::data_plane() -> Using the first of 1891 matching records for "LHTFL/L0":

So I reran to select only a single initialization and lead time:

data_plane LHTFL_SHTFL_October_2019.grib2 LHTFL.ps 'name="LHTFL"; level="L0"; init_time="20191015_00"; lead_time="36";'

Here's the resulting image:

That range of values makes me suspect that those very large negative values should actually be bad data. But I don't know anything about LHTFL data. If for example, anything less that 0 should be interpreted as bad data (-9999 in MET) you could censor the data like this:

data_plane LHTFL_SHTFL_October_2019.grib2 LHTFL.ps 'name="LHTFL"; level="L0"; init_time="20191015_00"; lead_time="36"; censor_thresh=[lt0]; censor_val[-9999];'

But that's just an example, its really up to you to determine how to handle the actual forecast values and make them comparable to the observation values.

In doing that, you may find the convert function to be useful. For example, if you need to divide the forecast values by 1000 before comparing them to the observations, you'd do so using the convert(x) function, like this:

data_plane LHTFL_SHTFL_October_2019.grib2 LHTFL.ps 'name="LHTFL"; level="L0"; init_time="20191015_00"; lead_time="36"; censor_thresh=[lt0]; censor_val[-9999]; convert(x)=x/1000;'

Hope that helps. Please give plot_data_plane a try and let me know if you're able to make progress.

[JohnHalleyGotway]

@YakelynRJ I'd like to mark this discussion as being answered and "lock it" to prevent future posts. That's how we encourage users to create new discussions for new questions. But I wanted to give you an opportunity to comment on it before doing so. Please feel free to select one of the responses as being the best answer to your original question. That'll help future users with similar questions find answers.