Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make IMSFV3 converter more robust #1497

Closed
CoryMartin-NOAA opened this issue Apr 16, 2024 · 6 comments · Fixed by #1510
Closed

Make IMSFV3 converter more robust #1497

CoryMartin-NOAA opened this issue Apr 16, 2024 · 6 comments · Fixed by #1510
Assignees
Labels
LAND Land DA OBS OBS processing, UFO

Comments

@CoryMartin-NOAA
Copy link
Contributor

https://github.com/JCSDA-internal/ioda-converters/blame/0bc00802515fe5834fb70311bfcac3f4cf12d349/src/land/imsfv3_scf2ioda.py#L101

This tries to get the date from the filename, and works fine if:

/path/to/file/20240416_sample.nc4
but if one is doing something like:

/path/to/jenny/86753099/work/20240416_sample.nc4
it will find the wrong 8 digit integer.

First found by @aerorahul as part of global-workflow CI, as the hash happened to be an 8 digit integer.

@jiaruidong2017
Copy link

We can add a line to find the basename (20240416_sample.nc4) first and then get the date from the basename.

@YoulongXia-NOAA
Copy link
Contributor

The best way is IMS processing code to output time into nc4 file, the ioda converter read time directly from there. This also happened for SMAP ioda converter. We finally solve it by reading date and time from hd5 file directly. @jiaruidong2017, if you have experience to add a line to find the basename and then get the date, that is great. It should not be hard to add time into the repocessed nc4 file from ims processing code. Here I include @ClaraDraper-NOAA here.

@CoryMartin-NOAA
Copy link
Contributor Author

I agree @YoulongXia-NOAA it's dangerous to infer from filenames and if we can put it in the file itself that would be the best solution.

@CoryMartin-NOAA
Copy link
Contributor Author

@WalterKolczynski-NOAA suggested here: NOAA-EMC/global-workflow#2484 (comment) some regex that would be a quick fix.

@WalterKolczynski-NOAA
Copy link

WalterKolczynski-NOAA commented Apr 16, 2024

My solution there only works if the date is always delimited by periods. If that isn't the case (as in the example above), it won't work as-is. I agree with @YoulongXia-NOAA reading the date directly from the file is the best long-term solution.

@BenjaminRuston BenjaminRuston added OBS OBS processing, UFO LAND Land DA labels Apr 22, 2024
@YoulongXia-NOAA
Copy link
Contributor

I am working on this issue. Example data has be produced and the PR (NOAA-PSL/land-SCF_proc#5) is under review. Offline modified imsfv3_scf2ioda.py which reads time directly from data file has been checked and it works. Now I am planning to work on iodacov within ioda-bundle to update imsfv3_scf2ioda.py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
LAND Land DA OBS OBS processing, UFO
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants