Basic HAPI CSV codec #167
Basic HAPI CSV codec #167
Conversation
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
IOBase can't know how to close and readline should directly call inner impl readline. Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
Most products have a fill value so it makes sense to expose it from SpeasyVariable interface. Checking if several SpeasyVariable share the same axis is a quite common operation when writing them into a file. Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
| for var_name in variables: | ||
| parameter = hapi_csv_file.get_parameter(var_name) | ||
| if parameter is not None: | ||
| loaded_vars[var_name] = SpeasyVariable(axes=[time_axis], values=DataContainer(parameter.values, |
There was a problem hiding this comment.
You only defined "time axis", but for a spectrogram you can have also a "VariableAxis".
There was a problem hiding this comment.
That's intended, I only wanted to handle simple 1D cases first.
|
|
||
| def _extract_data(file: io.IOBase) -> pds.DataFrame: | ||
| file.seek(0) | ||
| return pds.read_csv(file, comment='#', header=None, skiprows=0) |
There was a problem hiding this comment.
With Python 3.12.7 and pandas 2.2.3:
import pandas as pds
df = pds.read_csv("tests/resources/HAPI_sample_csv.csv", comment='#', header=None, skiprows=0)
print(df.index.values)The result is:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47])
=> Dataframe index is not the "time" column and there is something wrong with (cf. line 55 in load_hapi_csv method):
hapi_csv_file.create_parameter(data.index.to_numpy(dtype="datetime64[ns]"), meta=headers["parameters"][0])There was a problem hiding this comment.
import speasy as spz
from speasy.core.codecs import get_codec
hapi_csv_codec = get_codec('hapi/csv')
v = hapi_csv_codec.load_variable("Magnitude", "tests/resources/HAPI_sample_csv.csv", disable_cache=True)
print(v.time)Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
|
This is a basic CSV codec for HAPI. Many bits are still missing, in order to complete this codec we need to refactor and standardize SpeasyVariable metadata interface. @brenard-irap, we have two options:
I would be in favor of 2 mostly to avoid merge conflict nightmare and since this code doesn't break any user API.