Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update docs. #289

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 6 additions & 7 deletions docs/source/development.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
"- `xarray`: Provides the framework for defining and implementing data schemas.\n",
"- `dask` and `distributed`: Enable parallel execution for handling large datasets efficiently.\n",
"- `zarr` ([zarr specification](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html)): Used as a storage backend for scalable, chunked data.\n",
"- `python-casacore` ([Casacore Table Data System (CTDS) File Formats](https://casacore.github.io/casacore-notes/260.pdf)): Used as a storage backend, with ongoing development toward a lightweight, pure Python replacement.\n",
"- `python-casacore` ([Casacore Table Data System (CTDS) File Formats](https://casacore.github.io/casacore-notes/260.pdf)): Used to convert data from MS v2 to zarr MS v4, with ongoing development toward a lightweight, pure Python replacement.\n",
"- `pyasdm` (under development): A Python-based storage backend in progress, designed for accessing ASDM (Astronomy Science Data Model) data.\n"
]
},
Expand All @@ -52,19 +52,18 @@
"- [xarray Datasets](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html): A multi-dimensional, array database of labeled n-dimensional arrays.\n",
"- `XRADIO Processing Sets`: XRADIO-specific data structure, based on a Python dictionary, that consists of a collection of `xarray Datasets`. We will be looking into replacing the Processing set with [xarray Datatree](https://xarray-datatree.readthedocs.io/en/latest/) in the future.\n",
"\n",
"## Translating from a Table-based to an Xarray-based Schema\n",
"\n",
"## Coordinates and Data Variables\n",
"When creating an Xarray-based schema from a table-based schema, we use the following criteria to decide what type of Xarray structure is used:\n",
"\n",
"In XRADIO, we follow these conventions when defining a data schema:\n",
"\n",
"- **Coordinates**: Values used to label plots (e.g., numbers or strings). Coordinate names are always in lower case and use snake_case.\n",
"- **Data Variables**: Numerical values used for plotting. Data variable names are always in upper case and use snake_case.\n",
"- **Coordinates**: Values used to label plots (e.g., numbers or strings). Coordinate names are always in lowercase and use snake_case.\n",
"- **Data Variables**: Numerical values used for plotting. Data variable names are always in uppercase and use snake_case.\n",
"\n",
"For instance, in the [Measurement Set v4 schema](measurement_set/schema_and_api/measurement_set_schema.rst), `antenna_name` and `frequency` are coordinates, while `VISIBILITY` data are data variables.\n",
"\n",
"## Measures\n",
"\n",
"Both data variables and coordinates can have additional metadata, such as associated coordinates and units, stored in their attributes. XRADIO’s measures are based on [python-casacore measures](https://casacore.github.io/python-casacore/casacore_measures.html), with updates to align with [astropy coordinate](https://docs.astropy.org/en/stable/coordinates/index.html) naming conventions. The table below outlines the different types of XRADIO measures:\n",
"Both data variables and coordinates can have additional metadata, such as measures information, stored in their attributes. XRADIO’s measures are based on [python-casacore measures](https://casacore.github.io/python-casacore/casacore_measures.html), with updates to align with [astropy coordinate](https://docs.astropy.org/en/stable/coordinates/index.html) naming conventions. The table below outlines the different types of XRADIO measures:\n",
"\n",
".. autoclass:: xradio.measurement_set.schema.TimeArray()\n",
"\n",
Expand Down
2 changes: 1 addition & 1 deletion docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ To thoroughly review XRADIO and the Measurement Set v4.0.0 draft:
- Examples show how different telescopes' data can be represented.
- If your telescope isn't represented, open an issue and attach a Measurement Set v2 (10MB or smaller).

4. Examine the schema and API documentation in the Measurement Set v4.0.0 section.
4. Examine the [schema](https://xradio.readthedocs.io/en/latest/measurement_set/schema_and_api/measurement_set_schema.html) and [API documentation](https://xradio.readthedocs.io/en/latest/measurement_set/schema_and_api/measurement_set_api.html) in the Measurement Set v4.0.0 section.

- The schema is included in ReadTheDocs for versioning and accessibility.

Expand Down
2 changes: 1 addition & 1 deletion docs/source/measurement_set_overview.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
"\n",
"## Schema Layout\n",
"\n",
"The MS v4 contains data for a single observation, spectral window, polarization setup, observation mode, processor and beam per antenna (though finer partitioning, such as splitting by scan, is allowed if desired). This structure simplifies the MS v4 data representation, enabling it to be stored as n-dimensional arrays with consistent shapes over time (rare baseline dropouts are handled by NaN padding). To process data from multiple MS v4s, we've introduced the concept of a Processing Set (`ps`), which is a collection of MS v4s. Importantly, each MS v4 is fully self-describing. As shown in [Figure 1](#figure-1) (a simplified diagram; for full details, see the [Data Model Schema](measurement_set/schema_and_api/measurement_set_schema.rst)), the MS v4 is structured as a dataset (xds) of datasets comprising the `correlated_xds` along with `antenna_xds`, `pointing_xds`, `phase_calibration_xds`, `weather_xds`, `system_calibration_xds`, `gain_curve_xds`, and `phased_array_xds`, all stored in the attribute section. The `correlated_xds` contains the `VISIBILITY` (for interferometer data) or `SPECTRUM` (for single dish data), `UVW`, `WEIGHT`, and `FLAGS` data variables, along with info dictionaries in the attributes. The `field_and_source_xds` is specifically stored within the attributes of the `VISIBILITY`/`SPECTRUM` data variable.\n",
"An xarray dataset conforming to the MSv4 schema contains data for a single observation, spectral window, polarization setup, observation mode, processor and beam per antenna (though finer partitioning, such as splitting by scan, is allowed if desired). This structure simplifies the MS v4 data representation relative to the MS v2, enabling it to be stored as n-dimensional arrays with consistent shapes over time (rare baseline dropouts are handled by NaN padding). Related datasets can be grouped together into a Processing Set (`ps`), which is useful for processing them together. Importantly, each MS v4 is fully self-describing. As shown in [Figure 1](#figure-1) (a simplified diagram; for full details, see the [Data Model Schema](measurement_set/schema_and_api/measurement_set_schema.rst)), the MS v4 is structured as a dataset (xds) of datasets comprising the `correlated_xds` along with `antenna_xds`, `pointing_xds`, `phase_calibration_xds`, `weather_xds`, `system_calibration_xds`, `gain_curve_xds`, and `phased_array_xds`, all stored in the attribute section. The `correlated_xds` contains the `VISIBILITY` (for interferometer data) or `SPECTRUM` (for single dish data), `UVW`, `WEIGHT`, and `FLAGS` data variables, along with info dictionaries in the attributes. The `field_and_source_xds` is specifically stored within the attributes of the `VISIBILITY`/`SPECTRUM` data variable.\n",
"\n",
"<div style=\"text-align: center;\">\n",
" <figure id=\"figure-1\" style=\"display: inline-block;\">\n",
Expand Down
Loading