diff --git a/.github/workflows/buildbook.yaml b/.github/workflows/buildbook.yaml index 2d105df6..10178708 100644 --- a/.github/workflows/buildbook.yaml +++ b/.github/workflows/buildbook.yaml @@ -54,6 +54,25 @@ jobs: path: ${{ env.CONDA }}/envs key: conda-${{ runner.os }}--${{ runner.arch }}--${{ hashFiles('environment.yml') }}-${{ env.CONDA_CACHE_NUMBER }} + - name: install IPFS + run: | + wget -q https://dist.ipfs.tech/kubo/${IPFS_VERSION}/kubo_${IPFS_VERSION}_linux-amd64.tar.gz + tar -xvzf kubo_${IPFS_VERSION}_linux-amd64.tar.gz + pushd kubo + sudo bash install.sh + popd + ipfs --version + env: + IPFS_VERSION: v0.32.1 + - name: init IPFS + run: | + ipfs init --profile server,announce-off + - name: configure known IPFS peers + run: | + wget -q https://raw.githubusercontent.com/orcestra-campaign/ipfs_tools/refs/heads/main/scripts/add_peers.py -O - | python - -f https://raw.githubusercontent.com/orcestra-campaign/ipfs_tools/refs/heads/main/known_peers.yaml + - name: run IPFS in background + run: | + ipfs daemon 2>ipfs.log | grep -i -o -m1 'Daemon is ready' & tail -F --pid=$! ipfs.log - name: setting up notebook execution cache uses: actions/cache/restore@v4 with: diff --git a/environment.yml b/environment.yml index 1cbc20e2..3cfd7a74 100644 --- a/environment.yml +++ b/environment.yml @@ -33,5 +33,5 @@ dependencies: - pip - pip: - easygems>=0.0.11 - - orcestra>=0.0.25 + - orcestra>=0.0.28 - pybtex-apa-style diff --git a/orcestra_book/_config.yml b/orcestra_book/_config.yml index 80c79eee..4c9cfdba 100644 --- a/orcestra_book/_config.yml +++ b/orcestra_book/_config.yml @@ -10,6 +10,7 @@ exclude_patterns: ["_templates/*"] # See https://jupyterbook.org/content/execute.html execute: execute_notebooks: cache + timeout: 60 # Define the name of the latex output file for PDF builds latex: diff --git a/orcestra_book/_toc.yml b/orcestra_book/_toc.yml index e635a74c..23d18fd2 100644 --- a/orcestra_book/_toc.yml +++ b/orcestra_book/_toc.yml @@ -40,6 +40,7 @@ chapters: - file: hifs - file: water_vapour_overview - file: temperature_example + - file: halo_flight_segmentation - file: blog - file: lam title: Simulations diff --git a/orcestra_book/halo_flight_segmentation.md b/orcestra_book/halo_flight_segmentation.md new file mode 100644 index 00000000..73e77f2d --- /dev/null +++ b/orcestra_book/halo_flight_segmentation.md @@ -0,0 +1,203 @@ +--- +jupytext: + text_representation: + extension: .md + format_name: myst +kernelspec: + display_name: Python 3 (ipykernel) + language: python + name: python3 +execution: + timeout: 600 +--- + +# HALO's flight segmentation +To aid in the analysis of flight data, all HALO flights are segmented via timestamps into a system of hierarchical identifiers. Non-exclusive segments are defined by two (YYYY-MM-DD hh:mm:ss) timestamps, the first one defining the `start` of the segment and the second denoting the first time step after the `end` of the segment. Timestamps have a temporal resolution of 1 s, and times are given in UTC. Every segment belongs to a `kind` – a categorical type for segments defined below. It helps to think of segments as an interval of flight time and the corresponding kinds as describing how the aircraft was being operated during this time interval. + +Flight segmentation data are provided in a YAML (YAML Ain't Markup Language) file can be accessed at https://orcestra-campaign.github.io/flight_segmentation/all_flights.yaml or in python directly via the `pyorcestra` package. This section provides a description of the YAML file and some examples of how to use it with python. + ++++ + +## Loading the flight segmentation data + +```{code-cell} ipython3 +from orcestra import get_flight_segments + +meta = get_flight_segments() +``` + +### Which flights are included in the flight segmentation? + +```{code-cell} ipython3 +flight_ids = [flight_id + for flights in meta.values() + for flight_id in flights] +print(f"Totel number of flights: {len(flight_ids)}") +flight_ids +``` + +## Segments + +Sometimes it may be useful to have a flat list of segments, irrespective of which flight or platformt they belong to. So we build it here: + +```{code-cell} ipython3 +segments = [{**s, + "platform_id": platform_id, + "flight_id": flight_id + } + for platform_id, flights in meta.items() + for flight_id, flight in flights.items() + for s in flight["segments"] + ] +``` + +### List available segment `kinds` + +```{code-cell} ipython3 +kinds = set(k for s in segments for k in s["kinds"]) +kinds +``` + +### Plotting all segments of a specific flight + ++++ + +A supplement function is added here for loading HALO position and attitude data to show the application of flight segmentation information. + +```{code-cell} ipython3 +import xarray as xr +import matplotlib.pyplot as plt +from orcestra.flightplan import sal, tbpb + +def get_halo_position_attitude(flight_id): + root = "ipns://latest.orcestra-campaign.org/products/HALO/position_attitude" + return (xr.open_dataset(f"{root}/{flight_id}.zarr", engine="zarr") + .reset_coords().resample(time="1s").mean().load()) + +def kinds2color(kinds): + if "circle" and "atr_coordination" in kinds: + return "C2" + if "circle" in kinds: + return "C1" + if "ec_track" in kinds: + return "C3" + return "C0" +``` + +```{code-cell} ipython3 +flight = "HALO-20240811a" +ds = get_halo_position_attitude(flight) + +fig, ax = plt.subplots() +for s in meta["HALO"][flight]["segments"]: + t = slice(s["start"], s["end"]) + ax.plot(ds.lon.sel(time=t), ds.lat.sel(time=t), c=kinds2color(s["kinds"]))#, label=s["name"]) + +for k in ["circle", "straight_leg", "ec_track", "atr_coordination"]: + ax.plot([], [], color=kinds2color(k), label=k) + +plt.scatter(sal.lon, sal.lat, marker="o", color="k") +plt.annotate("SAL", (sal.lon, sal.lat)) + +ax.set_xlabel("longitude / °") +ax.set_ylabel("latitude / °") +ax.spines[['right', 'top']].set_visible(False) +ax.legend() +plt.title(flight); +``` + +### Plotting all segments of a specific kind + +```{code-cell} ipython3 +fig, ax = plt.subplots() +kind = "circle" + +for f in flight_ids: + ds = get_halo_position_attitude(f) + for s in segments: + if kind in s["kinds"] and f==s["flight_id"]: + t = slice(s["start"], s["end"]) + ax.plot(ds.lon.sel(time=t), ds.lat.sel(time=t), c=kinds2color(kind), label=s["name"]) +ax.set_xlabel("longitude / °") +ax.set_ylabel("latitude / °") + +ax.scatter(sal.lon, sal.lat, marker="o", color="k") +ax.annotate("SAL", (sal.lon, sal.lat)) +ax.scatter(tbpb.lon, tbpb.lat, marker="o", color="k") +ax.annotate("BARBADOS", (tbpb.lon, tbpb.lat)) +ax.spines[['right', 'top']].set_visible(False) +``` + +## Some statistics + +```{code-cell} ipython3 +segment_ids_by_kind = {kind: [segment["segment_id"] + for segment in segments + if kind in segment["kinds"]] + for kind in kinds} +``` + +Print the total number of circles. +Note that those are not necessarily with dropsondes and the circles can have different radii. + +```{code-cell} ipython3 +len(segment_ids_by_kind["circle"]) +``` + +List the segments that include the tag `atr_coordination` + +```{code-cell} ipython3 +segment_ids_by_kind["atr_coordination"] +``` + +### Total time spent on EarthCARE tracks + +```{code-cell} ipython3 +import datetime + +ec_time = sum([s["end"] - s["start"] + for s in segments + if "ec_track" in s["kinds"] + ], datetime.timedelta()) +print(ec_time) +``` + +## Events +Events are different from segments in having only **one** timestamp. Examples are the usual "EC meeting points" or station / ship overpasses. + +```{code-cell} ipython3 +events = [{**e, + "platform_id": platform_id, + "flight_id": flight_id + } + for platform_id, flights in meta.items() + for flight_id, flight in flights.items() + for e in flight["events"] + ] +``` + +### EarthCARE underpass events +How well did HALO manage to meet with EathCARE? + +```{code-cell} ipython3 +ec_dist = [e["distance"] for e in events if "ec_underpass" in e["kinds"]] +ec_dist +``` + +Which flights do not have an ec_event? + +```{code-cell} ipython3 +flight_ids_events = [e["flight_id"] for e in events] +set(flight_ids) - set(flight_ids_events) +``` + +#### Histogram of distance HALO - EarthCARE during meeting point + +```{code-cell} ipython3 +fig, ax = plt.subplots() + +ax.hist(ec_dist) +ax.set_xlabel("Distance / m") +ax.set_ylabel("Frequency") +ax.spines[['right', 'top']].set_visible(False) +``` diff --git a/orcestra_book/temperature_example.md b/orcestra_book/temperature_example.md index df3f1c9a..b263e502 100644 --- a/orcestra_book/temperature_example.md +++ b/orcestra_book/temperature_example.md @@ -8,8 +8,6 @@ kernelspec: display_name: Python 3 language: python name: python3 -execution: - timeout: 60 --- # Temperature comparison