Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Snyk vulnerabilities in documentation #74

Merged
merged 7 commits into from
Nov 1, 2023
Merged

Update Snyk vulnerabilities in documentation #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
53c87f4
Update Snyk dependencies for documentation pip requirements.
owenlittlejohns Oct 30, 2023
1c894a0
Update dependencies for Sphinx documentation.
owenlittlejohns Oct 30, 2023
031689d
Update basic.ipynb to ensure GeoTIFF outputs.
owenlittlejohns Oct 30, 2023
61d4237
Update progressbar2 dependency to avoid GitHub Action dependency conf…
owenlittlejohns Oct 30, 2023
6c39b84
Pin pillow in requirements/examples.txt
owenlittlejohns Oct 30, 2023
8809ac3
Update test dependencies to match Sphinx version, deprecate tests on …
owenlittlejohns Oct 30, 2023
fd9578f
Reduce Sphinx version for Python 3.8 compatibility.
owenlittlejohns Oct 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ jobs:
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: [3.7, 3.8, 3.9, '3.10', '3.11']
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The update to a recent version of nbconvert requires Python >= 3.8. I also checked and 3.7 was EOL back in June (see: https://devguide.python.org/versions/). It would have been great to not drop 3.7 in the testing, but this seems to be the answer to the rabbit hole of dependencies. (The root of the issue comes from the tests.yml workflow installing the core, test and docs dependencies, and therefore needing all three to be consistent with one another)

python-version: [3.8, 3.9, '3.10', '3.11']

steps:
- uses: actions/checkout@v3
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ We welcome feedback on Harmony-Py via [GitHub Issues](https://github.com/nasa/ha

## Prerequisites

* Python 3.7+
* Python 3.8+


## Installing
Expand Down
57 changes: 32 additions & 25 deletions examples/basic.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,8 @@
"\n",
"request = Request(\n",
" collection=collection,\n",
" spatial=BBox(-165, 52, -140, 77)\n",
" spatial=BBox(-165, 52, -140, 77),\n",
" format='image/tiff'\n",
")"
]
},
Expand Down Expand Up @@ -167,15 +168,17 @@
"metadata": {},
"outputs": [],
"source": [
"print(f'\\n{job1_id}')\n",
"print(f'\\nHarmony job ID: {job1_id}')\n",
"\n",
"print('\\nWaiting for the job to finish')\n",
"results = harmony_client.result_json(job1_id, show_progress=True)\n",
"\n",
"print('\\nDownloading results:')\n",
"futures = harmony_client.download_all(job1_id)\n",
"\n",
"for f in futures:\n",
" print(f.result()) # f.result() is a filename, in this case\n",
"\n",
"print('\\nDone downloading.')"
]
},
Expand All @@ -194,9 +197,9 @@
"metadata": {},
"outputs": [],
"source": [
"\n",
"futures = harmony_client.download_all(job1_id, overwrite=False)\n",
"filenames = [f.result() for f in futures]\n",
"\n",
"for filename in filenames:\n",
" helper.show_result(filename)"
]
Expand All @@ -221,7 +224,9 @@
" temporal={\n",
" 'start': dt.datetime(2020, 6, 1),\n",
" 'stop': dt.datetime(2020, 6, 30)\n",
" })\n",
" },\n",
" format='image/tiff'\n",
")\n",
"\n",
"job2_id = harmony_client.submit(request)"
]
Expand Down Expand Up @@ -268,7 +273,9 @@
" temporal={\n",
" 'start': dt.datetime(2010, 1, 1),\n",
" 'stop': dt.datetime(2020, 12, 30)\n",
" })\n",
" },\n",
" format='image/tiff'\n",
")\n",
"\n",
"job3_id = harmony_client.submit(request)"
]
Expand Down Expand Up @@ -339,12 +346,14 @@
"\n",
"request = Request(\n",
" collection=collection,\n",
" spatial=BBox(-183, 40, 10, 30)\n",
" spatial=BBox(-183, 40, 10, 30),\n",
" format='image/tiff'\n",
")\n",
"\n",
"print(f\"Request valid? {request.is_valid()}\")\n",
"print(f'Request valid? {request.is_valid()}')\n",
"\n",
"for m in request.error_messages():\n",
" print(\" * \" + m)"
" print(f' * {m}')"
]
},
{
Expand All @@ -369,12 +378,14 @@
" temporal={\n",
" 'start': dt.datetime(2020, 12, 30),\n",
" 'stop': dt.datetime(2010, 1, 1)\n",
" }\n",
" },\n",
" format='image/tiff'\n",
")\n",
"\n",
"print(f\"Request valid? {request.is_valid()}\")\n",
"print(f'Request valid? {request.is_valid()}')\n",
"\n",
"for m in request.error_messages():\n",
" print(\" * \" + m)"
" print(f' * {m}')"
]
},
{
Expand All @@ -400,12 +411,14 @@
" temporal={\n",
" 'start': dt.datetime(2020, 12, 30),\n",
" 'stop': dt.datetime(2010, 1, 1)\n",
" }\n",
" },\n",
" format='image/tiff'\n",
")\n",
"\n",
"print(f\"Request valid? {request.is_valid()}\")\n",
"print(f'Request valid? {request.is_valid()}')\n",
"\n",
"for m in request.error_messages():\n",
" print(\" * \" + m)"
" print(f' * {m}')"
]
},
{
Expand All @@ -426,7 +439,7 @@
"try:\n",
" harmony_client.submit(request)\n",
"except Exception as e:\n",
" print(\"Harmony Py raised an exception:\\n\")\n",
" print('Harmony Py raised an exception:\\n')\n",
" print(e)"
]
},
Expand Down Expand Up @@ -456,7 +469,8 @@
" 'start': dt.datetime(2021, 1, 1),\n",
" 'stop': dt.datetime(2021, 1, 10)\n",
" },\n",
" max_results=2\n",
" max_results=2,\n",
" format='image/tiff'\n",
")\n",
"request.is_valid()"
]
Expand Down Expand Up @@ -498,6 +512,7 @@
"outputs": [],
"source": [
"job_id = harmony_client.submit(request)\n",
"\n",
"for filename in [f.result() for f in harmony_client.download_all(job_id)]:\n",
" helper.show_result(filename)"
]
Expand Down Expand Up @@ -630,14 +645,6 @@
"for filename in [f.result() for f in harmony_client.download_all(job_id)]:\n",
" helper.show_result(filename)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cd883298-4642-4128-baff-e4291fb4d00d",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
Expand All @@ -656,7 +663,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.2"
"version": "3.10.13"
},
"vscode": {
"interpreter": {
Expand Down
29 changes: 13 additions & 16 deletions examples/job_stac.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,13 @@
"source": [
"import sys\n",
"sys.path.append('..')\n",
"\n",
"!{sys.executable} -m pip install -q -r ../requirements/examples.txt\n",
"\n",
"# Install harmony-py requirements. Not necessary if you ran `pip install harmony-py` in your kernel \n",
"!{sys.executable} -m pip install -q -r ../requirements/core.txt\n",
"\n",
"from harmony import BBox, Client, Collection, Request, Environment\n"
"from harmony import BBox, Client, Collection, Request, Environment"
]
},
{
Expand All @@ -37,7 +38,7 @@
"metadata": {},
"outputs": [],
"source": [
" harmony_client = Client(env=Environment.UAT) # assumes .netrc usage\n",
"harmony_client = Client(env=Environment.UAT) # assumes .netrc usage\n",
"\n",
"collection = Collection(id='C1234088182-EEDTEST')\n",
"request = Request(\n",
Expand All @@ -47,7 +48,7 @@
")\n",
"\n",
"job_id = harmony_client.submit(request)\n",
"job_id\n"
"job_id"
]
},
{
Expand All @@ -63,7 +64,7 @@
"metadata": {},
"outputs": [],
"source": [
"stac_catalog_url = harmony_client.stac_catalog_url(job_id, show_progress=True)\n"
"stac_catalog_url = harmony_client.stac_catalog_url(job_id, show_progress=True)"
]
},
{
Expand All @@ -81,16 +82,17 @@
"source": [
"from urllib.parse import urlparse\n",
"import requests\n",
"from pystac import STAC_IO\n",
"from pystac import stac_io\n",
"\n",
"\n",
"def requests_read_method(uri):\n",
" parsed = urlparse(uri)\n",
" if parsed.hostname.startswith('harmony.') or parsed.hostname.startswith('localhost'):\n",
" return harmony_client.read_text(uri)\n",
" else:\n",
" return STAC_IO.default_read_text_method(uri)\n",
" return stac_io.default_read_text_method(uri)\n",
"\n",
"STAC_IO.read_text_method = requests_read_method"
"stac_io.read_text_method = requests_read_method"
]
},
{
Expand All @@ -107,19 +109,14 @@
"outputs": [],
"source": [
"from pystac import Catalog\n",
"cat = Catalog.from_file(stac_catalog_url)\n",
"\n",
"\n",
"cat = Catalog.from_file(stac_catalog_url)\n",
"print(cat.title)\n",
"\n",
"for item in cat.get_all_items():\n",
" print(item.datetime, item.properties.get('expires'), [asset.href for asset in item.assets.values()])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
Expand All @@ -138,7 +135,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.2"
"version": "3.10.13"
}
},
"nbformat": 4,
Expand Down
16 changes: 9 additions & 7 deletions harmony/harmony.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1187,14 +1187,16 @@ def iterator(
Each iteration returns a dictionary, or `None` when all granules have been iterated.
The dictionary has the following form:

{
'path': Future
'bbox': BBox object containing the bounding box for the granule,
'temporal': {
'start': '2020-01-11T14:00:00.000Z',
'end': '2020-01-11T15:59:59.000Z'
.. code-block:: python

{
'path': Future
'bbox': BBox object containing the bounding box for the granule,
'temporal': {
'start': '2020-01-11T14:00:00.000Z',
'end': '2020-01-11T15:59:59.000Z'
}
}
}

The Future resolves to the path to the downloaded file.

Expand Down
2 changes: 1 addition & 1 deletion requirements/core.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
python-dateutil ~= 2.8.2
python-dotenv ~= 0.20.0
progressbar2 ~= 3.55.0
progressbar2 ~= 4.2.0
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to leave these dependencies entirely alone, but progressbar2 needs to be consistent in core.txt and docs.txt to ensure the tests.yml workflow can run. The sphinx dependency in dev.txt was bumped for the same reason.

requests ~= 2.28
sphinxcontrib-napoleon ~= 0.7
curlify ~= 2.2.1
2 changes: 1 addition & 1 deletion requirements/dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@ pytest-mock ~= 3.5
pytest-watch ~= 4.2
responses ~= 0.12
setuptools >= 54.2
sphinx ~= 5.3.0
sphinx ~= 7.1.2
wheel >= 0.36
10 changes: 5 additions & 5 deletions requirements/docs.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
nbconvert ~= 7.4.0
curlify ~= 2.2.1
Jinja2 ~= 3.1.2
load-dotenv ~=0.1.0
curlify ~= 2.2.1
progressbar2 ~= 3.55.0
sphinx ~= 5.3.0
sphinx-rtd-theme ~= 1.2.0
nbconvert ~= 7.10.0
progressbar2 ~= 4.2.0
sphinx ~= 7.1.2
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sphinx goes up to 7.2.6, but 7.2.0 onwards all require Python 3.9+. 7.1.2 was only released in August, and currently has no vulnerabilities.

sphinx-rtd-theme ~= 1.3.0
19 changes: 10 additions & 9 deletions requirements/examples.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
boto3 ~= 1.17
ipyplot ~= 1.1
ipywidgets ~= 7.6
jupyterlab ~= 3.0
matplotlib ~= 3.3
pystac ~= 1.7.3
rasterio ~= 1.2
boto3 ~= 1.28
intake-stac ~= 0.4.0
netCDF4
numpy
ipyplot ~= 1.1
ipywidgets ~= 8.1
jupyterlab ~= 4.0
matplotlib ~= 3.8
netCDF4 ~= 1.6
numpy ~= 1.26
pillow ~= 10.1 # A dependency of ipyplot, pinned to avoid critical vulnerability.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm guessing future releases of ipyplot will bump their own required version of pillow, so this line can be removed at a later date. 🤞

pystac ~= 1.9.0
rasterio ~= 1.3