Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenAQ v1 fixes + initial OpenAQ v2 reader #141

Merged
merged 82 commits into from
Sep 19, 2024
Merged
Show file tree
Hide file tree
Changes from 80 commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
2402338
WIP: new reader for single openaq-fetches file
zmoon Sep 9, 2023
4401a6a
Trying a larger file from Jordan's day
zmoon Sep 9, 2023
2f6c164
WIP: revising OPENAQ class
zmoon Sep 9, 2023
375d72a
WIP: want to try going through the JSON with Python
zmoon Sep 9, 2023
c06e8f2
Python processing reader mostly works
zmoon Sep 11, 2023
03a99e9
Merge remote-tracking branch 'noaa/develop' into openaq
zmoon Sep 11, 2023
10f0402
Get first attribution
zmoon Sep 12, 2023
5b20db8
Trying out getting attribution with larger dataset
zmoon Sep 12, 2023
1af59cb
Trying out OpenAQ API v2
zmoon Sep 12, 2023
0657d3e
todo
zmoon Sep 12, 2023
157be3c
Experimenting with smaller (finishable) queries
zmoon Sep 12, 2023
36c4189
NOx ppm; disable prints (for prog bar)
zmoon Sep 12, 2023
3ff4cd6
The species Jordan wants
zmoon Sep 12, 2023
b096dd7
Pass API key
zmoon Sep 15, 2023
af05335
Merge remote-tracking branch 'noaa/develop' into openaq
zmoon Oct 16, 2023
d41c645
Currently converting NOx to ppmv
zmoon Oct 18, 2023
7a69d06
WIP
zmoon Oct 18, 2023
f91a7c5
Testing get-locations
zmoon Oct 19, 2023
4ff270e
Locations as df
zmoon Oct 19, 2023
af86394
notes
zmoon Oct 19, 2023
d59e647
Get parameters list
zmoon Oct 19, 2023
919bb91
Trying lat/lon box method
zmoon Oct 19, 2023
8510ff5
Towards working add-data fn
zmoon Oct 19, 2023
97b68de
Initial search radius(es) option
zmoon Oct 19, 2023
3ab3102
docstring work
zmoon Oct 19, 2023
5cc7d08
No time split option; more input validation
zmoon Oct 19, 2023
916d7de
country option
zmoon Oct 19, 2023
6a8107b
consistency
zmoon Oct 19, 2023
e86d927
Use logger
zmoon Oct 19, 2023
8a11fe3
fix
zmoon Oct 19, 2023
725b746
Sites option
zmoon Oct 24, 2023
a6a3588
Start tests for OpenAQ v2
zmoon Oct 24, 2023
dd854e1
Test get locations
zmoon Oct 24, 2023
dee4d74
docstring
zmoon Oct 24, 2023
50059f4
docstrings
zmoon Oct 24, 2023
8fb1c1d
Site ID as string
zmoon Oct 24, 2023
5418685
Test get selected data
zmoon Oct 24, 2023
dd4a2de
docstrings
zmoon Oct 24, 2023
bdd932d
verbose false
zmoon Oct 24, 2023
ac63e9b
docstrings
zmoon Oct 24, 2023
7eea32b
Support using either reader
zmoon Oct 24, 2023
44102c0
More test; fix averaging period calc for other units
zmoon Oct 24, 2023
2e55bf8
Cap for faster test
zmoon Oct 24, 2023
c421aba
Refactor a bit
zmoon Oct 24, 2023
5299b3b
cleanup
zmoon Oct 24, 2023
f122df9
Test parameter accounting
zmoon Oct 24, 2023
a7f83af
WIP: dealing with time-site dupes
zmoon Oct 24, 2023
fc6ddd9
`wide_fmt` option
zmoon Oct 25, 2023
f733ad5
doc
zmoon Oct 25, 2023
a9e810b
Remove averaging period from pivot table index
zmoon Oct 26, 2023
cb000b5
Full openaq v1 test suite only in 3.7+
zmoon Oct 26, 2023
8284707
Merge remote-tracking branch 'noaa/develop' into openaq
zmoon Oct 26, 2023
462d488
`usedforsecurity` is 3.9+
zmoon Oct 26, 2023
aaebef6
Log a bit of info about dupe location IDs
zmoon Oct 27, 2023
419318d
Wait some time before retry
zmoon Oct 27, 2023
1bb0e5c
Add entity and sensor-type options
zmoon Aug 23, 2024
6f88124
Add some randomness to sleeps
zmoon Aug 23, 2024
cc79ec6
Pass endpoint to consume instead
zmoon Aug 23, 2024
149d2bf
Wait until usage to warn about API key
zmoon Aug 23, 2024
3c8890e
Validate search radii earlier
zmoon Aug 23, 2024
5e49e3c
Generator for params for the separate queries
zmoon Aug 23, 2024
ee81f6a
Initial multi-thread support
zmoon Aug 23, 2024
a62c16f
Allow passing single time
zmoon Aug 23, 2024
ee6dbf7
More renames
zmoon Aug 23, 2024
06b4909
Wait more when rate-limited
zmoon Aug 23, 2024
e497ab4
Neg -> NaN
zmoon Aug 28, 2024
427bddc
Initial wide format transformation
zmoon Aug 30, 2024
5c865a9
Document some API-related params
zmoon Sep 11, 2024
b6329f0
Import
zmoon Sep 11, 2024
a3e1da3
Seems like API key is required now
zmoon Sep 11, 2024
f1687e8
Ensure site has just one location name
zmoon Sep 11, 2024
2d3dfed
len
zmoon Sep 11, 2024
f855877
Ensure lat/lon unique for site as well
zmoon Sep 11, 2024
7dfe2e2
Merge remote-tracking branch 'noaa/develop' into openaq
zmoon Sep 11, 2024
70cd53a
Fix typos
zmoon Sep 11, 2024
5729d87
Provide OpenAQ API key
zmoon Sep 11, 2024
de87dd0
Skip on 3.6
zmoon Sep 11, 2024
8f0be93
Quote API key
zmoon Sep 11, 2024
bbc16ec
Check length
zmoon Sep 11, 2024
4490a50
Skip if on CI and API key is empty string
zmoon Sep 11, 2024
b0a77e6
Merge remote-tracking branch 'noaa/develop' into openaq
zmoon Sep 19, 2024
7d16a44
API key notes in module docstring
zmoon Sep 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ on:
schedule:
- cron: "0 12 * * 1"

env:
OPENAQ_API_KEY: "${{ secrets.OPENAQ_API_KEY }}"

jobs:
test:
name: Test (Py ${{ matrix.python-version }})
Expand Down
1 change: 1 addition & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ repos:
exclude: tdump\.[0-9]*
- id: end-of-file-fixer
- id: check-docstring-first
exclude: monetio/obs/openaq_v2\.py
- id: check-yaml

- repo: https://github.com/asottile/pyupgrade
Expand Down
16 changes: 15 additions & 1 deletion monetio/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,19 @@
from . import grids
from .models import camx, cmaq, fv3chem, hysplit, hytraj, ncep_grib, pardump, prepchem, raqms
from .obs import aeronet, airnow, aqs, cems, crn, improve, ish, ish_lite, nadp, openaq, pams
from .obs import (
aeronet,
airnow,
aqs,
cems,
crn,
improve,
ish,
ish_lite,
nadp,
openaq,
openaq_v2,
pams,
)
from .profile import geoms, gml_ozonesonde, icartt, tolnet
from .sat import goes

Expand Down Expand Up @@ -29,6 +42,7 @@
"ish_lite",
"nadp",
"openaq",
"openaq_v2",
"pams",
#
# profile obs
Expand Down
2 changes: 2 additions & 0 deletions monetio/obs/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
ish_lite,
nadp,
openaq,
openaq_v2,
pams,
)

Expand All @@ -25,6 +26,7 @@
"cems_mod",
"nadp",
"openaq",
"openaq_v2",
"pams",
]

Expand Down
Loading