vignettes/bowerbird.Rmd
bowerbird.RmdAs of bowerbird v0.17 (August 2024) it is possible to synchronize
+directly into an object store like AWS S3, rather than to a directory on
+your local file system. This works for data sources using the
+bb_handler_rget, bb_handler_earthdata, or
+bb_handler_oceandata handlers. To do this, provide
+target_s3_args as part of your data source definition:
my_source <- bb_example_sources("Australian Election 2016 House of Representatives data") %>%
+ bb_modify_source(method = list(target_s3_args = list(base_url = "my.objectstore.com", region = "",
+ bucket = "bucket", key = "KEY", secret = "SECRET"))where the entries in target_s3_args are those required
+by functions from the aws.s3 package (see
+aws.s3::s3HTTP() for an overview). Objects will be stored
+with keys that follow the directory hierarchy on the remote data source.
+For example,
+http://www.somewhere.org/monkey/banana/dataset.zip will be
+stored in the specified bucket with key
+www.somewhere.org/monkey/banana/dataset.zip. You can remove
+the host name from this key by adding no_host = TRUE to the
+data source’s method parameters; similarly, leading levels of the
+directory hierarchy can be dropped with the cut_dirs
+parameter: see bb_rget() for details.
target_s3_args can also be specified as part of
+bb_config() to avoid having to provide the same parameters
+to multiple data sources:
+cf <- bb_config(target_s3_args = list(base_url = "my.objectstore.com", region = "",
+ key = "KEY", secret = "SECRET")) %>%
+ bb_add(my_source_1 %>%
+ bb_modify_source(method = list(target_s3_args = list(bucket = "bucket1")))) %>%
+ bb_add(my_source_2 %>%
+ bb_modify_source(method = list(target_s3_args = list(bucket = "bucket2"))))+@@ -82,7 +82,7 @@mysrc <- bb_example_sources("CMEMS global gridded SSH reprocessed (1993-ongoing)") mysrc$user <- "yourusername" mysrc$password <- "yourpassword" @@ -740,7 +775,7 @@option. Here we use theReducing download sizesaccept
bb_modify_sourcehelper function to do so: -+mysrc <- bb_example_sources("CMEMS global gridded SSH reprocessed (1993-ongoing)") %>% bb_modify_source(user = "yourusername", password = "yourpassword", method = list(accept_follow = "/2017")) @@ -823,7 +858,7 @@CMEMS global gridded
Authentication note: Copernicus Marine login required, see http://marine.copernicus.eu/services-portfolio/register-now/
Approximate size: 310 GB
-Documentation link: http://marine.copernicus.eu/services-portfolio/access-to-products/?option=com_csw&view=details&product_id=SEALEVEL_GLO_PHY_L4_REP_OBSERVATIONS_008_047
+Documentation link: https://data.marine.copernicus.eu/product/SEALEVEL_GLO_PHY_L4_MY_008_047/description
@@ -927,7 +962,7 @@diff --git a/docs/articles/data_provenance.html b/docs/articles/data_provenance.html index 1ecfb61..a0a4570 100644 --- a/docs/articles/data_provenance.html +++ b/docs/articles/data_provenance.html @@ -6,7 +6,7 @@Bathymetry of Lake Superior -
Site built with pkgdown 2.0.7.
+Site built with pkgdown 2.0.9.
Data provenance • bowerbird - + @@ -33,7 +33,7 @@Data provenance
Ben Raymond, Michael Sumner
-2024-03-08
+2024-08-16
Source:vignettes/data_provenance.Rmd@@ -130,8 +130,8 @@data_provenance.RmdWhich files were used in an analys knitr::kable(data.frame(name = cf$data_sources$name, source_dir = source_dirs))
| name | @@ -140,7 +140,7 @@||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NOAA OI SST V2 | -/some/local/path/ftp.cdc.noaa.gov/Datasets/noaa.oisst.v2/ | +/some/local/path/downloads.psl.noaa.gov/Datasets/noaa.oisst.v2/ | ||||||||||||||||
Australian Election 2016 House of Representatives
@@ -150,11 +150,11 @@ Which files were used in an analys
CMEMS global gridded SSH reprocessed
(1993-ongoing) |
-/some/local/path/ftp.sltac.cls.fr/Core/SEALEVEL_GLO_PHY_L4_REP_OBSERVATIONS_008_047/dataset-duacs-rep-global-merged-allsat-phy-l4-v3/ |
+/some/local/path/data.marine.copernicus.eu/SEALEVEL_GLO_PHY_L4_MY_008_047 |
Oceandata SeaWiFS Level-3 mapped monthly 9km chl-a |
-/some/local/path/oceandata.sci.gsfc.nasa.gov/SeaWiFS/Mapped |
+/some/local/path/oceandata.sci.gsfc.nasa.gov/SeaWiFS/Mapped/Monthly/9km/chlor |
Sea Ice Trends and Climatologies from SMMR and
@@ -260,7 +260,7 @@ |
A word on digital object identifie
diff --git a/docs/articles/index.html b/docs/articles/index.html
index 75506d5..e97a553 100644
--- a/docs/articles/index.html
+++ b/docs/articles/index.html
@@ -1,5 +1,5 @@
-References@@ -95,7 +95,7 @@Examplesmy_source <- bb_source( name = "Sea Ice Trends and Climatologies from SMMR and SSM/I-SSMIS, Version 3", - id = "10.5067/EYICLBOAAJOU", + id = "10.5067/IJ0T7HFHB9Y6", description = "NSIDC provides this data set ... [truncated; see bb_example_data_sources()]", doc_url = "https://nsidc.org/data/NSIDC-0192/versions/3", citation = "Stroeve J, Meier WN (2018) ... [truncated; see bb_example_data_sources()]", @@ -104,8 +104,8 @@Examplesauthentication_note = "Requires Earthdata login, see https://urs.earthdata.nasa.gov/. Note that you will also need to authorize the application 'nsidc-daacdata' (see 'My Applications' at https://urs.earthdata.nasa.gov/profile)", - method = list("bb_handler_earthdata", recursive = TRUE, level = 4, no_parent = TRUE, - relative = TRUE), + method = list("bb_handler_earthdata", level = 4, relative = TRUE, + accept_download = "\\.(s|n|png|txt)$", allow_unrestricted_auth = TRUE), user = "your_earthdata_username", password = "your_earthdata_password", collection_size = 0.02) @@ -125,7 +125,7 @@Examples
+
+
+
+
+
+
+
+
+
+ Postprocessing: remove redundant NRT oceandata files+ Source:R/oceandata.R
+ bb_oceandata_cleanup.Rd
+
+
+ This function is not intended to be called directly, but rather is specified as a
+
+
+
+
+ Arguments+
+
+ Value+ + +a list, with components
+
+
+ Details+This function will remove near-real-time (NRT) files from an oceandata collection that have been superseded by their non-NRT versions. +logical: if logical: similar to logical: if string: if provided, then the named list: options to use with list: named list or arguments to provide to Value@@ -224,7 +234,7 @@Detailsbb_modify_source()
| Modify a data source |
+ + + | +Postprocessing: remove redundant NRT oceandata files |
@@ -173,7 +177,7 @@ | Generate a bowerbird data source object for a Zenodo data set |
- + | bowerbird |