- Fix bug in
s3exists
when usingaws-cli
.
- Fix typo in
last_modified
.
- Compatibility with
aws-cli
.
- Bugfixes and better compatibility with system environment variables.
- Allowing s3mpi.path and s3mpi.cache to be called via system environment variables.
- Better error output in
s3read
if path does not exist. (see issue #72)
- Fix incorrect
strptime
version specification.
- Remove overly complicated metaprogramming.
- Other fixes for s4cmd.
- allow choice of storage format in
s3read
ands3store
. Defaults toRDS
, and now you can choooseCSV
ortable
for data frames.
- Make
s3store
work with s4cmd.
- Don't set --bucket-location flag if
s4cmd
is detected as thes3cmd
binary.
- Revert the change in 0.2.21 in favor of using
s3cmd info
overs3cmd ls
to obtain updated_at information on files.
- Add exponential backoff logic to s3.put function.
- Turn off the LRU cache too when
cache = FALSE
ins3read
.
options(s3mpi.num_retries)
now allows you to specify default number of retries globally.
- Automatically adds ending slashes to paths if they are missing when using
s3store
,s3exists
, ors3delete
.
- Add the ability to delete an object in s3 using
s3delete
.
s3path()
is exported.
s3read()
(with no arguments) is no longer supported.
- Fixed an issue where reading files that have the same prefix as another file on the S3 bucket generates a warning.
- Fix a more serious problem where writing and reading within the same minute produces incorrect results due to the s3cmd utility having minute-level rather than second-level granularity.
- Workaround for the silent but oh-so-deadly sporadic failure of s3cmd's put. By default we now check for the existence of the object when issuing a put, with the option to retry a number of times.
- Keep AWS.tools on a remote.
- Add remotes to DESCRIPTION.
- Explicitly create the directory of a file given by
tempfile()
to prevent rare errors wherein the directory does not exist and yields a file connection error. (#41)
- Introduce an
s3mpi.disable_lru_cache
option as well as silently fail if storage to LRU does not succeed.
- Switch to
system2
, which should be more windows friendly, and allow the user to specify path to executable of s3cmd, by settingoptions(s3mpi.s3cmd_path = '/usr/local/bin/s3cmd')
- Fixup LRU cache to actually use size parameter option.
- A stylistic refactor of the package. The
.path
argument has been deprecated ins3read
ands3store
in favor of simplypath
.
- Remove the need to type a trailing slash in
.path
.
- A hotfix for cache corruption, where data exists without metadata. It can happen if writing metadata ever fails.
-
Remove the
s3mpi.memoize_cache
global option, since it makes no sense. A user could have overwritten an S3 key in a different R session. -
s3exists(NULL)
now returns FALSE. Fixes issue #22.
- The
s3mpi.memoize_cache
global controls whether or not caching is memoised. If set toTRUE
, it would have the effect of keeping a common object in the R session instead of retrieving it from the cached file for each given s3 key. This can significantly speed up code that reads from the same S3 key multiple times within a single R session.
- The safety check on
s3store
usessafe = FALSE
by default now. The new functions3put
is equivalent tos3store
and should be used going forward if one does not wish to overwrite existing keys. The other approach was causing too many breaking changes to existing codebases.
- Added a safety check for
s3store
. Now if you want to overwrite a key inside a bucket, you need to uses3store(key, safe = FALSE)
. By default safe is set toTRUE
.
- Added a caching mechanism that will keep copies of files downloaded and
uploaded to S3. Useful if local storage constraints are not an issue.
To enable, set
options(s3mpi.cache = '~/.s3cache')
in your~/.Rprofile
(or replace'~/.s3cache'
with a directory of your choice).