Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Switch to single IO thread for S3 downloads #619

Closed
jamesls opened this issue Jan 29, 2014 · 0 comments · Fixed by #620
Closed

Switch to single IO thread for S3 downloads #619

jamesls opened this issue Jan 29, 2014 · 0 comments · Fixed by #620

Comments

@jamesls
Copy link
Member

jamesls commented Jan 29, 2014

We have seen issues on some OSes that have "trouble" with
concurrent IO threads to different parts of the same file,
despite these being non overlapping parts of the file.

The fix is to switch to a single IO thread.

Pictorially:

From:

                   file
    +----+         +----+
    |T1  |         |    |
    +----++------->|    |
                   +----+
    +----+         |    |
    |T2  +-------->|    |
    +----+         |    |
                   +----+
    +----+         |    |
    |T3  +-------->|    |
    +----+         +----+

To:

                                 file
                                +---+
   +----+                       |   |
   | T1 +--------+              |   |
   +----+        |              |   |
                 v              |   |
   +----+      +---------+      |   |
   | T2 +----->|IO Thread|----->|   |
   +----+      +---------+      |   |
                 ^              |   |
   +----+        |              |   |
   | T3 +--------+              |   |
   +----+                       |   |
                                +---+

That way all IO to a particular file is serialized, avoiding any issues with the OS.
@jamesls jamesls mentioned this issue Jan 29, 2014
Merged
jamesls added a commit to jamesls/aws-cli that referenced this issue Feb 24, 2014
@jamesls
Fix hanging Ctrl+C on S3 downloads
4263d0d 
This is a regression that was introduced when the IO queue/thread
was introduced (aws#619).

The current shutdown order of the threads is:

1. Queue end sentinel for IO thread.
2. Queue end sentinel for result thread.
3. Queue end sentinel for worker threads.
4. .join() threads in this order:
   [io_thread, result_thread [, worker threads]]

Though the actual thread shutdown order is non-deterministic,
it's fairly common that the threads shutdown in roughly the above
order.  This means that the IO thread will generally shutdown before
all the worker threads have shutdown.

However, the download tasks can still be enqueueing writes to the
IO queue.  If the IO thread shutsdown there's nothing consuming
writes on the other end of the queue.  Given that the queue is
bounded in maxsize, .put() calls to the queue will block until space
becomes available.  This will never happen if the IO queue is already
shutdown.

The fix here is to ensure that the IO thread is always the last thing
to shutdown.  This means any remaining IO requests will be executed
before the IO thread shutsdown.  This will prevent deadlock.

Added unit tests demonstrates this issue.  I've also added an
integration test that actually sends a SIGINT to the process
and verifies it exits in a timely manner so can ensure we
don't regress on this again.

Note: some unit/integ tests needed updating because they were
using .call() multiple times.

Fixes aws#650
Fixes aws#657
@jamesls jamesls mentioned this issue Feb 24, 2014
Merged
thoward-godaddy pushed a commit to thoward-godaddy/aws-cli that referenced this issue Feb 12, 2022
@heitorlessa @jfuss
UX: Improves event names to reflect Lambda Event Sources (aws#619)
6e3740a
thoward-godaddy pushed a commit to thoward-godaddy/aws-cli that referenced this issue Feb 12, 2022
@jfuss
Release: 0.6.0 Release (aws#636)
87a4577 
* fix: Functional tests must run on localhost to work in Windows (aws#552)

* fix: spacing typo in Log statement in start-lambda (aws#559)

* docs: Fix syntax highlighting in README.md (aws#561)

* docs: Change jest to mocha in Nodejs init README (aws#564)

* docs: Fix @mhart link in README (aws#562)

* docs(README): removed cloudtrail, added SNS to generate-event (aws#569)

* docs: Update repo name references (aws#577)

* feat(debugging): Fixing issues around debugging Golang functions. (aws#565)

* fix(init): Improve current init samples around docs and fixes (aws#558)

* docs(README): Update launch config to SAM CLI from SAM Local (aws#587)

* docs(README): Update sample code for calling Local Lambda Invoke (aws#584)

* refactor(init): renamed handler for camel case, moved callback call up (aws#586)

* chore: aws-lambda-java-core 1.1.0 -> 1.2.0 for java sam init (aws#578)

* feat(validate): Add profile and region options (aws#582)

Currently, `sam validate` requires AWS Creds (due to the SAM Translator).
This commits adds the ability to pass in the credientials through a profile
that is configured through `aws configure`.

* docs(README): Update README prerequisites to include awscli (aws#596)

* fix(start-lambda): Remove Content-Type Header check (aws#594)

* docs: Disambiguation "Amazon Kinesis" (aws#599)

* docs: Adding instructions for how to add pyenv to your PATH for Windows (aws#600)

* docs: Update README with small grammar fix (aws#601)

* fix: Update link in NodeJS package.json (aws#603)

* docs: Creating instructions for Windows users to install sam (aws#605)

* docs: Adding a note directing Windows users to use pipenv (aws#606)

* fix: Fix stringifying λ environment variables when using Python2 (aws#579)

* feat(generate-event): Added support for 50+ events (aws#612)

* feat(invoke): Add region parameter to all invoke related commands (aws#608)

* docs: Breaking up README into separate files to make it easier to read (aws#607)

* chore: Update JVM size params to match docker-lambda (aws#615)

* feat(invoke): Invoke Function Without Parameters through --no-event (aws#604)

* docs: Update advanced_usage.rst with clarification on --env-vars usage (aws#610)

* docs: Remove an extra word in the sam packaging command (aws#618)

* UX: Improves event names to reflect Lambda Event Sources (aws#619)

* docs: Fix git clone typo in installation docs (aws#630)

* docs(README): Callout go1.x runtime support (aws#631)

* docs(installation): Update sam --version command (aws#634)

* chore(0.6.0): SAM CLI Version bump (aws#635)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use a single IO thread when downloading S3 objects jamesls/aws-cli
1 participant
@jamesls