Skip to content

Commit

Permalink
Add clarification on multithreading and multiprocessing for resources
Browse files Browse the repository at this point in the history
  • Loading branch information
ryansonshine committed May 14, 2021
1 parent 8a6bd89 commit e0bde37
Showing 1 changed file with 16 additions and 6 deletions.
22 changes: 16 additions & 6 deletions docs/source/guide/resources.rst
Original file line number Diff line number Diff line change
Expand Up @@ -199,22 +199,32 @@ keyword arguments. Examples of waiters include::


Multithreading and multiprocessing
--------------------------------
It is recommended to create a resource instance for each thread / process in a multithreaded or multiprocess application rather than sharing a single instance among the threads / processes. For example::
----------------------------------
SDK object instances are **not** thread safe in almost all cases and should not be shared across threads or processes. You should create a new SDK object for each thread or process::

import boto3
import boto3.session
import threading

class MyTask(threading.Thread):
def run(self):
# Here we create a new session per thread
session = boto3.session.Session()

# Next, we create a resource client using our thread's session object
s3 = session.resource('s3')
# ... do some work with S3 ...

# Put your thread-safe code here

In the example above, each thread would have its own Boto3 session and its own instance of the S3 resource. This is a good idea because resources contain shared data when loaded and calling actions, accessing properties, or manually loading or reloading the resource can modify this data.

.. note::
Resources are **not** thread safe. These special classes contain additional meta data that cannot be shared between threads. When using a Resource, it is recommended to instantiate a new Resource for each thread, as is shown in the example above.

Low-level clients **are** thread safe. When using a low-level client, it is recommended to instantiate your client then pass that client object to each of your threads.
Resources are **not** thread safe. These special classes contain additional meta data that cannot be shared between threads. When using a Resource, it is recommended to instantiate a new Resource for each thread, as is shown in the example above.

Sessions are **not** thread safe. This is due to shared meta data, similar to Resource.

Low-level clients **are** thread safe in a **threaded environment** in all cases when using the client's attached operation methods (i.e. ``s3.get_object``). However, clients are **not** thread safe if you are explicitly using the client to interact with the underlying Session (i.e. using ``s3.meta``, ``s3.exceptions``, or ``s3.waiter_names``).

Low-level clients are **not** safe in a **multiprocess environment**. This is due to `forking issues <https://github.com/psf/requests/issues/4323>`__ with urllib3's connection pool.


0 comments on commit e0bde37

Please sign in to comment.