Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRAFT] feat: improve error group tracebacks on < py11 #823

Closed
wants to merge 227 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
227 commits
Select commit Hold shift + click to select a range
748310b
chore(deps): Update nox in .kokoro/requirements.in [autoapprove] (#750)
gcf-owl-bot[bot] Mar 16, 2023
45d3e43
docs: Fix formatting of request arg in docstring (#756)
gcf-owl-bot[bot] Mar 23, 2023
505273b
fix: Pass the "retry" when calling read_rows. (#759)
Mariatta Mar 30, 2023
efe332c
chore(samples): Fixed a typo in the readme (#760)
shweta345 Apr 5, 2023
18b8f18
chore(deps): update all dependencies (#746)
renovate-bot Apr 6, 2023
128b4e1
docs: fix delete from column family example (#764)
billyjacobson Apr 14, 2023
b5d10d6
chore(deps): update all dependencies (#763)
renovate-bot Apr 18, 2023
1d02154
added initial implementation of mutate_rows
daniel-sanche Apr 24, 2023
ab63cba
implemented mutation models
daniel-sanche Apr 24, 2023
cf9daa5
added retries to mutate_row
daniel-sanche Apr 24, 2023
1247da4
return exception group if possible
daniel-sanche Apr 24, 2023
3b3ed8c
check for idempotence
daniel-sanche Apr 24, 2023
5d20037
initial implementation for bulk_mutations
daniel-sanche Apr 24, 2023
3d322a1
include successes in bulk mutation error message
daniel-sanche Apr 24, 2023
a31232b
fixed style checks
daniel-sanche Apr 24, 2023
8da2d65
added basic system tests
daniel-sanche Apr 24, 2023
2b89d9c
added unit tests for mutate_row
daniel-sanche Apr 25, 2023
47c5985
ran blacken
daniel-sanche Apr 25, 2023
38fdcd7
improved exceptions
daniel-sanche Apr 25, 2023
504d2d8
added bulk_mutate_rows unit tests
daniel-sanche Apr 25, 2023
b16067f
ran blacken
daniel-sanche Apr 25, 2023
3ab1405
support __new___ for exceptions for python3.11+
daniel-sanche Apr 25, 2023
0a6c0c6
added exception unit tests
daniel-sanche Apr 25, 2023
ec043cf
makde exceptions tuple
daniel-sanche Apr 26, 2023
518530e
got exceptions to print consistently across versions
daniel-sanche Apr 26, 2023
9624729
added test for 311 rich traceback
daniel-sanche Apr 27, 2023
3087081
moved retryable row mutations to new file
daniel-sanche Apr 27, 2023
9df588f
use index map
daniel-sanche Apr 27, 2023
7ed8be3
added docstring
daniel-sanche Apr 27, 2023
2536cc4
added predicate check to failed mutations
daniel-sanche Apr 27, 2023
1f6875c
added _mutate_rows tests
daniel-sanche Apr 27, 2023
1ea24e6
improved client tests
daniel-sanche Apr 27, 2023
25ca2d2
refactored to loop by raising exception
daniel-sanche Apr 28, 2023
c0787db
refactored retry deadline logic into shared wrapper
daniel-sanche Apr 28, 2023
3ed5c3d
ran black
daniel-sanche Apr 28, 2023
a91fbcb
pulled in table default timeouts
daniel-sanche Apr 28, 2023
df8a058
added tests for shared deadline parsing function
daniel-sanche Apr 28, 2023
b866b57
added tests for mutation models
daniel-sanche Apr 28, 2023
54a4d43
fixed linter errors
daniel-sanche Apr 28, 2023
bd51dc4
added tests for BulkMutationsEntry
daniel-sanche Apr 28, 2023
921b05a
improved mutations documentation
daniel-sanche Apr 28, 2023
53f7ad2
added basic implementation for batcher
daniel-sanche May 1, 2023
f65b2e5
added flush timer
daniel-sanche May 1, 2023
7051cd2
added flush on size and count
daniel-sanche May 1, 2023
9731f4c
improved queue
daniel-sanche May 1, 2023
34bf655
raise exceptions on close
daniel-sanche May 1, 2023
582457a
added args for flush
daniel-sanche May 1, 2023
eeb24a5
improved closing logic
daniel-sanche May 1, 2023
110cd96
use custom queue type to track size and count
daniel-sanche May 1, 2023
375a77d
added flow control
daniel-sanche May 1, 2023
5f9fa3f
added comments and todos
daniel-sanche May 1, 2023
e63cfc2
allow None for batcher limits
daniel-sanche May 2, 2023
d5e15aa
strip indices from exceptions
daniel-sanche May 2, 2023
82ea61f
refactored mutate_rows logic into helper function
daniel-sanche May 2, 2023
fa42b86
implemented callbacks for mutate_rows
daniel-sanche May 2, 2023
a47aeda
Merge branch 'mutate_rows' into mutate_rows_batch
daniel-sanche May 2, 2023
ee23721
release locks as mutations are completed, instead of waiting until end
daniel-sanche May 2, 2023
b69cc96
refacotr row processing
daniel-sanche May 2, 2023
9e1850c
FlowControl returns exceptions rather than raises them
daniel-sanche May 2, 2023
9e4d013
raise previous exceptions on flush with flag set
daniel-sanche May 2, 2023
2c7ec9c
disallow empty bulk mutation entries
daniel-sanche May 3, 2023
5c4f739
simplified staged mutation store
daniel-sanche May 3, 2023
fa4947f
ran blacken
daniel-sanche May 3, 2023
914b24b
replaced semaphores with condition
daniel-sanche May 3, 2023
8bda651
fixed mypy issues
daniel-sanche May 3, 2023
742194b
added raise_exceptions fn
daniel-sanche May 3, 2023
45fe1f5
fixed failing tests
daniel-sanche May 3, 2023
d8f3a4d
added skeleton for tests
daniel-sanche May 3, 2023
171fea6
feat: publish RateLimitInfo and FeatureFlag protos (#768)
gcf-owl-bot[bot] May 4, 2023
7faebc0
improved and tested flow control
daniel-sanche May 4, 2023
10da89a
refactored flushes to run sequentially
daniel-sanche May 5, 2023
0eca65e
added batcher tests
daniel-sanche May 5, 2023
2532b63
ran blacken
daniel-sanche May 5, 2023
8313b57
added additional tests
daniel-sanche May 5, 2023
01a16f3
made exceptions into a tuple
daniel-sanche May 5, 2023
da9b6cf
Merge branch 'mutate_rows' into mutate_rows_batch
daniel-sanche May 5, 2023
b7d3fcf
added warning for unflushed mutation entries
daniel-sanche May 5, 2023
7521a61
Feat: Threaded MutationsBatcher (#722)
Mariatta May 10, 2023
8f9de8d
chore(main): release 2.18.0 (#757)
release-please[bot] May 10, 2023
a767cff
fix: Revert "Feat: Threaded MutationsBatcher" (#773)
mutianf May 11, 2023
a4e7325
Revert "fix: Revert "Feat: Threaded MutationsBatcher" (#773)" (#775)
mutianf May 11, 2023
9e35ad2
chore(main): release 2.18.1 (#774)
release-please[bot] May 16, 2023
6140acb
remove aborted from retryable errors
daniel-sanche May 22, 2023
36ba2b6
improved SetCell mutation
daniel-sanche May 22, 2023
b3c9017
fixed mutations tests
daniel-sanche May 22, 2023
cac9e2d
SetCell timestamps use millisecond precision
daniel-sanche May 22, 2023
34b051f
renamed BulkMutationsEntry to RowMutationEntry
daniel-sanche May 22, 2023
63ac35c
Merge branch 'v3' into mutate_rows
daniel-sanche May 24, 2023
bb72ea8
Merge branch 'mutate_rows' into mutate_rows_batch
daniel-sanche May 24, 2023
cef70f2
feat: add ChangeStreamConfig to CreateTable and UpdateTable (#786)
gcf-owl-bot[bot] May 25, 2023
ae477b5
build(deps): bump requests from 2.28.1 to 2.31.0 in /synthtool/gcp/te…
gcf-owl-bot[bot] May 25, 2023
ecf539c
docs(samples): add region tags (#788)
Sita04 May 25, 2023
a51201c
added metadata to mutate rows and bulk mutate rows
daniel-sanche May 25, 2023
a21bebf
moved _convert_retry_deadline wrapper from exceptions into _helpers
daniel-sanche May 25, 2023
4ca89d9
fixed system tests
daniel-sanche May 25, 2023
b240ee1
only handle precision adjustment when creating timestamp
daniel-sanche May 26, 2023
cb0e951
added _from_dict for mutation models
daniel-sanche May 26, 2023
a9cf385
rpc timeouts adjust when approaching operation_timeout
daniel-sanche May 26, 2023
eddc1c9
pass table instead of request dict
daniel-sanche May 26, 2023
f8b26aa
refactoring mutate rows
daniel-sanche May 27, 2023
5b80dc5
made on_terminal_state into coroutine
daniel-sanche May 27, 2023
9e5b80a
fixed style issues
daniel-sanche May 27, 2023
f7539f6
moved callback rewriting into retryable attempt
daniel-sanche May 27, 2023
e77a4fa
fixed tests
daniel-sanche May 27, 2023
4e19ed0
pop successful mutations from error dict
daniel-sanche May 30, 2023
920e4b7
removed unneeded check
daniel-sanche May 30, 2023
725f5ff
refactoring
daniel-sanche May 30, 2023
1054bc4
pass list of exceptions in callback
daniel-sanche May 30, 2023
f39a891
raise error in unexpected state
daniel-sanche May 30, 2023
e5f65f2
Merge branch 'mutate_rows' into mutate_rows_batch
daniel-sanche May 31, 2023
8038662
added timeout to user facing flush
daniel-sanche May 31, 2023
c2ec26d
added comments
daniel-sanche May 31, 2023
1d97135
removed callback
daniel-sanche May 31, 2023
88e2bf5
refactoring mutation attempt into class
daniel-sanche May 31, 2023
a3c0166
use partial function
daniel-sanche May 31, 2023
70c35ef
renamed class
daniel-sanche May 31, 2023
e00f592
added comments
daniel-sanche Jun 1, 2023
18af78a
added tests
daniel-sanche Jun 1, 2023
23e84f5
improved helpers
daniel-sanche Jun 1, 2023
c4fd78f
build(deps): bump cryptography from 39.0.1 to 41.0.0 in /synthtool/gc…
gcf-owl-bot[bot] Jun 3, 2023
56fdf7c
refactored operation into class only
daniel-sanche Jun 5, 2023
aca31f0
restructured how remaining indices are tracked
daniel-sanche Jun 5, 2023
5a5d541
fixed tests
daniel-sanche Jun 5, 2023
afed731
added docstrings
daniel-sanche Jun 5, 2023
2396ec8
moved index deletion to end of block
daniel-sanche Jun 6, 2023
3d441a2
added comment to exception types
daniel-sanche Jun 6, 2023
0013c4d
Merge branch 'mutate_rows' into mutate_rows_batch
daniel-sanche Jun 6, 2023
8b41518
update flow control after batch, instead of after each entry
daniel-sanche Jun 6, 2023
5a8b9e9
Merge branch 'v3' into mutate_rows_batch
daniel-sanche Jun 6, 2023
bdd7f45
fixed test
daniel-sanche Jun 6, 2023
553bc4a
add kwargs docstring
daniel-sanche Jun 7, 2023
20156c6
allow mutations over flow limits
daniel-sanche Jun 7, 2023
b9c4467
added limits for mutate_rows mutation and entry counts
daniel-sanche Jun 7, 2023
8b8d8ab
refactored mutation system tests
daniel-sanche Jun 7, 2023
a1de714
added batcher system tests
daniel-sanche Jun 7, 2023
2382a04
allow batcher to append multiple entries
daniel-sanche Jun 7, 2023
1db0377
improved timeout handling
daniel-sanche Jun 7, 2023
589aa5d
fix: add a callback function on flush_rows (#796)
mutianf Jun 8, 2023
fbe6ace
chore(main): release 2.19.0 (#789)
release-please[bot] Jun 8, 2023
6a66c48
made batch rpcs concurrent
daniel-sanche Jun 14, 2023
e1718f0
allow single mutations in flow control public methods
daniel-sanche Jun 14, 2023
eeda2e9
mutation batcher only accepts single entries
daniel-sanche Jun 15, 2023
eeb483f
ensure that proper exceptions are raised for each flush
daniel-sanche Jun 15, 2023
853b5f2
surface unexpected exceptions in MutationsExceptionGroup
daniel-sanche Jun 15, 2023
2865acd
prefixed flow control attributes with underscore
daniel-sanche Jun 15, 2023
e568d92
batcher signature improvements
daniel-sanche Jun 15, 2023
bc7b850
use request mutation limit instead of entry limit
daniel-sanche Jun 15, 2023
ff0aed6
reduce repeated flush tasks when entries appended in a loop
daniel-sanche Jun 15, 2023
f71eebc
renamed variables
daniel-sanche Jun 15, 2023
c0dc760
added test; renamed variables
daniel-sanche Jun 15, 2023
a6b12dc
removed outdated comment
daniel-sanche Jun 16, 2023
d5128eb
added comments
daniel-sanche Jun 16, 2023
335391c
renamed variable
daniel-sanche Jun 16, 2023
836171b
made append async; removed _pending_entries
daniel-sanche Jun 20, 2023
06f16bc
use futures instead of tasks for type hints
daniel-sanche Jun 20, 2023
09a5d51
refactored asyncio calls out to make sync implementation easier
daniel-sanche Jun 20, 2023
0b0f30e
renamed flow_control_max_count
daniel-sanche Jun 21, 2023
283a949
removed None as a flow_control_max_* option
daniel-sanche Jun 21, 2023
834db1a
fixed type annotation
daniel-sanche Jun 21, 2023
524f2d8
fixed tests
daniel-sanche Jun 21, 2023
df8d2d5
improved docstrings
daniel-sanche Jun 21, 2023
eaed9ab
added timeouts to batcher init
daniel-sanche Jun 21, 2023
e0b5e27
fixed 3.11 test issue
daniel-sanche Jun 22, 2023
d50d89b
fixed 3.7 unit test issues
daniel-sanche Jun 22, 2023
32d39fd
fixed lint issue
daniel-sanche Jun 22, 2023
1343b34
fixed lint
daniel-sanche Jun 22, 2023
4f3aa3a
fixed up unit tests
daniel-sanche Jun 22, 2023
0b99b89
added ping and warm to system test
daniel-sanche Jun 22, 2023
6a35be4
fixed broken ping and warm
daniel-sanche Jun 22, 2023
aa35901
added table and app profile to warming params
daniel-sanche Jun 22, 2023
4e1ec6f
fixed _remove_instance_registration
daniel-sanche Jun 22, 2023
8f4b34f
improved tests
daniel-sanche Jun 22, 2023
9c8df9f
allow warming for single instance
daniel-sanche Jun 22, 2023
584f94d
updated tests
daniel-sanche Jun 22, 2023
3c605ba
fixed metadata format
daniel-sanche Jun 22, 2023
12099f5
remove index from exceptions
daniel-sanche Jun 23, 2023
6833707
added test for memory leaks
daniel-sanche Jun 23, 2023
4c40299
clear task after completion
daniel-sanche Jun 23, 2023
2931aa8
added exception list truncation
daniel-sanche Jun 23, 2023
1dc760e
removed index information
daniel-sanche Jun 23, 2023
18fd309
Merge branch 'v3' into warm_improvements
daniel-sanche Jun 23, 2023
badbb83
fixed existing tests
daniel-sanche Jun 23, 2023
97e6680
Merge branch 'v3' into mutate_rows_batch
daniel-sanche Jun 23, 2023
6da9a64
removed manual flushing
daniel-sanche Jun 23, 2023
0ecbbcd
added tests
daniel-sanche Jun 23, 2023
61b367a
pr comment improvements
daniel-sanche Jun 23, 2023
0ce16c6
fixed metadata
daniel-sanche Jun 23, 2023
697cefd
Merge branch 'mutate_rows_batch' into restructure_paths
daniel-sanche Jun 23, 2023
9a407f0
moved v3 client into /data folder
daniel-sanche Jun 23, 2023
c1587e5
moved legacy client back into root
daniel-sanche Jun 23, 2023
a6a6b7b
moved async classes into own sub-folder
daniel-sanche Jun 23, 2023
7e7a667
moved ReadRowsIterator into _read_rows
daniel-sanche Jun 23, 2023
66afd15
moved unit tests into data folder
daniel-sanche Jun 23, 2023
fe4e280
fixed v2 unit tests
daniel-sanche Jun 23, 2023
badaf65
rewrote some test paths
daniel-sanche Jun 24, 2023
48b2d2c
renamed async subpath to _async
daniel-sanche Jun 24, 2023
c00b395
fixed imports
daniel-sanche Jun 24, 2023
e4a6546
split up test files for state machine class
daniel-sanche Jun 24, 2023
1191444
got unit tests running
daniel-sanche Jun 24, 2023
2ca05d8
fixed batcher tests
daniel-sanche Jun 24, 2023
6f12cda
got unit tests passing
daniel-sanche Jun 24, 2023
67f5a3d
fixed system tests
daniel-sanche Jun 24, 2023
d7b6dca
fixed v2 system tests
daniel-sanche Jun 24, 2023
14a3d3d
pin to api_core with streaming retries
daniel-sanche Jun 24, 2023
563a133
fixed type errors
daniel-sanche Jun 24, 2023
b030fb5
updated submodule
daniel-sanche Jun 24, 2023
ac0ef64
fixed batcher comments
daniel-sanche Jun 26, 2023
0b8a651
Merge branch 'main' into restructure_paths
daniel-sanche Jun 26, 2023
d90430a
Merge branch 'v3' into restructure_paths
daniel-sanche Jun 26, 2023
4122a16
updated docs folder
daniel-sanche Jun 26, 2023
ff325ef
changed mypy to only check data client
daniel-sanche Jun 26, 2023
e996946
fixed some test issues
daniel-sanche Jun 26, 2023
d63d68c
reverted gapic tests
daniel-sanche Jun 26, 2023
5b7db3e
added verbose logging for unit tests
daniel-sanche Jun 26, 2023
9947afd
fixed atexit test that was breaking coverage reports
daniel-sanche Jun 26, 2023
9e8a0b3
removed external api-core installation for prerelease-deps
daniel-sanche Jun 26, 2023
30b9b11
improve timeouts
daniel-sanche Jun 27, 2023
b6ad653
fixed docstrings
daniel-sanche Jun 27, 2023
eb55460
chore: remove pinned Sphinx version [autoapprove] (#818)
gcf-owl-bot[bot] Jun 28, 2023
10d12b5
Merge branch 'v3' into improve-timeouts
daniel-sanche Jun 28, 2023
12c388c
Merge branch 'main' into improve-timeouts
daniel-sanche Jun 28, 2023
a723cb2
improve exception messages
daniel-sanche Jun 28, 2023
ccc9ae4
attach cause in more general way
daniel-sanche Jun 28, 2023
f0b3558
fixed tests for 3.11
daniel-sanche Jun 28, 2023
e2f263c
added improved output tests for 3.11
daniel-sanche Jun 28, 2023
6f269a8
got 3.10 tests passing
daniel-sanche Jun 28, 2023
e090689
added tests for 3.10- output
daniel-sanche Jun 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/.OwlBot.lock.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@
# limitations under the License.
docker:
image: gcr.io/cloud-devrel-public-resources/owlbot-python:latest
digest: sha256:240b5bcc2bafd450912d2da2be15e62bc6de2cf839823ae4bf94d4f392b451dc
# created: 2023-06-03T21:25:37.968717478Z
digest: sha256:ddf4551385d566771dc713090feb7b4c1164fb8a698fe52bbe7670b24236565b
# created: 2023-06-27T13:04:21.96690344Z
6 changes: 3 additions & 3 deletions google/cloud/bigtable/data/_async/_mutate_rows.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,15 +52,15 @@ def __init__(
table: "TableAsync",
mutation_entries: list["RowMutationEntry"],
operation_timeout: float,
per_request_timeout: float | None,
attempt_timeout: float | None,
):
"""
Args:
- gapic_client: the client to use for the mutate_rows call
- table: the table associated with the request
- mutation_entries: a list of RowMutationEntry objects to send to the server
- operation_timeout: the timeout t o use for the entire operation, in seconds.
- per_request_timeout: the timeoutto use for each mutate_rows attempt, in seconds.
- attempt_timeout: the timeoutto use for each mutate_rows attempt, in seconds.
If not specified, the request will run until operation_timeout is reached.
"""
# check that mutations are within limits
Expand Down Expand Up @@ -99,7 +99,7 @@ def __init__(
self._operation = _convert_retry_deadline(retry_wrapped, operation_timeout)
# initialize state
self.timeout_generator = _attempt_timeout_generator(
per_request_timeout, operation_timeout
attempt_timeout, operation_timeout
)
self.mutations = mutation_entries
self.remaining_indices = list(range(len(self.mutations)))
Expand Down
6 changes: 3 additions & 3 deletions google/cloud/bigtable/data/_async/_read_rows.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,14 +63,14 @@ def __init__(
client: BigtableAsyncClient,
*,
operation_timeout: float = 600.0,
per_request_timeout: float | None = None,
attempt_timeout: float | None = None,
):
"""
Args:
- request: the request dict to send to the Bigtable API
- client: the Bigtable client to use to make the request
- operation_timeout: the timeout to use for the entire operation, in seconds
- per_request_timeout: the timeout to use when waiting for each individual grpc request, in seconds
- attempt_timeout: the timeout to use when waiting for each individual grpc request, in seconds
If not specified, defaults to operation_timeout
"""
self._last_emitted_row_key: bytes | None = None
Expand All @@ -79,7 +79,7 @@ def __init__(
self.operation_timeout = operation_timeout
# use generator to lower per-attempt timeout as we approach operation_timeout deadline
attempt_timeout_gen = _attempt_timeout_generator(
per_request_timeout, operation_timeout
attempt_timeout, operation_timeout
)
row_limit = request.get("rows_limit", 0)
# lock in paramters for retryable wrapper
Expand Down
336 changes: 206 additions & 130 deletions google/cloud/bigtable/data/_async/client.py

Large diffs are not rendered by default.

28 changes: 11 additions & 17 deletions google/cloud/bigtable/data/_async/mutations_batcher.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
from google.cloud.bigtable.data.mutations import RowMutationEntry
from google.cloud.bigtable.data.exceptions import MutationsExceptionGroup
from google.cloud.bigtable.data.exceptions import FailedMutationEntryError
from google.cloud.bigtable.data._helpers import _validate_timeouts

from google.cloud.bigtable.data._async._mutate_rows import _MutateRowsOperationAsync
from google.cloud.bigtable.data._async._mutate_rows import (
Expand Down Expand Up @@ -189,7 +190,7 @@ def __init__(
flow_control_max_mutation_count: int = 100_000,
flow_control_max_bytes: int = 100 * _MB_SIZE,
batch_operation_timeout: float | None = None,
batch_per_request_timeout: float | None = None,
batch_attempt_timeout: float | None = None,
):
"""
Args:
Expand All @@ -203,25 +204,18 @@ def __init__(
- flow_control_max_bytes: Maximum number of inflight bytes.
- batch_operation_timeout: timeout for each mutate_rows operation, in seconds. If None,
table default_operation_timeout will be used
- batch_per_request_timeout: timeout for each individual request, in seconds. If None,
table default_per_request_timeout will be used
- batch_attempt_timeout: timeout for each individual request, in seconds. If None,
table default_attempt_timeout will be used
"""
self._operation_timeout: float = (
batch_operation_timeout or table.default_operation_timeout
batch_operation_timeout or table.default_mutate_rows_operation_timeout
)
self._per_request_timeout: float = (
batch_per_request_timeout
or table.default_per_request_timeout
self._attempt_timeout: float = (
batch_attempt_timeout
or table.default_mutate_rows_attempt_timeout
or self._operation_timeout
)
if self._operation_timeout <= 0:
raise ValueError("batch_operation_timeout must be greater than 0")
if self._per_request_timeout <= 0:
raise ValueError("batch_per_request_timeout must be greater than 0")
if self._per_request_timeout > self._operation_timeout:
raise ValueError(
"batch_per_request_timeout must be less than batch_operation_timeout"
)
_validate_timeouts(self._operation_timeout, self._attempt_timeout)
self.closed: bool = False
self._table = table
self._staged_entries: list[RowMutationEntry] = []
Expand Down Expand Up @@ -346,7 +340,7 @@ async def _execute_mutate_rows(

Args:
- batch: list of RowMutationEntry objects to send to server
- timeout: timeout in seconds. Used as operation_timeout and per_request_timeout.
- timeout: timeout in seconds. Used as operation_timeout and attempt_timeout.
If not given, will use table defaults
Returns:
- list of FailedMutationEntryError objects for mutations that failed.
Expand All @@ -361,7 +355,7 @@ async def _execute_mutate_rows(
self._table,
batch,
operation_timeout=self._operation_timeout,
per_request_timeout=self._per_request_timeout,
attempt_timeout=self._attempt_timeout,
)
await operation.start()
except MutationsExceptionGroup as e:
Expand Down
27 changes: 27 additions & 0 deletions google/cloud/bigtable/data/_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,3 +109,30 @@ def wrapper(*args, **kwargs):
handle_error()

return wrapper_async if iscoroutinefunction(func) else wrapper


def _validate_timeouts(
operation_timeout: float, attempt_timeout: float | None, allow_none: bool = False
):
"""
Helper function that will verify that timeout values are valid, and raise
an exception if they are not.

Args:
- operation_timeout: The timeout value to use for the entire operation, in seconds.
- attempt_timeout: The timeout value to use for each attempt, in seconds.
- allow_none: If True, attempt_timeout can be None. If False, None values will raise an exception.
Raises:
- ValueError if operation_timeout or attempt_timeout are invalid.
"""
if operation_timeout <= 0:
raise ValueError("operation_timeout must be greater than 0")
if not allow_none and attempt_timeout is None:
raise ValueError("attempt_timeout must not be None")
elif attempt_timeout is not None:
if attempt_timeout <= 0:
raise ValueError("attempt_timeout must be greater than 0")
if attempt_timeout > operation_timeout:
raise ValueError(
"attempt_timeout must not be greater than operation_timeout"
)
54 changes: 39 additions & 15 deletions google/cloud/bigtable/data/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,27 @@ def __init__(self, message, excs):
if len(excs) == 0:
raise ValueError("exceptions must be a non-empty sequence")
self.exceptions = tuple(excs)
super().__init__(message)
# simulate an exception group in Python < 3.11 by adding exception info
# to the message
first_line = "--+---------------- 1 ----------------"
last_line = "+------------------------------------"
message_parts = [message + "\n" + first_line]
for idx, e in enumerate(excs[:15]):
if idx != 0:
message_parts.append(f"+---------------- {str(idx+1).rjust(2)} ----------------")
cause = e.__cause__
if cause is not None:
message_parts.extend(f"| {type(cause).__name__}: {cause}".splitlines())
message_parts.append("| ")
message_parts.append("| The above exception was the direct cause of the following exception:")
message_parts.append("| ")
message_parts.extend(f"| {type(e).__name__}: {e}".splitlines())
if len(excs) > 15:
message_parts.append("+---------------- ... ---------------")
message_parts.append(f"| and {len(excs) - 15} more")
if last_line not in message_parts[-1]:
message_parts.append(last_line)
super().__init__("\n ".join(message_parts))

def __new__(cls, message, excs):
if is_311_plus:
Expand All @@ -83,11 +103,19 @@ def __new__(cls, message, excs):
return super().__new__(cls)

def __str__(self):
if is_311_plus:
# don't return built-in sub-exception message
return self.args[0]
return super().__str__()

def __repr__(self):
"""
String representation doesn't display sub-exceptions. Subexceptions are
described in message
repr representation should strip out sub-exception details
"""
return self.args[0]
if is_311_plus:
return super().__repr__()
message = self.args[0].split("\n")[0]
return f"{self.__class__.__name__}({message!r}, {self.exceptions!r})"


class MutationsExceptionGroup(_BigtableExceptionGroup):
Expand Down Expand Up @@ -200,14 +228,12 @@ def __init__(
idempotent_msg = (
"idempotent" if failed_mutation_entry.is_idempotent() else "non-idempotent"
)
index_msg = f" at index {failed_idx} " if failed_idx is not None else " "
message = (
f"Failed {idempotent_msg} mutation entry{index_msg}with cause: {cause!r}"
)
index_msg = f" at index {failed_idx}" if failed_idx is not None else ""
message = f"Failed {idempotent_msg} mutation entry{index_msg}"
super().__init__(message)
self.__cause__ = cause
self.index = failed_idx
self.entry = failed_mutation_entry
self.__cause__ = cause


class RetryExceptionGroup(_BigtableExceptionGroup):
Expand All @@ -217,10 +243,8 @@ class RetryExceptionGroup(_BigtableExceptionGroup):
def _format_message(excs: list[Exception]):
if len(excs) == 0:
return "No exceptions"
if len(excs) == 1:
return f"1 failed attempt: {type(excs[0]).__name__}"
else:
return f"{len(excs)} failed attempts. Latest: {type(excs[-1]).__name__}"
plural = "s" if len(excs) > 1 else ""
return f"{len(excs)} failed attempt{plural}"

def __init__(self, excs: list[Exception]):
super().__init__(self._format_message(excs), excs)
Expand Down Expand Up @@ -268,8 +292,8 @@ def __init__(
failed_query: "ReadRowsQuery" | dict[str, Any],
cause: Exception,
):
message = f"Failed query at index {failed_index} with cause: {cause!r}"
message = f"Failed query at index {failed_index}"
super().__init__(message)
self.__cause__ = cause
self.index = failed_index
self.query = failed_query
self.__cause__ = cause
3 changes: 1 addition & 2 deletions noxfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -366,10 +366,9 @@ def docfx(session):

session.install("-e", ".")
session.install(
"sphinx==4.0.1",
"gcp-sphinx-docfx-yaml",
"alabaster",
"recommonmark",
"gcp-sphinx-docfx-yaml",
)

shutil.rmtree(os.path.join("docs", "_build"), ignore_errors=True)
Expand Down
4 changes: 2 additions & 2 deletions tests/unit/data/_async/test__mutate_rows.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ def _make_one(self, *args, **kwargs):
kwargs["table"] = kwargs.pop("table", AsyncMock())
kwargs["mutation_entries"] = kwargs.pop("mutation_entries", [])
kwargs["operation_timeout"] = kwargs.pop("operation_timeout", 5)
kwargs["per_request_timeout"] = kwargs.pop("per_request_timeout", 0.1)
kwargs["attempt_timeout"] = kwargs.pop("attempt_timeout", 0.1)
return self._target_class()(*args, **kwargs)

async def _mock_stream(self, mutation_list, error_dict):
Expand Down Expand Up @@ -267,7 +267,7 @@ async def test_run_attempt_single_entry_success(self):
mock_gapic_fn = self._make_mock_gapic({0: mutation})
instance = self._make_one(
mutation_entries=[mutation],
per_request_timeout=expected_timeout,
attempt_timeout=expected_timeout,
)
with mock.patch.object(instance, "_gapic_fn", mock_gapic_fn):
await instance._run_attempt()
Expand Down
2 changes: 1 addition & 1 deletion tests/unit/data/_async/test__read_rows.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ def test_ctor(self):
request,
client,
operation_timeout=expected_operation_timeout,
per_request_timeout=expected_request_timeout,
attempt_timeout=expected_request_timeout,
)
assert time_gen_mock.call_count == 1
time_gen_mock.assert_called_once_with(
Expand Down
Loading