Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable question chaining #660

Merged
merged 215 commits into from
Jul 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
215 commits
Select commit Hold shift + click to select a range
788fb24
REF: Factor out number of questions in `Child.ask_multiple`
cortadocodes May 29, 2024
8725ce9
ENH: Log ultimately failed questions in `Child.ask_multiple`
cortadocodes May 29, 2024
00dfb21
ENH: Improve logging of errors in `Child.ask_multiple`
cortadocodes May 29, 2024
e01738f
TST: Test logging errors from `Child.ask_multiple`
cortadocodes May 29, 2024
fa7162f
ENH: Allow existing subscriptions in `create_push_subscription`
cortadocodes May 29, 2024
f47dd96
ENH: Add `overrides` option to `Runner.from_configuration`
cortadocodes May 29, 2024
b306a20
ENH: Make datasets recursive by default in `Dataset.__init__`
cortadocodes May 29, 2024
ce42be2
WIP: Temporarily provide updated schema locally
cortadocodes May 31, 2024
6f9ab99
FEA: Add parent question UUID to questions and provide it in `Runner`
cortadocodes May 31, 2024
02aa462
WIP: Temporarily add hardcoded schema version number
cortadocodes May 31, 2024
6580487
REF: Make `question_uuid` an explicit argument of `Service._emit_event`
cortadocodes May 31, 2024
7ed32af
ENH: Update emulators with parent question UUID
cortadocodes May 31, 2024
5a669f4
FIX: Move question UUID to correct place in log handler
cortadocodes May 31, 2024
7811eee
FIX: Add missing `parent_question_uuid` parameters
cortadocodes May 31, 2024
dc08f4a
WIP: Add `originator_question_uuid` to local schema
cortadocodes May 31, 2024
838c305
FEA: Add originator question UUID to question attributes
cortadocodes May 31, 2024
27cbd10
REF: Use keyword arguments in `Service`
cortadocodes May 31, 2024
039de99
TST: Update `EventReplayer` tests with new question UUID fields
cortadocodes Jun 3, 2024
782531b
REF: Rename `originator` event attribute to `parent`
cortadocodes Jun 3, 2024
7d6302e
FEA: Add new `originator` event attribute
cortadocodes Jun 3, 2024
569d404
ENH: Add `originator` argument to `Child` and `ChildEmulator`
cortadocodes Jun 3, 2024
200ccdc
ENH: Set originator question UUID and originator from question in runner
cortadocodes Jun 3, 2024
64e894f
FIX: Fix sending exceptions in `answer_pub_sub_question`
cortadocodes Jun 3, 2024
e70114f
FIX: Add missing arguments in cloud emulators
cortadocodes Jun 3, 2024
8db120a
FIX: Serialise `None` value pub/sub attributes
cortadocodes Jun 3, 2024
2f5ca06
OPS: Update event handler and its bigquery table
cortadocodes Jun 3, 2024
8e45737
TST: Use new example service SRUID
cortadocodes Jun 3, 2024
50e510f
ENH: Remove unnecessary `sender` argument from `get_events`
cortadocodes Jun 4, 2024
72308ae
FEA: Update `get_events` to work with parent and originator questions
cortadocodes Jun 4, 2024
fc25aa6
ENH: Order events retrieved by `get_events` by `datetime` not `order`
cortadocodes Jun 4, 2024
fd0edb8
FIX: Fix typo in events query
cortadocodes Jun 4, 2024
6344808
FIX: Unflatten new event attributes in `get_events`
cortadocodes Jun 4, 2024
af16f54
ENH: Provide SRUID to `Child` internal service instead of name
cortadocodes Jun 4, 2024
c8ad230
TST: Fix template app tests
cortadocodes Jun 4, 2024
7daf015
ENH: Remove `name` argument from `Service`
cortadocodes Jun 4, 2024
afe1984
FIX: Set analysis ID at start of `Runner.run`
cortadocodes Jun 4, 2024
354509f
REF: Make parent question UUID setting clearer in `Runner`
cortadocodes Jun 4, 2024
3797199
REF: Factor out finalising and cleaning up in `Runner`
cortadocodes Jun 4, 2024
45e0b6d
DEP: Loosen `Sphinx` range required
cortadocodes Jun 4, 2024
78dce5c
DEP: Loosen other docs package ranges
cortadocodes Jun 4, 2024
855d8c8
DOC: Update docs with new `get_events` usage
cortadocodes Jun 4, 2024
b28ce04
DOC: Update `Dataset` docs
cortadocodes Jun 4, 2024
faadc5b
DOC: Clarify `Child.ask_multiple` docs
cortadocodes Jun 4, 2024
b0b1036
DOC: Update `ChildEmulator` docs
cortadocodes Jun 4, 2024
47b641c
ENH: Update `internal_service_name` argument in `ChildEmulator`
cortadocodes Jun 4, 2024
51c493a
REF: Move test emulator files into test data directory
cortadocodes Jun 4, 2024
8e39453
DOC: Clarify question asking docs
cortadocodes Jun 4, 2024
79c46e6
DOC: Add missing docstring params
cortadocodes Jun 4, 2024
384b048
FIX: Set originator in `MockService.ask`
cortadocodes Jun 4, 2024
7535787
DOC: Adjust docstrings
cortadocodes Jun 4, 2024
7c3dbda
REF: Factor out routing metadata in `Service.answer`
cortadocodes Jun 4, 2024
fe817ef
DOC: Update docstrings
cortadocodes Jun 4, 2024
ca2aeb4
REF: Simplify error logging in `Child.ask_multiple`
cortadocodes Jun 4, 2024
02cf1e0
REF: Avoid using `gen_uuid` in `Runner`
cortadocodes Jun 4, 2024
6c489e0
REF: Simplify default args for originators
cortadocodes Jun 4, 2024
d71b1dc
DOC: Improve docstrings
cortadocodes Jun 4, 2024
982949c
REF: Use more appropriate error and improve docstring in `get_events`
cortadocodes Jun 5, 2024
eaeb238
REF: Factor out validation in `get_events`
cortadocodes Jun 5, 2024
8b62261
REF: Factor out fields in `get_events`
cortadocodes Jun 5, 2024
b143141
FIX: Query correct question UUID in `get_events`
cortadocodes Jun 5, 2024
d92c61d
FEA: Use version `0.11.0` of service communication schema
cortadocodes Jun 5, 2024
a0b2f58
FIX: Deserialise `parent_question_uuid` properly in `get_events`
cortadocodes Jun 5, 2024
927f90a
TST: Update deployment test SRUID
cortadocodes Jun 5, 2024
8c9b54a
ENH: Give feedback on (un)successful push subscription creation in CLI
cortadocodes Jun 5, 2024
2d9317b
FEA: Add retry count event attribute
cortadocodes Jun 5, 2024
a64aefe
WIP: Temporarily include new schema locally
cortadocodes Jun 5, 2024
71c731c
FEA: Add `retry_count` kwarg to `Service.ask`
cortadocodes Jun 5, 2024
935e797
FIX: Extract `retry_count` attribute for all events
cortadocodes Jun 5, 2024
fcabadf
FIX: Add `retry_count` arg to log handler
cortadocodes Jun 5, 2024
452808f
TST: Update tests
cortadocodes Jun 5, 2024
42b22b1
ENH: Allow retried questions to have the same UUID
cortadocodes Jun 5, 2024
997f7b8
FIX: Increment retry count when retrying in `Child.ask_multiple`
cortadocodes Jun 5, 2024
7afd15b
ENH: Log the retry number for retried questions
cortadocodes Jun 5, 2024
e999532
REF: Factor out dropping redelivered questions in flask app
cortadocodes Jun 5, 2024
41c82f0
ENH: Allow explicit question retries in flask app
cortadocodes Jun 5, 2024
5cc468d
REF: Change `_acknowledge_and_drop_redelivered_questions` signature
cortadocodes Jun 5, 2024
9e9e5fb
REF: Move service accounts into separate terraform file
cortadocodes Jun 5, 2024
0304161
FIX: Format log messages correctly in flask app
cortadocodes Jun 5, 2024
ca34c42
FIX: Add missing argument
cortadocodes Jun 5, 2024
b9f01dd
FIX: Fix getting `retry_count` attribute
cortadocodes Jun 5, 2024
ce5f8c3
ENH: Avoid errors being logged multiple times in `Child.ask_multiple`
cortadocodes Jun 5, 2024
d1571d1
ENH: Log a warning if a dataset is empty at instantiation
cortadocodes Jun 6, 2024
eccd459
FEA: Add `RegisteredTemporaryDirectory` class
cortadocodes Jun 6, 2024
edb5ae6
ENH: Delete registered temporary directories at `Runner.run` finish
cortadocodes Jun 6, 2024
d543587
ENH: Use `RegisteredTemporaryDirectory` when downloading datasets
cortadocodes Jun 6, 2024
e561f42
ENH: Delete temporary directories before deleting temporary files
cortadocodes Jun 6, 2024
39f4e49
ENH: Log which retry `Child.ask_multiple` is carrying out
cortadocodes Jun 6, 2024
f074f69
ENH: Log warning when `PYTHONUNBUFFERED` envvar is unset
cortadocodes Jun 6, 2024
91c9eae
ENH: Use `RegisteredTemporaryDirectory` in template app
cortadocodes Jun 6, 2024
e708c19
ENH: Remove "analysis-" from start of question UUIDs in log context
cortadocodes Jun 7, 2024
ea51c97
ENH: Set question UUIDs in advance in `Child.ask_multiple`
cortadocodes Jun 7, 2024
6c2702c
FIX: Return question UUID alongside error from `Child.ask_multiple`
cortadocodes Jun 9, 2024
d574b5a
ENH: Allow exceptions to pass into `convert_exception_to_primitives`
cortadocodes Jun 9, 2024
c09f477
FIX: Fix manifest deserialisation in `get_events`
cortadocodes Jun 9, 2024
59d7580
TST: Update deployment test
cortadocodes Jun 9, 2024
05502b9
TST: Update tests
cortadocodes Jun 9, 2024
18cbef1
TST: Fix test
cortadocodes Jun 9, 2024
ba9c9ed
TST: Test event validity in deployment test
cortadocodes Jun 9, 2024
ce433ec
FIX: Use service communication schema 0.13.0 to allow non-boolean for…
thclark Jun 11, 2024
dc6bf36
FIX: Use the schema ref for service communication
thclark Jun 11, 2024
30de609
FIX: Hack deserialisation of event attributes from bigquery to correc…
thclark Jun 11, 2024
19c95cf
ENH: Make limited get_events function tail by default
thclark Jun 12, 2024
cc69c9b
FIX: Remove erroneous count output
thclark Jun 12, 2024
c1ebcb6
FIX: Fix row deserialisation in `get_events`
cortadocodes Jun 24, 2024
74252ca
FIX: Remove manifest deserialisation from `get_events`
cortadocodes Jun 24, 2024
914c6ca
FIX: Convert numpy arrays to lists in `get_events`
cortadocodes Jun 24, 2024
5a97393
TST: Update deployment test
cortadocodes Jun 24, 2024
e00ad96
ENH: Make non-tail ordering in `get_events` explicit
cortadocodes Jun 24, 2024
6039548
REF: Factor out base query in `get_events`
cortadocodes Jun 24, 2024
051c762
DOC: Document default tail behaviour in `get_events`
cortadocodes Jun 24, 2024
17558a4
FIX: Remove incorrect unpacking of string
cortadocodes Jun 24, 2024
c6d7b38
ENH: Add comment explaining multiple ordering in `tail` sql
cortadocodes Jun 24, 2024
6838aba
TST: Update `get_events` tests with tail query
cortadocodes Jun 24, 2024
f45e67a
TST: Fix `convert_exception_to_primitives` test
cortadocodes Jun 24, 2024
8a23fce
TST: Test using `get_events` with `tail=False`
cortadocodes Jun 24, 2024
72e4d4a
FIX: Order queries before applying limit
cortadocodes Jun 24, 2024
9a2557b
TST: Use event replayer in async question deployment test
cortadocodes Jun 24, 2024
d25ebf8
FIX: Fix `forward_logs` type conversion in `get_events`
cortadocodes Jun 24, 2024
5295db4
REF: Avoid using pandas dataframe when getting events from event store
cortadocodes Jun 25, 2024
7b3685f
DEP: Remove unneeded `db-dtypes` package
cortadocodes Jun 25, 2024
3e8a7f9
REF: Simplify event retrieval in `get_events`
cortadocodes Jun 25, 2024
1ed5289
ENH: Allow setting of event store table ID in service configuration
cortadocodes Jun 25, 2024
7b8f842
FEA: Stop redelivered questions by checking event store
cortadocodes Jun 25, 2024
e9b44b7
TST: Make flask app tests clearer
cortadocodes Jun 25, 2024
f3b8557
TST: Test warning logged if event store not specified
cortadocodes Jun 25, 2024
1f443d9
REF: Factor out `DEFAULT_SERVICE_CONFIGURATION_PATH`
cortadocodes Jun 25, 2024
c1fa156
DEP: Make `google-cloud-bigquery` a mandatory dependency
cortadocodes Jun 25, 2024
8fa3e68
FIX: Add `question` kind to valid event kinds
cortadocodes Jun 25, 2024
5cae9c3
FIX: Get retry count correctly in flask app
cortadocodes Jun 25, 2024
1eb330d
FIX: Catch no-events error from `get_events`
cortadocodes Jun 25, 2024
a4637be
FIX: Use (meta-)generation agnostic retry strategy with cloud storage
cortadocodes Jun 25, 2024
8588a52
ENH: Add child SRUID and question UUID to no heartbeat error info
cortadocodes Jun 25, 2024
bf8cdca
FIX: Enable `EventReplayer` to handle question events
cortadocodes Jun 25, 2024
39a2303
REF: Use events' actual attributes instead of first event's
cortadocodes Jun 25, 2024
038f624
TST: Update tests to use new stored events format
cortadocodes Jun 25, 2024
8a491b8
REF: Remove redundant `recipient` argument from event handlers
cortadocodes Jun 26, 2024
bc19236
ENH: Update validation functions to expect SRUID instead of `Service`
cortadocodes Jun 26, 2024
85ace7f
FIX: Allow questions to continue if event store can't be found
cortadocodes Jun 26, 2024
1b5dc3c
DEP: Upgrade `google-cloud-secret-manager`
cortadocodes Jun 26, 2024
6764312
FIX: Always return 2-tuple from `_skip_to_earliest_waiting_event`
cortadocodes Jun 26, 2024
b1b6d63
FIX: Return correct question UUIDs with failed questions
cortadocodes Jun 27, 2024
1d60258
ENH: Improve failed question logging in `Child.ask_multiple`
cortadocodes Jun 27, 2024
8d08fbe
DOC: Update docstring of `_skip_to_earliest_waiting_event`
cortadocodes Jun 27, 2024
92d5acb
REF: Rename dockerfile to reflect its python version
cortadocodes Jun 28, 2024
5d2168d
ENH: Return empty list from `get_events` if no events for question
cortadocodes Jun 28, 2024
1f41527
ENH: Log when a new question has been received
cortadocodes Jun 28, 2024
e2cccd0
FIX: Use correct formatting in log message
cortadocodes Jun 28, 2024
677160b
FIX: Use correct formatting in log message
cortadocodes Jun 28, 2024
80b136d
ENH: Improve error logging in `Child.ask_multiple`
cortadocodes Jun 28, 2024
8ddf652
TST: Rename variable for consistency
cortadocodes Jun 28, 2024
99676d7
FIX: Avoid race condition between event handler and cloud run service
cortadocodes Jun 28, 2024
9c583a9
TST: Use string `retry_count` in tests to mirror production
cortadocodes Jun 28, 2024
8d17744
REF: Rename variable for clarity
cortadocodes Jun 28, 2024
c1839e8
REF: Factor out question acknowledgement response in flask app
cortadocodes Jun 28, 2024
4ef076c
FIX: Avoid logging exception twice per failure in `Child.ask_multiple`
cortadocodes Jun 28, 2024
1e7feb1
ENH: Use envvar to specify service configuration location by default
cortadocodes Jul 1, 2024
e68c07e
REF: Remove ordering from event handlers
cortadocodes Jul 1, 2024
d4659fd
TST: Replace "message" with "event"
cortadocodes Jul 1, 2024
c06cef6
TST: Increase wait time in asynchronous question test
cortadocodes Jul 1, 2024
e5e936b
FIX: Fix skipping invalid events in pub/sub event handler
cortadocodes Jul 1, 2024
d106a03
ENH: Order events by datetime in `EventReplayer`
cortadocodes Jul 1, 2024
f95e917
ENH: Promote log message from debug to warning level
cortadocodes Jul 1, 2024
d3c986a
REV: Revert "ENH: Order events by datetime in `EventReplayer`"
cortadocodes Jul 1, 2024
aed347f
FIX: Return empty list if no events pulled from subscription
cortadocodes Jul 2, 2024
9af23dc
ENH: Improve docstring and warning message
cortadocodes Jul 2, 2024
0497eb6
REF: Refactor question redelivery check in flask app
cortadocodes Jul 2, 2024
ad8a8dd
DOC: Add param string for `retry_count`
cortadocodes Jul 2, 2024
2d4929a
REF: Remove redundant datetime from delivery ack and heartbeat events
cortadocodes Jul 2, 2024
4b8db3b
FEA: Add ability to not include service metadata in logs in replayer
cortadocodes Jul 2, 2024
7d5f64a
FIX: Update delivery ack event handler
cortadocodes Jul 2, 2024
adb7c66
TST: Update service tests
cortadocodes Jul 2, 2024
9a67f36
ENH: Use version `0.14.0` of service communication schema
cortadocodes Jul 2, 2024
327cf54
TST: Finish event replayer test
cortadocodes Jul 2, 2024
634f7dd
ENH: Order pub/sub messages using question UUID ordering key
cortadocodes Jul 2, 2024
71ecfe2
FIX: Enable message ordering in pub/sub publisher
cortadocodes Jul 2, 2024
1615a9e
FIX: Avoid logging that app failed when it didn't
cortadocodes Jul 4, 2024
37d2a5d
ENH: Emit correct logs when no diagnostics with `octue get-diagnostics`
cortadocodes Jul 4, 2024
43590e9
WIP: Attempt to solve POSIX-noncompliant filesystem interaction
cortadocodes Jul 4, 2024
9387401
WIP: Stop using `os.getcwd`
cortadocodes Jul 4, 2024
8cfcec4
REF: Remove workaround for `abspath` non-posix failures
cortadocodes Jul 8, 2024
2829c63
FIX: Cache metadata against datafile/dataset instead of path
cortadocodes Jul 8, 2024
55a8349
ENH: Use temporary directory for `Dataset` path by default
cortadocodes Jul 8, 2024
1e1ddb3
FIX: Allow setting of `max_workers` when CPU count is indeterminate
cortadocodes Jul 9, 2024
791dc5e
ENH: Log max threads used by `Child.ask_multiple`
cortadocodes Jul 9, 2024
e67a30a
WIP: Log registered temporary directories
cortadocodes Jul 10, 2024
e350960
WIP: Temporarily disable local file deletion in runner
cortadocodes Jul 10, 2024
4d0a299
REV: Revert "WIP: Log registered temporary directories"
cortadocodes Jul 10, 2024
3164def
REV: Revert "WIP: Temporarily disable local file deletion in runner"
cortadocodes Jul 10, 2024
9d5a282
FIX: Disable `delete_local_files` by default and provide in config
cortadocodes Jul 10, 2024
cf32a29
TST: Update test
cortadocodes Jul 10, 2024
510bcc7
REF: Remove misleading log message
cortadocodes Jul 11, 2024
5a1452b
DOC: Add missing param in docstring
cortadocodes Jul 11, 2024
a21d295
REF: Remove `order` event attribute
cortadocodes Jul 11, 2024
a76158b
REF: Simplify `answer_question`
cortadocodes Jul 11, 2024
b06710c
REF: Simplify attributes dict construction in `Service._emit_event`
cortadocodes Jul 11, 2024
c3bda0b
DOC: Update docstring
cortadocodes Jul 11, 2024
51f7c19
REF: Make argument name clearer
cortadocodes Jul 11, 2024
0d33220
DOC: Add docstring to `Metadata.metadata_path`
cortadocodes Jul 11, 2024
ed11d0f
REV: Revert "ENH: Use temporary directory for `Dataset` path by default"
cortadocodes Jul 11, 2024
0b14c5b
REF: Rename `temporary_directories`
cortadocodes Jul 11, 2024
a07b2c4
REF: Simplify `Child.ask_multiple`
cortadocodes Jul 11, 2024
d2be372
ENH: Ensure app failure log message is emitted
cortadocodes Jul 11, 2024
8f815f1
REF: Simplify `get_events`
cortadocodes Jul 11, 2024
07e6d48
FIX: Add missing space in log message
cortadocodes Jul 11, 2024
7752da7
TST: Improve flask app tests
cortadocodes Jul 11, 2024
4eb8dba
TST: Fix typo
cortadocodes Jul 11, 2024
7970e57
OPS: Use new event handler function
cortadocodes Jul 15, 2024
25db0af
OPS: Remove `order` column from event store
cortadocodes Jul 15, 2024
f2df9e4
FIX: Remove `order` from `get_events`
cortadocodes Jul 15, 2024
b9d7a6c
CHO: Add version compatibility data
cortadocodes Jul 15, 2024
1ecae5b
TST: Update deployment test
cortadocodes Jul 15, 2024
2868526
DEP: Require `Sphinx>=5`
cortadocodes Jul 15, 2024
51db351
DOC: Improve `Manifest.download` documentation
cortadocodes Jul 15, 2024
fb3444d
DOC: Improve `Child.ask_multiple` documentation
cortadocodes Jul 15, 2024
95fc245
DOC: Improve docstring
cortadocodes Jul 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Required by the python script for building documentation
poetry==1.2.0b2
Sphinx>=4,<5
Sphinx>=5,<8
sphinx-rtd-theme>=1,<2
sphinx-tabs>=3,<4
sphinx-toolbox==3.0.0
sphinx-tabs>=3.4.0,<4
sphinx-toolbox>=3,<4
git+https://github.com/octue/octue-sdk-python.git@main
43 changes: 20 additions & 23 deletions docs/source/asking_questions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -96,17 +96,23 @@ access the event store and run:

events = get_events(
table_id="your-project.your-dataset.your-table",
sender="octue/test-service:1.0.0",
question_uuid="53353901-0b47-44e7-9da3-a3ed59990a71",
)


**Options**

- ``question_uuid`` - Retrieve events from this specific question
- ``parent_question_uuid`` - Retrieve events from questions triggered by the same parent question (this doesn't include the parent question's events)
- ``originator_question_uuid`` - Retrieve events for the entire tree of questions triggered by an originator question (a question asked manually through ``Child.ask``; this does include the originator question's events)
- ``kind`` - Only retrieve this kind of event if present (e.g. "result")
- ``include_backend_metadata`` - If ``True``, retrieve information about the service backend that produced the event
- ``limit`` - If set to a positive integer, limit the number of events returned to this

.. note::

Only one of ``question_uuid``, ``parent_question_uuid``, and ``originator_question_uuid`` can be provided at one time.


.. collapse:: See an example output here...

Expand All @@ -116,8 +122,8 @@ access the event store and run:
[
{
"event": {
"datetime": "2024-03-06T15:44:18.156044",
"kind": "delivery_acknowledgement"
"kind": "delivery_acknowledgement",
"datetime": "2024-03-06T15:44:18.156044"
},
},
{
Expand Down Expand Up @@ -149,21 +155,10 @@ access the event store and run:
},
{
"event": {
"datetime": "2024-03-06T15:46:18.167424",
"kind": "heartbeat"
"kind": "heartbeat",
"datetime": "2024-03-06T15:46:18.167424"
},
"attributes": {
"datetime": "2024-04-11T10:46:48.236064",
"uuid": "a9de11b1-e88f-43fa-b3a4-40a590c3443f",
"order": "7",
"question_uuid": "d45c7e99-d610-413b-8130-dd6eef46dda6",
"originator": "octue/test-service:1.0.0",
"sender": "octue/test-service:1.0.0",
"sender_type": "CHILD",
"sender_sdk_version": "0.51.0",
"recipient": "octue/another-service:3.2.1"
}
}
},
{
"event": {
"kind": "result",
Expand Down Expand Up @@ -203,23 +198,25 @@ raised and no answers are returned.
{"input_values": {"height": 7, "width": 32}},
)
>>> [
{"output_values": {"some": "output"}, "output_manifest": None},
{"output_values": {"another": "result"}, "output_manifest": None},
{"output_values": {"different": "result"}, "output_manifest": None},
({"output_values": {"some": "output"}, "output_manifest": None}, '2681ef4e-4ab7-4cf9-8783-aad982d5e324'),
({"output_values": {"another": "result"}, "output_manifest": None}, '474923bd-14b6-4f4c-9bfe-8148358f35cd'),
({"output_values": {"different": "result"}, "output_manifest": None}, '9a50daae-2328-4728-9ddd-b2252474f118'),
]

This method uses multithreading, allowing all the questions to be asked at once instead of one after another.

**Options**

- If ``raise_errors=False`` is provided, answers are returned for all successful questions while unraised errors are
returned for unsuccessful ones
logged and returned for unsuccessful ones
- If ``raise_errors=False`` is provided with ``max_retries > 0``, failed questions are retried up to this number of
times
- If ``raise_errors=False`` is provided with ``max_retries > 0`` and ``prevent_retries_when`` is set to a list of
exception types, failed questions are retried except for those whose exception types are in the list
- The maximum number of threads that can be used to ask questions in parallel can be set via the ``max_workers``
argument. This has no effect on the total number of questions that can be asked via ``Child.ask_multiple``.
- ``max_workers``: The maximum number of threads that can be used to ask questions in parallel can be set via this
argument. It has no effect on the total number of questions that can be asked via ``Child.ask_multiple``.
- ``log_errors``: If `True` and ``raise_errors=False``, any errors remaining once retries are exhausted are logged in
addition to being returned


Asking a question within a service
Expand Down
9 changes: 7 additions & 2 deletions docs/source/dataset.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,14 @@ Working with a dataset is the same whether it's local or cloud-based.

from octue.resources import Dataset

dataset = Dataset(path="path/to/dataset", recursive=True)
dataset = Dataset(path="path/to/dataset")

dataset = Dataset(path="gs://my-bucket/path/to/dataset", recursive=True)
dataset = Dataset(path="gs://my-bucket/path/to/dataset")


.. warning::

Datasets recurse all subdirectories by default unless ``recursive=False`` is set.


Upload a dataset
Expand Down
Loading
Loading