Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add start time to change feed query #34694

Closed
wants to merge 11 commits into from

Conversation

bambriz
Copy link
Member

@bambriz bambriz commented Mar 7, 2024

Description

Adds Start Time option for change feed queries in Python SDK. A utc datetime is passed into the options when calling a change feed pull. If the datetime is not utc but has a timezone defined it will convert it to utc, otherwise it is assumed to be in UTC. An incorrect value results in an AttributeError.

The following Example creates 10 items at two different times. By using start time you can filter out the items created after the time recorded.

    batchSize = 10
    container = database.get_container_client(container_name)
    create_random_items(container, batchSize)
    sleep(1)
    now = datetime.now(timezone.utc)
    sleep(1)
    create_random_items(container, batchSize)
    change_feed_iter = container.query_items_change_feed(start_time=now)

This PR also fixes a bug where change feed pulls in async clients were failing to fetch remaining pages and only returning the first one by removing the casting to Dict for response headers and keeping them as CIMultiDict type.

This PR was merged into PR 35090 Please refer to that PR.

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

Adds start time, at the moment has a work around for getting all the documents.
@github-actions github-actions bot added the Cosmos label Mar 7, 2024
bambriz added 3 commits March 28, 2024 10:33
Start time change feed queries always result in the initial fetch results being empty, which is a deviation in behaviour from other change feed options. To accommodate that we check if we have the start time change feed option and check if it is the first fetch request in order to grab the continuation value from etag to get the rest of the results.
Added a new header to remove ambuiguity between 'etag' meant for sync clients and 'Etag' meant for async clients in our code for header strings.
@azure-sdk
Copy link
Collaborator

API change check

APIView has identified API level changes in this PR and created following API reviews.

azure-cosmos

bambriz added 2 commits March 29, 2024 11:09
added negative test for start time option of change feed.
@bambriz bambriz marked this pull request as ready for review March 29, 2024 18:13
sdk/cosmos/azure-cosmos/azure/cosmos/aio/_container.py Outdated Show resolved Hide resolved
@@ -342,6 +345,8 @@ def query_items_change_feed(
feed_options["partitionKey"] = self._set_partition_key(partition_key)
if is_start_from_beginning is not None:
feed_options["isStartFromBeginning"] = is_start_from_beginning
if start_time is not None and is_start_from_beginning is False:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this logic different from the async client?
Additionally - it looks like these parameters are mutually exclusive, so what happens if I pass both a start_time as well as start_from_beginning=True? Will the service raise an exception?

sdk/cosmos/azure-cosmos/azure/cosmos/http_constants.py Outdated Show resolved Hide resolved
bambriz added 2 commits April 5, 2024 08:34
Fixes Etag issue with response headers by making the response headers to be type CIMultDict instead of Dict, so that it is case insensitive. This also adds other recommended changes.
Checks for datetime object for start time option to avoid error when start time is None
@bambriz bambriz closed this Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants