Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Storage-Blob] Quick Query API #11991

Merged
merged 29 commits into from
Jun 30, 2020
Merged

[Storage-Blob] Quick Query API #11991

merged 29 commits into from
Jun 30, 2020

Conversation

annatisch
Copy link
Member

No description provided.

xiafu-msft
xiafu-msft reviewed Jun 12, 2020
View reviewed changes
sdk/storage/azure-storage-blob/azure/storage/blob/_blob_client.py Outdated
This keyword arg could be set for delimited (CSV) serialization or JSON serialization.
When the input_serialization is set for JSON records, only a record separator in str format is needed.
:paramtype output_serialization: ~azure.storage.blob.DelimitedTextConfiguration or str.
:keyword Union[str, Callable[Exception]] errors:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does operation_on_error sound better?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've modelled the parameter naming based on this:
https://docs.python.org/3.8/library/codecs.html#error-handlers

Copy link
Contributor

@xiafu-msft xiafu-msft Jun 19, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kasobol-msft thoughts? how about error handler

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I posted comment next to implementation below. I believe this change doesn't follow our feature specification and I'd recommend to change in back.

@annatisch annatisch mentioned this pull request Jun 12, 2020
Closed
@annatisch
Copy link
Member Author

/azp run python - storage - ci

@azure-pipelines
Copy link

Pull request contains merge conflicts.

@annatisch annatisch marked this pull request as ready for review June 12, 2020 22:02
@annatisch annatisch changed the title [Storage-Blob] Quick Query API prototype [Storage-Blob] Quick Query API Jun 12, 2020
xiafu-msft
xiafu-msft reviewed Jun 30, 2020
View reviewed changes
sdk/storage/azure-storage-blob/azure/storage/blob/_blob_client.py
a custom DelimitedTextDialect, or alternatively a DelimitedJSON.
:paramtype blob_format: ~azure.storage.blob.DelimitedTextDialect or ~azure.storage.blob.DelimitedJSON
:keyword output_format:
Optional. Defines the output serialization for the data stream. By default the data will be returned
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry that I told you wrong about this"By default the data will be returned as it is represented in the blob", I tried again and the it shows the default output is also Delimited Text data. While I think it makes more sense to return the format that the blob presently have. I will add a commit to make output format default to the existing blob format.

if output_format:
    # existing code
else:
    output = input_format

xiafu-msft
xiafu-msft reviewed Jun 30, 2020
View reviewed changes
sdk/storage/azure-storage-blob/azure/storage/blob/_blob_client.py
expression=query_expression,
input_serialization=serialize_query_format(input_format),
output_serialization=serialize_query_format(output_format)
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
)
)
else:
output_format = input_format

@xiafu-msft xiafu-msft merged commit 17c1284 into Azure:feature/storage-stg73 Jun 30, 2020
xiafu-msft added a commit that referenced this pull request Jul 7, 2020
@xiafu-msft @kasobol-msft
@zezha-msft @annatisch
Feature/storage stg73 (#12272)
900aa7d 
* [Storage]STG73

* [Blob][Swagger]Update Swagger (#10943)

* [Blob][Swagger]Regenerate Swagger Code

* fix container test failure caused by list_containers include type change

* [Storage] Internal avro parser. (#10764)

* initial avro parser

* try fixing test...

* falling in love with python compatibility...

* make linter happy.

* raise StopIteration when there is no more bytes instead of tracking file length.

* async avro parser

* fix syntax for Python 3.5

* get rid of 'readers_schema' as we only honor schema that has been written to file ('writer_schema').

* pr feedback

* trim unused code.

* pr feedback.

* simplify skip sync in next.

* move avro tests from _shared.

* Jumbo blob support (#11176)

* wip

* initial test coverage.

* wip.

* wip

* single upload.

* add async tests.

* disable 50k block tests.

* datalake append.

* async datalake

* disable tests that send large payload over network.

* pr feedback.

* Undelete share (#11394)

* Undelete container (#11339)

* [Storage][Blob] Added support for Object Replication (#11525)

* Blob versioning (#11154)

* [Blob][QuickQuery]Add Quick Query Support (#10946)

* [Blob][STG73]Blob Tags (#11418)

* regenerate code (#11964)

* fix the bug which caused only showing fatal error (#11997)

* [Storage][STG73]Address API Review Comments (#12111)

* [Storage][STG73]Address API Review Comments

* [Storage][STG73]dict<policy, rules> -> list(ObjectReplicationPolicy)

* fix blob tag_value test

* expose ObjectReplicationPolicy and ObjectReplicationRule, fix test

* fix test

* Changefeed (#10755)

* [ChangeFeed]Add ChangeFeed Package

* test_avro failure

* update dev_requirement.txt

* change namespace to azure.storage.blob.changefeed

* address comments

* optimize memory when reading changefeed events

* namespace change

* set up package change

* fix failed tests

* readme and kwargs

* Update sdk/storage/azure-storage-blob-changefeed/azure/storage/blob/changefeed/_change_feed_client.py

Co-authored-by: Rakshith Bhyravabhotla <sabhyrav@microsoft.com>

* address comments

* 'azure-storage-blob>=12.3.0' which does not match the frozen requirement 'azure-storage-blob~=1.3'

Co-authored-by: Rakshith Bhyravabhotla <sabhyrav@microsoft.com>

* [Storage-Blob] Quick Query API (#11991)

* Renamed query error

* Renamed query reader

* Updated config models

* Updated format request params

* Updated iterator

* fix the bug which caused only showing fatal error

* Updated Error message

* Fixed query helper

* Started test conversion

* small fix

* Fixed tests

* Updated error handling + json model

* Updated recordings

* Removed old recording

* Added iter tests

* Iter test recordings

* Fix test

* Remove extra recording

* Fix pylint

* Some docs cleanup

* Renamed iter_records -> iter_stream

* Review feedback

* Updated tests

* Missing commas

* Fix syntax

* Fix pylint

Co-authored-by: xiafu <xiafu@microsoft.com>

* tag sas (#12258)

* tag sas

* disable undelete_container

* pylint

* skip undelete_container tests

* [Blob][Versioning]Disable Versioning Live Test (#12281)

* [Blob][QQ]Default output_format to input_format (#12283)

* [Storage][Jumbo]Remove super (#12314)

* [Storage][JumboBlob]remove empty super()

* pypy3

* change sas version to latest

* set tags account location to central canada

* re-recording queue

* changefeed paths generator

* mark tests for vid as playback only

* fix changefeed

* fix pylint
make the test account location for tags to central canada

* add a delay before calling find_blobs_by_tags

* remove tags header

* mark a large file test playback only

* revert "mark a large file test playback only"
skip upload large file test
address comment

* move tag permission and filter_by_tags permission to kwargs

Co-authored-by: Kamil Sobol <61715331+kasobol-msft@users.noreply.github.com>
Co-authored-by: Ze Qian Zhang <zezha@microsoft.com>
Co-authored-by: Rakshith Bhyravabhotla <sabhyrav@microsoft.com>
Co-authored-by: annatisch <antisch@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@seanmcc-msft seanmcc-msft seanmcc-msft left review comments

@rakshith91 rakshith91 rakshith91 left review comments

@kasobol-msft kasobol-msft kasobol-msft approved these changes

@xiafu-msft xiafu-msft xiafu-msft approved these changes

@amishra-dev amishra-dev Awaiting requested review from amishra-dev

@zezha-msft zezha-msft Awaiting requested review from zezha-msft

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@annatisch @rakshith91 @seanmcc-msft @xiafu-msft @kasobol-msft