Skip to content

Commit

Permalink
Updating Storage API Autosharding documentation to include that it do…
Browse files Browse the repository at this point in the history
…esn't work on Runner V2 (#28233)

* add documentation

* doc for python too
  • Loading branch information
ahmedabu98 committed Sep 18, 2023
1 parent 0d37c6f commit d6068ad
Showing 1 changed file with 4 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -788,6 +788,8 @@ BigQuery Storage Write API for Python SDK currently has some limitations on supp
{{< paragraph class="language-py" >}}
**Note:** If you want to run WriteToBigQuery with Storage Write API from the source code, you need to run `./gradlew :sdks:java:io:google-cloud-platform:expansion-service:build` to build the expansion-service jar. If you are running from a released Beam SDK, the jar will already be included.

**Note:** Auto sharding is not currently supported for Python's Storage Write API.

{{< /paragraph >}}

#### Exactly-once semantics
Expand Down Expand Up @@ -877,6 +879,8 @@ explicitly enable this using [`withAutoSharding`](https://beam.apache.org/releas

***Note:*** `STORAGE_WRITE_API` will default to dynamic sharding when
`numStorageWriteApiStreams` is set to 0 or is unspecified.

***Note:*** Auto sharding with `STORAGE_WRITE_API` is supported on Dataflow's legacy runner, but **not** on Runner V2
{{< /paragraph >}}

When using `STORAGE_WRITE_API`, the PCollection returned by
Expand Down

0 comments on commit d6068ad

Please sign in to comment.