[SparkConnect] Initial Protobuf Definitions #37075
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,155 @@ | ||
| // Protocol Buffers - Google's data interchange format | ||
|
There was a problem hiding this comment. Shall we change the header here? |
||
| // Copyright 2008 Google Inc. All rights reserved. | ||
| // https://developers.google.com/protocol-buffers/ | ||
| // | ||
| // Redistribution and use in source and binary forms, with or without | ||
| // modification, are permitted provided that the following conditions are | ||
| // met: | ||
| // | ||
| // * Redistributions of source code must retain the above copyright | ||
| // notice, this list of conditions and the following disclaimer. | ||
| // * Redistributions in binary form must reproduce the above | ||
| // copyright notice, this list of conditions and the following disclaimer | ||
| // in the documentation and/or other materials provided with the | ||
| // distribution. | ||
| // * Neither the name of Google Inc. nor the names of its | ||
| // contributors may be used to endorse or promote products derived from | ||
| // this software without specific prior written permission. | ||
| // | ||
| // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS | ||
| // "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT | ||
| // LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR | ||
| // A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT | ||
| // OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, | ||
| // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT | ||
| // LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, | ||
| // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY | ||
| // THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | ||
| // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE | ||
| // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
|
|
||
| syntax = "proto3"; | ||
|
|
||
| package google.protobuf; | ||
|
|
||
| option csharp_namespace = "Google.Protobuf.WellKnownTypes"; | ||
| option go_package = "github.com/golang/protobuf/ptypes/any"; | ||
| option java_package = "com.google.protobuf"; | ||
| option java_outer_classname = "AnyProto"; | ||
| option java_multiple_files = true; | ||
| option objc_class_prefix = "GPB"; | ||
|
|
||
| // `Any` contains an arbitrary serialized protocol buffer message along with a | ||
| // URL that describes the type of the serialized message. | ||
| // | ||
| // Protobuf library provides support to pack/unpack Any values in the form | ||
| // of utility functions or additional generated methods of the Any type. | ||
| // | ||
| // Example 1: Pack and unpack a message in C++. | ||
| // | ||
| // Foo foo = ...; | ||
| // Any any; | ||
| // any.PackFrom(foo); | ||
| // ... | ||
| // if (any.UnpackTo(&foo)) { | ||
| // ... | ||
| // } | ||
| // | ||
| // Example 2: Pack and unpack a message in Java. | ||
| // | ||
| // Foo foo = ...; | ||
| // Any any = Any.pack(foo); | ||
| // ... | ||
| // if (any.is(Foo.class)) { | ||
| // foo = any.unpack(Foo.class); | ||
| // } | ||
| // | ||
| // Example 3: Pack and unpack a message in Python. | ||
| // | ||
| // foo = Foo(...) | ||
| // any = Any() | ||
| // any.Pack(foo) | ||
| // ... | ||
| // if any.Is(Foo.DESCRIPTOR): | ||
| // any.Unpack(foo) | ||
| // ... | ||
| // | ||
| // Example 4: Pack and unpack a message in Go | ||
| // | ||
| // foo := &pb.Foo{...} | ||
| // any, err := ptypes.MarshalAny(foo) | ||
| // ... | ||
| // foo := &pb.Foo{} | ||
| // if err := ptypes.UnmarshalAny(any, foo); err != nil { | ||
| // ... | ||
| // } | ||
| // | ||
| // The pack methods provided by protobuf library will by default use | ||
| // 'type.googleapis.com/full.type.name' as the type URL and the unpack | ||
| // methods only use the fully qualified type name after the last '/' | ||
| // in the type URL, for example "foo.bar.com/x/y.z" will yield type | ||
| // name "y.z". | ||
| // | ||
| // | ||
| // JSON | ||
| // ==== | ||
| // The JSON representation of an `Any` value uses the regular | ||
| // representation of the deserialized, embedded message, with an | ||
| // additional field `@type` which contains the type URL. Example: | ||
| // | ||
| // package google.profile; | ||
| // message Person { | ||
| // string first_name = 1; | ||
| // string last_name = 2; | ||
| // } | ||
| // | ||
| // { | ||
| // "@type": "type.googleapis.com/google.profile.Person", | ||
| // "firstName": <string>, | ||
| // "lastName": <string> | ||
| // } | ||
| // | ||
| // If the embedded message type is well-known and has a custom JSON | ||
| // representation, that representation will be embedded adding a field | ||
| // `value` which holds the custom JSON in addition to the `@type` | ||
| // field. Example (for message [google.protobuf.Duration][]): | ||
| // | ||
| // { | ||
| // "@type": "type.googleapis.com/google.protobuf.Duration", | ||
| // "value": "1.212s" | ||
| // } | ||
| // | ||
| message Any { | ||
| // A URL/resource name that uniquely identifies the type of the serialized | ||
| // protocol buffer message. This string must contain at least | ||
| // one "/" character. The last segment of the URL's path must represent | ||
| // the fully qualified name of the type (as in | ||
| // `path/google.protobuf.Duration`). The name should be in a canonical form | ||
| // (e.g., leading "." is not accepted). | ||
| // | ||
| // In practice, teams usually precompile into the binary all types that they | ||
| // expect it to use in the context of Any. However, for URLs which use the | ||
| // scheme `http`, `https`, or no scheme, one can optionally set up a type | ||
| // server that maps type URLs to message definitions as follows: | ||
| // | ||
| // * If no scheme is provided, `https` is assumed. | ||
| // * An HTTP GET on the URL must yield a [google.protobuf.Type][] | ||
| // value in binary format, or produce an error. | ||
| // * Applications are allowed to cache lookup results based on the | ||
| // URL, or have them precompiled into a binary to avoid any | ||
| // lookup. Therefore, binary compatibility needs to be preserved | ||
| // on changes to types. (Use versioned type names to manage | ||
| // breaking changes.) | ||
| // | ||
| // Note: this functionality is not currently available in the official | ||
| // protobuf release, and it is not used for type URLs beginning with | ||
| // type.googleapis.com. | ||
| // | ||
| // Schemes other than `http`, `https` (or the empty scheme) might be | ||
| // used with implementation specific semantics. | ||
| // | ||
| string type_url = 1; | ||
|
|
||
| // Must be a valid serialized protocol buffer of the above specified type. | ||
| bytes value = 2; | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| // Protocol Buffers - Google's data interchange format | ||
| // Copyright 2008 Google Inc. All rights reserved. | ||
| // https://developers.google.com/protocol-buffers/ | ||
| // | ||
| // Redistribution and use in source and binary forms, with or without | ||
| // modification, are permitted provided that the following conditions are | ||
| // met: | ||
| // | ||
| // * Redistributions of source code must retain the above copyright | ||
| // notice, this list of conditions and the following disclaimer. | ||
| // * Redistributions in binary form must reproduce the above | ||
| // copyright notice, this list of conditions and the following disclaimer | ||
| // in the documentation and/or other materials provided with the | ||
| // distribution. | ||
| // * Neither the name of Google Inc. nor the names of its | ||
| // contributors may be used to endorse or promote products derived from | ||
| // this software without specific prior written permission. | ||
| // | ||
| // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS | ||
| // "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT | ||
| // LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR | ||
| // A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT | ||
| // OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, | ||
| // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT | ||
| // LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, | ||
| // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY | ||
| // THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | ||
| // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE | ||
| // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
|
|
||
| syntax = "proto3"; | ||
|
|
||
| package google.protobuf; | ||
|
|
||
| option csharp_namespace = "Google.Protobuf.WellKnownTypes"; | ||
| option go_package = "github.com/golang/protobuf/ptypes/empty"; | ||
| option java_package = "com.google.protobuf"; | ||
| option java_outer_classname = "EmptyProto"; | ||
| option java_multiple_files = true; | ||
| option objc_class_prefix = "GPB"; | ||
| option cc_enable_arenas = true; | ||
|
|
||
| // A generic empty message that you can re-use to avoid defining duplicated | ||
| // empty messages in your APIs. A typical example is to use it as the request | ||
| // or the response type of an API method. For instance: | ||
| // | ||
| // service Foo { | ||
| // rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); | ||
| // } | ||
| // | ||
| // The JSON representation for `Empty` is empty JSON object `{}`. | ||
| message Empty {} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we should shade and relocation
protobufin spark to avoid potential conflicts with other third-party libraries, such as hadoopThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Absolutely, the shading and relocation rules haven't been updated yet. I'm still a bit unclear what the best way to progress is to avoid conflicts with third-party packages or Spark consumers. I've discussed some approaches and one way would be to produce a shaded spark connect artifact that is then consumed in it's shaded version by spark itself or to shade and relocate after the build.
However, this PR is mostly for discussing the proto interface.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hadoop hdfs has its own shaded copy now so doesn't care(*); a protobuf upgrade is incompatible with any code compiled against the later version, so must be shaded somehow.
(*) more specifically, has a new problem, how to safely upgrade that shaded hadoop-thirdparty jar with guava, protobuf etc
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@steveloughran what does this mean? could you point to more details?
If I shaded 'com.google.protobuf', in my jar, I can update it anytime, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shaded, yes. if unshaded, then if you update protobuf.jar, all .class files compiled with the older version of protobuf are unlikely to link. let alone work