Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-40654][SQL] Protobuf support for Spark - from_protobuf AND to_protobuf #37972

Closed
wants to merge 47 commits into from
Closed

[SPARK-40654][SQL] Protobuf support for Spark - from_protobuf AND to_protobuf #37972

wants to merge 47 commits into from

Commits on Aug 11, 2022

  1. spark support for protobuf from_proto

    Sandish Kumar Hebbani Naga committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    419dd78 View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2022

  1. read .pb fileformat

    Sandish Kumar Hebbani Naga committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    870dbbf View commit details
    Browse the repository at this point in the history

  2. read .pb fileformat

    Sandish Kumar Hebbani Naga committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    077ac9e View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2022

  1. to_proto version

    Sandish Kumar Hebbani Naga committed Aug 21, 2022
    Configuration menu
    Copy the full SHA
    d605bc5 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2022

  1. Working copy of from_proto and to_proto

    Sandish Kumar Hebbani Naga committed Aug 27, 2022
    Configuration menu
    Copy the full SHA
    1ea816a View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'apache-github/master' into SPARK_FROM_P… 

    …ROTO
    Sandish Kumar Hebbani Naga committed Aug 27, 2022
    Configuration menu
    Copy the full SHA
    ac94e14 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2022

  1. Initial from_proto and to_proto support

    Sandish Kumar Hebbani Naga committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    d2710c6 View commit details
    Browse the repository at this point in the history

  2. Abstracted the array operations into newArrayWriter 

    for clarity
    @mposdev21
    mposdev21 committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    c7fca1c View commit details
    Browse the repository at this point in the history

  3. Merge pull request #1 from SandishKumarHN/redo-array 

    array type support for from_proto function.
    @SandishKumarHN
    SandishKumarHN authored Aug 30, 2022
    Configuration menu
    Copy the full SHA
    4c9bd74 View commit details
    Browse the repository at this point in the history

  4. Added test cases. 

    1) Basic Message repeated once
2) Basic Message repeated twice
    @mposdev21
    mposdev21 committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    7d60f9e View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2022

  1. Merge commit 'refs/pull/2/head' of https://github.com/SandishKumarHN/… 

    …spark into SPARK_PROTO_1
    Sandish Kumar Hebbani Naga committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    0435dd7 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #2 from SandishKumarHN/redo-array 

    unit tests for protobuf repeated message
    @SandishKumarHN
    SandishKumarHN authored Aug 31, 2022
    Configuration menu
    Copy the full SHA
    85884f7 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'SPARK_PROTO_1' of https://github.com/SandishKumarHN/spark… 

    … into SPARK_PROTO_1
    Sandish Kumar Hebbani Naga committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    0e0a4d5 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2022

  1. remove use of proto compiler, ProtoSerdeSuite, ProtoFunctionsSuite tests

    Sandish Kumar Hebbani Naga committed Sep 3, 2022
    Configuration menu
    Copy the full SHA
    5d57331 View commit details
    Browse the repository at this point in the history

  2. Added new test cases for the following: 

    1) Primitive types (TBD: Decimal)
2) Deserialization (push down filters not working yet)

Formatting errors to keep the IDE happy
    @mposdev21
    mposdev21 committed Sep 3, 2022
    Configuration menu
    Copy the full SHA
    c8ac0d4 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'SPARK_PROTO_1' of github.com:SandishKumarHN/spark into … 

    …SPARK_PROTO_1
    @mposdev21
    mposdev21 committed Sep 3, 2022
    Configuration menu
    Copy the full SHA
    6aeb274 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2022

  1. Fixed formatting problems and removed unused argument (ProtoDeseriali… 

    …zer.scala)
    @mposdev21
    mposdev21 committed Sep 4, 2022
    Configuration menu
    Copy the full SHA
    63d648d View commit details
    Browse the repository at this point in the history

  2. Filter push down works now after making sure the sql schema 

    matches with the message structure
    @mposdev21
    mposdev21 committed Sep 4, 2022
    Configuration menu
    Copy the full SHA
    d24f873 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2022

  1. Don't handle exceptions locally. Spark tests need to see them.

    @mposdev21
    mposdev21 committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    9185cc8 View commit details
    Browse the repository at this point in the history

  2. serde suite sql type

    Sandish Kumar Hebbani Naga committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    8a2f493 View commit details
    Browse the repository at this point in the history

  3. Remove the failing test (multiple_messages.desc not found)

    @mposdev21
    mposdev21 committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    75a4e5f View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2022

  1. Added a test for bad schema. DynamicMessage parseFrom parses 

    successfully but also creates an UnkonwnFieldSet and use that
to infer that there was a problem with the schema.
    @mposdev21
    mposdev21 committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    c86c5d8 View commit details
    Browse the repository at this point in the history

  2. fixing scala style issues

    @SandishKumarHN
    sandishkumarhn authored and SandishKumarHN committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    be82d92 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2022

  1. Fix the typo in the test case

    @mposdev21
    mposdev21 committed Sep 19, 2022
    Configuration menu
    Copy the full SHA
    849213e View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2022

  1. implementing to_proto without descriptor funtion

    @SandishKumarHN
    SandishKumarHN committed Sep 20, 2022
    Configuration menu
    Copy the full SHA
    adbfaf1 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'SPARK_PROTO_1' of https://github.com/SandishKumarHN/spark… 

    … into SPARK_PROTO_1
    @SandishKumarHN
    SandishKumarHN committed Sep 20, 2022
    Configuration menu
    Copy the full SHA
    a418407 View commit details
    Browse the repository at this point in the history

  3. fixing style check issues

    @SandishKumarHN
    SandishKumarHN committed Sep 20, 2022
    Configuration menu
    Copy the full SHA
    f7ca9e7 View commit details
    Browse the repository at this point in the history

  4. adding PROTO_REBASE_MODE_IN_READ support

    @SandishKumarHN
    SandishKumarHN committed Sep 20, 2022
    Configuration menu
    Copy the full SHA
    a65c131 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2022

  1. CleanMessage to BasicMessage SerdeSuite

    @SandishKumarHN
    SandishKumarHN committed Sep 21, 2022
    Configuration menu
    Copy the full SHA
    e0c22d2 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2022

  1. some nit changes

    @SandishKumarHN
    SandishKumarHN committed Sep 22, 2022
    Configuration menu
    Copy the full SHA
    780d119 View commit details
    Browse the repository at this point in the history

  2. nit picks

    @SandishKumarHN
    SandishKumarHN committed Sep 22, 2022
    Configuration menu
    Copy the full SHA
    e22b763 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' of https://github.com/apache/spark into SPARK_P… 

    …ROTO_1
    @SandishKumarHN
    SandishKumarHN committed Sep 22, 2022
    Configuration menu
    Copy the full SHA
    e1fde67 View commit details
    Browse the repository at this point in the history

  4. including proto to SparkBuild

    @SandishKumarHN
    SandishKumarHN committed Sep 22, 2022
    Configuration menu
    Copy the full SHA
    1901cd4 View commit details
    Browse the repository at this point in the history

  5. scala 13 compilation changes

    @SandishKumarHN
    SandishKumarHN committed Sep 22, 2022
    Configuration menu
    Copy the full SHA
    121630c View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2022

  1. fixing build issues

    @SandishKumarHN
    SandishKumarHN committed Sep 23, 2022
    Configuration menu
    Copy the full SHA
    e8a7c2b View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2022

  1. Added a new test case for from_proto->filter->to_proto

    @mposdev21
    mposdev21 committed Sep 25, 2022
    Configuration menu
    Copy the full SHA
    6cfad63 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'remote-spark/master' into SPARK_PROTO_1 

    # Conflicts:
#	pom.xml
    @SandishKumarHN
    SandishKumarHN committed Sep 25, 2022
    Configuration menu
    Copy the full SHA
    4b824bd View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2022

  1. spark-proto: shading google protobuf jars

    @SandishKumarHN
    SandishKumarHN committed Sep 28, 2022
    Configuration menu
    Copy the full SHA
    2d2a822 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2022

  1. mapsupport, rename proto-protobuf, schema-evolution, catch-recursive … 

    …case, CI fixes
    @SandishKumarHN
    SandishKumarHN committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    7cdf9dd View commit details
    Browse the repository at this point in the history

  2. Nit, cleanup unused

    @SandishKumarHN
    SandishKumarHN committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    a90800c View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2022

  1. stylecheck, remove positional match, handle null values

    @SandishKumarHN
    SandishKumarHN committed Oct 6, 2022
    Configuration menu
    Copy the full SHA
    6f447e8 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'remote-spark/master' into SPARK_PROTO_1 

    # Conflicts:
#	pom.xml
#	project/SparkBuild.scala
    @SandishKumarHN
    SandishKumarHN committed Oct 6, 2022
    Configuration menu
    Copy the full SHA
    7aa678c View commit details
    Browse the repository at this point in the history

  3. required field, remove tukaani, github labeler

    @SandishKumarHN
    SandishKumarHN committed Oct 6, 2022
    Configuration menu
    Copy the full SHA
    c6d45b7 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2022

  1. google protobuf shading changes

    @SandishKumarHN
    SandishKumarHN committed Oct 7, 2022
    Configuration menu
    Copy the full SHA
    58151dc View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2022

  1. Timestamp support and use latest error msg framework

    @SandishKumarHN
    SandishKumarHN committed Oct 11, 2022
    Configuration menu
    Copy the full SHA
    03d791c View commit details
    Browse the repository at this point in the history

  2. DayTimeIntervalType and Duration support

    @SandishKumarHN
    SandishKumarHN committed Oct 11, 2022
    Configuration menu
    Copy the full SHA
    d7c40ac View commit details
    Browse the repository at this point in the history

  3. move arraywriter mapping to single line

    @SandishKumarHN
    SandishKumarHN committed Oct 11, 2022
    Configuration menu
    Copy the full SHA
    330d01e View commit details
    Browse the repository at this point in the history