Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-24251][SQL] Add AppendData logical plan. #21305

Closed
wants to merge 10 commits into from
Closed

[SPARK-24251][SQL] Add AppendData logical plan. #21305

wants to merge 10 commits into from

Commits on Aug 7, 2018

  1. SPARK-24251: Add AppendData logical plan. 

    This adds a new logical plan, AppendData, that was proposed in
SPARK-23521. This also adds an analyzer rule to validate data written
with AppendData against the target table. DataFrameWriter is also
updated so that v2 writes use the new AppendData logical plan.
    @rdblue
    rdblue committed Aug 7, 2018
    Configuration menu
    Copy the full SHA
    d0d2de8 View commit details
    Browse the repository at this point in the history

  2. SPARK-24251: Fix deprecation annotation for old plan.

    @rdblue
    rdblue committed Aug 7, 2018
    Configuration menu
    Copy the full SHA
    98bf4df View commit details
    Browse the repository at this point in the history

  3. SPARK-24251: Use source name in unknown v2 table names.

    @rdblue
    rdblue committed Aug 7, 2018
    Configuration menu
    Copy the full SHA
    ae38187 View commit details
    Browse the repository at this point in the history

  4. SPARK-24251: Update for review comments. 

    * Require NamedRelation in AppendData
* Check each column's compatibility recursively
    @rdblue
    rdblue committed Aug 7, 2018
    Configuration menu
    Copy the full SHA
    504be8a View commit details
    Browse the repository at this point in the history

  5. SPARK-24251: Remove "DataSourceV2" from AppendData docs.

    @rdblue
    rdblue committed Aug 7, 2018
    Configuration menu
    Copy the full SHA
    d62e1b6 View commit details
    Browse the repository at this point in the history

  6. SPARK-24251: Add better error message for nested fields.

    @rdblue
    rdblue committed Aug 7, 2018
    Configuration menu
    Copy the full SHA
    4127833 View commit details
    Browse the repository at this point in the history

  7. SPARK-24251: Add catch-all case to DataType.canWrite.

    @rdblue
    rdblue committed Aug 7, 2018
    Configuration menu
    Copy the full SHA
    d22ebf1 View commit details
    Browse the repository at this point in the history

  8. SPARK-24251: Implement strict matching for structs, add canWrite tests.

    @rdblue
    rdblue committed Aug 7, 2018
    Configuration menu
    Copy the full SHA
    36c0237 View commit details
    Browse the repository at this point in the history

  9. SPARK-24251: Do not allow any missing fields. 

    Previously missing fields were allowed if the read type's corresponding
field was optional.
    @rdblue
    rdblue committed Aug 7, 2018
    Configuration menu
    Copy the full SHA
    e81790d View commit details
    Browse the repository at this point in the history

  10. Merge branch 'master' into SPARK-24251-add-append-data

    @rdblue
    rdblue authored Aug 7, 2018
    Configuration menu
    Copy the full SHA
    42d86e1 View commit details
    Browse the repository at this point in the history