Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Day 1 #1

Merged
merged 2 commits into from
Sep 14, 2021
Merged

Day 1 #1

merged 2 commits into from
Sep 14, 2021

Conversation

yjshen
Copy link
Collaborator

@yjshen yjshen commented Sep 13, 2021 •
edited
Loading

I didn't have much time to work on it today. Sync what I have currently.

yjshen
yjshen commented Sep 13, 2021
View reviewed changes
datafusion/src/logical_plan/plan.rs
@@ -31,18 +31,6 @@ use std::{
sync::Arc,
};

use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicates

datafusion/src/optimizer/constant_folding.rs
@@ -32,7 +31,6 @@ use crate::optimizer::optimizer::OptimizerRule;
use crate::optimizer::utils;
use crate::physical_plan::functions::BuiltinScalarFunction;
use crate::scalar::ScalarValue;
use arrow::compute::{kernels, DEFAULT_CAST_OPTIONS};
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-valid uses;

datafusion/src/physical_plan/expressions/binary.rs
@@ -776,25 +776,6 @@ mod tests {
Ok(())
}

#[test]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicates

datafusion/src/physical_plan/datetime_expressions.rs
@@ -27,8 +27,7 @@ use arrow::{
array::*,
compute::cast,
datatypes::{
ArrowPrimitiveType, DataType, TimeUnit, TimestampMicrosecondType,
TimestampMillisecondType, TimestampNanosecondType, TimestampSecondType,
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-exists

datafusion/src/physical_plan/expressions/in_list.rs
@@ -41,6 +41,17 @@ macro_rules! compare_op_scalar {
}};
}

// TODO: primitive array currently doesn't have `values_iter()`, it may
// worth adding one there, and this specialized case could be removed.
macro_rules! compare_primitive_op_scalar {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check TODO above

datafusion/src/physical_plan/hash_utils.rs
@@ -116,7 +116,7 @@ fn combine_hashes(l: u64, r: u64) -> u64 {
}

macro_rules! hash_array {
($array_type:ident, $column: ident, $ty: ident, $hashes: ident, $random_state: ident, $multi_col: ident) => {
($array_type:ty, $column: ident, $ty: ident, $hashes: ident, $random_state: ident, $multi_col: ident) => {
let array = $column.as_any().downcast_ref::<$array_type>().unwrap();
Copy link
Collaborator Author

@yjshen yjshen Sep 13, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Utf8Array::<i32> is not a ident

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ty or path

@yjshen yjshen changed the title wip Day 1 Sep 13, 2021
@houqp houqp merged commit b9f12d0 into houqp:arrow2-merge Sep 14, 2021
houqp pushed a commit that referenced this pull request Jan 13, 2022
@Igosuki
fix tests #1
257a7c5
houqp pushed a commit that referenced this pull request May 27, 2024
@yyy1000 @alamb
@Weijun-H @Jefffrey
Support compute return types from argument values (not just their Dat…
92d9274 
…aTypes) (apache#8985)

* ScalarValue return types from argument values

* change file name

* try using ?Sized

* use Ok

* move method default impl outside trait

* Use type trait for ExprSchemable

* fix nit

* Proposed Return Type from Expr suggestions (#1)

* Improve return_type_from_args

* Rework example

* Update datafusion/core/tests/user_defined/user_defined_scalar_functions.rs

---------

Co-authored-by: Junhao Liu <junhaoliu2023@gmail.com>

* Apply suggestions from code review

Co-authored-by: Alex Huang <huangweijun1001@gmail.com>

* Fix tests + clippy

* rework types to use dyn trait

* fmt

* docs

* Apply suggestions from code review

Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

* Add docs explaining what happens when both `return_type` and `return_type_from_exprs` are called

* clippy

* fix doc -- comedy of errors

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Alex Huang <huangweijun1001@gmail.com>
Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
houqp pushed a commit that referenced this pull request May 27, 2024
@peter-toth @berkaysynnada
@mustafasrepo @ozankabak
Consolidate TreeNode transform and rewrite APIs (apache#8891)
a84e5f8 
* refactor `TreeNode::rewrite()`

* use handle_tree_recursion in `Expr`

* use macro for transform recursions

* fix api

* minor fixes

* fix

* don't trust `t.transformed` coming from transformation closures, keep the old way of detecting if changes were made

* rephrase todo comment, always propagate up `t.transformed` from the transformation closure, fix projection pushdown closure

* Fix `TreeNodeRecursion` docs

* extend Skip (Prune) functionality to Jump as it is defined in https://synnada.notion.site/synnada/TreeNode-Design-Proposal-bceac27d18504a2085145550e267c4c1

* fix Jump and add tests

* jump test fixes

* fix clippy

* unify "transform" traversals using macros, fix "visit" traversal jumps, add visit jump tests, ensure consistent naming `f` instead of `op`, `f_down` instead of `pre_visit` and `f_up` instead of `post_visit`

* fix macro rewrite

* minor fixes

* minor fix

* refactor tests

* add transform tests

* add apply, transform_down and transform_up tests

* refactor tests

* test jump on both a and e nodes in both top-down and bottom-up traversals

* better transform/rewrite tests

* minor fix

* simplify tests

* add stop tests, reorganize tests

* fix previous merges and remove leftover file

* Review TreeNode Refactor (#1)

* Minor changes

* Jump doesn't ignore f_up

* update test

* Update rewriter

* LogicalPlan visit update and propagate from children flags

* Update tree_node.rs

* Update map_children's

---------

Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>

* fix

* minor fixes

* fix f_up call when f_down returns jump

* simplify code

* minor fix

* revert unnecessary changes

* fix `DynTreeNode` and `ConcreteTreeNode` `transformed` and `tnr` propagation

* introduce TransformedResult helper

* fix docs

* restore transform as alias to trassform_up

* restore transform as alias to trassform_up 2

* Simplifications and comment improvements (#2)

---------

Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>
Co-authored-by: Mustafa Akur <mustafa.akur@synnada.ai>
Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
datafusion
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@yjshen @houqp