Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

functions: Add lower support #3521

Merged
merged 4 commits into from
Dec 17, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions common/functions/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ hex = "0.4.3"
base64 = "0.13.0"
itertools = "0.10.3"
num-format = "0.4"
bstr = "0.2.17"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's this crate for?

Copy link
Member Author

@Xuanwo Xuanwo Dec 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bstr is a byte string library. I use the ByteSlice trait so that we don't need to implement s.char_indices() on [u8] again.


[dev-dependencies]
bumpalo = "3.8.0"
Expand Down
32 changes: 32 additions & 0 deletions common/functions/src/scalars/strings/lower.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
// Copyright 2021 Datafuse Labs.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

use bstr::ByteSlice;

use super::string2string::String2StringFunction;
use super::string2string::StringOperator;

#[derive(Clone, Default)]
pub struct Lower;

impl StringOperator for Lower {
#[inline]
fn apply_with_no_null<'a>(&'a mut self, s: &'a [u8], buffer: &mut [u8]) -> usize {
buffer.copy_from_slice(&s.to_lowercase());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some research, I found:

  • to_lowercase_into requires a &mut Vec<u8> and we can't fit the requirement.

One possible solution is to implement the to_lowercase_into by hand. What's your idea?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just use the codes inside to_lowercase_into:

 #[inline]
    fn to_lowercase_into(&self, buf: &mut Vec<u8>) {
        // TODO: This is the best we can do given what std exposes I think.
        // If we roll our own case handling, then we might be able to do this
        // a bit faster. We shouldn't roll our own case handling unless we
        // need to, e.g., for doing caseless matching or case folding.

        // TODO(BUG): This doesn't handle any special casing rules.

        buf.reserve(self.as_bytes().len());
        for (s, e, ch) in self.char_indices() {
            if ch == '\u{FFFD}' {
                buf.push_str(&self.as_bytes()[s..e]);
            } else if ch.is_ascii() {
                buf.push_char(ch.to_ascii_lowercase());
            } else {
                for upper in ch.to_lowercase() {
                    buf.push_char(upper);
                }
            }
        }
    }


s.len()
}
}

pub type LowerFunction = String2StringFunction<Lower>;
2 changes: 2 additions & 0 deletions common/functions/src/scalars/strings/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ mod insert;
mod leftright;
mod length;
mod locate;
mod lower;
mod oct;
mod octet_length;
mod ord;
Expand Down Expand Up @@ -67,6 +68,7 @@ pub use length::LengthFunction;
pub use locate::InstrFunction;
pub use locate::LocateFunction;
pub use locate::PositionFunction;
pub use lower::LowerFunction;
pub use oct::OctFunction;
pub use octet_length::OctetLengthFunction;
pub use ord::OrdFunction;
Expand Down
3 changes: 3 additions & 0 deletions common/functions/src/scalars/strings/string.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ use crate::scalars::LeftFunction;
use crate::scalars::LeftPadFunction;
use crate::scalars::LengthFunction;
use crate::scalars::LocateFunction;
use crate::scalars::LowerFunction;
use crate::scalars::OctFunction;
use crate::scalars::OctetLengthFunction;
use crate::scalars::OrdFunction;
Expand Down Expand Up @@ -98,5 +99,7 @@ impl StringFunction {
factory.register("find_in_set", FindInSetFunction::desc());
factory.register("length", LengthFunction::desc());
factory.register("format", FormatFunction::desc());
factory.register("lower", LowerFunction::desc());
factory.register("lcase", LowerFunction::desc());
}
}
74 changes: 74 additions & 0 deletions common/functions/tests/it/scalars/strings/lower.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
// Copyright 2021 Datafuse Labs.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use common_datavalues::prelude::*;
use common_exception::Result;
use common_functions::scalars::LowerFunction;

use super::run_tests;
use super::Test;

#[test]
fn test_lower_function() -> Result<()> {
let schema = DataSchemaRefExt::create(vec![DataField::new("a", DataType::String, false)]);

let tests = vec![
Test {
name: "lower-abc-passed",
display: "lower",
nullable: true,
arg_names: vec!["a"],
columns: vec![Series::new(vec!["Abc"]).into()],
func: LowerFunction::try_create("lower")?,
expect: DataColumn::Constant(DataValue::String(Some("abc".as_bytes().to_vec())), 1),
error: "",
},
Test {
name: "lower-utf8-passed",
display: "lower",
nullable: true,
arg_names: vec!["a"],
columns: vec![Series::new(vec!["Dobrý den"]).into()],
func: LowerFunction::try_create("lower")?,
expect: DataColumn::Constant(
DataValue::String(Some("dobrý den".as_bytes().to_vec())),
1,
),
error: "",
},
Test {
name: "lcase-utf8-passed",
display: "lcase",
nullable: true,
arg_names: vec!["a"],
columns: vec![Series::new(vec!["Dobrý den"]).into()],
func: LowerFunction::try_create("lcase")?,
expect: DataColumn::Constant(
DataValue::String(Some("dobrý den".as_bytes().to_vec())),
1,
),
error: "",
},
Test {
name: "lcase-null-passed",
display: "lcase",
nullable: true,
arg_names: vec!["a"],
columns: vec![Series::new(vec![Option::<Vec<u8>>::None]).into()],
func: LowerFunction::try_create("lcase")?,
expect: DataColumn::Constant(DataValue::String(None), 1),
error: "",
},
];
run_tests(tests, schema)
}
2 changes: 2 additions & 0 deletions common/functions/tests/it/scalars/strings/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,10 @@
// limitations under the License.

mod locate;
mod lower;
mod substring;
mod trim;

mod utils;

pub use utils::*;
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
hello,world!
здравствуйте
NULL
3 changes: 3 additions & 0 deletions tests/suites/0_stateless/02_0042_function_strings_lower.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
SELECT LOWER('Hello,World!');
SELECT LOWER('Здравствуйте');
SELECT LOWER(NULL);
11 changes: 11 additions & 0 deletions website/databend/docs/user/sqlstatement/string-functions/lcase.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
title: LCASE
---

Synonym for LOWER(str).

## Syntax

```sql
LCASE(str);
```
33 changes: 33 additions & 0 deletions website/databend/docs/user/sqlstatement/string-functions/lower.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
title: LOWER
---

Returns the string str with all characters changed to lowercase.

## Syntax

```sql
LOWER(str);
```

## Arguments

| Arguments | Description |
|-----------|----------------------------|
| str | The string to be lowercase |


## Return Type

A string data type value.

## Examples

```txt
SELECT LOWER('Hello, World!')
+----------------------------+
| substring('Hello, World!') |
+----------------------------+
| hello, world! |
+----------------------------+
```