Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Speed up starts_with for small prefixes #19904

Merged
merged 10 commits into from
Nov 22, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions crates/polars-arrow/src/array/binview/view.rs
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,28 @@ impl View {
}
}

/// Checks if the string starts with the prefix
stijnherfst marked this conversation as resolved.
Show resolved Hide resolved
/// When the prefix is smaller than View::MAX_INLINE_SIZE then this will be very fast
pub fn starts_with(&self, prefix: &str, buffers: &[Buffer<u8>]) -> bool {
unsafe {
if self.length <= View::MAX_INLINE_SIZE {
self.get_inlined_slice_unchecked()
.starts_with(prefix.as_bytes())
} else {
let starts = self
.prefix
.to_le_bytes()
.starts_with(&prefix.as_bytes()[0..4]);
stijnherfst marked this conversation as resolved.
Show resolved Hide resolved
if starts {
return self
.get_slice_unchecked(buffers)
.starts_with(prefix.as_bytes());
}
false
}
}
}

/// Constructs a byteslice from this view.
///
/// # Safety
Expand Down
31 changes: 30 additions & 1 deletion crates/polars-ops/src/chunked_array/strings/namespace.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use arrow::array::ValueSize;
use arrow::array::{Array, ValueSize};
use arrow::legacy::kernels::string::*;
#[cfg(feature = "string_encoding")]
use base64::engine::general_purpose;
Expand Down Expand Up @@ -216,6 +216,35 @@ pub trait StringNameSpaceImpl: AsString {
}
}

/// Check if strings starts with a substring
fn starts_with(&self, sub: &str) -> BooleanChunked {
let ca = self.as_string();

let iter = ca.downcast_iter().map(|arr| {
let out: <BooleanType as PolarsDataType>::Array = arr
.views()
stijnherfst marked this conversation as resolved.
Show resolved Hide resolved
.iter()
.map(|view| view.starts_with(sub, arr.data_buffers()))
.collect_arr_with_dtype(DataType::Boolean.to_arrow(CompatLevel::newest()));
out.with_validity_typed(arr.validity().cloned())
});

ChunkedArray::from_chunk_iter(ca.name().clone(), iter)
}

/// This is more performant than the BinaryChunked version because we use the inline prefix
/// Use the BinaryChunked::ends_with as there is no specialization here for that
fn starts_with_chunked(&self, prefix: &StringChunked) -> BooleanChunked {
let ca = self.as_string();
match prefix.len() {
1 => match prefix.get(0) {
Some(s) => self.starts_with(s),
None => BooleanChunked::full_null(ca.name().clone(), ca.len()),
},
_ => broadcast_binary_elementwise_values(ca, prefix, |s, sub| s.starts_with(sub)),
}
}

/// Get the length of the string values as number of chars.
fn str_len_chars(&self) -> UInt32Chunked {
let ca = self.as_string();
Expand Down
5 changes: 2 additions & 3 deletions crates/polars-plan/src/dsl/function_expr/strings.rs
Original file line number Diff line number Diff line change
Expand Up @@ -502,9 +502,8 @@ pub(super) fn ends_with(s: &[Column]) -> PolarsResult<Column> {
}

pub(super) fn starts_with(s: &[Column]) -> PolarsResult<Column> {
let ca = &s[0].str()?.as_binary();
let prefix = &s[1].str()?.as_binary();

let ca = s[0].str()?;
let prefix = s[1].str()?;
Ok(ca.starts_with_chunked(prefix).into_column())
}

Expand Down
Loading