Support using simdutf8 for validate_string_view and other utf8 validation

[Dandandan]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

We have some different sources were std::str::from_utf8 is used for utf8 validation, such as validate_string_view.
We can speed it up by supporting simdutf8 as well here.

Describe the solution you'd like

Similar for #6668 , optionally switch to using simdutf8 instead.

Describe alternatives you've considered

Additional context

[etseidl]

I mentioned this in #6668, but I was thinking we could just globally replace calls to from_utf8 with an internal version. Something like

#[cfg(feature = "simdutf8")]
#[inline(always)]
pub fn from_utf8(val: &[u8]) -> Result<&str, simdutf8::compat::Utf8Error> {
    match simdutf8::basic::from_utf8(val) {
        Ok(result) => Ok(result),
        Err(_) => simdutf8::compat::from_utf8(val),
    }
}

#[cfg(not(feature = "simdutf8"))]
#[inline(always)]
pub fn from_utf8(val: &[u8]) -> Result<&str, std::str::Utf8Error> {
    std::str::from_utf8(val)
}

[alamb]

I am embarassed to admit I missed the StringView one -- I will make a PR to switch

[alamb]

Update I don't think I was going crazy. I can't find any remaining hot location that does utf8 validation that I think will make any different for performance

• Experimental parquet decoder with first-class selection pushdown support #6921 (comment)

To be clear, I don't think @etseidl 's idea is a bad one, just I don't think it will help much

[etseidl]

I don't think it will help much

That is entirely possible. I'm still testing to see if I have any use cases that would benefit.

[alamb]

Maybe I should have used the positive rather than double negative: " @etseidl 's idea is a good one, I just don't think it will help much " 🤦

[Dandandan]

There are places that still do std::str::from_utf8 like validate_string_view.

arrow-rs/arrow-data/src/byte_view.rs

Line 145 in 3bf29a2

pub fn validate_string_view(views: &[u128], buffers: &[Buffer]) -> Result<(), ArrowError> {

This still has std::str::from_utf8 ?
I got it from @alamb profile in apache/datafusion#13983 (comment) where it has ~5% of samples, so I expect that query to benefit from it by a couple of percentage points.

I think there is also some other places which can benefit by a bit from this (e.g. reading csv/json, casting...).
https://github.com/search?q=repo%3Aapache%2Farrow-rs%20std%3A%3Astr%3A%3Afrom_utf8&type=code

[Dandandan]

So I think we can follow the idea of @etseidl to replace all std::str::from_utf8 calls.