create dictionary based WordSegmenter returns MissingDataKey, segmenter/dictionary/w_auto@1

[xshadowlegendx]

hello everyone, I am not able to initialize the word segmenter using dictionary mode, but using lstm mode it works fine. with dictionary mode it throws me this error word segmenter init error - Data(DataError { kind: MissingDataKey, key: Some(DataKey{segmenter/dictionary/w_auto@1}), str_context: None, silent: true })

I used this command to generate the provider data icu4x-datagen --keys all --locales km --format blob --out my_data_blob.postcard --overwrite --trie-type fast and checking the output with grep shows that the dictionary key are there

icu4x-datagen ...  | grep segmenter
INFO  [icu_datagen] Writing key: segmenter/lstm/wl_auto@1
INFO  [icu_datagen] Writing key: segmenter/dictionary/wl_ext@1
INFO  [icu_datagen] Writing key: segmenter/dictionary/w_auto@1
INFO  [icu_datagen] Writing key: segmenter/grapheme@1
INFO  [icu_datagen] Writing key: segmenter/word@1
INFO  [icu_datagen] Writing key: segmenter/sentence@1
INFO  [icu_datagen] Writing key: segmenter/line@1

below is the main.rs and Cargo.toml

use icu::segmenter::WordSegmenter;
use icu_provider_blob::BlobDataProvider;

#[derive(Debug)]
struct CharPositionBound(usize, usize);

#[derive(Debug)]
struct CharPositionBoundConstruct(Option<usize>, Vec<CharPositionBound>);

fn main() {
    let blob = std::fs::read("my_data_blob.postcard").expect("Failed to read file");

    let buffer_provider = BlobDataProvider::try_new_from_blob(blob.into_boxed_slice())
        .expect("Failed to initialize Data Provider.");

    let d = WordSegmenter::try_new_dictionary_with_buffer_provider(&buffer_provider);

    if let Err(err) = d {
        println!("word segmenter init error - {:?}", err);
    } else if let Ok(res) = d {
        //let w = String::from("ប្រទេសកម្ពុជាជាប្រទេសល្អ");
        // let w = "ភាសាខ្មែរ";
        // let w = "កម្ពុជាសប្យាយ";
        let w = "សួស្តីពិភពលោក";

        let CharPositionBoundConstruct(_, res) = res.segment_str(&w).fold(
            CharPositionBoundConstruct(None, Vec::new()),
            |mut acc, elem| {
                if acc.0.is_some() {
                    acc.1.push(CharPositionBound(acc.0.unwrap(), elem));
                    acc.0 = Some(elem);
                }

                acc.0 = Some(elem);

                acc
            },
        );

        println!("{:?}", res);

        for CharPositionBound(start, end) in res {
            println!("{}", &w[start..end]);
        }
    }
}

# Cargo.toml

[package]
name = "rust_icu4x_demo"
version = "0.1.0"
edition = "2021"

[dependencies]
icu = { version = "1.2.0", features = ["serde"] }
icu_provider_blob = "1.2.0"

[robertbastian]

This is a bug in BlobDataProvider, whose current data model is not capable of storing keys without any values (such as segmenter/dictionary/w_auto@1 for --locales km), and therefore returns MissingDataKey instead of MissingLocale (which try_new_dictionary_with_buffer_provider would handle).

[robertbastian]

There are two workarounds I can think of if you need to use BlobDataProvider:

• Use --locales km ja to populate the key (it only supports a ja value)
• Write a thin wrapper around BlobDataProvider that returns the correct error:

struct FixProvider(BlobDataProvider);

use icu_provider::prelude::*;
impl BufferProvider for FixProvider {
    fn load_buffer(&self, key: DataKey, req: DataRequest) -> Result<DataResponse<BufferMarker>, DataError> {
        if key == icu::segmenter::provider::DictionaryForWordOnlyAutoV1Marker::KEY {
            Err(DataErrorKind::MissingLocale.with_req(key, req))
        } else {
            self.0.load_buffer(key, req)
        }
    }
}

let buffer_provider = FixProvider(BlobDataProvider::try_new_from_blob(blob.into_boxed_slice())
        .expect("Failed to initialize Data Provider."));

[Manishearth]

For such keys can we store some dummy entry that uses an empty-string locale or something similar as a sentinel value?

Another fix would be to fix dictionary segmenter specifically so that it is ok with MissingDataKey for some of these cases.

[sffc]

I'm not actually convinced that failing on MissingDataKey is the best behavior in the segmenter, but the proposed change to BlobDataProvider seems fine in a vacuum.