fix(dict): Remove only corrections if a space could be inserted as well

[not-my-profile]

The typo dictionary words.csv previously contained a bunch of problematic entries such as:

abouta,about
algorithmi,algorithm
attachen,attach
shouldbe,should

Which resulted in wrong corrections if the following spaces (indicated by ␣) were accidentally missed:

about␣a
algorithm␣i developed
attach␣en masse
should␣be

Many of these entries were introduced by taking entries from the codespell-dict and removing corrections containing spaces (since typos currently doesn't support them), e.g the codespell dictionary contains:

abouta->about a, about,
shouldbe->should, should be,

This commit updates tests/verify.rs to automatically remove entries in the form of {correction}{common_word},{correction}, where {common_word} is one of the 1000 most frequent English words.

The top-1000-most-frequent-words.csv file was generated by running:

curl https://norvig.com/ngrams/count_1w.txt \
  | head -n1024 \
  | awk '{print $1;}' \
  | grep -vE '^([^ia]|al|re)$' \
  > top-1000-most-frequent-words.csv

[epage]

Let's leave this off for now because we'd need to workout cases like "extrememe"

[not-my-profile]

I worked these cases out by adding the check:

                        if only_correction.ends_with(suffix) {
                            // We still want to correct e.g. "extrememe" to "extreme".
                            return true;
                        }

[epage]

I think I'd still prefer not be constrained by this very mechanical process. It can provide insight but I don't trust it to automatically be applied

[not-my-profile]

The current process now seems to make exactly the changes we want to all our 63,200 entries, which does inspire some confidence in me. Besides the process is very easy to adapt so I think we can just do so when we figure out that it too eagerly filters out something.

[epage]

The current process now seems to make exactly the changes we want to all our 63,200 entries

Except this isn't exactly what I want (see the other thread). Arbitrarily combining words that don't make sense when combined leads to us losing corrections we would have otherwise.

[not-my-profile]

I have updated the PR to now also detect {common_word}{correction},{correction} and not just {correction}{common_word},{correction}.

[epage]

The redundant a's is a separate thing and we should be correcting these

The challenge with blindly checking concatenated words is it doesn't filter out for when they don't make sense.

[epage]

In general, I'd feel better if we just looked at what changed due to the spaces and applied it to those. We could then separate decide which of these changes might make sense. As is, I'm seeing a lot that don't and don't want to take the time to decide that.

[not-my-profile]

Oh the logic I had did actually already detect these with

            if only_correction.starts_with(prefix) {
                return false;
            }

I just must have omitted resetting words.csv to run SNAPSHOTS=overwrite cargo test verify again.

[not-my-profile]

So the many cases have just become:

aequidistant
aequivalent
afor
amuch
anumber
ascripts
asudo
imakes
isimilar
itheir
itheirs
iwithout

which I think make sense to not correct automatically.

[epage]

iwithout and many of these don't make sense from a "word combining" perspective. Things we can correct in master will become uncorrectable with this change.