Use the unicode-ident crate

[MichaReiser]

Summary

This PR replaces the unic-ucd-ident with the much faster and smaller unicode-ident crate.

I don't expect this PR to improve performance in our benchmarks because

a) We have no test files with unicode identifiers
b) Our lexer has a fast path that bypasses the is_identifier_start for ascii characters.

This PR further removes the emoji identifier support which is not part of Python's identifier specification

Test Plan

cargo test

Wait for the ecosystem check

Performance

I extended the unicode-ident 's benchmark to also include unic-ucd-ident . unicode-ident outperforms it significantly

Running benches/ident.rs (target/release/deps/ident-ad9d9d2d593f7661)
Gnuplot not found, using plotters backend
0%-nonascii/baseline    time:   [217.56 µs 217.84 µs 218.14 µs]
0%-nonascii/unicode-ident
                        time:   [417.51 µs 418.02 µs 418.55 µs]
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe
0%-nonascii/unicode-xid time:   [1.5056 ms 1.5061 ms 1.5066 ms]
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe
0%-nonascii/unic-ucd-ident
                        time:   [9.8530 ms 9.8653 ms 9.8804 ms]
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) high mild
  4 (4.00%) high severe

1%-nonascii/baseline    time:   [239.63 µs 240.05 µs 240.60 µs]
1%-nonascii/unicode-ident
                        time:   [442.01 µs 443.87 µs 446.10 µs]
1%-nonascii/unicode-xid time:   [1.5937 ms 1.5941 ms 1.5946 ms]
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe
1%-nonascii/unic-ucd-ident
                        time:   [10.102 ms 10.145 ms 10.187 ms]

10%-nonascii/baseline   time:   [493.61 µs 493.94 µs 494.35 µs]
Found 14 outliers among 100 measurements (14.00%)
  1 (1.00%) low mild
  9 (9.00%) high mild
  4 (4.00%) high severe
10%-nonascii/unicode-ident
                        time:   [671.83 µs 672.38 µs 673.02 µs]
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe
Benchmarking 10%-nonascii/unicode-xid: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 10.0s. You may wish to increase target time to 13.7s, enable flat sampling, or reduce sample count to 60.
10%-nonascii/unicode-xid
                        time:   [2.6001 ms 2.6043 ms 2.6094 ms]
10%-nonascii/unic-ucd-ident
                        time:   [10.434 ms 10.448 ms 10.460 ms]
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) low severe
  1 (1.00%) high mild
  2 (2.00%) high severe

100%-nonascii/baseline  time:   [609.36 µs 611.90 µs 614.95 µs]
Found 25 outliers among 100 measurements (25.00%)
  25 (25.00%) low mild
100%-nonascii/unicode-ident
                        time:   [1.4087 ms 1.4127 ms 1.4161 ms]
Found 14 outliers among 100 measurements (14.00%)
  12 (12.00%) low severe
  2 (2.00%) low mild
100%-nonascii/unicode-xid
                        time:   [11.733 ms 11.742 ms 11.754 ms]
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) high mild
  3 (3.00%) high severe
100%-nonascii/unic-ucd-ident
                        time:   [10.895 ms 10.897 ms 10.900 ms]
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) low mild

[MichaReiser]

Current dependencies on/for this PR:

• main
  • PR Use the unicode-ident crate #7212 👈

This comment was auto-generated by Graphite.

[konstin]

This PR further removes the emoji identifier support which is not part of Python's identifier specification

A test for that would be good

[github-actions]

PR Check Results

Ecosystem

✅ ecosystem check detected no changes.