Skip to content

Latest commit

 

History

History
243 lines (212 loc) · 5 KB

大模型词汇语言分布.md

File metadata and controls

243 lines (212 loc) · 5 KB

大模型词汇语言分布

Language 360Zhinao2-7B-Chat-4K Llama-3.3-70B-Instruct MiniCPM3-4B MiniMax-Text-01 Mistral-7B-Instruct-v0.3 Phi-3.5-mini-instruct Yi-1.5-34B-Chat deepseek_v3 gemma-2-9b-it glm-4-9b-chat gpt-4o internlm3-8b-instruct qwen2.5-72b telechat-7B
NULL 4582 7211 6843 5786 2166 2062 1796 6456 16809 4673 23900 5141 11374 7332
arabic 1623 3780 38 6123 70 54 15 3185 6131 1990 8036 13111 3634 0
chinese 46093 4265 28322 38420 1459 700 21356 35184 21762 28478 7449 10364 24966 29919
control 3489 3565 4 1809 1 1 9 1011 64 3489 3381 0 3488 790
digits 10 1110 10 1110 10 10 10 1110 10 411 1110 10 10 1365
english 68175 67300 21836 92564 16086 15627 23866 49324 112870 69196 95471 44979 65279 90312
greek 557 1375 143 128 58 63 24 616 1229 829 1498 71 127 174
hebrew 20 20 10 54 38 36 1 622 1301 20 2373 10 3160 60
japanese 883 1077 140 11918 134 137 51 880 7061 915 806 35 2064 210
korean 1447 2246 92 4522 346 111 28 1120 2295 544 2365 6394 3473 337
pure_english 28170 28158 15832 33489 10612 10204 15682 21994 72065 29915 37839 23291 27376 28863
russian 2226 6507 158 4037 1731 2951 1151 5251 13116 9734 14211 16082 4122 725
thai 58 1386 12 40 57 44 3 1247 1287 58 1561 9081 2570 48
Total 157333 128000 73440 200000 32768 32000 63992 128000 256000 150252 200000 128569 151643 160135

Summary

360Zhinao2-7B-Chat-4K (Total: 157333)

  • english: 68175
  • chinese: 46093
  • pure_english: 28170
  • NULL: 4582
  • control: 3489
  • russian: 2226
  • arabic: 1623
  • korean: 1447
  • japanese: 883
  • greek: 557
  • thai: 58
  • hebrew: 20
  • digits: 10

Yi-1.5-34B-Chat (Total: 63992)

  • english: 23866
  • chinese: 21356
  • pure_english: 15682
  • NULL: 1796
  • russian: 1151
  • japanese: 51
  • korean: 28
  • greek: 24
  • arabic: 15
  • digits: 10
  • control: 9
  • thai: 3
  • hebrew: 1

gemma-2-9b-it (Total: 256000)

  • english: 112870
  • pure_english: 72065
  • chinese: 21762
  • NULL: 16809
  • russian: 13116
  • japanese: 7061
  • arabic: 6131
  • korean: 2295
  • hebrew: 1301
  • thai: 1287
  • greek: 1229
  • control: 64
  • digits: 10

telechat-7B (Total: 160135)

  • english: 90312
  • chinese: 29919
  • pure_english: 28863
  • NULL: 7332
  • digits: 1365
  • control: 790
  • russian: 725
  • korean: 337
  • japanese: 210
  • greek: 174
  • hebrew: 60
  • thai: 48

Llama-3.3-70B-Instruct (Total: 128000)

  • english: 67300
  • pure_english: 28158
  • NULL: 7211
  • russian: 6507
  • chinese: 4265
  • arabic: 3780
  • control: 3565
  • korean: 2246
  • thai: 1386
  • greek: 1375
  • digits: 1110
  • japanese: 1077
  • hebrew: 20

Mistral-7B-Instruct-v0.3 (Total: 32768)

  • english: 16086
  • pure_english: 10612
  • NULL: 2166
  • russian: 1731
  • chinese: 1459
  • korean: 346
  • japanese: 134
  • arabic: 70
  • greek: 58
  • thai: 57
  • hebrew: 38
  • digits: 10
  • control: 1

Phi-3.5-mini-instruct (Total: 32000)

  • english: 15627
  • pure_english: 10204
  • russian: 2951
  • NULL: 2062
  • chinese: 700
  • japanese: 137
  • korean: 111
  • greek: 63
  • arabic: 54
  • thai: 44
  • hebrew: 36
  • digits: 10
  • control: 1

deepseek_v3 (Total: 128000)

  • english: 49324
  • chinese: 35184
  • pure_english: 21994
  • NULL: 6456
  • russian: 5251
  • arabic: 3185
  • thai: 1247
  • korean: 1120
  • digits: 1110
  • control: 1011
  • japanese: 880
  • hebrew: 622
  • greek: 616

qwen2.5-72b (Total: 151643)

  • english: 65279
  • pure_english: 27376
  • chinese: 24966
  • NULL: 11374
  • russian: 4122
  • arabic: 3634
  • control: 3488
  • korean: 3473
  • hebrew: 3160
  • thai: 2570
  • japanese: 2064
  • greek: 127
  • digits: 10

MiniCPM3-4B (Total: 73440)

  • chinese: 28322
  • english: 21836
  • pure_english: 15832
  • NULL: 6843
  • russian: 158
  • greek: 143
  • japanese: 140
  • korean: 92
  • arabic: 38
  • thai: 12
  • digits: 10
  • hebrew: 10
  • control: 4

internlm3-8b-instruct (Total: 128569)

  • english: 44979
  • pure_english: 23291
  • russian: 16082
  • arabic: 13111
  • chinese: 10364
  • thai: 9081
  • korean: 6394
  • NULL: 5141
  • greek: 71
  • japanese: 35
  • digits: 10
  • hebrew: 10

gpt-4o (Total: 200000)

  • english: 95471
  • pure_english: 37839
  • NULL: 23900
  • russian: 14211
  • arabic: 8036
  • chinese: 7449
  • control: 3381
  • hebrew: 2373
  • korean: 2365
  • thai: 1561
  • greek: 1498
  • digits: 1110
  • japanese: 806

MiniMax-Text-01 (Total: 200000)

  • english: 92564
  • chinese: 38420
  • pure_english: 33489
  • japanese: 11918
  • arabic: 6123
  • NULL: 5786
  • korean: 4522
  • russian: 4037
  • control: 1809
  • digits: 1110
  • greek: 128
  • hebrew: 54
  • thai: 40

glm-4-9b-chat (Total: 150252)

  • english: 69196
  • pure_english: 29915
  • chinese: 28478
  • russian: 9734
  • NULL: 4673
  • control: 3489
  • arabic: 1990
  • japanese: 915
  • greek: 829
  • korean: 544
  • digits: 411
  • thai: 58
  • hebrew: 20