This is a tokenizer play ground implemented with python.
- Step by Step
pip install -r requirements.txt
cd src
python app.py- Quick launch
Linux/Mac
bash run.shWindows
double click `run.bat`
-
Notifications
You must be signed in to change notification settings - Fork 0
tokenizer play ground and visualization for different models
License
thinkall/tokenizer-play-ground
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
tokenizer play ground and visualization for different models
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published