Online playground for OpenAPI tokenizers

go version of tiktoken

Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion

An elegant LLM chat UI forked from chatgpt-demo of @anse-app. Index site at https://free-chat.asia

Analysis of OpenAI's ChatGPT

OpenShield is a new generation security layer for AI models

This project implements token calculation for OpenAI's gpt-4 and gpt-3.5-turbo model, specifically using `cl100k_base` encoding.

The ChatGPT Long Term Memory package is a powerful tool designed to empower your projects with the ability to handle a large number of simultaneous users and external sources.

A fast RWKV Tokenizer written in Rust

GPT4 Tokenizer Visualizer

Kotlin multiplatform BPE tokenizer library for OpenAI models

Predict the exact openai token usage of functions

A custom GPT based on [Zero To Hero](https://karpathy.ai/zero-to-hero.html) utilizing tiktoken with the intent to augment AI Transformer-model education and reverse engineer GPT models from scratch.

✂️ OpenAI's tiktoken tokenizer written in Go

C++ implementation of tokenizers, including tiktoken.

📐 GPT token estimation and context size utilities without a full tokenizer

🧰 The AutoTokenizer that TikToken always needed -- Load any tokenizer with TikToken now! ✨

C API for tiktoken-rs

The Tiktoken API is a tool that enables developers to calculate the token usage of their OpenAI API requests before sending them, allowing for more efficient use of tokens.

PHP port for openai/tiktoken (most)