tokendagger
last updated: Jun 30, 2025
https://github.com/M4THYOU/TokenDagger
A fast, drop-in implementation of OpenAI's TikToken, designed for large-scale text processing. 2x Throughput and 4x faster on code sample tokenization.
https://github.com/M4THYOU/TokenDagger
A fast, drop-in implementation of OpenAI's TikToken, designed for large-scale text processing. 2x Throughput and 4x faster on code sample tokenization.