Skip to content

Instantly share code, notes, and snippets.

View xenova's full-sized avatar

Joshua Lochner xenova

View GitHub Profile
@xenova
xenova / tiktoken-to-hf.ipynb
Last active August 29, 2024 03:05
Convert tiktoken tokenizers to the Hugging Face tokenizers format
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.