tiktoken

Elixir bindings for Tiktoken tokenizer

connorjacobsen
Natural Language Processing (NLP)

Package Resources

tiktoken
connorjacobsen/tiktoken-elixir
Documentation
No Changelog

Stars

25

Unreleased Activity

Up to date

First Release

2

years ago

Last Release

21

days ago

Downloads

7.1k

last 90 days

Last Stable Version

0.4.2

Activity

last year

Pull Requests

Open 0
Merged 5

Latest Merged Pull Requests

Issues

Open 0
Closed 1

Version 0.4.2

Published Oct 2025 •
Elixir Version Requirement ~> 1.13
Documentation for 0.4.2

Dependencies

2
rustler ~> 0.36.0

Mix compiler and runtime helpers for Rustler.

rustler_precompiled ~> 0.8

Make the usage of precompiled NIFs easier for projects using Rustler

No changelog for 0.4.2
Inspect source for 0.4.2

Related Packages

20 packages

inflex

2.1.0

An Elixir library for handling word inflections.

tokenizers

0.5.1

Bindings to Hugging Face Tokenizers for Elixir

text_chunker

0.5.2

An Elixir library for semantic text chunking.

gpt3_tokenizer

0.1.1

OpenAI GPT-3 Tokenizer