tiktoken

Elixir bindings for Tiktoken tokenizer

connorjacobsen
Natural Language Processing (NLP)

Package Resources

tiktoken
connorjacobsen/tiktoken-elixir
Documentation
No Changelog

Stars

25

Unreleased Activity

Up to date

First Release

2

years ago

Last Release

2

months ago

Downloads

4.5k

last 90 days

Last Stable Version

0.4.2

Activity

last year

Pull Requests

Open 0
Merged 5

Latest Merged Pull Requests

Issues

Open 0
Closed 1

Version 0.4.2

Published Oct 2025 •
Elixir Version Requirement ~> 1.13
Documentation for 0.4.2

Dependencies

2
rustler ~> 0.36.0

Mix compiler and runtime helpers for Rustler.

rustler_precompiled ~> 0.8

Make the usage of precompiled NIFs easier for projects using Rustler

No changelog for 0.4.2
Inspect source for 0.4.2

Community

Looks like this space is still empty… Want to be the first to contribute?

Share an article, talk, or podcast by adding resources to this package.

Related Packages

16 packages

inflex

2.1.0

An Elixir library for handling word inflections.

tokenizers

0.5.1

Bindings to Hugging Face Tokenizers for Elixir

text_chunker

0.5.2

An Elixir library for semantic text chunking.

stemmer

1.2.0

An English (Porter2) stemming implementation in Elixir.