tiktoken

Elixir bindings for Tiktoken tokenizer

connorjacobsen
Natural Language Processing (NLP)

Package Resources

tiktoken
connorjacobsen/tiktoken-elixir
Documentation
No Changelog

Stars

24

Unreleased Activity

14

commits

First Release

2

years ago

Last Release

11

months ago

Downloads

14.1k

last 90 days

Last Stable Version

0.4.1

Activity

last year

Pull Requests

Open 0
Merged 4

Latest Merged Pull Requests

Issues

Open 0
Closed 1

Version 0.4.1

Published Sep 2024 •
Elixir Version Requirement ~> 1.13
Documentation for 0.4.1

Dependencies

2
rustler ~> 0.30.0

Mix compiler and runtime helpers for Rustler.

rustler_precompiled ~> 0.7

Make the usage of precompiled NIFs easier for projects using Rustler

No changelog for 0.4.1
Inspect source for 0.4.1

Related Packages

22 packages

inflex

2.1.0

An Elixir library for handling word inflections.

tokenizers

0.5.1

Bindings to Hugging Face Tokenizers for Elixir

text_chunker

0.4.0

An Elixir library for semantic text chunking.

fast_ngram

1.2.0

A fast and unicode aware letter & word N-gram library written in Elixir.