tokenizers

Bindings to Hugging Face Tokenizers for Elixir

elixir_nx
Natural Language Processing (NLP)

Package Resources

tokenizers
elixir-nx/tokenizers
Documentation
Changelog

Stars

100

Unreleased Activity

5

commits

First Release

3

years ago

Last Release

1

years ago

Downloads

28.0k

last 90 days

Last Stable Version

0.5.1

Activity

last year

Pull Requests

Open 0
Merged 2

Latest Merged Pull Requests

Issues

Open 0
Closed 1

Version 0.5.1

Published Oct 2024 •
Elixir Version Requirement ~> 1.13
Documentation for 0.5.1

Dependencies

2
castore ~> 0.1 or ~> 1.0

Up-to-date CA certificate store.

rustler_precompiled ~> 0.6

Make the usage of precompiled NIFs easier for projects using Rustler

Optional Dependencies

Dependencies that are not required by default but can be installed or enable additional features or functionality

Changelog for 0.5.1
Inspect source for 0.5.1

Community

Looks like this space is still empty… Want to be the first to contribute?

Share an article, talk, or podcast by adding resources to this package.

Related Packages

16 packages

inflex

2.1.0

An Elixir library for handling word inflections.

text_chunker

0.5.2

An Elixir library for semantic text chunking.

stemmer

1.2.0

An English (Porter2) stemming implementation in Elixir.

fast_ngram

1.2.0

A fast and unicode aware letter & word N-gram library written in Elixir.