Package: tok
Title: Fast Text Tokenization
Version: 0.2.2
Authors@R: c(
    person("Tomasz", "Kalinowski", , "tomasz@posit.co", c("ctb", "cre")),
    person("Daniel", "Falbel", , "dfalbel@gmail.com", c("aut")),
    person("Regouby", "Christophe", , "christophe.regouby@free.fr", c("ctb")),
    person(family = "Posit", role = c("cph"))
  )
Description: 
  Interfaces with the 'Hugging Face' tokenizers library to provide implementations
  of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm 
  <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both 
  training new vocabularies and tokenizing texts.
License: MIT + file LICENSE
SystemRequirements: Cargo (Rust's package manager), rustc >= 1.91
Encoding: UTF-8
RoxygenNote: 7.3.3
Depends: R (>= 4.2.0)
Imports: R6, cli
Suggests: rmarkdown, testthat (>= 3.0.0), hfhub (>= 0.1.1), withr
Config/testthat/edition: 3
URL: https://github.com/mlverse/tok
BugReports: https://github.com/mlverse/tok/issues
Config/rextendr/version: 0.5.0
NeedsCompilation: yes
Packaged: 2026-04-21 23:57:56 UTC; dfalbel
Author: Tomasz Kalinowski [ctb, cre],
  Daniel Falbel [aut],
  Regouby Christophe [ctb],
  Posit [cph]
Maintainer: Tomasz Kalinowski <tomasz@posit.co>
Repository: CRAN
Date/Publication: 2026-04-22 14:40:02 UTC
Built: R 4.5.3; x86_64-w64-mingw32; 2026-04-29 02:01:22 UTC; windows
Archs: x64
