Wheelodex » Projects » ai21-tokenizer

ai21-tokenizer

View on PyPI — Reverse Dependencies (2)

0.12.0 ai21_tokenizer-0.12.0-py3-none-any.whl

Wheel Details

Project: ai21-tokenizer
Version: 0.12.0
Filename: ai21_tokenizer-0.12.0-py3-none-any.whl
Download: [link]
Size: 2675582
MD5: c616bba971bd67d9480cc8cdeba7d21f
SHA256: 7fd37b9093894b30b0f200e5f44fc8fb8772e2b272ef71b6d73722b4696e63c4
Uploaded: 2024-08-21 11:42:03 +0000

dist-info

METADATA · WHEEL · RECORD

METADATA

Metadata-Version: 2.1
Name: ai21-tokenizer
Version: 0.12.0
Author: AI21 Labs
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.8,<4.0
Requires-Dist: anyio (<5.0.0,>=4.4.0)
Requires-Dist: sentencepiece (<1.0.0,>=0.2.0)
Requires-Dist: tokenizers (<1.0.0,>=0.15.0)
Description-Content-Type: text/markdown
[Description omitted; length: 5929 characters]

WHEEL

Wheel-Version: 1.0
Generator: poetry-core 1.9.0
Root-Is-Purelib: true
Tag: py3-none-any

RECORD

Path Digest Size
ai21_tokenizer/__init__.py sha256=xRUqXVTILL0t1524Mly6Fma6umWae5NT4WqorvPvidM 807
ai21_tokenizer/base_jamba_tokenizer.py sha256=8vKD1Cg-sOcbgkE2KlzobMzmQ6hCnHDZTx5N45ACgbw 1625
ai21_tokenizer/base_jurassic_tokenizer.py sha256=I6N4EzUqNfgYXpo5Sebsu5nNt6nw2aA8PE_JXucIQ78 7979
ai21_tokenizer/base_tokenizer.py sha256=-2eDNnj93YBHxz_dp_OzlDxNwitf4nOix8-1z7fKdT0 4507
ai21_tokenizer/file_utils.py sha256=Pfm7WfSzK3I_TPpzXdWaJeAKD6J2H94ntGWwLMhkBIQ 1106
ai21_tokenizer/jamba_1_5_tokenizer.py sha256=I4WIDukKck3o2mxkaQy1scaMqdq-Yn5StXK8X9KeQHY 5473
ai21_tokenizer/jamba_instruct_tokenizer.py sha256=SlnAK0ZMwknHorzuYraqRGlAAfK765S6d2tl1Sr-x6I 5481
ai21_tokenizer/jurassic_tokenizer.py sha256=ASGQv2hbnonBNJFtMtxTXJ6h89UcRlkUdxYobBwrD28 7772
ai21_tokenizer/resources/j2-tokenizer/config.json sha256=vM7OEN5e2bQeb9qsR09wgybrK1NI9_28SUWDnHerko8 223
ai21_tokenizer/resources/j2-tokenizer/j2-tokenizer.model sha256=Dadce3WQgG5ut5Hssr7rid957yuUcogJqR0TSfqbPYI 6659225
ai21_tokenizer/tokenizer_factory.py sha256=CVjmvlpXF1NtyAnlWdiiDp18JX21cKs7KZ5RBsntwok 3933
ai21_tokenizer/version.py sha256=oFXJ29j3C1ptyR4srl_tl-0-F7jzcbzJxfN7qHOHwv0 19
ai21_tokenizer-0.12.0.dist-info/LICENSE sha256=xx0jnfkXJvxRnG63LTGOxlggYnIysveWIZ6H3PNdCrQ 11357
ai21_tokenizer-0.12.0.dist-info/METADATA sha256=kOt9NYSYFVZMzBUcLbEArNNl-W7ZTVe5eHNcGqhMP0Q 6514
ai21_tokenizer-0.12.0.dist-info/WHEEL sha256=sP946D7jFCHeNz5Iq4fL4Lu-PrWrFsgfLXbbkciIZwg 88
ai21_tokenizer-0.12.0.dist-info/RECORD — —