Wheelodex » Projects » autotiktokenizer

autotiktokenizer

View on PyPI — Reverse Dependencies (1)

0.2.1 autotiktokenizer-0.2.1-py3-none-any.whl

Wheel Details

Project: autotiktokenizer
Version: 0.2.1
Filename: autotiktokenizer-0.2.1-py3-none-any.whl
Download: [link]
Size: 8852
MD5: ba92610482fde78c8403d15ec571d387
SHA256: 0b32feff5c294afaffc8245c6a9228b93c5607b245fb2a5aacfaa5fa6eb1fff5
Uploaded: 2024-11-11 21:16:01 +0000

dist-info

METADATA · WHEEL · RECORD · top_level.txt

METADATA

Metadata-Version: 2.1
Name: autotiktokenizer
Version: 0.2.1
Summary: 🧰 The AutoTokenizer that TikToken always needed -- Load any tokenizer with TikToken now! ✨
Author-Email: Bhavnick Minhas <bhavnicksm[at]gmail.com>
Project-Url: Homepage, https://github.com/bhavnicksm/autotiktokenizer
License: MIT License Copyright (c) 2024 Bhavnick Minhas Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Keywords: tokenizer,transformers,huggingface,tiktoken
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Python: >=3.8
Requires-Dist: tiktoken (>=0.2.0)
Requires-Dist: huggingface-hub (>=0.0.12)
Requires-Dist: pytest (>=6.2.0); extra == "dev"
Provides-Extra: dev
Description-Content-Type: text/markdown
License-File: LICENSE
[Description omitted; length: 5790 characters]

WHEEL

Wheel-Version: 1.0
Generator: setuptools (75.4.0)
Root-Is-Purelib: true
Tag: py3-none-any

RECORD

Path Digest Size
autotiktokenizer/__init__.py sha256=GS-u_T0Trbysi7UdIBR_iXw45-Bvm6AZ1It_0STdKUw 139
autotiktokenizer/autotiktokenizer.py sha256=dX2StkzVVIcs849chIc6p1qs9sL2Bx25BQSZeGJN8k8 11741
autotiktokenizer-0.2.1.dist-info/LICENSE sha256=9p4m9unVoO-4D3nvK8wxLd0IPACqs5o-M801lIGW-BI 1072
autotiktokenizer-0.2.1.dist-info/METADATA sha256=hl6kE-3JusRm2uvv3dcPSWY9e5_M_XsFkQkxugm-djM 7989
autotiktokenizer-0.2.1.dist-info/WHEEL sha256=a7TGlA-5DaHMRrarXjVbQagU3Man_dCnGIWMJr5kRWo 91
autotiktokenizer-0.2.1.dist-info/top_level.txt sha256=ALiR-_aEI6LnyqyaFYfkOh5LHhWvpvADOEoKSpL_sds 17
autotiktokenizer-0.2.1.dist-info/RECORD — —

top_level.txt

autotiktokenizer