| 接口 | 说明 |
|---|---|
| Tokenizer |
| 类 | 说明 |
|---|---|
| QwenTokenizer |
BPE encode and decode, implementation reference https://github.com/openai/tiktoken and
https://github.com/karpathy/minbpe
|
| Tokenization | |
| Tokenization.Models | |
| TokenizationOutput | |
| TokenizationResult | |
| TokenizationUsage | |
| TokenizerFactory |
Copyright © 2025. All rights reserved.