Token Counter
Compare the number of tokens of various big prediction model word participanters hosted by Hugging Face, and preload some models for quick use by users.
Paste text in the input box, or upload one or more text files; select the models you want to compare in the "Built-in Model Word of" Advanced Options ", and then click" Start Counting "to view the number of tokens for each model. If the remote microservice cannot be connected to the Internet, you can expand "Advanced Options", enter the "Custom Model Word Segmentation" pop-up window, and import the tokenizer file downloaded locally through "Offline Cache Preloading"; or click "Roll Back to Resource Pack" to restore the application of built-in word segmentation. You can also right-click a text file from the Lazy Cat network disk, use this application to directly open and count.
