<Compare vocabulary size of WordPiece and BPE toke...
# community-feeds
r
Compare vocabulary size of WordPiece and BPE tokenizer algorithm I have a text file. I used hugging face tokenizer library to ran WordPiece and BPE tokenizer algorithm on my text file. I trained them on the file and got the vocabulary size. In spite of my expectation, vocabulary size of WordPiece became bigger. The result was: WordPiece vocabulary size: 17555 Byte-Pair Encoding vocabulary size: 16553 Anyone knows what is the reason of this result? Here is my code for WordPiece algorithm: from tokenizers import Tokenizer from tokenizers.models import...