Compare vocabulary size of WordPiece and BPE tokenizer algorithm
I have a text file. I used hugging face tokenizer library to ran WordPiece and BPE tokenizer algorithm on my text file. I trained them on the file and got the vocabulary size. In spite of my expectation, vocabulary size of WordPiece became bigger. The result was:
WordPiece vocabulary size: 17555
Byte-Pair Encoding vocabulary size: 16553
Anyone knows what is the reason of this result?
Here is my code for WordPiece algorithm:
from tokenizers import Tokenizer
from tokenizers.models import...