研究子词分词在语言模型训练中的优势解耦

Today we release a study on decoupling the benefits of subword tokenization for language model training, by simulating each suspected benefit one at a time inside a 1.7B byte-level pretraining pipeline.

We formulate seven hypotheses for why subword LLMs outperform byte-level https://t.co/nT9VkG50aq