The brand new tokenizer has 200,000 tokens in whole, and about 25% are in non-English languages, says Deedy Das, an AI investor at Menlo Ventures. He used language filters to depend the variety of tokens in several languages, and the highest languages, moreover English, are Russian, Arabic, and Vietnamese.
“So the tokenizer’s predominant influence, in my view, is you get the associated fee down in these languages, not that the standard in these languages goes dramatically up,” Das says. When an LLM has higher and longer tokens in non-English languages, it might probably analyze the prompts quicker and cost customers much less for a similar reply. With the brand new tokenizer, “you’re taking a look at virtually 4 occasions price discount,” he says.
Das, who additionally speaks Hindi and Bengali, took a take a look at the longest tokens in these languages. The tokens replicate discussions taking place in these languages, in order that they embrace phrases like “Narendra” or “Pakistan,” however widespread English phrases like “Prime Minister,” “college,” and “worldwide” additionally come up incessantly. Additionally they don’t exhibit the problems surrounding the Chinese language tokens.
That probably displays the coaching knowledge in these languages, Das says: “My working concept is the web sites in Hindi and Bengali are very rudimentary. It’s like [mostly] information articles. So I might count on this to be the case. There aren’t many spam bots and porn web sites attempting to occur in these languages. It’s largely going to be in English.”
Polluted knowledge and an absence of cleansing
Nevertheless, issues are drastically completely different in Chinese language. Based on a number of researchers who’ve seemed into the brand new library of tokens used for GPT-4o, the longest tokens in Chinese language are virtually completely spam phrases utilized in pornography, playing, and scamming contexts. Even shorter tokens, like three-character-long Chinese language phrases, replicate these matters to a big diploma.
“The issue is evident: the corpus used to coach [the tokenizer] isn’t clear. The English tokens appear high quality, however the Chinese language ones aren’t,” says Cai from Princeton College. It isn’t uncommon for a language mannequin to crawl spam when gathering coaching knowledge, however normally there can be vital effort taken to scrub up the information earlier than it’s used. “It’s potential that they didn’t do correct knowledge clearing in relation to Chinese language,” he says.
The content material of those Chinese language tokens may counsel that they’ve been polluted by a particular phenomenon: web sites hijacking unrelated content material in Chinese language or different languages to spice up spam messages.
These messages are sometimes ads for pornography movies and playing web sites. They might be actual companies or merely scams. And the language is inserted into content material farm web sites or typically respectable web sites to allow them to be listed by serps, circumvent the spam filters, and are available up in random searches. For instance, Google listed one search consequence web page on a US National Institutes of Health website, which lists a porn web site in Chinese language. The identical web site identify additionally appeared in at the least 5 Chinese language tokens in GPT-4o.
