OpenAI’s newest blunder reveals the challenges dealing with Chinese language AI fashions

In truth, among the many few lengthy Chinese language tokens in GPT-4o that aren’t both pornography or playing nonsense, two are “socialism with Chinese language traits” and “Individuals’s Republic of China.” The presence of those phrases suggests {that a} vital a part of the coaching knowledge truly is from Chinese language state media writings, the place formal, lengthy expressions are extraordinarily widespread.

OpenAI has traditionally been very tight-lipped concerning the knowledge it makes use of to coach its fashions, and it in all probability won’t ever inform us how a lot of its Chinese language coaching database is state media and the way a lot is spam. (OpenAI didn’t reply to MIT Expertise Evaluation’s detailed questions despatched on Friday.)

However it’s not the one firm scuffling with this drawback. Individuals inside China who work in its AI trade agree there’s an absence of high quality Chinese language textual content knowledge units for coaching LLMs. One purpose is that the Chinese language web was once, and largely stays, divided up by huge firms like Tencent and ByteDance. They personal a lot of the social platforms and aren’t going to share their knowledge with rivals or third events to coach LLMs. 

In truth, that is additionally why engines like google, together with Google, kinda suck on the subject of looking out in Chinese language. Since WeChat content material can solely be searched on WeChat, and content material on Douyin (the Chinese language TikTok) can solely be searched on Douyin, this knowledge is just not accessible to a third-party search engine, not to mention an LLM. However these are the platforms the place precise human conversations are occurring, as a substitute of some spam web site that retains attempting to attract you into on-line playing.

The shortage of high quality coaching knowledge is a a lot larger drawback than the failure to filter out the porn and common nonsense in GPT-4o’s token-training knowledge. If there isn’t an current knowledge set, AI firms need to put in vital work to establish, supply, and curate their very own knowledge units and filter out inappropriate or biased content material. 

It doesn’t appear OpenAI did that, which in equity makes some sense, given that folks in China can’t use its AI fashions anyway. 

Nonetheless, there are various individuals dwelling outdoors China who wish to use AI providers in Chinese language. They usually deserve a product that works correctly as a lot as audio system of some other language do. 

How can we clear up the issue of the shortage of excellent Chinese language LLM coaching knowledge? Inform me your concept at zeyi@technologyreview.com.