Baidu recently announced PLATO-XL, an AI model for dialog generation, which was trained on over a billion samples collected from social media conversations in both English and Chinese. PLATO-XL achieves state-of-the-art performance on several conversational benchmarks, outperforming currently available commercial chatbots.
The model and several experiments were described in a paper published on arXiv. PLATO-XL is based on a Unified Transformer architecture, which allows for simultaneous learning of both language understanding and response generation. The model uses multi-party-aware pre-training to distinguish the utterances of different participants in a conversation, which improves consistency of the bot's responses. When evaluated by human judges in open-domain conversation, PLATO-XL outperformed other chatbot models, including Facebook's Blender. PLATO-XL also set new performance records on benchmarks for knowledge grounded dialog and task-oriented conversation. According to the Baidu team,
PLATO-XL expands new horizons in open-domain conversations, one of the most challenging tasks in natural language processing. As the largest pre-training model for Chinese and English dialogue, PLATO-XL hits new levels of conversation consistency and factuality, one step closer to the future of finally human-like learning and chatting abilities.
Natural language processing (NLP) AI models have been shown to achieve improvements in performance from scale. These larger models are pre-trained on massive datasets, often scraped from the web, before being fine-tuned for specific NLP tasks. However, the Baidu researchers pointed out that it is currently unclear whether the dialog-generation models used by chatbots always benefit from increased scale, citing both Microsoft's DialoGPT and Facebook's Blender, where medium-sized models outperformed the largest models for those architectures. The key to achieving performance improvements with increased scale, according to Baidu, is the pre-training process.
PLATO-XL builds on the original PLATO model and the improved PLATO-2 released in 2020. The core of the model is a Unified Transformer, instead of the more common encoder-decoder architecture; this allows the model to share parameters between the tasks of language understanding and response generation, making it more efficient. Like many other chatbots, PLATO-XL is pre-trained using conversations scraped from social media websites---in this case, Reddit comments. However, because these conversations have multiple participants as well as threading hierarchy, the models often mix up information from different participants, producing inconsistent responses. To address this, Baidu added type and role embedding components to the training text inputs, which are used to distinguish different types of responses and participants in the conversation.
Using their PaddlePaddle deep-learning platform, Baidu trained PLATO-XL on both English and Chinese language datasets consisting of context/response pairs; the English data contained 811M samples and the Chinese, 1.2B. To evaluate its performance, the team collected transcripts of English and Chinese conversations between humans and several different chatbots, including PLATO-XL as well as DialoGPT, Blender, and PLATO-2. The judges scored the conversations on their coherence, informativeness, engagingness, inconsistency, and hallucination; PLATO-XL outscored all the other bots. The team also evaluated PLATO-XL on three benchmark datasets: DuConv, DSTC9-Track1, and MultiWOZ; the bot set new state-of-the-art performance, outscoring previous leading models by several percentage points.
InfoQ recently covered Baidu's ERNIE 3.0 model, which exceeded the human baseline performance on the SuperGLUE language-understanding benchmark. Several other large Chinese-language NLP models have also been developed recently. Earlier this year, Huawei announced its 200B-parameter PanGu-Alpha model, trained on 1.1TB of Chinese data, and cloud company Inspur announced its 245B-parameter Yuan model, trained on 5TB of data, that the company claims is the "current largest high-quality Chinese corpus."
Baidu says that they plan to release the PLATO-XL source code and English model "before the end of November 2021" as part of their Knover toolkit, which is available on GitHub.