Mengzi Pre-training Model
Text Generation
Machine Translation
Industry SearchMengzi Pre-training Model
Introduction to Technology
The Mengzi pre-training model is a large-scale pre-training language model developed based on the team's self-developed technology. It can handle multilingual and multimodal data, and supports multiple text understanding and text generation tasks. The Mengzi model is based on Transformer architecture, contains 1 billion parameters, and is based on hundreds of gigabytes of high-quality corpus covering Internet web pages, communities, news, e-commerce, finance and other fields.
Mengzi Pre-training Model
Introduction to Technology
The Mengzi pre-training model is a large-scale pre-training language model developed based on the team's self-developed technology. It can handle multilingual and multimodal data, and supports multiple text understanding and text generation tasks. The Mengzi model is based on Transformer architecture, contains 1 billion parameters, and is based on hundreds of gigabytes of high-quality corpus covering Internet web pages, communities, news, e-commerce, finance and other fields.
Technical Solutions
Support Multiple Model Architectures
- Autoregressive models: such as GPT
- Self-encoding models: such as BERT
- Encoder-Decoder model: T5
Lightweight Model Performance Enhancement
- Fusion of multiple pre-training tasks
- SMART adverserial training
- Knowledge distillation
Knowledge Graph Based Enhancement
- Enhancements with entity extraction
- Knowledge graph enhancement (isa relationship)
- Knowledge graph to text conversion
Linguistic Knowledge Based Enhancement
- Mask mechanism enhanced by syntactic information
- Semantic role embedding enhancement
- Attention weight constrained pruning of dependencies
Few-Shot/Zero-Shot Learning
- Prompt template construction
- Multi-task learning technique
- Common information extraction scenarios, out of the box
Retrieval Based Enhancement
- Knowledge decoupling
- Strong interpretability
- External knowledge components are updated in real time
Technical Advantages
It has achieved better performance than conventional models in multiple tasks
It supports BERT, GPT, T5 and other architectures, with different scenarios covered
It supports image and text dual-mode input, which better handles image and text related tasks
It supports rapid optimization for vertical domains, and offers models scaling from 10M to 1B parameters
CLUE Leaderboards
*Ranking as of July 30, 2021
| Ranking | 1 | 2 | 3 | |
|---|---|---|---|---|
| Model | Mengzi | Motian | BETRTSG | Human Level |
| Scale | 1B | 1B | 10B | |
| Total Score | 82.90 | 82.15 | 81.80 | 86.68 |
| AFQMC | 79.82 | 78.30 | 79.85 | 81.00 |
| TNEWS | 64.68 | 57.42 | 57.42 | 71.00 |
| IFLYTEK | 65.08 | 65.46 | 64.54 | 80.30 |
| OCNLI | 81.87 | 84.97 | 85.93 | 90.30 |
| WSC2020 | 96.55 | 94.83 | 95.17 | 98.00 |
| CSL | 89.87 | 90.17 | 89.00 | 84.00 |
| CMRC2018 | 82.25 | 85.30 | 83.80 | 92.40 |
| CHID | 96.00 | 94.43 | 93.06 | 87.10 |
| C3 | 89.98 | 88.49 | 87.44 | 96.00 |
| Ranking | Model | Scale | Total Score | AFQMC | TNEWS | IFLYTEK | OCNLI | WSC2020 | CSL | CMRC2018 | CHID | C3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Mengzi | 1B | 82.90 | 79.82 | 64.68 | 65.08 | 81.87 | 96.55 | 89.87 | 82.25 | 96.00 | 89.98 |
| 2 | Motian | 1B | 82.15 | 78.30 | 57.42 | 65.46 | 84.97 | 94.83 | 90.17 | 85.30 | 94.43 | 88.49 |
| 3 | BETRTSG | 10B | 81.80 | 79.85 | 57.42 | 64.54 | 85.93 | 95.17 | 89.00 | 83.80 | 93.06 | 87.44 |
| Human Level | 86.68 | 81.00 | 71.00 | 80.30 | 90.30 | 98.00 | 84.00 | 92.40 | 87.10 | 96.00 |
Application Scenarios
Bulletin Extraction
Fiction Generation
Sentiment Classification
Research Reports Classification
News Digest
Knowledge Map Construction
Q&A System
Image-text Mutual Inspection
Customer Stories
Hithink RoyalFlush Information Network Co., Ltd.
Together with RoyalFlush, Langboat Technology focuses on the field of cognitive intelligence, jointly innovates NLP technology, upgrades products and services in the financial technology field, and brings better user experience to customers.
Business Cooperation Email
Address
11F, Block A, Dinghao DH3 Building, No.3 Haidian Street, Haidian District, Beijing, China
Business Cooperation:
bd@langboat.com
Address:
11F, Block A, Dinghao DH3 Building, No.3 Haidian Street, Haidian District, Beijing, China
Official Accounts:
