Basics of Natural Language Processing NLP Advanced Course#

This part covers the basics of natural language processing. Please understand the important principles and models of natural language processing, and be familiar with the use of Hugging Face-transformers, a Python library for natural language processing. Please pay particular attention to the following key points:

• Common pre-trained models (BERT, RoBERTa, GPT)
• Fine-tune method

If you have never touched NLP#

We recommend you refer to the following tutorials:

• Chinese video course: NLP course with Shusen Wang from SIT
• English document course:Lena Voita’s NLP Tutorial
• (Optional) English video course: CMU’s Neural Networks for NLP（more details and requires a certain foundation, recommended for students who have enough energy to learn）

Transformer and pre-trained model related resources#

• Attention Is All You Need essay analysis
• Diagram Transformer and important pre-trained models
  • Transformer
  • BERT
  • GPT-2
• Review of pre-trained models and original English papers
• Pre-training model Pytorch implementation and Demo
• Collection of pre-trained model papers

Hugging Face Related Tutorials#

• What is Hugging Face?

Hugging Face provides a Python-based library Transformers for NLP practitioners. Transformers provides an API that can use many Transformer models, such as BERT, RoBERTa, GPT-2, etc. These models are able to achieve SOTA (state-of-the-art) results in various NLP downstream tasks.

Getting Started with Hugging Face#

• Chinese documentation: Hugging Face beginner's guide mainly for users in mainland China
• Official documentation in English: Using Hugging Face Transformers Library with Pytorch

Detailed explanation of the main functions of Hugging Face-Transformers#

• Text Preprocessing (Tokenizer, Padding, Truncation)
• Training and Fine-tuning
• Hugging Face Transformers Official Demo Notebooks
• Hugging Face Transformers Official Documentation