YUNOVA · Natural Language Processing
Module 9: Advanced transformers
Fine-tuning BERT for text classification
Key points
- A classification head is added on top of a pre-trained BERT model.
- Text is tokenised into input IDs and attention masks before training.
- Fine-tuning trains the whole model for a few epochs with a small learning rate.
Hugging Face Transformers pipelines
Key points
- pipeline() wraps a model and its tokenizer for one task in a single call.
- Ready tasks include sentiment-analysis, ner, summarization and translation.
- You can pass any model name from the Hugging Face Hub.
Sequence-to-sequence models: T5 and BART
Key points
- Seq2seq models have an encoder that reads input and a decoder that writes output.
- T5 frames every task as text-to-text, using a prefix like "summarize:".
- BART is pre-trained by reconstructing corrupted text.
Named entity recognition with transformers
Key points
- NER labels words as entities such as person, organisation or location.
- It is token classification, often with BIO tags.
- Sub-word tokens must be aligned back to the original words.