YUNOVA · Natural Language Processing
Module 11: Search and generation
Semantic search with sentence transformers
Key points
- Sentence transformers turn text into vectors that capture meaning.
- Similar texts get vectors that are close, often measured by cosine similarity.
- Search embeds the query and finds the nearest document vectors.
Vector databases for NLP
Key points
- A vector database stores embeddings and finds the nearest ones fast.
- Approximate nearest neighbour indexes such as HNSW trade a little accuracy for speed.
- Examples include FAISS, Pinecone, Weaviate and pgvector.
Decoding strategies: beam search, temperature and top-p
Key points
- Greedy decoding always picks the most likely next token.
- Beam search keeps several candidate sequences at once.
- Temperature and top-p sampling control how random the output is.
Evaluating generated text: BLEU and ROUGE
Key points
- BLEU measures n-gram precision against reference texts and is common for translation.
- ROUGE measures n-gram recall and is common for summarisation.
- Both miss meaning, so human review is still needed.