LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

TL;DR

LLM2Vec is an unsupervised method that converts decoder-only LLMs into powerful text encoders, outperforming existing encoder-only models.

Problem

Decoder-only LLMs excel at generation but are not suitable for text embedding tasks. Existing methods rely on encoder-decoder architectures or large-scale synthetic data; LLM2Vec aims to utilize decoder-only models for embeddings without such costs.

Approach

Three steps: (1) enabling bidirectional attention, (2) masked next token prediction, (3) unsupervised contrastive learning. This encourages the LLM to understand context bidirectionally and learn sentence-level representations.

Results & Contribution

Applied to 4 LLMs ranging from 1.3B to 8B parameters, it significantly outperforms encoder-only models on word-level tasks and achieves state-of-the-art unsupervised performance on the MTEB benchmark. Combined with supervised contrastive learning using only publicly available data, it achieves state-of-the-art results. The approach is parameter-efficient and does not require GPT-4 generated data.