Embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering. Recently, there has been a surge of interest in developing universal text embedding models that can generalize across tasks (e.g., MTEB). However, progress in learning universal multimodal embedding models has been relatively slow despite their importance. In this work, we aim to explore the potential for building universal embeddings capable of handling a wide range of downstream tasks. Our contributions are twofold: (1) MMEB (Massive Multimodal Embedding Benchmark), which covers 4 meta-tasks including classification, question answering, retrieval, and visual grounding and 36 datasets, including 20 training and 16 evaluation datasets, and (2) VLM2Vec (Vision-Language Model -> Vector), a contrastive training framework that converts any state-of-the-art vision-language model into an embedding model via training on MMEB. Unlike previous models such as CLIP and BLIP, VLM2Vec can process any combination of images and text to generate a fixed-dimensional vector based on task instructions. We build a series of VLM2Vec models on SoTA VLMs like Phi-3.5-V, LLaVA-1.6 and evaluate them on MMEB's evaluation split. Our results show that VLM2Vec achieves an absolute average improvement of 10% to 20% over existing multimodal embedding models on both in-distribution and out-of-distribution datasets in MMEB. We show that VLMs are secretly strong embedding models.

We propose VLM2Vec framework to learn a single multimodal embedding model that can encode a series of images and text for any downstream task. Unlike traditional CLIP or BLIP embeddings, VLM2Vec can handle images with any resolution and text with any length. It can also follow instruction to produce instruction-guided representation, which fits the downstream tasks better than other task-agnostic multimodal emebddings.

The model was trained with contrastive learning on a massive amount of examples we compiled from 36 datatsets spanning 4 tasks. We name this benchmark as MMEB, which has the train and eval splits separately. We hold out 15 datasets for out-of-distribution evaluation.

We evaluated a wide range of multimodal embeddings on MMEB benchmarks. We show our results below. VLM2Vec outperforms all the baselines by a huge margin. The improvement on out-of-distribution evaluation demonstrates the generalization capability of VLM2Vec framework.

We show the detailed breakdown evaluation results for VLM2Vec Llava1.6 Model as follows:

We ablate different factors like batch size, step size, and image resolution to understand their impact on the final results.