State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks. Generally, such models are often either cross-modal (contrastive) or multi-modal (with earlier fusion) but not both; and they often only target specific modalities or tasks. A promising direction would be to use a single holistic universal model, as a "foundation", that targets all modalities at once -- a true vision and language foundation model should be good at vision tasks, language tasks, and cross- and multi-modal vision and language tasks. We introduce FLAVA as such a model and demonstrate impressive performance on a wide range of 35 tasks spanning these target modalities.

FLAVA, a language and vision alignment model learns strong representations from multimodal (imagetext pairs) and unimodal data (unpaired images and text) and can be applied to target a broad scope of tasks from three domains (visual recognition, language understanding, and multimodal reasoning) under a common transformer model architecture.

Source code is available at this link and pretrained models can also be find on Hugging Face. Follow Amanpreet on Twitter for latest updates.

@inproceedings{singh2022flava,
  author = {
    Amanpreet Singh and
    Ronghang Hu and
    Vedanuj Goswami and
    Guillaume Couairon and
    Wojciech Galuba and
    Marcus Rohrbach and
    Douwe Kiela
  },
  title = {
    {FLAVA:} {A} Foundational Language And 
    Vision Alignment Model
  },
  booktitle={CVPR},
  year={2022}
}