Foundation models
A foundation model1 is an AI model that is trained on a large amount of unlabeled data using self-supervised learning techniques, resulting in a highly adaptable model that can be fine-tuned to a variety of downstream tasks. Since their introduction in 2018, foundation models have revolutionized the field of AI and opened up new possibilities for natural language processing (NLP), image recognition, and other applications. Early examples of foundation models were large pre-trained language models, such as BERT and GPT-3, which demonstrated the power of unsupervised learning and transfer learning.
Domain-specific models based on different kinds of tokens have also been developed, including medical codes. Multimodal foundation models such as DALL-E, Flamingo, Florence, and NOOR have also been produced. The term “foundation model”2 was popularized by Stanford University’s Human-Centered Artificial Intelligence’s (HAI) Center for Research...
