Chapter 2, Understanding the BERT Model

1. BERT stands for Bidirectional Encoder Representation from Transformer. It is a state-of-the-art embedding model published by Google. BERT is a context-based embedding model, unlike other popular embedding models such as word2vec, which are context-free.
2. The BERT-base model consists of , , , and 110 million parameters, and the BERT-large model consists of , , , and 340 million parameters.
3. The segment embedding is used to distinguish between the two given sentences. The segment embedding layer returns only either of the two embeddings and as the output. That is, if the input token belongs to sentence A, then the token will be mapped to the embedding , and if the token belongs to sentence B, then it will be mapped to the embedding .
4. BERT is pre-trained using two tasks, namely masked language modeling and next-sentence prediction.
5. In the masked language modeling task, in a given input sentence, we randomly mask 15% of the words and train the network...