Chapter 4, BERT Variants I – ALBERT, RoBERTa, ELECTRA, SpanBERT

1. In the next-sentence prediction task, we train the model to predict whether a sentence pair belongs to the isNext or notNext class, whereas in the sentence order prediction task, we train the model to predict whether a sentence order in a given sentence pair is swapped or not.
2. ALBERT uses the following two techniques to reduce the number of parameters: cross-layer parameter sharing and factorized embedding layer parameterization.
3. In cross-layer parameter sharing, instead of learning the parameters of all the encoder layers, we only learn the parameters of the first encoder layer, and then we just share the parameters of the first encoder layer with all the other encoder layers.
4. In a shared feedforward network, we only share the parameters of the feedforward network of the first encoder layer with the feedforward networks of other encoder layers. In shared attention, we only share the parameters of the multi-head...