Implementation of Megatron-LM
In this section, we will briefly discuss how to use Megatron-LM. For more details, you can read the official user manual of Megatron-LM here: https://github.com/NVIDIA/Megatron-LM. Here are the steps we need to follow:
- To use Megatron-LM, we first need to install some pre-trained checkpoints, as follows:
# Download checkpoints # Terminal wget --content-disposition \ Â Â Â Â models/nvidia/megatron_lm_345m/ \ Â Â Â Â versions/ \ Â Â Â Â v0.1/zip \ Â Â Â Â -0 \ Â Â Â Â megatron_lm_345m_v0.1.zip
- After that, you can use
megatron-lm
to preprocess the data, as follows:# pre-process data python3 preprocess_data.py \ Â Â Â Â Â Â Â Â --input xxx.json \ Â Â Â Â Â Â Â Â --output-prefix my-model \ Â Â Â Â Â Â Â Â --vocab bert-vercab.txt \ Â Â Â Â Â Â Â ...