The rise of small language models (SLMs)

Following the popularity of LLMs, we have seen a rise in SLMs. Researchers began exploring SLMs as a response to the challenges posed by their larger counterparts. While large models offer impressive performance, they also bring substantial demands in terms of computational resources, energy consumption, and data requirements. These factors limit accessibility and practicality, especially for individuals and organizations with constrained resources.

The architecture of SLMs is fundamentally similar to that of LLMs, with both based on the transformer architecture (for example, Llama). The differences mainly lie in the scale and some specific optimizations tailored to their respective use cases. Language models in the range of millions and the order of 10 billion parameters or less are considered to be SLMs. They are streamlined versions of language models that are designed to deliver a balance between performance and efficiency. Unlike their...