Common challenges in developing ML applications

Companies typically run into common kinds of challenges when they embark on an AI/ML development journey, and it is often a key requirement of an architect’s role to understand common challenges in a given problem space. As an architect, if you are not aware of challenges and how to address them, it’s unlikely that you will design an appropriate solution. In this section, we introduce the most frequently encountered challenges and pitfalls at a high level, and in later sections of this book, we discuss ways to address or alleviate some of these hurdles of AI/ML development.

Gathering, processing, and labeling data

Data is the key ingredient in ML because, in general, ML models cannot function without data. There’s an often-quoted adage that data scientists spend up to 80% of their time working on finding, cleaning, and processing data before they can begin to make use of it for analytical or data science purposes...