In this chapter, we will cover the fundamental techniques of machine learning. We will use these throughout the book to solve interesting cybersecurity problems. We will cover both foundational algorithms, such as clustering and gradient boosting trees, and solutions to common data challenges, such as imbalanced data and false-positive constraints. A machine learning practitioner in cybersecurity is in a unique and exciting position to leverage enormous amounts of data and create solutions in a constantly evolving landscape.
This chapter covers the following recipes:
- Train-test-splitting your data
- Standardizing your data
- Summarizing large data using principal component analysis (PCA)
- Generating text using Markov chains
- Performing clustering using scikit-learn
- Training an XGBoost classifier
- Analyzing time series using statsmodels
- Anomaly detection using Isolation Forest
- Natural language processing (NLP) using hashing vectorizer and tf-idf with scikit-learn
- Hyperparameter tuning with scikit-optimize
