Book Image

Python Data Analysis - Third Edition

By : Avinash Navlani, Ivan Idris
5 (1)
Book Image

Python Data Analysis - Third Edition

5 (1)
By: Avinash Navlani, Ivan Idris

Overview of this book

Data analysis enables you to generate value from small and big data by discovering new patterns and trends, and Python is one of the most popular tools for analyzing a wide variety of data. With this book, you’ll get up and running using Python for data analysis by exploring the different phases and methodologies used in data analysis and learning how to use modern libraries from the Python ecosystem to create efficient data pipelines. Starting with the essential statistical and data analysis fundamentals using Python, you’ll perform complex data analysis and modeling, data manipulation, data cleaning, and data visualization using easy-to-follow examples. You’ll then understand how to conduct time series analysis and signal processing using ARMA models. As you advance, you’ll get to grips with smart processing and data analytics using machine learning algorithms such as regression, classification, Principal Component Analysis (PCA), and clustering. In the concluding chapters, you’ll work on real-world examples to analyze textual and image data using natural language processing (NLP) and image analytics techniques, respectively. Finally, the book will demonstrate parallel computing using Dask. By the end of this data analysis book, you’ll be equipped with the skills you need to prepare data for analysis and create meaningful data visualizations for forecasting values from data.

Related Content you might be interested in

Current Title:

Small book image
Python Data Analysis - Third Edition
Avinash Navlani | Ivan Idris
Feb, 2021 | 478 pages
Small book image
Python Data Analysis
Ivan Idris
Mar, 2017 | 330 pages
Small book image
Hands-On Data Visualization with Bokeh
Kevin Jolly
Jun, 2018 | 174 pages
Small book image
Building Statistical Models in Python
Paul N Adams | Stuart J Miller | Huy Hoang Nguyen
Aug, 2023 | 420 pages
Table of Contents (20 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
1
Section 1: Foundation for Data Analysis
Section 1: Foundation for Data Analysis
Free Chapter
2
Getting Started with Python Libraries
Getting Started with Python Libraries
Understanding data analysis
The standard process of data analysis
The KDD process
SEMMA
CRISP-DM
Comparing data analysis and data science
The skillsets of data analysts and data scientists
Installing Python 3
Software used in this book
Using IPython as a shell
Using JupyterLab
Using Jupyter Notebooks
Advanced features of Jupyter Notebooks
Summary
3
NumPy and pandas
NumPy and pandas
Technical requirements
Understanding NumPy arrays
NumPy array numerical data types
Manipulating array shapes
The stacking of NumPy arrays
Partitioning NumPy arrays
Changing the data type of NumPy arrays
Creating NumPy views and copies
Slicing NumPy arrays
Boolean and fancy indexing
Broadcasting arrays
Creating pandas DataFrames
Understanding pandas Series
Reading and querying the Quandl data
Describing pandas DataFrames
Grouping and joining pandas DataFrame
Working with missing values
Creating pivot tables
Dealing with dates
Summary
References
4
Statistics
Statistics
Technical requirements
Understanding attributes and their types
Measuring central tendency
Measuring dispersion
Skewness and kurtosis
Understanding relationships using covariance and correlation coefficients
Central limit theorem
Collecting samples
Performing parametric tests
Performing non-parametric tests
Summary
5
Linear Algebra
Linear Algebra
Technical requirements
Fitting to polynomials with NumPy
Determinant
Finding the rank of a matrix
Matrix inverse using NumPy
Solving linear equations using NumPy
Decomposing a matrix using SVD
Eigenvectors and Eigenvalues using NumPy
Generating random numbers
Binomial distribution
Normal distribution
Testing normality of data using SciPy
Creating a masked array using the numpy.ma subpackage
Summary
6
Section 2: Exploratory Data Analysis and Data Cleaning
Section 2: Exploratory Data Analysis and Data Cleaning
7
Data Visualization
Data Visualization
Technical requirements
Visualization using Matplotlib
Advanced visualization using the Seaborn package
Interactive visualization with Bokeh
Summary
8
Retrieving, Processing, and Storing Data
Retrieving, Processing, and Storing Data
Technical requirements
Reading and writing CSV files with NumPy
Reading and writing CSV files with pandas
Reading and writing data from Excel
Reading and writing data from JSON
Reading and writing data from HDF5
Reading and writing data from HTML tables
Reading and writing data from Parquet
Reading and writing data from a pickle pandas object
Lightweight access with sqllite3
Reading and writing data from MySQL
Reading and writing data from MongoDB
Reading and writing data from Cassandra
Reading and writing data from Redis
PonyORM
Summary
9
Cleaning Messy Data
Cleaning Messy Data
Technical requirements
Exploring data
Filtering data to weed out the noise
Handling missing values
Handling outliers
Feature encoding techniques
Feature scaling
Feature transformation
Feature splitting
Summary
10
Signal Processing and Time Series
Signal Processing and Time Series
Technical requirements
The statsmodels modules
Moving averages
Window functions
Defining cointegration
STL decomposition
Autocorrelation
Autoregressive models
ARMA models
Generating periodic signals
Fourier analysis
Spectral analysis filtering
Summary
11
Section 3: Deep Dive into Machine Learning
Section 3: Deep Dive into Machine Learning
12
Supervised Learning - Regression Analysis
Supervised Learning - Regression Analysis
Technical requirements
Linear regression
Understanding multicollinearity
Dummy variables
Developing a linear regression model
Evaluating regression model performance
Fitting polynomial regression
Regression models for classification
Logistic regression
Implementing logistic regression using scikit-learn
Summary
13
Supervised Learning - Classification Techniques
Supervised Learning - Classification Techniques
Technical requirements
Classification
Naive Bayes classification
Decision tree classification
KNN classification
SVM classification
Splitting training and testing sets
Evaluating the classification model performance
ROC curve and AUC
Summary
14
Unsupervised Learning - PCA and Clustering
Unsupervised Learning - PCA and Clustering
Technical requirements
Unsupervised learning
Reducing the dimensionality of data
Clustering
Partitioning data using k-means clustering
Hierarchical clustering
DBSCAN clustering
Spectral clustering
Evaluating clustering performance
Summary
15
Section 4: NLP, Image Analytics, and Parallel Computing
Section 4: NLP, Image Analytics, and Parallel Computing
16
Analyzing Textual Data
Analyzing Textual Data
Technical requirements
Installing NLTK and SpaCy
Text normalization
Tokenization
Removing stopwords
Stemming and lemmatization
POS tagging
Recognizing entities
Dependency parsing
Creating a word cloud
Bag of Words
TF-IDF
Sentiment analysis using text classification
Text similarity
Summary
17
Analyzing Image Data
Analyzing Image Data
Technical requirements
Installing OpenCV
Understanding image data
Color models
Drawing on images
Writing on images
Resizing images
Flipping images
Changing the brightness
Blurring an image
Face detection
Summary
18
Parallel Computing Using Dask
Parallel Computing Using Dask
Parallel computing using Dask
Dask data types
Dask Delayed
Preprocessing data at scale
Machine learning at scale
Summary
19
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Installing Python 3

The installer file for installing Python 3 can easily be downloaded from the official website (https://www.python.org/downloads/) for Windows, Linux, and Mac 32-bit or 64-bit systems. The installer can be installed by double-clicking on it. This installer also has an IDE named "IDLE" that can be used for development. We will dive deeper into each of the operating systems in the next few sections.

Python installation and setup on Windows

This book is based on the latest Python 3 version. All the code that will be used in this book is written in Python 3, so we need to install Python 3 before we can start coding. Python is an open source, distributed, and freely available language. It is also licensed for commercial use. There are many implementations of Python, including commercial implementations and distributions. In this book, we will focus on the standard Python implementation, which is guaranteed to be compatible with NumPy.

You can download Python 3.9.x from the Python official website: https://www.python.org/downloads/. Here, you can find installation files for Windows, Linux, Mac OS X, and other OS platforms. You can find instructions for installing and using Python for various operating systems at https://docs.python.org/3.7/using/index.html.

You need to have Python 3.5.x or above installed on your system. The sunset date for Python 2.7 was moved from 2015 to 2020, but at the time of writing, Python 2.7 will not be supported and maintained by the Python community.

At the time of writing this book, we had Python 3.8.3 installed as a prerequisite on our Windows 10 virtual machine: https://www.python.org/ftp/python/3.8.3/python-3.8.3.exe.

Python installation and setup on Linux

Installing Python on Linux is significantly easier compared to the other OSes. To install the foundational libraries, run the following command-line instruction:

$ pip3 install numpy scipy pandas matplotlib jupyter notebook

It may be essential to run the sudo command before the preceding command if you don't have sufficient rights on the machine that you are using.

Python installation and setup on Mac OS X with a GUI installer

Python can be installed via the installation file from the Python official website. The installer file can be downloaded from its official web page (https://www.python.org/downloads/mac-osx/) for macOS. This installer also has an IDE named "IDLE" that can be used for development.

Python installation and setup on Mac OS X with brew

For Mac systems, you can use the Homebrew package manager to install Python. It will make it easier to install the required applications for developers, researchers, and scientists. The brew install command is used to install another application, such as installing python3 or any other Python package, such as NLTK or SpaCy.

To install the most recent version of Python, you need to execute the following command in a Terminal:

$ brew install python3

After installation, you can confirm the version of Python you've installed by running the following command:

$ python3 --version
Python 3.7.4

You can also open the Python Shell from the command line by running the following command:

$ python3

Now that we know how to install Python on our system, let's dive into the actual tools that we will need to start data analysis.

Previous Section
End of Section 9
Next Section