Book Image

Python Data Analysis - Third Edition

By : Avinash Navlani, Ivan Idris
5 (1)
Book Image

Python Data Analysis - Third Edition

5 (1)
By: Avinash Navlani, Ivan Idris

Overview of this book

Data analysis enables you to generate value from small and big data by discovering new patterns and trends, and Python is one of the most popular tools for analyzing a wide variety of data. With this book, you’ll get up and running using Python for data analysis by exploring the different phases and methodologies used in data analysis and learning how to use modern libraries from the Python ecosystem to create efficient data pipelines. Starting with the essential statistical and data analysis fundamentals using Python, you’ll perform complex data analysis and modeling, data manipulation, data cleaning, and data visualization using easy-to-follow examples. You’ll then understand how to conduct time series analysis and signal processing using ARMA models. As you advance, you’ll get to grips with smart processing and data analytics using machine learning algorithms such as regression, classification, Principal Component Analysis (PCA), and clustering. In the concluding chapters, you’ll work on real-world examples to analyze textual and image data using natural language processing (NLP) and image analytics techniques, respectively. Finally, the book will demonstrate parallel computing using Dask. By the end of this data analysis book, you’ll be equipped with the skills you need to prepare data for analysis and create meaningful data visualizations for forecasting values from data.

Related Content you might be interested in

Current Title:

Small book image
Python Data Analysis - Third Edition
Avinash Navlani | Ivan Idris
Feb, 2021 | 478 pages
Small book image
Python Data Analysis
Ivan Idris
Mar, 2017 | 330 pages
Small book image
Hands-On Data Visualization with Bokeh
Kevin Jolly
Jun, 2018 | 174 pages
Small book image
Building Statistical Models in Python
Paul N Adams | Stuart J Miller | Huy Hoang Nguyen
Aug, 2023 | 420 pages
Table of Contents (20 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
1
Section 1: Foundation for Data Analysis
Section 1: Foundation for Data Analysis
Free Chapter
2
Getting Started with Python Libraries
Getting Started with Python Libraries
Understanding data analysis
The standard process of data analysis
The KDD process
SEMMA
CRISP-DM
Comparing data analysis and data science
The skillsets of data analysts and data scientists
Installing Python 3
Software used in this book
Using IPython as a shell
Using JupyterLab
Using Jupyter Notebooks
Advanced features of Jupyter Notebooks
Summary
3
NumPy and pandas
NumPy and pandas
Technical requirements
Understanding NumPy arrays
NumPy array numerical data types
Manipulating array shapes
The stacking of NumPy arrays
Partitioning NumPy arrays
Changing the data type of NumPy arrays
Creating NumPy views and copies
Slicing NumPy arrays
Boolean and fancy indexing
Broadcasting arrays
Creating pandas DataFrames
Understanding pandas Series
Reading and querying the Quandl data
Describing pandas DataFrames
Grouping and joining pandas DataFrame
Working with missing values
Creating pivot tables
Dealing with dates
Summary
References
4
Statistics
Statistics
Technical requirements
Understanding attributes and their types
Measuring central tendency
Measuring dispersion
Skewness and kurtosis
Understanding relationships using covariance and correlation coefficients
Central limit theorem
Collecting samples
Performing parametric tests
Performing non-parametric tests
Summary
5
Linear Algebra
Linear Algebra
Technical requirements
Fitting to polynomials with NumPy
Determinant
Finding the rank of a matrix
Matrix inverse using NumPy
Solving linear equations using NumPy
Decomposing a matrix using SVD
Eigenvectors and Eigenvalues using NumPy
Generating random numbers
Binomial distribution
Normal distribution
Testing normality of data using SciPy
Creating a masked array using the numpy.ma subpackage
Summary
6
Section 2: Exploratory Data Analysis and Data Cleaning
Section 2: Exploratory Data Analysis and Data Cleaning
7
Data Visualization
Data Visualization
Technical requirements
Visualization using Matplotlib
Advanced visualization using the Seaborn package
Interactive visualization with Bokeh
Summary
8
Retrieving, Processing, and Storing Data
Retrieving, Processing, and Storing Data
Technical requirements
Reading and writing CSV files with NumPy
Reading and writing CSV files with pandas
Reading and writing data from Excel
Reading and writing data from JSON
Reading and writing data from HDF5
Reading and writing data from HTML tables
Reading and writing data from Parquet
Reading and writing data from a pickle pandas object
Lightweight access with sqllite3
Reading and writing data from MySQL
Reading and writing data from MongoDB
Reading and writing data from Cassandra
Reading and writing data from Redis
PonyORM
Summary
9
Cleaning Messy Data
Cleaning Messy Data
Technical requirements
Exploring data
Filtering data to weed out the noise
Handling missing values
Handling outliers
Feature encoding techniques
Feature scaling
Feature transformation
Feature splitting
Summary
10
Signal Processing and Time Series
Signal Processing and Time Series
Technical requirements
The statsmodels modules
Moving averages
Window functions
Defining cointegration
STL decomposition
Autocorrelation
Autoregressive models
ARMA models
Generating periodic signals
Fourier analysis
Spectral analysis filtering
Summary
11
Section 3: Deep Dive into Machine Learning
Section 3: Deep Dive into Machine Learning
12
Supervised Learning - Regression Analysis
Supervised Learning - Regression Analysis
Technical requirements
Linear regression
Understanding multicollinearity
Dummy variables
Developing a linear regression model
Evaluating regression model performance
Fitting polynomial regression
Regression models for classification
Logistic regression
Implementing logistic regression using scikit-learn
Summary
13
Supervised Learning - Classification Techniques
Supervised Learning - Classification Techniques
Technical requirements
Classification
Naive Bayes classification
Decision tree classification
KNN classification
SVM classification
Splitting training and testing sets
Evaluating the classification model performance
ROC curve and AUC
Summary
14
Unsupervised Learning - PCA and Clustering
Unsupervised Learning - PCA and Clustering
Technical requirements
Unsupervised learning
Reducing the dimensionality of data
Clustering
Partitioning data using k-means clustering
Hierarchical clustering
DBSCAN clustering
Spectral clustering
Evaluating clustering performance
Summary
15
Section 4: NLP, Image Analytics, and Parallel Computing
Section 4: NLP, Image Analytics, and Parallel Computing
16
Analyzing Textual Data
Analyzing Textual Data
Technical requirements
Installing NLTK and SpaCy
Text normalization
Tokenization
Removing stopwords
Stemming and lemmatization
POS tagging
Recognizing entities
Dependency parsing
Creating a word cloud
Bag of Words
TF-IDF
Sentiment analysis using text classification
Text similarity
Summary
17
Analyzing Image Data
Analyzing Image Data
Technical requirements
Installing OpenCV
Understanding image data
Color models
Drawing on images
Writing on images
Resizing images
Flipping images
Changing the brightness
Blurring an image
Face detection
Summary
18
Parallel Computing Using Dask
Parallel Computing Using Dask
Parallel computing using Dask
Dask data types
Dask Delayed
Preprocessing data at scale
Machine learning at scale
Summary
19
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Reading and writing data from MySQL

MySQL is a fast, open-source, and easy-to-use relational or tabular database. It is suitable for small and large business applications. It is very friendly with database-driven web development applications. There are lots of ways to access data in Python from MySQL. Connectors such as MySQLdb, mysqlconnector, and pymysql are available for MySQL database connectivity. For this connectivity purpose, you should install a MySQL relational database and the mysql-python connector. The MySQL setup details are available on its website: https://www.mysql.com/downloads/.

You can use the pymysql connector as the client library and it can be installed using pip:

pip install pymysql

We can establish a connection with the following steps:

  1. Import the library.
  2. Create a database connection.
  3. Create a cursor object.
  4. Execute the SQL query.
  5. Fetch the records or response for the update or insert the record.
  6. Close the connection.

In our examples, we are trying database...

Previous Section
End of Section 12
Next Section