Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Free Learning
Arrow right icon
Hands-On Data Analysis with Pandas
Hands-On Data Analysis with Pandas

Hands-On Data Analysis with Pandas: Efficiently perform data collection, wrangling, analysis, and visualization using Python

Arrow left icon
Profile Icon Stefanie Molin
Arrow right icon
R$50 per month
Full star icon Full star icon Full star icon Full star icon Half star icon 4.7 (11 Ratings)
Paperback Jul 2019 740 pages 1st Edition
eBook
R$80 R$218.99
Paperback
R$272.99
Subscription
Free Trial
Renews at R$50p/m
Arrow left icon
Profile Icon Stefanie Molin
Arrow right icon
R$50 per month
Full star icon Full star icon Full star icon Full star icon Half star icon 4.7 (11 Ratings)
Paperback Jul 2019 740 pages 1st Edition
eBook
R$80 R$218.99
Paperback
R$272.99
Subscription
Free Trial
Renews at R$50p/m
eBook
R$80 R$218.99
Paperback
R$272.99
Subscription
Free Trial
Renews at R$50p/m

What do you get with a Packt Subscription?

Free for first 7 days. $19.99 p/m after that. Cancel any time!
Product feature icon Unlimited ad-free access to the largest independent learning library in tech. Access this title and thousands more!
Product feature icon 50+ new titles added per month, including many first-to-market concepts and exclusive early access to books as they are being written.
Product feature icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Product feature icon Thousands of reference materials covering every tech concept you need to stay up to date.
Subscribe now
View plans & pricing
Table of content icon View table of contents Preview book icon Preview Book

Hands-On Data Analysis with Pandas

Introduction to Data Analysis

Before we can begin our hands-on introduction to data analysis with pandas, we need to learn about the fundamentals of data analysis. Those who have ever looked at the documentation for a software library know how overwhelming it can be if you have no clue what you are looking for. Therefore, it is essential that we not only master the coding aspect, but also the thought process and workflow required to analyze data, which will prove the most useful in augmenting our skill set in the future.

Much like the scientific method, data science has some common workflows that we can follow when we want to conduct an analysis and present the results. The backbone of this process is statistics, which gives us ways to describe our data, make predictions, and also draw conclusions about it. Since prior knowledge of statistics is not a prerequisite, this chapter...

Chapter materials

All the files for this book are on GitHub at https://github.com/stefmolin/Hands-On-Data-Analysis-with-Pandas. While having a GitHub account isn't necessary to work through this book, it is a good idea to create one, as it will serve as a portfolio for any data/coding projects. In addition, working with Git will provide a version control system and make collaboration easy.

In order to get a local copy of the files, we have a few options (ordered from least useful to most useful):

  • Download the ZIP file and extract the files locally
  • Clone the repository without forking it
  • Fork the repository and then clone it

This book includes exercises for every chapter; therefore, for those who want to keep a copy of their solutions...

Fundamentals of data analysis

Data analysis is a highly iterative process involving collection, preparation (wrangling), exploratory data analysis (EDA), and drawing conclusions. During an analysis, we will frequently revisit each of these steps. The following diagram depicts a generalized workflow:

In practice, this process is heavily skewed towards the data preparation side. Surveys have found that, although data scientists enjoy the data preparation side of their job the least, it makes up 80% of their work (https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#419ce7b36f63). This data preparation step is where pandas really shines.

Data collection

...

Statistical foundations

When we want to make observations about the data we are analyzing, we are often, if not always, turning to statistics in some fashion. The data we have is referred to as the sample, which was observed from (and is a subset of) the population. Two broad categories of statistics are descriptive and inferential statistics. With descriptive statistics, as the name implies, we are looking to describe the sample. Inferential statistics involves using the sample statistics to infer, or deduce, something about the population, such as the underlying distribution.

The sample statistics are used as estimators of the population parameters, meaning that we have to quantify their bias and variance. There are a multitude of methods for this; some will make assumptions on the shape of the distribution (parametric) and others won't (non-parametric). This is all well...

Setting up a virtual environment

This book was written using Python 3.6.4, but the code should work for Python 3.6+, which is available on all major operating systems. In this section, we will go over how to set up the virtual environment in order to follow along with this book. If Python isn't already installed on your computer, read through the following sections on virtual environments first, and then decide whether to install Anaconda, since it will also install Python. To install Python without Anaconda, download it here: https://www.python.org/downloads/. Then, continue with the section on venv.

To check if Python is already installed, run where python3 from the command line on Windows or which python3 from the command line on Linux/macOS. If this returns nothing, try running it with just python (instead of python3). If Python is installed, check the version by running...

Summary

In this chapter, we learned about the main processes in conducting data analysis: data collection, wrangling, EDA, and drawing conclusions. We followed that up with an overview of descriptive statistics and learned how to describe the central tendency and spread of our data; how to summarize it both numerically and visually using the 5-number summary, box plots, histograms, and kernel density estimates; how to scale our data; and how to quantify relationships between variables in our dataset.

We got an introduction to prediction and time series analysis. Then, we had a very brief overview of some core topics in inferential statistics that can be explored after mastering the contents of this book. Note that, while all the examples in this chapter were of one or two variables, real-life data is often high-dimensional. Chapter 10, Making Better Predictions – Optimizing...

Exercises

Run through the introduction_to_data_analysis.ipynb notebook for a review of this chapter's content, and then complete the following exercises to practice working with JupyterLab and calculating summary statistics in Python:

  1. Explore the JupyterLab interface and look at some of the shortcuts that are available. Don't worry about memorizing them for now (eventually, they will become second nature and save you a lot of time)—just get comfortable using Jupyter Notebooks.
  2. Is all data normally distributed? Explain why or why not.
  1. When would it make more sense to use the median instead of the mean for the measure of center?
  2. Run the code in the first cell of the exercises.ipynb notebook. It will give you a list of 100 values to work with for the rest of the exercises in this chapter.
  3. Using the data from exercise #4, calculate the following statistics without...

Further reading

The following are some resources that you can use to become more familiar with Jupyter:

The following resource shows you how to use conda to manage virtual environments instead of the venv solution that was explained earlier in this chapter:

Left arrow icon Right arrow icon

Key benefits

  • Perform efficient data analysis and manipulation tasks using pandas
  • Apply pandas to different real-world domains with the help of step-by-step demonstrations
  • Get accustomed to using pandas as an effective data exploration tool.

Description

Data analysis has become a necessary skill in a variety of domains where knowing how to work with data and extract insights can generate significant value. Hands-On Data Analysis with Pandas will show you how to analyze your data, get started with machine learning, and work effectively with Python libraries often used for data science, such as pandas, NumPy, matplotlib, seaborn, and scikit-learn. Using real-world datasets, you will learn how to use the powerful pandas library to perform data wrangling to reshape, clean, and aggregate your data. Then, you will be able to conduct exploratory data analysis by calculating summary statistics and visualizing the data to find patterns. In the concluding chapters, you will explore some applications of anomaly detection, regression, clustering, and classification using scikit-learn to make predictions based on past data. By the end of this book, you will be equipped with the skills you need to use pandas to ensure the veracity of your data, visualize it for effective decision-making, and reliably reproduce analyses across multiple datasets.

Who is this book for?

This book is for data analysts, data science beginners, and Python developers who want to explore each stage of data analysis and scientific computing using a wide range of datasets. You will also find this book useful if you are a data scientist looking to implement pandas in machine learning. Working knowledge of Python programming language will be beneficial.

What you will learn

  • Understand how data analysts and scientists gather and analyze data
  • Perform data analysis and data wrangling using Python
  • Combine, group, and aggregate data from multiple sources
  • Create data visualizations with pandas, matplotlib, and seaborn
  • Apply machine learning (ML) algorithms to identify patterns and make predictions
  • Use Python data science libraries to analyze real-world datasets
  • Use pandas to solve common data representation and analysis problems
  • Build Python scripts, modules, and packages for reusable analysis code

Product Details

Country selected
Publication date, Length, Edition, Language, ISBN-13
Publication date : Jul 26, 2019
Length: 740 pages
Edition : 1st
Language : English
ISBN-13 : 9781789615326
Category :
Languages :
Concepts :
Tools :

What do you get with a Packt Subscription?

Free for first 7 days. $19.99 p/m after that. Cancel any time!
Product feature icon Unlimited ad-free access to the largest independent learning library in tech. Access this title and thousands more!
Product feature icon 50+ new titles added per month, including many first-to-market concepts and exclusive early access to books as they are being written.
Product feature icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Product feature icon Thousands of reference materials covering every tech concept you need to stay up to date.
Subscribe now
View plans & pricing

Product Details

Publication date : Jul 26, 2019
Length: 740 pages
Edition : 1st
Language : English
ISBN-13 : 9781789615326
Category :
Languages :
Concepts :
Tools :

Packt Subscriptions

See our plans and pricing
Modal Close icon
R$50 billed monthly
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Simple pricing, no contract
R$500 billed annually
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just R$25 each
Feature tick icon Exclusive print discounts
R$800 billed in 18 months
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just R$25 each
Feature tick icon Exclusive print discounts

Frequently bought together


Stars icon
Total R$240.00 R$683.97 R$443.97 saved
Mastering pandas
R$272.99
Python Machine Learning
R$306.99
Hands-On Data Analysis with Pandas
R$272.99
Total R$240.00R$683.97 R$443.97 saved Stars icon
Banner background image

Table of Contents

19 Chapters
Section 1: Getting Started with Pandas Chevron down icon Chevron up icon
Introduction to Data Analysis Chevron down icon Chevron up icon
Working with Pandas DataFrames Chevron down icon Chevron up icon
Section 2: Using Pandas for Data Analysis Chevron down icon Chevron up icon
Data Wrangling with Pandas Chevron down icon Chevron up icon
Aggregating Pandas DataFrames Chevron down icon Chevron up icon
Visualizing Data with Pandas and Matplotlib Chevron down icon Chevron up icon
Plotting with Seaborn and Customization Techniques Chevron down icon Chevron up icon
Section 3: Applications - Real-World Analyses Using Pandas Chevron down icon Chevron up icon
Financial Analysis - Bitcoin and the Stock Market Chevron down icon Chevron up icon
Rule-Based Anomaly Detection Chevron down icon Chevron up icon
Section 4: Introduction to Machine Learning with Scikit-Learn Chevron down icon Chevron up icon
Getting Started with Machine Learning in Python Chevron down icon Chevron up icon
Making Better Predictions - Optimizing Models Chevron down icon Chevron up icon
Machine Learning Anomaly Detection Chevron down icon Chevron up icon
Section 5: Additional Resources Chevron down icon Chevron up icon
The Road Ahead Chevron down icon Chevron up icon
Solutions Chevron down icon Chevron up icon
Other Books You May Enjoy Chevron down icon Chevron up icon

Customer reviews

Top Reviews
Rating distribution
Full star icon Full star icon Full star icon Full star icon Half star icon 4.7
(11 Ratings)
5 star 90.9%
4 star 0%
3 star 0%
2 star 9.1%
1 star 0%
Filter icon Filter
Top Reviews

Filter reviews by




Ali Assad Apr 06, 2021
Full star icon Full star icon Full star icon Full star icon Full star icon 5
Excellent book! If you are looking to sharpen up on pandas and data analysis in python in general (not only pandas) this is a very solid. In my case I needed a strong refresher and this book was a great choice going beyond that. The accompanying notebooks are great too.
Amazon Verified review Amazon
Felipe M. Sep 21, 2019
Full star icon Full star icon Full star icon Full star icon Full star icon 5
If you are new to Pandas and looking for a way to get up to speed fast this is a great book. Lots of good examples to see how to apply the concepts in real life, which helps a lot to understand how the different dataframe methods work. You can see the author payed a lot of attention to the detail, from the way to breakdown the chapters to the choice of exercises at the end of each chapter.As a Python programmer Learning Pandas I also appreciate a lot the quality of the code. Usually the code you find in data science examples get things done but it’s far from being of good quality. It’s very refreshing to see good examples written in a very pythonic way.I keep this book next to my desk and I have the feeling I will be coming back to use it as a reference a lot in the coming weeks. The only way this could be improved would be to have the author sit next to you every time you need to use Pandas :)
Amazon Verified review Amazon
Amazon Customer Nov 16, 2021
Full star icon Full star icon Full star icon Full star icon Full star icon 5
Having just started on my Pandas journey and coming from a non technical background, I find this book simple and the exercises relatable .
Amazon Verified review Amazon
CryptoKnight Aug 30, 2019
Full star icon Full star icon Full star icon Full star icon Full star icon 5
I really enjoyed this book. This book is great for python programmers who want to learn/implement something cool with data using Pandas (of-course). With some programming background in python/R/C, this book is perfect for the data scientist wannabe. Every chapter has a very interesting exercise set. Easy to follow. I am glad that the author has provided the github repo for this too, way better than reading code from the book alone! The book starts from easy steps e.g., import data, transform, basic computations, more fancy topics like plotting, to more advanced and application-specific like finance, machine learning (yes the book also talked about this in practical aspect!, using sklearn), security analytics!I can't say enough good things about this book and about how effective it is. I started 10 days ago with a skill of python but no data science background. Now, nearly two weeks later, I'm making my scripts do all kinds of things that are complicated if you don't know Pandas e.g., aggregate statistics based on duration (day, week, ..). I genuinely found this book and the exercises a truly beneficial experience/resource. Honestly I usually don't trust this publisher. Most Packt books I own I think weren't worth their price. This book is surprisingly good quality. Finally they have a good author.
Amazon Verified review Amazon
Adam Mar 23, 2021
Full star icon Full star icon Full star icon Full star icon Full star icon 5
Amazing read. Very thorough, covering a broad range of topics in the data science space. I would highly recommend starting with this book for anyone interested in building a foundation in Data Science.
Amazon Verified review Amazon
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

What is included in a Packt subscription? Chevron down icon Chevron up icon

A subscription provides you with full access to view all Packt and licnesed content online, this includes exclusive access to Early Access titles. Depending on the tier chosen you can also earn credits and discounts to use for owning content

How can I cancel my subscription? Chevron down icon Chevron up icon

To cancel your subscription with us simply go to the account page - found in the top right of the page or at https://subscription.packtpub.com/my-account/subscription - From here you will see the ‘cancel subscription’ button in the grey box with your subscription information in.

What are credits? Chevron down icon Chevron up icon

Credits can be earned from reading 40 section of any title within the payment cycle - a month starting from the day of subscription payment. You also earn a Credit every month if you subscribe to our annual or 18 month plans. Credits can be used to buy books DRM free, the same way that you would pay for a book. Your credits can be found in the subscription homepage - subscription.packtpub.com - clicking on ‘the my’ library dropdown and selecting ‘credits’.

What happens if an Early Access Course is cancelled? Chevron down icon Chevron up icon

Projects are rarely cancelled, but sometimes it's unavoidable. If an Early Access course is cancelled or excessively delayed, you can exchange your purchase for another course. For further details, please contact us here.

Where can I send feedback about an Early Access title? Chevron down icon Chevron up icon

If you have any feedback about the product you're reading, or Early Access in general, then please fill out a contact form here and we'll make sure the feedback gets to the right team. 

Can I download the code files for Early Access titles? Chevron down icon Chevron up icon

We try to ensure that all books in Early Access have code available to use, download, and fork on GitHub. This helps us be more agile in the development of the book, and helps keep the often changing code base of new versions and new technologies as up to date as possible. Unfortunately, however, there will be rare cases when it is not possible for us to have downloadable code samples available until publication.

When we publish the book, the code files will also be available to download from the Packt website.

How accurate is the publication date? Chevron down icon Chevron up icon

The publication date is as accurate as we can be at any point in the project. Unfortunately, delays can happen. Often those delays are out of our control, such as changes to the technology code base or delays in the tech release. We do our best to give you an accurate estimate of the publication date at any given time, and as more chapters are delivered, the more accurate the delivery date will become.

How will I know when new chapters are ready? Chevron down icon Chevron up icon

We'll let you know every time there has been an update to a course that you've bought in Early Access. You'll get an email to let you know there has been a new chapter, or a change to a previous chapter. The new chapters are automatically added to your account, so you can also check back there any time you're ready and download or read them online.

I am a Packt subscriber, do I get Early Access? Chevron down icon Chevron up icon

Yes, all Early Access content is fully available through your subscription. You will need to have a paid for or active trial subscription in order to access all titles.

How is Early Access delivered? Chevron down icon Chevron up icon

Early Access is currently only available as a PDF or through our online reader. As we make changes or add new chapters, the files in your Packt account will be updated so you can download them again or view them online immediately.

How do I buy Early Access content? Chevron down icon Chevron up icon

Early Access is a way of us getting our content to you quicker, but the method of buying the Early Access course is still the same. Just find the course you want to buy, go through the check-out steps, and you’ll get a confirmation email from us with information and a link to the relevant Early Access courses.

What is Early Access? Chevron down icon Chevron up icon

Keeping up to date with the latest technology is difficult; new versions, new frameworks, new techniques. This feature gives you a head-start to our content, as it's being created. With Early Access you'll receive each chapter as it's written, and get regular updates throughout the product's development, as well as the final course as soon as it's ready.We created Early Access as a means of giving you the information you need, as soon as it's available. As we go through the process of developing a course, 99% of it can be ready but we can't publish until that last 1% falls in to place. Early Access helps to unlock the potential of our content early, to help you start your learning when you need it most. You not only get access to every chapter as it's delivered, edited, and updated, but you'll also get the finalized, DRM-free product to download in any format you want when it's published. As a member of Packt, you'll also be eligible for our exclusive offers, including a free course every day, and discounts on new and popular titles.