You're reading from Hands-On Web Scraping with Python Extract quality data from the web using effective Python techniques

Product type Paperback

Published in Oct 2023

Publisher Packt

ISBN-13 9781837636211

Length 324 pages

Edition 2nd Edition

Languages

Python

Tools

Pandas

Concepts

Web Programming

Author (1):

Anish Chapagain

View More author details

Table of Contents (20) Chapters

Preface

1. Part 1:Python and Web Scraping

2. Chapter 1: Web Scraping Fundamentals FREE CHAPTER

3. Chapter 2: Python Programming for Data and Web

4. Part 2:Beginning Web Scraping

5. Chapter 3: Searching and Processing Web Documents

6. Chapter 4: Scraping Using PyQuery, a jQuery-Like Library for Python

7. Chapter 5: Scraping the Web with Scrapy and Beautiful Soup

8. Part 3:Advanced Scraping Concepts

9. Chapter 6: Working with the Secure Web

10. Chapter 7: Data Extraction Using Web APIs

11. Chapter 8: Using Selenium to Scrape the Web

12. Chapter 9: Using Regular Expressions and PDFs

13. Part 4:Advanced Data-Related Concepts

14. Chapter 10: Data Mining, Analysis, and Visualization

15. Chapter 11: Machine Learning and Web Scraping

16. Part 5:Conclusion

17. Chapter 12: After Scraping – Next Steps and Data Analysis

18. Index

Why subscribe?

19. Other Books You May Enjoy

Web scraping using APIs

Technically, obtaining data from APIs is easy, and is also different from the web scraping scenarios. Most of the data is in JSON format, and there is no use of XPath, CSS Selector, or any other parsing libraries. Some data found using APIs might contain a lot of HTML code. To deal with this content, XPath and CSS Selector might be required.

Important note

In the examples in this section, we have tried to omit APIs that require an API key (walmartlabs.com, nasa.gov, nytimes.com, maps.googleapis.com) or user authentication tools. An API key is an authenticated value provided by service providers (after user registration) that identifies the user using the API. Freely available APIs are being used in these examples.

With the help of a few examples, we will collect data returned via some APIs. This collected data will then be exported as JSON and CSV files.