7. Advanced Web Scraping and Data Gathering

Activity 7.01: Extracting the Top 100 e-books from Gutenberg

Solution:

These are the steps to complete this activity:

1. Import the necessary libraries, including regex and BeautifulSoup:

   import urllib.request, urllib.parse, urllib.error
   import requests
   from bs4 import BeautifulSoup
   import ssl
   import re

2. Read the HTML from the URL:

   top100url = 'https://www.gutenberg.org/browse/scores/top'
   response = requests.get(top100url)

3. Write a small function to check the status of the web request:

   def status_check(r):
       if r.status_code==200:
           print("Success!")
           return 1
       else:
           print("Failed!")
           return -1

4. Check the status of response:

   status_check(response)

   The output is as follows...