Scraping using Selenium
Selenium is used for automation – primarily web testing – using various browsers and coding in different languages. Along with automation, the benefits or features provided are quite handy and can be utilized in tasks such as web scraping.
In this section, we will use and explore quite a few features from the selenium library for web scraping.
Example 1 – book information
In this example, we will collect some details from the books listed in the Childrens category at the URL http://books.toscrape.com, which are available in the fictional bookstore at the https://toscrape.com URL.
In particular, we are searching for the anchor element <a>, which contains the bookstore text (partial text or a portion of the text) after loading mainUrl. With element <a> being traced, the href attribute from <a> can be collected using the get_attribute() method for link. The click() method clicks the element that contains the bookstore...
