In this chapter, we saw how we can write a custom crawler from scratch. This task is made easier using Python's modules, such as requests, BeautifulSoup, and so on. Feel free to download the whole code base and test the crawler with various other websites in order to examine its coverage. There may be occasions in which the crawler does not give 100% coverage. Take a look and see for yourself the limitations of the crawler and how it can be improved.
Germany
Slovakia
Canada
Brazil
Singapore
Hungary
Philippines
Mexico
Thailand
Ukraine
Luxembourg
Estonia
Lithuania
Norway
Chile
United States
Great Britain
India
Spain
South Korea
Ecuador
Colombia
Taiwan
Switzerland
Indonesia
Cyprus
Denmark
Finland
Poland
Malta
Czechia
New Zealand
Austria
Turkey
France
Sweden
Italy
Egypt
Belgium
Portugal
Slovenia
Ireland
Romania
Greece
Argentina
Malaysia
South Africa
Netherlands
Bulgaria
Latvia
Australia
Japan
Russia