Summary

In this chapter, we covered two easier ways to scrape text data from the internet. Newspaper3k made short work of scraping news websites, returning clean text, headlines, keywords, and more. It allowed us to skip steps we’d done using BeautifulSoup and get to clean data much quicker. We used this clean text and NER to create and visualize networks. Finally, we used the Twitter Python library and V2 API to scrape tweets and connections, and we also used tweets to create and visualize networks. Between what you learned in this chapter and the previous one, you now have a lot of flexibility in scraping the web and converting text into networks so that you can explore embedded and hidden relationships.

Here is some good news: collecting and cleaning data is the most difficult part of what we are going to do, and this marks the end of data collection and most of the cleanup. After this chapter, we will mostly be having fun with networks!

In the next chapter, we will...