Web Scraping with BeautifulSoup and Scrapingdog API

In this note, we'll look at dynamic websites that use JavaScript libraries such as React.js, Vue.js, Angular.js, etc., which require extra effort. It's a simple but lengthy process if you plan to install all the libraries like Selenium, Puppeteer, and headless browsers like Phantom.js. However, there is a tool that can handle all this load on its own. It is a Web Scraping Tool that offers an API and tools for web scraping.

Why this tool specifically? This tool helps us scrape dynamic websites using millions of rotating proxies to avoid being blocked. It also provides CAPTCHA solving. For scraping dynamic websites, it uses a headless Chrome.

What do we need?

Web scraping consists of two simple parts:

• Fetching data by making an HTTP request
• Extracting important data by parsing the HTML DOM.

We will use Python and the ScrapingDog API:

• Beautiful Soup is a Python library for extracting data from HTML and XML files.
• Requests makes it very easy to send HTTP requests.

Setup

Our setup is quite simple. Just create a folder and install Beautiful Soup and requests into it. To create the folder and install the libraries, enter the commands below. I assume you already have Python 3.x installed.

mkdir scraper
pip install beautifulsoup4
pip install requests

Now create a file in this folder with any name. I'm using scraping.py.

First, you need to register on the website of this scraping tool. It will give you 1000 FREE credits. Then simply import Beautiful Soup & requests into your file.

from bs4 import BeautifulSoup
import requests

Scraping Dynamic Content

Now we are familiar with Scrapingdog and how it works. But for reference, you should read the API documentation. This will give you a clear understanding of how the API works. Now we will work with Amazon to find titles of books about Python.

Now we have 16 books on this page. We will extract the HTML from the Scrapingdog API, and then use BeautifulSoup to generate a JSON response. Now, in just one line, we can scrape Amazon. For the API request, I'll use requests.

r = requests.get("https://api.scrapingdog.com/scrape?api_key=<your-api-key>&url=https://www.amazon.com/s?k=python+books&ref=nb_sb_noss_2&dynamic=true").text

This will return the HTML of the target URL.

Now we need to use BeautifulSoup to parse the HTML.

soup = BeautifulSoup(r, 'html.parser')

Each title has a "class" attribute named "a-size-mini a-spacing-none a-color-base s-line-clamp-2" and an "h2" tag. You can see this in the figure below.

First, let's find all these tags using the soup variable.

allbooks = soup.find_all("h2", {"class": "a-size-mini a-spacing-none a-color-base s-line-clamp-2"})

Then we'll run a loop to get the title of each book on this page, using the length of the "allbooks" variable.

l ={}
u = list()
for i in range(0, len(allbooks)):
    l["title"]= allbooks[i].text.replace("\n", "")
    u.append(l)
    l ={}
print({"Titles":u})

The list "u" contains all the titles, and we just need to print it. Now after printing the list "u" from the for loop, we get a JSON response. It looks like this.

Output -

{
 "Titles": [
 { "title": "Python for Beginners: 2 Books in 1: Python Programming for Beginners, Python Workbook"},
 { "title": "Python Tricks: A Buffet of Awesome Python Features" },
 { "title": "Python Crash Course, 2nd Edition: A Hands-On, Project-Based Introduction to Programming" },
 { "title": "Learning Python: Powerful Object-Oriented Programming" },
 { "title": "Python: 4 Books in 1: Ultimate Beginner's Guide, 7 Days Crash Course, Advanced Guide, and Data Science, Learn Computer Programming and Machine Learning with Step-by-Step Exercises" },
 { "title": "Intro to Python for Computer Science and Data Science: Learning to Program with AI, Big Data and The Cloud" },
 { "title": "Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython" },
 { "title": "Automate the Boring Stuff with Python: Practical Programming for Total Beginners" },
 { "title": "Python: 2 Books in 1: The Crash Course for Beginners to Learn Python Programming, Data Science and Machine Learning + Practical Exercises Included. (Artificial Intelligence, Numpy, Pandas)" },
 { "title": "Python for Beginners: 2 Books in 1: The Perfect Beginner's Guide to Learning How to Program with Python with a Crash Course + Workbook" },
 { "title": "Python: 2 Books in 1: The Crash Course for Beginners to Learn Python Programming, Data Science and Machine Learning + Practical Exercises Included. (Artificial Intelligence, Numpy, Pandas)" },
 { "title": "The Warrior-Poet's Guide to Python and Blender 2.80" },
 { "title": "Python: 3 Manuscripts in 1 book: — Python Programming For Beginners — Python Programming For Intermediates — Python Programming for Advanced" },
 { "title": "Python: 2 Books in 1: Basic Programming & Machine Learning — The Comprehensive Guide to Learn and Apply Python Programming Language Using Best Practices and Advanced Features." },
 { "title": "Learn Python 3 the Hard Way: A Very Simple Introduction to the Terrifyingly Beautiful World of Computers and Code (Zed Shaw's Hard Way Series)" },
 { "title": "Python Tricks: A Buffet of Awesome Python Features" },
 { "title": "Python Pocket Reference: Python In Your Pocket (Pocket Reference (O'Reilly))" },
 { "title": "Python Cookbook: Recipes for Mastering Python 3" },
 { "title": "Python (2nd Edition): Learn Python in One Day and Learn It Well. Python for Beginners with Hands-on Project. (Learn Coding Fast with Hands-On Project Book 1)" },
 { "title": "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems" },
 { "title": "Hands-On Deep Learning Architectures with Python: Create deep neural networks to solve computational problems using TensorFlow and Keras" },
 { "title": "Machine Learning: 4 Books in 1: Basic Concepts + Artificial Intelligence + Python Programming + Python Machine Learning. A Comprehensive Guide to Build Intelligent Systems Using Python Libraries" }
 ]
}