E-Commerce Product Data Parsing
In this tutorial, we will look at how to extract product data from any e-commerce website using Java. There are many different use cases for extracting product data, such as:
- E-commerce price monitoring
- Price comparison
- Availability monitoring
- Review extraction
- Market research
- MAP violation detection
What You'll Need
We will use HtmlUnit to make HTTP requests and parse the DOM. Add this dependency to your pom.xml.
<dependency>
<groupId>net.sourceforge.htmlunit</groupId>
<artifactId>htmlunit</artifactId>
<version>2.19</version>
</dependency>
We will also use the Jackson library:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.8</version>
</dependency>
Schema.org
To extract the fields we are interested in, we will parse the https://schema.org metadata from the HTML markup.
Schema is a semantic vocabulary that can be added to any web page. Implementing Schema has many benefits. Most search engines use it to understand what a page is about (product, article, review, and much more).
According to schema.org, about 10 million sites worldwide use it. That's a lot! There are different types of Schema, and today we will look at the Product type.
This is very convenient because if you write a parser that extracts certain data from the schema, it will work on any other site using the same schema. No more writing specific XPath / CSS selectors!
There are three main types of schema markup:
JSON-LD
<script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "ItemList",
"url": "http://multivarki.ru?filters%5Bprice%5D%5BLTE%5D=39600",
"numberOfItems": "315",
"itemListElement": [
{
"@type": "Product",
"image": "http://img01.multivarki.ru.ru/c9/f1/a5fe6642-18d0-47ad-b038-6fca20f1c923.jpeg",
"url": "http://multivarki.ru/brand_502/",
"name": "Brand 502",
"offers": {
"@type": "Offer",
"price": "4399 p."
}
},
{
"@type": "Product",
"name": "..."
}
]
}
</script>
RDF-A
<div vocab="http://schema.org/" typeof="ItemList">
<link property="url" href="http://multivarki.ru?filters%5Bprice%5D%5BLTE%5D=39600"><span property="numberOfItems">315</span>
<div property="itemListElement" typeof="Product">
<img property="image" alt="Photo of product" src="http://img01.multivarki.ru.ru/c9/f1/a5fe6642-18d0-47ad-b038-6fca20f1c923.jpeg"> <a property="url" href="http://multivarki.ru/brand_502/"><span property="name">BRAND 502</span></a>
<div property="offers" typeof="http://schema.org/Offer">
<meta property="schema:priceCurrency" content="RUB">руб
<meta property="schema:price" content="4399.00">4 399,00
<link property="schema:itemCondition" href="http://schema.org/NewCondition">
</div>...
<div property="itemListElement" typeof="Product">
...
</div>
</div>
</div>
And the one used in our example — Microdata:
<div class="schema-org">
<div itemscope="" itemtype="https://schema.org/Product">
<img itemprop="image" src="https://images.asos-media.com/products/the-north-face-vault-backpack-28-litres-in-black/10253008-1-black" alt="Image 1 of The North Face Vault Backpack 28 Litres in Black">
<link itemprop="itemCondition" href="https://schema.org/NewCondition">
<span itemprop="productID">10253008</span>
<span itemprop="sku">10253008</span>
<span itemprop="brand" itemscope="" itemtype="https://schema.org/Brand">
<span itemprop="name">The North Face</span>
</span>
<span itemprop="name">The North Face Vault Backpack 28 Litres in Black</span>
<span itemprop="description">Shop The North Face Vault Backpack 28 Litres in Black at ASOS. Discover fashion online.</span>
<span itemprop="offers" itemscope="" itemtype="https://schema.org/Offer">
<link itemprop="availability" href="https://schema.org/InStock">
<meta itemprop="priceCurrency" content="GBP">
<span itemprop="price">60</span>
<span itemprop="eligibleRegion">GB</span>
<span itemprop="seller" itemscope="" itemtype="https://schema.org/Organization">
<span itemprop="name">ASOS</span>
</span>
</span>
</div></div>
Note that you can include multiple offers on a single page.
Extracting Data
public class Product {private BigDecimal price;private String name;private String sku;private URL imageUrl;private String currency;// ...getters and setters
WebClient client = new WebClient();client.getOptions().setCssEnabled(false);client.getOptions().setJavaScriptEnabled(false);String productUrl = "https://www.asos.com/the-north-face/the-north-face-vault-backpack-28-litres-in-black/prd/10253008";HtmlPage page = client.getPage(productUrl);HtmlElement productNode = ((HtmlElement) page.getFirstByXPath("//*[@itemtype='https://schema.org/Product']"));URL imageUrl = new URL((((HtmlElement) productNode.getFirstByXPath("./img"))).getAttribute("src"));HtmlElement offers = ((HtmlElement) productNode.getFirstByXPath("./span[@itemprop='offers']"));BigDecimal price = new BigDecimal(((HtmlElement) offers.getFirstByXPath("./span[@itemprop='price']")).asText());String productName = (((HtmlElement) productNode.getFirstByXPath("./span[@itemprop='name']")).asText());String currency = (((HtmlElement) offers.getFirstByXPath("./*[@itemprop='priceCurrency']")).getAttribute("content"));String productSKU = (((HtmlElement) productNode.getFirstByXPath("./span[@itemprop='sku']")).asText());
In the first lines, I created the HtmlUnit HTTP client and disabled JavaScript since we don't need it to fetch the Schema markup.
Then it's just basic XPath expressions to select the interesting DOM nodes we need.
This parser is far from perfect; it doesn't extract everything and doesn't handle multiple offers. However, it gives you an idea of how to extract Schema data.
Then we can create a Product object and output it as a JSON string:
Product product = new Product(price, productName, productSKU, imageUrl, currency);
ObjectMapper mapper = new ObjectMapper();
String jsonString = mapper.writeValueAsString(product) ;
System.out.println(jsonString);
Avoiding Blocking
Now that we can extract the product data we need, we must ensure that we aren't blocked.
For various reasons, sites sometimes implement anti-bot mechanisms. The most obvious reason for protecting sites from bots is to prevent large automated traffic from impacting site performance (so you need to be careful with concurrent requests, add delays...). Another reason is to prevent malicious bot behavior, such as spam.
There are different protection mechanisms. Sometimes your bot will be blocked if it makes too many requests per second/hour/day. Sometimes there is a limit on the number of requests per IP address. The most sophisticated protection involves analyzing user behavior. For example, a site might analyze the time between requests if the same IP makes requests simultaneously.
The simplest solution for hiding parsers is to use proxies. Combined with a random user agent, using proxies is a powerful way to hide our parsers and scrape rate-limited web pages. Of course, it's better not to get blocked at all, but sometimes sites only allow a certain number of requests per day/hour.
In such cases, you need to use proxies. There are many free proxy lists, but I don't recommend using them because they are often slow and unreliable, and the sites offering such lists are not always transparent about where these proxies are located. Sometimes public proxy lists are maintained by legitimate companies offering premium proxies, and sometimes not...
Setting a proxy for HtmlUnit is very simple:
ProxyConfig proxyConfig = new ProxyConfig("host", myPort);
client.getOptions().setProxyConfig(proxyConfig);