XPath vs. CSS Selectors

Purpose of XPath Expressions and CSS Selectors

Yes, we just said we assume you already have an idea of what they do, but if we were mistaken, allow us to indulge our readers with a brief overview.

The purpose of XPath expressions and CSS selectors is to provide an expressive language for finding and selecting elements in XML and HTML documents. Suppose we have the following web page:

<!doctype html>
<html>
<head>
<title>What is the DOM ?</title>
<meta charset="utf-8" />
</head>

<body>
<h1>DOM 101</h1>
<p>Webscraping is awsome !</p>
<p>Here is my <a href="https://www.scrapingbee.com/blog/">blog</a></p>
</body>
</html>

Now we want to extract the URL of our anchor tag pointing to ScrapingBee. How can we do this?

Of course, you could manually parse the entire HTML string (e.g., tokenize everything and check for <a) or — if more advanced — use regular expressions (e.g., <a\s+href="([^"]+)), but all of these are manual string parsing, which is rarely interesting.

This is where XPath and CSS selectors come to the rescue. In our example, we can access the anchor tag using the following syntax in XPath and CSS respectively.

History of XPath and CSS Selectors

Both technologies emerged quite a long time ago. They appeared around the mid-to-late 1990s, and although in many ways they are very similar — they can be considered cousins or even siblings — they nevertheless have different origins and different initial purposes.

XPath: Although the boundaries between HTML and XML have admittedly always been somewhat fluid, HTML has always been focused on the web, while XML was primarily a structured document format for data. This is where XPath came in. XPath was introduced alongside XML and aimed to provide a query language for XML. It allows quick and easy access to information within a given XML document, whether on the web or in a known XML database.

CSS: CSS selectors were not an independent invention but are part of the overall CSS ecosystem. However, they are an integral part of it, as they provide the core functionality of finding the element(s) to which a rule's declaration block will be applied. In this context, CSS selectors have always been very strongly (and almost exclusively) focused on the web and web pages.

While XPath has remained (mostly) true to its original purpose of being a query language for XML document structures (which includes HTML), CSS selectors have evolved beyond use only for styling and become a standalone web document query technology.

As we said, both technologies are practically identical in their capabilities and share syntactic features; however, there are still quite a few differences, as well as advantages and limitations. This is what we will examine next.

Advantages and Limitations of XPath

Generally speaking, XPath is a more complex language. If you can implement a query using CSS selectors, you can do the same with XPath. The reverse is not always true.

For example, one of the main advantages of XPath is its ability to support bidirectional tree traversal. You can find a specific element and then, using the parent axis, select its parent or even ancestors. Although CSS selectors offer a semi-equivalent in the form of the :has pseudo-class, it is still far from XPath, as it covers only a limited set of use cases, and browser support is still quite questionable.

Although performance usually doesn't play a big role in this context, sometimes XPath performance can be higher than that of CSS selectors. This is because some CSS selectors use XPath internally and first convert their selectors into XPath expressions.

One drawback of XPath is certainly its more verbose syntax. Things like HTML classes must be specified as attributes, while CSS supports them as first-class citizens in its syntax.

Pros and Cons
+ Support for bidirectional tree traversal (access to parent or ancestor element)
+ Support for selecting not only document elements (i.e., attributes and content)
+ Built-in functions for various use cases (e.g., text(), count())
+ Support for absolute and relative search paths

- More verbose syntax

Advantages and Limitations of CSS Selectors

One of the main advantages of CSS selectors is their "naturalness" in the web context. If you are familiar with web development, you will immediately understand CSS selectors, as you have been using them for styling pages for a long time.

Unlike XPath expressions, CSS selectors perfectly understand HTML identifiers and classes, and provide elements with their own syntax (e.g., #id & .class). For this reason, CSS selectors can be considered slightly less verbose (a#id vs //a[@id='id']).

Limitations include the fact that CSS selectors only work at the element level. That is, if you need to extract a link from an <a> tag, CSS selectors can only find all matching tags, and you have to access the href attribute in a second step, outside the context of the CSS selector (e.g., tag.getAttribute('href')). XPath expressions allow direct access to attributes here (e.g., //a/@href).

Another significant limitation is the top-down approach, meaning the DOM tree can only be traversed in one direction and you cannot select parent elements (e.g., `... for file systems). Although this may not be the most common use case, there are times when it could be useful, and with CSS selectors you have to find a workaround.

Pros and Cons
+ Web developers are familiar with the syntax
+ HTML identifiers and classes are first-class citizens (hash and dot syntax)

- Can only select elements (no attributes or content)
- Does not support upward traversal (access to parent or ancestor element)
- More difficult to specify absolute search path

Data Extraction with CSS Selectors and ScrapingBee

Just in case you are not yet familiar with the ScrapingBee data extraction service: it is a SaaS platform for data extraction with a lightweight REST API, native SDK packages for some languages, and support for headless browser integration.

Although XPath is undoubtedly more advanced for some use cases, the world of web scraping is largely dominated by CSS selectors, and users, especially web developers, often feel more comfortable with them, so ScrapingBee decided to support CSS selectors as the primary expression language in its API.

Launching a data extraction job is quite simple, and it suffices to provide a JSON object with the fields you want to extract from the page as the extract_rules parameter. For example, the following object defines two extraction items ("title" and "subtitle") along with their corresponding CSS selectors (h1 for title and #subtitle for subtitle).

{

"title": "h1",

"subtitle": "#subtitle"

}

In response, you will receive a similar JSON object containing the fields with their values from the page.

Using XPath and CSS Selectors in the Browser

Although there are many tools (e.g., https://try.jsoup.org) that can help you with XPath expressions and CSS selectors, there is actually one such tool already installed on your machine, the one you are using right now: your browser.

Developer tools in Chrome and Firefox not only make it easy to evaluate and debug XPath expressions and CSS selectors, but also help you find the right expressions. Simply right-click on the element of interest and select Inspect. Now you should have the page's DOM tree open, and you can access individual elements.

Finding the Right XPath and CSS Selector Using the Browser

In our example on example.com, we right-clicked on the "More information" link and now have a context menu with all the necessary options. Let's select 'Copy'. The highlighted entries will be of particular interest to us.

If we select Copy Selector, the browser will save the CSS selector path of the element to the clipboard, for example, the following expression:

body > div > p:nth-child(3) > a

If we select Copy XPath or Copy full XPath, the browser will do the same with the following XPath expression:

/html/body/div/p[2]/a

In any case, an expression for the element should appear in the clipboard. However, keep in mind that even Google and Mozilla probably can't work miracles (yet), and these expressions may only be applicable for this page, and perhaps only for this load. Often, fine-tuning of the expression may be required.

Fine-Tuning Expressions

In the two previous examples, we used absolute paths, and they worked perfectly in our context, but there is always room for optimization. For instance, given that example.com has only one single link, we could easily shorten our expressions to the following:

# XPath
//a

# CSS
a

On the one hand, unfortunately, this is not only not very readable, but it may also not work on reload due to changes in HTML classes.

On the other hand, fortunately, in this case we have a very specific HTML element that simplifies things greatly -> <time>.

We simply use time (or //time for XPath) and get "only" the element we need. Quoted because Twitter displays related tweets, and each of them will have a <time> element, so make sure to use only the first one.

However, not only specific element types can help narrow down the expression; we should also pay attention to HTML IDs, classes, attributes, and sometimes even content.

IDs should be unique, so a quick #myid or (//*[@id='myid']) will get the element faster and more reliably than an absolute path.
Although classes are not necessarily unique, they can still work wonders, especially if placed in the context of the document hierarchy: h1 > b.price

Last but not least, devtools also allow evaluating XPath expressions and CSS selectors.

Simply press F12 again, select the Elements tab, and press Ctrl/⌘ + F to open the search box.

Search Box

Here you can try various XPath expressions and CSS selectors and cycle through the found elements (if any).

Quite convenient for quickly testing expressions.

Conclusion

We hope you enjoyed the article and got a first idea of what these two languages are and what their strengths and weaknesses are.

Since in this article we tried to focus on the main differences between XPath and CSS selectors, we did not delve into the technical details of the element syntax of each language.