Using Watir for Web Browser Automation with Ruby

For many years, it has been possible to automate simple computer tasks performed via the command line. This is known as parsing. However, a more complex task is controlling a browser, because the graphical interface provides many more opportunities for altering the behavior of elements.

Browser automation describes the process of programmatically performing certain actions in a browser (or delegating those actions to robots) that would otherwise be tedious or repetitive for a human to perform manually.

One tool created for this purpose and used by Ruby developers is Watir.

What is Browser Automation?

Browser automation is useful for various use cases, such as data entry, data extraction, or uploading files to remote servers. It allows actions like entering text and clicking buttons on web pages to be performed with greater speed, accuracy, and at scale than manual operation.

Imagine a government website containing socio-demographic data but lacking an API. Many such websites exist. With browser automation, you can scrape data from such a site (legally, of course) and clean it for your own use.

Browser automation can also be used to automatically fill out forms and documents, which can be very useful when you need to export information for a large number of clients.

Automation can be applied both for testing local applications and for conducting full-scale tests across multiple platforms and browsers. For example, you can check how well your website performs in different browsers (e.g., Chrome, IE, Firefox, Safari) or on different operating systems (e.g., Linux, Mac, Windows). A manual approach would be slow and error-prone, while browser automation eases the process of detecting and fixing broken links, identifying errors, spotting browser incompatibilities, measuring response times, and more.

Implementing Watir

Watir stands for Web Application Testing In Ruby. It is a collection of open-source Ruby libraries used for automating web browsers. Watir is a wrapper around another popular framework, Selenium. Watir includes some enhancements and allows developers to use the Ruby language for automation tasks.

In this tutorial, the Watir framework will be used to explore the popular open-source website Wikipedia. You will create a browser instance, visit the Wikipedia landing page, and navigate to the English version of the site. On the landing page, you will find and fill in the search box, submit a query, and navigate to the result page. Along the way, you will retrieve some data and print output to the terminal. Finally, you will take a screenshot of the search results and close the browser.

Installing Gems

This guide assumes you already have Ruby installed on your computer and are familiar with the command line and IRB.

First, open a console and check if the watir and webdrivers gems are installed on your system. To view all gems on your computer, run the following command:

gem list

Review the console output. If watir or webdrivers (or both) are not available, install them:

gem install watir
gem install webdrivers

Next, launch IRB on Linux or Mac, or start an interactive Ruby session on Windows. You can type the code below line by line or run it as a script. Entering code in an IRB session allows you to observe changes in real time.

Setting Up Watir

To begin, require the watir and webdrivers gems:

#watir_testfile.rb
require 'watir'
require 'webdrivers'

Watir provides the necessary automation libraries, and webdrivers allows Watir to easily integrate with any supported browser. Without the webdrivers gem, you would have to download various packages for each browser you want to use.

Starting the Browser

To start the browser, run the following commands:

browser = Watir::Browser.new
browser.goto ("wikipedia.org")

Here, the command Watir::Browser.new initializes a browser instance and launches what is called a "headless browser," and the goto command directs the new browser instance to the specified URL.

By default, the new command opens the Chrome browser, but other browsers can be used if specified as options. Compatible browsers are Firefox, Internet Explorer, Edge, and Safari. For example, this command opens Firefox: Watir::Browser.new :firefox.

The Chrome instance displays a helpful message: "Chrome is being controlled by automated test software."

Finding and Interacting with Elements

The links method can be used to retrieve all links present on a page (as opposed to the link method, which only gets the first link). The data collection operation is performed by chaining links to the Ruby count method and printing the result:

puts browser.links.count
english_link = browser.div(class: "lang1")
english_link.present?
english_link.click
puts browser.url

The Wikipedia landing page contains links to various language versions of the site. To find the class and attributes of the link to the English version, use the target link (you can use browser developer tools to inspect classes and attributes). Clicking the target link navigates to the Wikipedia main page. Here and in subsequent steps, orientation can be maintained by having the browser call the url method of the watir program, which returns the current page URL.

Extracting Data from a Web Page

You can find elements on a page using the element method to refer to CSS selectors. In this case, we are interested in an element with the id "articlecount".

puts browser.element(id:'articlecount').text.strip

Once the element of interest is found, you can perform a data capture operation using the text method to get the text associated with the tagged element. Finally, calling the strip function removes any trailing newlines from the result.

Locating the Text Field

Locating form fields is done using the text_field method, which works similarly to the element method discussed above. Text_field allows targeting form fields using selectors. In this case, we are interested in a class named "vector-search-box-input".

search_box = browser.text_field(class: "vector-search-box-input")

This information is captured and stored in the variable search_box, which can be used in subsequent operations.

Setting Text and Waiting

Now that you have the ability to accept data, it's time to send it. Watir not only provides an API to send desired data but also ensures conditions are met before processing.

search_box.wait_until(&:present?).set("sailboats")

Here, the set method ensures the form field is filled with data. In this example, we use "sailboats" as the search query. You can change it to anything you like.

Wait_until serves a special purpose. You may have noticed that after executing the respective commands, the page takes some time to load. Calling set or performing any other operation on an element that has not loaded or is not enabled will cause an error. To avoid this, you must ensure that the set method is called only when the element (here, the text field) is ready. That is the role of wait_until.

Wait_until is one of two methods defined in the Waitable module, which is included in the Element, Alert, Window, Browser, ElementCollection, and WindowCollection classes. Wait_until takes a block (or proc) specifying conditions that must be met, and it will execute that block until a truthy value is returned. Here, the present? method is passed as a proc, which checks for the element's presence. Wait_until evaluates the condition until it returns a truthy value.

The other method available in Waitable is wait_while. Wait_while is similar to wait_until, but it evaluates the given condition until it becomes false. Neither wait_until nor wait_while run indefinitely; both have a default timeout of thirty seconds, which can be configured as needed.

Clicking Buttons

Clicking a button on a web page can perform a search or other action. Here, you locate a button element using the button class and the value attribute. Then, you call a click on that element so the browser navigates to a new page. These lines of code locate the button element associated with the search box and execute the search:

button = browser.button(value: "Go")
button.click
puts browser.url

Executing JavaScript

Watir can do more than just find page elements and send data. It also provides the ability to execute JavaScript in the browser using the execute_script method. For example, the following code creates a modal alert box with a custom message that appears on the current page:

browser.execute_script("alert('On the sailboats page!')")
browser.alert.wait_until { |a| a.text == "On the sailboats page!" }
sleep 3
browser.alert.ok

The first line creates a modal window with an embedded message. The next line uses the now-familiar wait_until method. For reasons similar to those discussed earlier, you need to synchronize page loading with requests, so wait_until is used. Here, wait_until checks for a match with the entered text ("On the sailboats page!"), and when that text appears on the page, the alert modal fires. Then alert.ok closes the modal.

The sleep method pauses execution of the current thread for a specified time. Calling sleep is not strictly necessary, and in many contexts of browser automation, using it is considered sloppy coding because the methods provided by the Waitable module are usually more efficient. Here, it is added for user convenience. The automation software does not care, but a human appreciates visual cues, and sleep keeps the modal on screen. Without it, the modal alert would close almost instantly after being triggered.

Taking a Screenshot

Finally, you can take a screenshot using the screenshot method, save it to a local file, and close the browser. Use the following code:

browser.screenshot.save("./sailboats_screenshot.png")
browser.close

That's it! You have successfully automated all actions related to browsing the website.

Conclusion

Watir is a family of libraries for testing and automating web browsers. It is highly regarded in the Ruby developer community, easy to learn, and simple to use. In this tutorial, you learned how to set it up and use its most common features. These include selecting elements, sending and capturing data, executing JavaScript, and taking screenshots. However, Watir is capable of much more. It can be used for advanced web scraping or even paired with popular testing frameworks like minitest and rspec for automated headless testing.