Alternatives to Portia for Web Scraping
Portia was a tool that allowed users to scrape websites without programming knowledge. It was a hosted service, but due to a decline in the use of visual scrapers, it was taken down and is no longer used today.
How did people use Portia when it existed? To use Portia, you would enter a pattern of URLs to visit, and then select elements on those web pages using point-and-click gestures, or via CSS or XPath. Despite its ease of use, Portia's biggest problems were the following:
- Compared to other open-source scraping tools, managing this tool was time-consuming.
- Navigating websites was difficult.
- To prevent Portia from visiting unnecessary URLs, you had to specify target pages when starting the crawl.
- There was no way to connect a database to save collected data.
What Are the Advantages of Visual Parsers?
If you need one-time scraping, you can use a visual parser, but using it as part of a workflow is not recommended. If you run a business where you need to collect data from a very small number of static web pages, and even then very rarely (say, once a month), you can assign someone on your team who knows what data to collect to learn how to use the visual parser in a few hours, and then occasionally extract data from web pages.
Visual parsers are especially useful for small businesses that don't have a technical team and have minimal scraping requirements.
Such parsers are almost like someone clicking 'inspect element' on a web page and copying data from the HTML content. Instead, when you use a visual parser, you click on a part of the web page, and the program copies the data for you to a location of your choice.
Where Do Visual Parsers Fail?
Visual parsers, however, do not handle serious tasks well.
You may need to incorporate scraping of certain data into a workflow (it needs to be automated).
You may need to scrape data from hundreds or thousands of pages and update it frequently.
There may be a need to stream the collected data directly to a specific business module. In most of these cases, a code-based parser will be much more convenient than a visual one.
Most mass scraping projects face the need to browse through tons of similar web pages to extract data about various items. This data can range from flight information on booking sites to product details on e-commerce sites.
The logic applied in such scenarios is that you try to understand the data pattern on web pages using a few pages, and then use code that can not only scrape pages with the exact same structure but even pages with similar structure. Moreover, when scraping all pages on a site, pages with a specific structure can be ignored. All these customizations are not possible in a visual parser, so it is not recommended to scrape too many pages with one.
On the other hand, because website layouts change every few weeks or months, you may have to retrain your visual parser every time the website's UI changes. On the other hand, when using a code-based parser, often a UI change may not even require any changes to the script, as the site's structure may remain unchanged.
Even if some UI changes require modifications to the parser, these changes are usually minimal, and adjusting the parser to those changes is fairly simple.
What Other Alternatives Are There?
There are many alternatives to Portia. Languages like Python, R, and Golang are used by developers and scraping teams around the world to extract data from web pages. New ways are being developed to make this process faster. For example, using parallel programming and caching in Golang with a package called Colly, you can use custom settings such as the following:
- The number of pages you want to crawl simultaneously at any given time.
- The maximum depth the parser should go when it starts scraping a web page. (For example, if you set the maximum depth to 3, the parser will scrape the top page, follow a URL found on it, scrape that, then follow a URL found on that page, and scrape that too, but on the third page, if it finds a URL, it will not go further.)
- You can set URL word filtering: if a word is present in the URL, then the web page at that URL should be scraped. Or you can set exclusions: URLs containing a certain word should not be accessible to the parser.
These are just a few examples of the hundreds of small features you get when building your own web scraper.
DaaS Providers vs. In-House Team?
Most companies that do not have a team of technical specialists, or even employees without a basic understanding of any scripting language, should avoid starting to build their own scraping team. The reason is simple. The money you would spend on hiring developers, and then having them build and maintain a completely new scraping system for your business needs, would be enormous.
And in the end, if you are a small company and web scraping is not the fuel for your business (i.e., your business is not centered on data you collect from the web), then there is no point in building an in-house team.
A simple solution in this case is DaaS providers, who take your requirements and provide you with data in your chosen format. Our team at ESK Solutions is proud to have reduced the scraping process to two steps for businesses and enterprises.
Conclusion
Although visual tools are useful for business teams, we can say that scraping is not a simple business task. It is a task that should be efficient, fast, and fully customizable. If you have large-scale scraping requirements or want to extract data at much larger scales, it is recommended to use scraping services.
If you are not tech-savvy or your requirements are complex, you can use our services to get clean data in an automated way without any technical difficulties or learning any tools.