How to develop data-intensive applications: Examples and practice

In today's fast-paced world, where data is king, the ability to leverage vast amounts of information has become a critical component for businesses striving to stay ahead of competitors. According to an IDC report, by 2025 the global data sphere will reach 175 zettabytes, with most of this data generated by enterprises. With such enormous volumes of information, software development capable of efficiently processing, storing, and analyzing data becomes a crucial task for organizations.

The challenges of developing applications for large volumes of data are not only technical but also involve ensuring that applications meet business requirements and are useful for end users.

What is a data-intensive application?

A data-intensive application is an application that processes and analyzes large volumes of data, typically in real time or near real time. Such applications are designed to quickly and efficiently handle, store, and manage vast amounts of constantly changing data in order to gain insights, make data-driven decisions, and support complex business operations.

Industries that require such software typically have a large number of users, high data volume, and frequent interaction. Here are some of them:

• Banking and financial services process and analyze large volumes of data to detect fraud, manage risk, and make data-driven investment decisions. Data-intensive applications help these organizations manage and analyze information from various sources, including customer data, transaction data, market data, and regulatory data.
• The Data Science and Data Mining industry needs data-intensive applications to improve data management, develop more accurate predictive models, machine learning, create better visualizations and real-time analytics. 
• In digital marketing, data-intensive applications are used to analyze consumer behavior, target advertising, and evaluate the effectiveness of marketing campaigns. These applications process data from various sources such as social media, website traffic, and customer data, and provide insights that help marketers optimize their strategies.
• E-commerce companies generate data on online sales, customer behavior, and website traffic. Data-intensive applications help them manage and analyze this data to personalize the shopping experience, optimize pricing and inventory, and improve supply chain management.

To learn the basics of extracting data from web sources, we recommend reading The Complete Guide to Web Data Extraction.

To provide real business value and help stakeholders make informed decisions faster, a data application must be able to collect, process, and present data in an easy-to-understand format. This circumstance creates various challenges in designing big data applications. Let's talk about them.

Common challenges of big data applications and how to solve them

Every time we design big data applications, we encounter various difficulties: some are completely unique to each case, while others have common patterns that repeat from project to project. Here we consider typical problems we usually face.

Challenge 1. Constantly changing data that often needs to be processed in real time

Managing the overwhelming volume of information is complex and tedious for users. Too many graphical elements can distract and confuse them. That is why designing for real-time data processing requires careful user interface (UI) design. 

The UI design for data-driven applications must display real-time data in a clear and concise manner so that users can quickly and easily understand what is being shown. Additionally, the application should provide users with the ability to customize how data is presented, such as through filters, appropriate dashboard design, and visualization.

Challenge 2. Designing for different types of users

Data-intensive applications may have multiple types of users, each with their own needs and skill levels. It is important to design an application that caters to different users, with clear audience segmentation and the ability to provide either high-level summaries or deep data analysis. 

The problem we encountered was that the program has four main user groups, each working with data in their own way and needing different features and different levels of data access. Therefore, we needed to design the interface for each user category considering their needs.

How did we successfully handle this task? Only through thorough user research and persona mapping.

To create a design that meets the expectations of each user type, we had to analyze the existing product and its users, as well as their problems, needs, and goals.

• First, we conducted interviews with the CTO of Involi to learn about the product's purpose, capabilities, design and legal constraints, stakeholder expectations, and of course its consumers.
• Then we performed our own analysis of the software to see how customers use it, identify what works well and what doesn't, and suggest possible improvements.
• Based on the information obtained, we were able to create user personas that showed what data each user type needs, what features they use, and what problems they face.

Challenge 3. Scalability and maintenance

Data-intensive applications typically experience growth in data volume, complexity, and traffic. This requires a robust architecture and design that can be maintained and updated as the application evolves.

Furthermore, since data-intensive applications continuously evolve, new features and functionalities are added over time. It is important to ensure that these new features are consistent with the existing design and interaction so that end users can use them without issues.

It is no exaggeration to say that we care about scalability in every one of our projects. But of course, it is especially relevant for data-intensive projects. Take the example of Gamaya. 

Gamaya is a SaaS platform (an example of cloud development / integration) that processes and analyzes agricultural data. 

The Gamaya application uses React, a front-end tool for building web user interfaces with a component architecture that allows developers to create reusable components and assemble them in various combinations. Here is how we leveraged this technology to make the application easily scalable:

• We used Ant Design, a ready-made React UI library, and adapted our design to this library. 
• We prepared a set of UI and guidelines to help developers implement the necessary design components using Ant Design. 
• We also conducted a demo session explaining how to implement complex components. 

Challenge 4. Data presentation

This challenge involves presenting complex and large volumes of data in a visual and understandable form for users.

• A common problem is choosing the right level of detail when presenting data. While data detail can provide a more complete understanding of the information, it can also overwhelm users, making data interpretation difficult. On the other hand, excessive simplification of data may lead to loss of important information.

The general dashboard shows important information for developers without overwhelming them. It provides an overview for a selected time period, including the number of newly discovered vulnerabilities, resolved issues, etc.