Data Manipulation and Cleaning with Python: Preparing for Machine Learning Projects

Before using data, it must be cleaned. This complex process includes everything from removing duplicate values to handling missing data outliers. The more accurate the cleaning, the easier it becomes to extract value from datasets during the stages of data manipulation, algorithm training, and modeling.

In fact, the bulk of any data science project primarily requires effective data cleaning and manipulation.

What is Data Manipulation/Cleaning?

As a data scientist, when starting a project, you should begin by collecting various datasets, either by extracting them from external websites or obtaining them from various internal sources, depending on your requirements.

Not all collected data will be relevant to your task. To separate relevant data from irrelevant, you need to clean the collected datasets. In other words, you may need to remove or modify columns, delete duplicate values, handle missing values, outliers, etc. You may also need to normalize and scale data to fit within a certain range.

Data cleaning also involves visualizing data using graphs and statistical functions to find 'baseline data', also known as 'mean', 'median', 'range', 'distribution', etc.

Why is Data Manipulation/Cleaning Important for Data Scientists?

Before a data scientist can focus on modeling, they need to master data cleaning. The complexity of modeling depends on how effectively you clean the data. The more orderly the datasets are during the cleaning stage, the simpler the learning algorithms will be during modeling. Data structure also directly affects prediction accuracy.

In short, data cleaning is no less important than creating the algorithms themselves. If you master data cleaning, you can expect the following:

Reduced data processing time
More accurate predictions
Simplified algorithm functionality
Improved model training efficiency

Why Python?

Python is becoming the most popular coding language in data science for many reasons. First, it provides many computational libraries that can be used for data science projects, including data manipulation and cleaning. In this article, we'll use the Python Pandas library.

6 Steps for Data Manipulation and Cleaning with Python:

#1: Imputing Missing Values – This is a standard statistical imputation constant using KNN imputation.

Outlier/anomaly detection is performed using: Isolation Forest, One Class SVM, Local Outlier Factor, and/or anomaly detection algorithms.

#2: Conducting Outlier/Anomaly Detection – For this, you can use outlier detection algorithms such as Isolation Forest, One-Class SVM, and/or Local Outlier Factor.
#3: Using Cleaning Methods for X-Variables – In this case, you need to apply custom functions, remove duplicates, and replace critical values.
#4: Using Cleaning Methods for Y-Variables – Here, it's important to perform label encoding, one-hot encoding, and dictionary mapping.
#5: DataFrames Should Be Combined – This step includes concatenation, merging, and joining.
#6: The Last Step – Parsing Dates – Here, you need to use strings that define auto-format to perform 'DateTime' conversion, including converting 'DateTime' objects into numbers.

Let's look at each step in detail.

First: Imputing Missing Values

One of the most common problems you may encounter when working with raw datasets is missing values. If there are not too many, they can be easily filled at this stage.

Simple imputation methods such as mean, median, mode can be used to fill missing values (NaN) with the statistical measure of each column. The parameter can be set to 'mean', 'median', 'most_frequent' or mode, or 'constant', which is a manually specified value.
KNN Imputation – a more advanced method for imputing missing values. The KNN algorithm is used to find data points similar to the missing values in the dataset.

It is important to note that for KNN imputation, the data must be normalized to eliminate scale differences. To use KNN imputation, you need to:

Normalize the data
Perform KNN imputation to fill missing values
Reverse scale/normalize the data

Second: Outlier and Anomaly Detection

Isolation Forest – an algorithm used to obtain an anomaly score for datasets. The algorithm selects a feature and isolates observations by randomly choosing a split value. Then paths are created representing the normality of the value. The shorter the path, the more anomalous. 'Trees' with shorter paths for these samples form a 'forest' where anomalies are likely to be detected.
One-Class SVM – another method for finding outliers. It is suitable when Isolation Forest cannot be applied due to excessive variance.
Local Outlier Factor – a third method used for anomaly detection. The local outlier factor measures the density deviation in each dataset compared to others. Samples with lower density than their neighbors are likely outliers. This algorithm is distance-based, so data must be normalized before use. This method is an alternative to Isolation Forest for high variance.

When using any of these three methods, it is important to ensure that anomalies are not just data clusters. PCA visualization can be used for double-checking.

THIRD: Cleaning Methods Using X-Variables

Applying custom functions is necessary when cleaning cannot be performed using built-in functions. In this case, you may need to write your own functions, but first try using external built-in functions.
Removing duplicates is an important part of data cleaning. This can be done using the data.drop_duplicates() function, which removes rows with identical values. Careful checking is needed to ensure duplicates are not errors, especially in small datasets.
Sampling data points is important for large datasets. To sample random data points, use the data.sample(number_of_samples) command.
Renaming columns is done using the .rename command, where the key is the original column name and the value is the renamed value.
Replacing values is done using the data.replace() function, which takes two values from the frame that are replaced with other values. This is useful for imputing missing values so that imputation algorithms can work effectively.

FOURTH: Cleaning Methods for Y-Variables

Label encoding is necessary for categorical y-variables. If the data has two classes, they must be converted to 0 and 1, since machine learning algorithms can only operate with mathematical symbols. To do this, you can use the .map() function, which transforms a dictionary or original names and replaces values with numbers. If there are too many classes for manual mapping, you can use sklearn's automated method. This method is convenient because data can be easily returned to the original format using encoder.inverse_transform(array).
One-hot encoding may be preferred in special cases where you have many classes and do not want to impose quantitative assessments on the data. In one-hot encoding, each y value is a vector whose length equals the number of classes, with '1' representing the index within the vector and the rest set to '0'. Pandas has a built-in function get_dummies that can automatically take shapes and output DataFrames in one-hot encoding.

FIFTH: Combining DataFrames

Concatenation – a downward method of combining DataFrames
Merging – the process of merging two DataFrames left to right.
Joining is for other types of merges. Merging only combines rows that have a common key in both DataFrames. Joining includes left outer join, where all keys in the left DataFrame are included, and rows in the right DataFrame are included only if their keys are in the left DataFrame.

SIXTH: Parsing Dates

Dates can be very complex datasets, yet they are among the most important. Therefore, it is crucial to understand how to work with this type of data correctly.
Automatic format detection and conversion of strings to dates is a vital skill, since datasets rarely come with easily accessible date objects. You can use the dateutil program to automatically detect days, months, and years.
Converting dates to numbers is necessary so that models can understand the concept of time. DateTime objects are converted to numbers. In other words, each date represents the number of days elapsed since the earliest date in your dataset. The function is applied to the date column using .apply()

Conclusion

Data manipulation and cleaning are critical stages that all data scientists must go through before starting any machine learning project. This article provides a step-by-step guide to help you avoid confusion and save time. Python libraries allow you to manipulate data now to achieve more accurate results later.