What is Data Analysis of Machine Learning

Data Analysis, Cleaning and visualisation

Exploratory Data Analysis:

• Data Cleaning and Preprocessing

• Data Visualization Techniques

• Feature Extraction and Feature Selection

What is Data Analysis?

Data analysis is the process of looking through, purifying, manipulating, and modelling data to glean valuable information and insights. 

Exploratory Data Analysis (EDA) is the process of analyzing and visualizing data to extract insights and patterns. It is an essential step in the machine learning pipeline that helps to identify data quality issues, understand the distribution of data, detect anomalies, and gain a deeper understanding of the relationships between variables.

Some common techniques used in EDA include:

Summary statistics: calculating measures like mean, median, mode, and standard deviation to get a sense of the data's central tendency and spread.

Data visualization: using charts, graphs, and plots to visualize data patterns and relationships, such as scatter plots, histograms, box plots, and heat maps.

Dimensionality reduction: using techniques like principal component analysis (PCA) or t-SNE to reduce the dimensionality of high-dimensional datasets and visualize them in lower dimensions.

Outlier detection: identifying observations that fall outside the expected range of values, which can be an indication of data quality issues or interesting patterns.

Example:

Consider a dataset of customer purchases from an e-commerce website. To perform EDA on this dataset, we might calculate summary statistics such as the mean and standard deviation of purchase amounts, visualize the distribution of purchase amounts using a histogram, and use PCA to reduce the dimensionality of the dataset and visualize it in two dimensions. We might also use outlier detection to identify unusual purchase behaviour that could be indicative of fraud.

Data Cleaning and Preprocessing:

Data cleaning and preprocessing are essential steps in the machine learning pipeline that involve preparing data for analysis by transforming it into a suitable format. This involves cleaning up data quality issues, dealing with missing values, transforming variables, and scaling the data.

Some common techniques used in data cleaning and pre-processing include:

Handling missing values: filling in missing data points using methods such as mean imputation or regression imputation.

Feature scaling: scaling the data to ensure that all features have similar ranges, such as normalizing data to have a mean of zero and a standard deviation of one.

Feature transformation: transforming features to make them more suitable for analysis, such as applying a logarithmic transformation to a feature that has a skewed distribution.

Encoding categorical variables: converting categorical variables into numerical values that can be used in machine learning models, such as using one-hot encoding or label encoding.

Example:

Consider a dataset of housing prices. To prepare this dataset for machine learning analysis, we might handle missing values by imputing them with the mean value of the feature. We might scale the data using min-max scaling to ensure that all features have similar ranges. We might also transform the target variable using a logarithmic transformation to make it more normally distributed. Finally, we might encode categorical variables such as neighbourhoods using one-hot encoding.

Data Visualization Techniques

Data visualization is a method of displaying information graphically or visually. Data visualization techniques help in understanding data and making sense of it. It also helps in finding patterns, trends, and relationships within the data.

There are various techniques used for data visualization, such as:

Scatter plots: A scatter plot is a graph that shows the relationship between two variables by displaying data points on a two-dimensional plane.

Histograms: A histogram is a graph that shows how a dataset is distributed. It shows the number of observations that fall within various ranges.

Box plots: A box plot is a method for graphically displaying groups of numerical data. The box represents the middle 50% of the data, the line inside the box represents the median, and the whiskers represent the range of the data.

Heat maps: A heat map is a two-dimensional data visualization in which colours stand in for values. Heat maps are commonly used to represent gene expression data.

Python libraries such as Matplotlib, Seaborn, and Plotly can be used for data visualization.

Feature Extraction and Feature Selection:

Feature extraction is the process of selecting a subset of relevant features from a larger set of features to reduce the dimensionality of the data. Feature selection is the process of selecting a subset of features that are most relevant to the problem at hand.

There are various techniques used for feature extraction and feature selection, such as:

Principal Component Analysis (PCA): PCA is a method for reducing the dimensionality of a dataset by projecting it onto a lower-dimensional subspace.

Linear Discriminant Analysis (LDA): LDA is a method for feature extraction that maximizes the distance between classes while mini space distance within classes.

Recursive Feature Elimination (RFE): RFE is a method for feature selection that recursively eliminates features with the least importance until the desired number of features is reached.

SelectKBest: SelectKBest is a method for feature selection that selects the K best features based on a scoring function.

Python libraries such as Scikit-learn and PyTorch can be used for feature extraction and feature selection.

The example implementation of PCA in Python:

Python Code      

import numpy as np

from sklearn.decomposition import PCA

# Example  data

X = np. array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

# Initialize PCA object and fit data

pca = PCA(n_components=2)

pca.fit(X)

# Transform data to lower-dimensional space

X_transformed = pca.transform(X)

print(X_transformed)

An example implementation of SelectKBest in Python:

Python code

import numpy as np

from sklearn.feature_selection import SelectKBest, chi2

# Example data

X = np. array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

y = np.array([0, 1, 1])

# Initialize SelectKBest object anit to data

selector = SelectKBest(chi2, k=2)

selector.fit(X, y)

# Transform data to selected features

X_selected = selector.transform(X)

print(X_selected)

Previous ( Machine Language Syllabus)

                                                                               Continue (Supervised Learning)