It is one of the machine learning techniques that fall under supervised learning. Linear regression is an algorithm that provides a linear relationship between an independent variable and a dependent variable to predict the outcome of future events. It is a statistical method used in data science and machine learning for predictive analysis.

Linear Regression is generally classified into two types:

In Simple Linear Regression, we try to find the relationship between a single independent variable (input) and a corresponding dependent variable (output). This can be expressed in the form of a straight line.

Mathematically, we can represent a simple linear regression as:

intercept of the line (Gives an additional degree of freedom)

Linear regression coefficient (scale factor to each input value).

In Multiple Linear Regression, we try to find the relationship between 2 or more independent variables (inputs) and the corresponding dependent variable (output).

The independent variables can be continuous or categorical.

What is Linear Regression

Random forest is a supervised machine learning algorithm that is used for both classification and regression problems. It is an ensemble learning algorithm, which means that it creates a model by combining the predictions of multiple decision trees.

A random forest is a collection of decision trees. Each decision tree is trained on a different subset of the data, and each tree makes a prediction. The final prediction of the random forest is the majority vote of the decision trees.

variable selection criterion in Random Forest can be done some measure of impurity via two approaches:

Gini impurity is a measure of how mixed a dataset is.

where p is the probability of a data point in the dataset belonging to a particular class.

The Gini impurity is used to determine which feature to split on in a decision tree. The feature with the highest Gini impurity is the one that will most effectively split the dataset into two groups.

Information gain is another measure of how mixed a dataset is.

where H(T) is the entropy of the dataset T, and H(T|A) is the conditional entropy of the dataset T given the feature A.

The information gain is used to determine which feature to split on in a decision tree. The feature with the highest information gain is the one that will most effectively reduce the entropy of the dataset.

The random forest algorithm works by first creating a set of decision trees. Each decision tree is created by randomly selecting a subset of features from the dataset. The decision trees are then grown to their maximum depth, and the predictions of the trees are averaged to produce the final prediction.

The Random Forest algorithm works as follows:

Randomly select “k” features from the total “m” features.

Among the “k” features, calculate the node “d” using the best-split point.

Split the node into daughter nodes using the best split.

Repeat steps 1 to 3 until the “l” number of nodes has been reached.

Build a forest by repeating steps 1 to 4 for “n” number of times to create “n” number of trees.

The Random Forest algorithm uses two types of randomness when building each tree:

A random sampling of training data points when building trees.

Random subsets of features are considered when splitting nodes.

Random forest is a very accurate machine learning algorithm. It is often used for tasks where accuracy is critical, such as fraud detection and medical diagnosis.

Random forest is a robust algorithm that is not easily affected by noise or outliers in the data. This makes it a good choice for problems where the data is not perfectly clean.

Random forest can be used to calculate the importance of each feature in the dataset. This information can be used to improve the model by removing unimportant features or by weighting the features more heavily in the model.

Random forest is a scalable algorithm that can be used to solve problems with large datasets.

Random forest is not as interpretable as some other machine learning algorithms, such as decision trees. This can make it difficult to understand why the model makes the predictions that it does.

Random forests can be computationally expensive to train, especially for large datasets.

Random forests can be used to classify data into different categories. For example, it can be used to classify images as spam or ham or to classify customers as likely to churn or not.

 Random forest can be used to predict a continuous value. For example, it can be used to predict the price of a house or the amount of sales that a company will make.

 Random forest can be used to detect outliers in the dataset. Outliers are data points that are significantly different from the rest of the data. They can be caused by errors in the data collection process, or they can be genuine data points that are just outside the normal range

Random Forest

A decision tree is a flowchart-like tree structure where each internal node denotes the feature, branches denote the rules and the leaf nodes denote the result of the algorithm. 

machine learning algorithm that can be used for both regression and classification tasks.

Classification trees are applied to data when the outcome is discrete in nature or is categorical such as the presence or absence of students in a class, a person who died or survived, approval of a loan etc.

Regression trees are used when the outcome of the data is continuous in nature such as prices, age of a person, length of stay in a hotel, etc

The goal of the algorithm is to predict a target variable from a set of input variables. The algorithm works by recursively splitting the data into subsets based on the values of the input features. At each node of the tree, the algorithm selects the feature that best separates the data into different classes.

variable selection criterion in Decision Trees can be done some measure of impurity via two approaches:

Entropy is the measure of impurity or disorder in a set of examples. In decision trees, entropy is used to control how the tree decides to split the data. The goal is to minimize the entropy at each node of the tree by selecting the feature that best separates the data into different classes.

Suppose we have a dataset with two classes, A and B. There are 10 examples in class A and 10 examples in class B. The entropy of this dataset is calculated as follows:



Entropy = - (10/20) * log2(10/20) - (10/20) * log2(10/20) = 1

Now suppose we split the dataset based on a feature X into two subsets. Subset 1 contains 5 examples of class A and 5 examples of class B. Subset 2 contains 5 examples of class A and 5 examples of class B. The entropy of each subset is:

Entropy(subset1) = - (5/10) * log2(5/10) - (5/10) * log2(5/10) = 1
Entropy(subset2) = - (5/10) * log2(5/10) - (5/10) * log2(5/10) = 1

The entropy here is 1. This is considered a high entropy, a low level of purity. 

is a measure of the reduction in entropy or Gini index achieved by splitting the data based on a particular feature. The goal is to select the feature that maximizes the information gain. 

The information gain from splitting on feature X is:

Information Gain = Entropy(parent) - [Weighted Average]Entropy(children)

This means that splitting on feature X does not reduce the entropy of the dataset, so it is not a good feature to split on.

It's another measure of impurity used in decision trees. It is similar to entropy but is based on the probability of misclassification. The Gini index is calculated as follows:

How Gini index is used in decision trees:

where p1 and p2 are the probabilities of classes 1 and 2, respectively. The Gini index ranges from 0 to 1, with 0 indicating perfect classification and 1 indicating complete randomness.

Suppose we have a dataset with two classes, A and B. There are 10 examples in class A and 10 examples in class B. The Gini index of this dataset is calculated as follows:

Gini Index = 1 - (10/20)^2 - (10/20)^2 = 0.5

Now suppose we split the dataset based on a feature X into two subsets. Subset 1 contains 5 examples of class A and 5 examples of class B. Subset 2 contains 5 examples of class A and 5 examples of class B. The Gini index of each subset is:

Gini Index(subset1) = 1 - (5/10)^2 - (5/10)^2 = 0.5
Gini Index(subset2) = 1 - (5/10)^2 - (5/10)^2 = 0.5

The information gain from splitting on feature X using the Gini index is:

Information Gain = Gini Index(parent) - [Weighted Average]Gini Index(children)

This means that splitting on feature X does not reduce the Gini index of the dataset, so it is not a good feature to split on.

Begin the tree with the root node, which contains the complete dataset.

Find the best attribute in the dataset using Entropy and Gini Index.

Divide the dataset into subsets that contain possible values for the best attributes.

Generate the decision tree node, which contains the best attribute.

Recursively repeat steps 2-4 on each subset until all data is classified.

I also like to write

Check out my latest posts!