Regression Algorithms and Examples

Applications: Predict Stock prices, house prices, sales, etc.

Linear Regression: 

This refers to a linear relationship between two or more variables. If we draw this relationship in a two-dimensional space (between two variables), we get a straight line.
Y = 𝛉1 + 𝛉2X
where X is the explanatory variable 
           Y is the dependent variable. 
            The slope of the line is 𝛉2, and 𝛉1 is the intercept (the value of Y when X = 0).

Disadvantage:  It assumes that a single straight line is appropriate as a summary of the data.
Example: Linear Regression in R
                    Linear Regression Model using Boston Housing Dataset

Ordinary Least Square (OLS):

Ordinary Least Squares (OLS) is one of the methods to find the best fit line for a dataset using linear regression. It calculates the difference between the predicted and the actual value, squares it, and repeats this step for all data points. A sum of all the errors is computed. This sum is the overall representation of how accurate the model is. 

Next, the parameters of the model are tweaked such that this squared error is minimized until there is no scope for improvement.

Disadvantage: Sensitivity to Outliers. 
                               Data has to be normally distributed. 
Example: OLS Model using Boston Housing Dataset

Polynomial Regression:

For most of the datasets, the data cannot be summarized by a straight line. It results in Underfitting. Now, to overcome underfitting, we need to increase the complexity of the Model. So, instead of representing the data by a straight line, we can generate a higher-order equation by adding powers to the original features as new features.
Thus, Y = 𝛉1 + 𝛉2X can be transformed into 
           Y = 𝛉1 + 𝛉2X + 𝛉3X²

Disadvantage: 

- The presence of one or two outliers in the data can seriously affect the results of the nonlinear analysis.
- There are fewer model validation tools for the detection of outliers in the nonlinear regression model.

Example: Polynomial Regression Model Using Boston Housing Dataset

Gradient Descent Regression:

Gradient Descent Regression uses gradient descent to optimize the model.  Gradient Descent Function aims at minimizing the value of the cost function J(𝛉1,𝛉2). We will start with some value of  𝛉1 and 𝛉2 and keep on changing the values until we get the Minimum value of J(𝛉1,𝛉2) i.e. best fit for the line that passes through the data points.
Gradient Descent equation for a linear regression model is as below:
θ0:=θ1:=}θ0−α1m∑i=1m(hθ(xi)−yi)θ1−α1m∑i=1m((hθ(xi)−yi)xi)

Disadvantage:
If the learning rate for gradient descent is too fast, we are going to skip the true local minimum to optimize for time. If it is too slow, the gradient descent may never converge.
Example: Gradient Descent Model in R



Decision Tree Regression:
A Decision Tree is mostly used for a classification problem. But,Decision trees can also be applied to regression problems, using DecistionTreeRegressor Class. 
The decision trees is used to fit a sine curve with addition to noisy observation. As a result, it learns local linear regressions approximating the sine curve.

We can see that if the maximum depth of the tree (controlled by the max_depth parameter) is set too high, the decision trees learn too fine details of the training data and learn from the noise, i.e. they overfit.

Disadvantage: Overfit: Overfitting is one of the most practical difficulties for decision tree models. This problem gets solved by the use of random forests.

Example: Decision Tree Regression Model using Boston Housing Dataset