Section: 17 🔖 Linear Regression
- Lecture slides:
Linear regression is a linear approach to modeling the relationship between a numerical response (\(Y\)) and one or more independent variables (\(X_i\)).
Usually in linear regression, models are used to predict only one scalar variable. But there are two subtype if these models: - First when there is only one explanatory variable and one output variable. This type of linear regression model known as simple linear regression. - Second, when there are multiple predictors, i.e. explanatory/dependent variables for the output variable. This type of linear regression model known as multiple linear regression.
Linear models fitted to various different type of data spread. This illustrates the pitfalls of relying solely on a fitted model to understand the relationship between variables. Credits: Wikipedia.
17.1 Linear regression using lm() function
Syntax for building the regression model using the lm() function is as follows:
lm(formula, data, ...)- formula: here we mention the prediction column and the other related columns(predictors) on which the prediction will be based on.
prediction ~ predictor1 + predictor2 + predictor3 + ...
- data: here we provide the dataset on which the linear regression model is to be trained.
- formula: here we mention the prediction column and the other related columns(predictors) on which the prediction will be based on.
For more info on the lm() function visit lm()
Lets look at the example on the Moody dataset.
| Midterm | Project | FinalExam | ClassScore |
|---|---|---|---|
| 73 | 8 | 70 | 39.60000 |
| 61 | 100 | 20 | 68.20000 |
| 58 | 88 | 38 | 67.00000 |
| 93 | 41 | 46 | 52.47565 |
| 85 | 52 | 85 | 68.50000 |
| 97 | 48 | 19 | 49.10000 |
| 26 | 59 | 22 | 41.30000 |
| 58 | 62 | 25 | 50.10000 |
| 53 | 56 | 27 | 46.70000 |
| 66 | 27 | 17 | 34.80494 |
Now we can build a simple linear regression model to predict the ClassScore attribute based on the various other attributes present in the dataset, as shown above.
Since we will be predicting only one attribute values, this model will be called simple linear regression model.
17.1.1 Snippet 1: How much do Midterm, Project and Final Exam count?
We can see that,
- The summary of the lm model give us information about the parameters of the model, the residuals and coefficients, etc.
- The predicted values are obtained from the predict function using the trained model and the test data.
17.2 Calculating the Error using mse()
As was the simple case in the categorical predictions of the classification models, where we could just compare the predicted categories and the actual categories, this type of direct comparison as an accuracy test won’t prove useful now in our numerical predictions scenario.
We don’t want to eyeball every time we predict, to find the accuracy of our predictions each row by row, so lets see a method to calculate the accuracy of our predictions, using some statistical technique.
To do this we will use the Mean Squared Error(MSE).
- The MSE is a measure of the quality of an predictor/estimator
- It is always non-negative
- Values closer to zero are better.
The equation to calculate the MSE is as follows:
\[\begin{equation} MSE=\frac{1}{n} \sum_{i=1}^{n}{(Y_i - \hat{Y_i})^2} \\ \text{where $n$ is the number of data points, $Y_i$ are the observed value}\\ \text{and $\hat{Y_i}$ are the predicted values} \end{equation}\]
To implement this, we will use the mse() function present in the Metrics Package, so remember to install the Metrics package and use library(Metrics) in the code for local use.
The syntax for mse() function is very simple:
mse(actual,predicted)- actual: vector of the actual values of the attribute we want to predict.
- predicted: vector of the predicted values obtained using our model.
17.3 Snippet 2: Cross Validate your prediction
We can see that,
- The summary of the lm model give us information about the parameters of the model, the residuals and coefficients, etc.
- The predicted values are obtained form the predict function using the trained model and the test data. In comparison to the previous model we are using the cross validation technique to check that if we have more accurate predictions, thus increasing the overall accuracy of the model.