Linear Regression Error Analysis

Linear Regression error Analysis: 

As mentioned earlier that there is an error in the solution obtained by a linear regression method.

We are assuming input from p-dimensional space ( p number of the features) and mapping it to R (real numbers).

We can write linear regression simply as $$\hat{\beta} = (X^TX)^{-1}X^Ty$$ $$f(X) = \beta_0 + \Sigma_{j=1}^{p} X_j\beta_j$$

Now Residual sum of squares is given by $$RSS(\beta) = \Sigma_{i=1}^N (y_i - f(x_i))^2$$ $$= \Sigma(y_i-\beta_0 - \Sigma_{j=1}^{p} x_{ij}\beta_j)^2$$

Here it is assumed that input data points are also independent of each other.

Now, the aim is to minimize the error in the result. Let X denote the matrix of order Nx(p+1) where N is the number of data points, p is the number of the features and 1 is added because of intercept, and y is N-vector of outputs.

Hence we can write Residual sum of squares as  $$RSS(\beta) = (y-X\beta)^T(y-X\beta)$$

Differentiate it with respect to β, we will obtain $$\frac{\partial RSS}{\partial \beta} = -2X^T(y-X\beta)$$ $$\frac{\partial^2 RSS}{\partial \beta \partial \beta^T} = X^TX$$

So a unique solution to minimize the error is given by finding β, which is obtained by putting the first derivative equal to zero. So the solution now is given by,  $$\hat{\beta} = (X^TX)^{-1}X^Ty$$