Understanding the Covariance of Estimated Coefficients and Residuals in Linear Regression

In the realm of linear regression, understanding the relationships between different components is crucial. This article will delve into the covariance of two key expressions: Estimate coefficients (hat{beta}) and unexplained variance (yI - XX^{-1}X). Through a step-by-step explanation and the application of properties from regression analysis, we'll explore the underlying mathematical principles and their practical implications.

Introduction to Covariance and Linear Model

Covariance is a statistical measure describing the relationship between two random variables. In the context of a linear regression model, (y) represents the response variable, and (X) is the design matrix. The estimated coefficients (hat{beta}) and the residuals provide valuable insights into how well the model fits the data.

Key Definitions

Covariance: The covariance between two random variables (A) and (B) is defined as:

[ text{cov}A B E[AB] - E[A]E[B] ]

Linear Model: In a linear regression context, with (y) being the response variable vector of size (n times 1), (X) is the design matrix of size (n times p), (hat{y} XX^{-1}Xy) is the vector of predicted values, and (e y - hat{y}) is the vector of residuals.

Expressions and Notations

We need to evaluate the covariance between two expressions: (hat{beta} XX^{-1}Xy) and (yI - XX^{-1}X).

First Expression

(hat{beta} XX^{-1}Xy) is the estimated coefficients in a linear regression model.

Second Expression

(yI - XX^{-1}X y - hat{y}) represents the unexplained variance, also known as the residuals of the model.

Covariance Calculation

To compute the covariance, let's denote:

(A XX^{-1}Xy) (B yI - XX^{-1}X)

The covariance is given by:

[ text{cov}(A, B) E[AB] - E[A]E[B] ]

Step 1: Calculate E[A]

[ E[A] E[XX^{-1}Xy] XX^{-1}XE[y] XX^{-1}XXbeta beta ]

where (beta) is the true coefficient vector.

Step 2: Calculate E[B]

[ E[B] E[yI - XX^{-1}X] E[yy] - E[yXX^{-1}Xy] ]

The first term (E[yy]) is the variance of (y), and the second term relates to the error structure.

Step 3: Calculate E[AB]

This stage involves expanding (A) and (B) and using properties of expectations and variances.

Conclusion

Under the assumptions of the classical linear regression model (normality, homoscedasticity, etc.), the covariance (text{cov}(XX^{-1}Xy, yI - XX^{-1}X)) is generally zero. This is because the estimated coefficients (XX^{-1}Xy) are derived from the projection of (y) onto the column space of (X), while (yI - XX^{-1}X) captures the variation in (y) not explained by the model.

In summary, this indicates that the estimated coefficients and unexplained variance are uncorrelated in the context of a linear regression model.