Proving the Law of Total Expectation: E[X] E[E[XY]]

The law of total expectation is a fundamental concept in probability theory and is widely used in statistics, econometrics, and machine learning. The statement of this law asserts that the expected value of a random variable X can be expressed as the expected value of the conditional expectation of X given another random variable Y. Mathematically, this can be written as:

E[X] E[E[XY]]

In this article, we will provide a detailed proof of this law and explore its implications. We will cover both the continuous and discrete versions of this proof, along with a video explanation for the discrete case.

Continuos Case: Proof of E[X] E[E[XY]]

Let's start with the continuous case. We denote X and Y as random variables with joint probability density function fXY(x, y) and marginal probability density functions fX(x) and fY(y).

The law of total expectation can be proven as follows:

$$ E[E[XY]] int_{-infty}^{infty} E[XY mid Y y] f_Y(y) dy $$

Given that E[XY] is the expected value of the product of X and Y, we can express this as:

$$ E[XY mid Y y] int_{-infty}^{infty} x f_{XY}(x, y) dx $$

Substituting this back into our original equation, we get:

$$ E[E[XY]] int_{-infty}^{inft} left(int_{-infty}^{infty} x f_{XY}(x, y) dxright) f_Y(y) dy $$

We can now change the order of integration:

$$ E[E[XY]] int_{-infty}^{infty} int_{-infty}^{infty} x f_{XY}(x, y) f_Y(y) dx dy $$

Noting that fXY(x, y) fY(y) fXY|Yy (x | y) and fY(y) is a constant with respect to x, we can simplify the expression:

$$ E[E[XY]] int_{-infty}^{infty} int_{-infty}^{infty} x f_{XY}(x, y) dy dx $$

We can separate the integrations:

$$ E[E[XY]] int_{-infty}^{infty} x left(int_{-infty}^{infty} f_{XY}(x, y) dyright) dx $$

By definition, the inner integral is the marginal probability density function of X, denoted as fX(x):

$$ E[E[XY]] int_{-infty}^{infty} x f_X(x) dx $$

This is precisely the definition of the expected value E[X], thus completing the proof:

E[X] E[E[XY]]

Discrete Case: Proof of E[X] E[E[XY]]

Now let's consider the discrete case. Here, we use the joint probability mass function pXY(x, y) and marginal probability mass functions pX(x) and pY(y).

The proof in this case can be simplified using the definition of expected value for discrete random variables:

$$ E[E[XY]] sum_y sum_x xy p_{XY}(x, y) p_Y(y) $$

We can rewrite this expression by recognizing that the inner sum represents the conditional expectation of X given Y:

$$ E[E[XY]] sum_y p_Y(y) left(sum_x x p_{XY}(x, y)right) $$

This inner sum is the expected value of X given Y y, E[X|Yy]. Therefore:

$$ E[E[XY]] sum_y p_Y(y) E[X|Yy] $$

By the law of total expectation, the above expression is equal to E[X]:

$$ E[X] sum_x x p_X(x) $$

Thus, we have shown that in the discrete case as well, E[X] E[E[XY]] holds true.

Video Explanation and Example

If you prefer a more visual understanding, we have provided a video that explains the proof of the discrete version of the law of total expectation, complete with examples and step-by-step reasoning.

This video covers the key concepts, including the definition of conditional expectation, the manipulation of joint and marginal probability functions, and the application of the law of total expectation in practical scenarios.

Conclusion

The law of total expectation is a powerful tool in probability and statistics. Understanding its proof and application can greatly enhance your ability to analyze and model complex systems involving multiple random variables. Whether you are a student, researcher, or practitioner, this law serves as a foundational concept that underpins many advanced statistical techniques.

In addition to the proof and video, other resources such as textbooks and online tutorials can further aid in mastering this topic. For those looking to dive deeper, we recommend exploring topics such as conditional variance and law of total variance.