# Expectation, Variance, Covariance

## Expectation

### Definition

If $X$ is a random variable with output space $H$ and with  density $f(x)$.

Then, $$E[X]=\int_H x f(x) dx$$

### Linearity

#### Sum of random variables

If $X$ and $Y$ are two random variables with respective outputs spaces $H$ and $K$, with respective densities $f$ and $g$, with a joint density $f(x,y)$ and with a finite expectation. 

We have $$E[X+Y]=E[X]+E[Y]$$

##### Idea of the proof

Let consider the function $q$ defined by $q(x,y)=x+y$

$E[q(X,Y)] = \int_{H}\int_{K} q(x,y) f(x,y)dx dy$

$\phantom{E[Z]} = \int_{H}\int_{K} (x+y) f(x,y)dx dy$

$\phantom{E[Z]} = \int_{H}\int_{K} x f(x,y)dx dy + \int_{H}\int_{K} y f(x,y)dx dy$

$\phantom{E[Z]} = \int_{H} x\int_{K} f(x,y)dy dx + \int_{K} y\int_{H}  f(x,y)dx dy$

$\phantom{E[Z]} = \int_{H} xf(x) dx + \int_{K} y(y)g(y)dy$

$\phantom{E[Z]} = E[X]+E[Y]$

#### Product by a constant

We also have

$$E[aX]=aE[X]$$

### Positivity

If $X$ is a positive random variable i.e $$P(X\geq 0)=1$$ then 

$$E[X]\geq 0$$

Where $X$ is a positive random variable, if $E[X]=0$, then 

$$P(X=0)=1$$

### Constant

$$E[a] = a$$

## Covariance and Variance

### Definition

$$\textrm{Cov}(X,Y) = E[(X-E[X])(Y-E[Y])]$$

### Properties 

#### Symmetry

$$\textrm{Cov}(X,Y)=\textrm{Cov}(Y,X)$$

##### Other expression

Sometimes more convenient

$$\textrm{Cov}(X,Y) = E[XY] - E[X]E[Y]$$

In particular

$$\textrm{Var}(X)  = \textrm{Cov}(X,X) = E[X^2] - E[X]^2$$

#### Linearity

$$\textrm{Cov}(aX+bY,Z) = a\textrm{Cov}(X,Z)+b\textrm{Cov}(Y,Z)$$

#### Variance and covariance

$$\textrm{Cov}(X,X) = E[(X-E[X])(X-E[X])]=\textrm{Var}(X)$$

Note that the variance is always positive (as the expectation of a square of a random variable).

#### Covariance between a variable and a constant

$$\textrm{Cov}(X,a) = 0$$

Consequence :

$$\textrm{Var}(a)=0$$

Reciprocally, if a random variable has a variance equal to 0, then the variable is constant.

#### Variance of a linear combination

$$\textrm{Var}\left(\sum_{i=1}^n\lambda_i Z_i\right) = \sum_{i=1}^n\sum_{j=1}^n \lambda_i\lambda_j\textrm{Cov}(Z_i,Z_j)$$

##### Applications

$$\textrm{Var}(aX) = a^2 \textrm{Var}(X)$$

$$\textrm{Var}(aX+bY)=a^2\textrm{Cov}(X,X)+2ab\textrm{Cov}(X,Y)+b^2\textrm{Cov}(Y,Y)$$

$$\textrm{Var}(aX-bY)=a^2\textrm{Cov}(X,X)-2ab\textrm{Cov}(X,Y)+b^2\textrm{Cov}(Y,Y)$$

$$\textrm{Var}(X+a) = \textrm{Var}(X)$$


### Covariance matrix

When we have a set of random variables $Z_1,\dots,Z_n$.

For each pair $(k,l)$, if we denote $$c_{kl} = \textrm{Cov}(Z_k,Z_l)$$

We can store the $c_{kl}$'s in a matrix $$\Sigma = \left[\begin{array}{ccc}c_{11} &\dots & c_{1n}\\
c_{21} & \dots & c_{2n}\\
\vdots & \ddots & \vdots\\
c_{n1} & \dots & c_{nn}\end{array}\right]$$

$\Sigma$ is named the covariance matrix of the random vector $$Z=\left[\begin{array}{c}Z_1\\ \vdots\\ Z_n\end{array}\right]$$

Note that we can rewrite 

$$\textrm{Var}\left(\sum_{i=1}^n\lambda_iZ_i\right) = \lambda^T \Sigma \lambda$$

where $$\lambda =\left[\begin{array}{c}\lambda_1\\ \vdots\\\lambda_n\end{array}\right]$$

and $^T$ designates the transposition

$$\lambda^T =\left[\begin{array}{ccc}\lambda_1& \dots & \lambda_n\end{array}\right]$$

Since a variance is always positive, the variance of any linear combination as to be positive. Therefore, a covariance matrix is always (semi-)positive definite, i.e

For each $\lambda$ $$\lambda^T \Sigma \lambda\geq 0$$

#### Cross-covariance matrix

Let consider two random vectors $X=(X_1,\dots,X_n)$ and $Y=(Y_1,\dots,Y_p)$.

We can consider the cross-covariance matrix $\textrm{Cov}(X,Y)$ where element corresponding to the row $i$ and the column $j$ is $\textrm{Cov}(X_i,Y_j)$

If $A$ and $B$ are some matrices (of constants)

$$\textrm{Cov}(AX,BY) = A\textrm{Cov}(X,Y)B^T$$

#### Exercise

Suppose that we want to estimate a quantity modeled by a random variable $Z_0$ as a linear combination of known quanties
$Z_1,\dots, Z_n$ stored in a vector $$Z=\left[\begin{array}{c}Z_1\\ \vdots\\ Z_n\end{array}\right]$$

We will denote $$Z_0^\star = \sum_{i=1}^n \lambda_i Z_i = \lambda^T Z$$ this (random) estimator.

We know the covariance matrix of the full vector $(Z_0,Z_1,\dots,Z_n)$ that we write with blocks for convenience:

$$\left[\begin{array}{cc}\sigma_0^2 & c_0^T \\
c_0 & C\end{array}\right]$$


where 

* $\sigma^2_0 = \textrm{Var}(Z_0)$
* $c_0 = \textrm{Cov}(Z,Z_0)$
* $C$ is the covariance matrix of $Z$.

Compute the variance of the error $$Z_0^\star-Z_0$$

##### Solution

$\textrm{Var}(Z_0^\star-Z_0) = \textrm{Cov}(Z_0^\star-Z_0,Z_0^\star-Z_0)$

$\phantom{\textrm{Var}(Z_0^\star-Z_0)} = \textrm{Var}(Z_0) -2 \textrm{Cov}(Z_0^\star,Z_0) + \textrm{Var}(Z_0)$

$\phantom{\textrm{Var}(Z_0^\star-Z_0)} = \textrm{Var}(\lambda^TZ) -2 \textrm{Cov}(\lambda^T Z,Z_0) + \sigma_0^2$

$\phantom{\textrm{Var}(Z_0^\star-Z_0)} = \lambda^T\textrm{Var}(Z)\lambda -2 \lambda^T\textrm{Cov}( Z,Z_0) + \sigma_0^2$

$\phantom{\textrm{Var}(Z_0^\star-Z_0)} = \lambda^TC\lambda -2 \lambda^Tc_0 + \sigma_0^2$

### Correlation coefficient

The covariance is a measure of the link between two variables. However it depends on the scale of each variable. To have a similar measure which is invariant by rescaling, we can use the correlation coefficient:

$$\rho(X,Y)=\frac{\textrm{Cov}(X,Y)}{\sqrt{\textrm{Var}(X)\textrm{Var}(Y)}}$$

When the correlation coefficient is equal to $1$ or $-1$, we have

$$Y=aX+b$$ 

with 

* $a>0$ if $\rho(X,Y)=1$ 
* $a<0$ if $\rho(X,Y)=-1$

Note that $\rho(X,Y)$ can be equal to $0$ even if the variables are strongly linked.

The usual example is a variable $X$ with a pair density ($f(-x)=f(x)$) and $Y=X^2$:

$$\textrm{Cov}(X,Y)=\textrm{Cov}(X,X^2)=E[X^3]-E[X]E[X^2]=E[X^3]=\int_{\mathbb{R}} x^3f(x)dx =0$$