The linear regression model

The normal linear regression model specifies that, in addition to $E[y|x]$ being linear, the sampling variability around the mean is $i.i.d.$ from a normal distribution:

$$ \epsilon_1,\cdots \epsilon_n\sim i.i.d. \;normal(0, \sigma^2) \\ Y_i = \beta^T x_i=\epsilon_i $$

This model provides a complete specification of the joint probability density of observed data$y_1,\cdots, y_n$ conditional upon $x_1,\cdots x_n$ and values of $\beta$ and $\sigma^2$

$$ p(y_1 \cdots y_n|x_1,\cdots, x_n, \beta, \sigma^2) \\ = \prod_{i=1}^n p(y_i|x_i,\beta,\sigma^2)\;\;\;\;\;\;\;\;\;\;\; \\ \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; =(2\pi\sigma^2)^{-n/2} exp \left\{ -\frac{1}{2\sigma^2} \sum_{i=1}^n (y_i-\beta^T x_i)^2 \right\} \\ $$

$x=(x_{i1}, x_{i2}, \cdots x_{ip})^T$

$y_i = \beta^Tx_i+\epsilon_i,\;\;\epsilon_i \sim N(0,\sigma^2)$

$\begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} = \begin{bmatrix} \beta^T x_1 \\ \beta^T x_2 \\ \vdots \\ \beta^T x_n \end{bmatrix} + \begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_n \end{bmatrix} = \begin{bmatrix} \beta_1 x_{11} +\beta_1 x_{11} + \cdots + \beta_p x_{1p} \\ \beta_1 x_{21} +\beta_2 x_{22} + \cdots + \beta_p x_{2p} \\ \vdots \\ \beta_n x_{n1} +\beta_2 x_{n2} + \cdots + \beta_p x_{np} \end{bmatrix} + \epsilon$

                                            $= \begin{bmatrix}x_{11}  & \cdots & x_{1p} \\ 

\vdots & \ddots & \vdots \\ x_{n1} & \cdots & x_{np} \end{bmatrix} + \epsilon$

                                            $\Rightarrow Y = X \beta + \epsilon, \;Y \sim N(X\beta, \sigma^2 I )$쏘

To find the value of $\beta$ at whcih this minimum occurs it is helpful to rewrite $SSR(\beta )$ in matrix motation:

$$ SSR(β) = \sum_{i=1}^n

(y_i − β^ T x_i)^ 2 = (y − Xβ)^ T (y − Xβ) \\ \; = y^Ty-2\beta^TX^Ty+\beta^TX^TX\beta $$

Recall from calculus that

  1. a minimum of a function $g(z)$ occurs at a value $z$ such that $\frac{d}{d(z)}g(z)=0$
  2. the derivative of $g(z)=az$ is $a$ and the derivative of $g(z)=bz^2$ is $2bz$.