Nonconjugate priors

Generalized linear models

$Y|x \sim Poisson(\theta_x),\;\theta_x > 0 \Leftrightarrow -\infin<log\theta_x<\infin$

$f(\theta_x):log\theta_x=\beta_1 +\beta_2x+\beta_3x^2$

                $\theta_x = exp(\beta_1+\beta_2x+\beta_3x^2) \ge0$

$Y:0 \; or\; 1$, x has no interact with Y

$Y|x \sim Bernoulli(\theta_x), 0 \le \theta_x \le 1$

$f(\theta_x):log(\theta_x)=log(\frac{\theta_x}{1-\theta_x})=\beta_1+\beta_2x+\beta_3x^2$

$0\le\theta_x-\frac{exp(\beta^T_x)}{1+exp(\beta^T_x)}\le 1$

The Metropolis algorithm

In most cases, the problems were occurred in sampling model $Y\sim p(y|\theta)$ and a prior distribution $p(\theta)$. Although in most problems $p(y|θ)$ and $p(θ)$ can be calculated for any values of $y$ and $θ$, $p(θ|y) = p(θ)p(y|θ)/ \int p(θ ^{'} )p(y|θ ^{'} ) dθ^{'}$ is often hard to calculate due to the integral in the denominator. If we were able to sample from $p(θ|y)$, then we could generate $θ^{(1)}, . . . , θ^{(S)} ∼ i.i.d$. $p(θ|y)$ and obtain Monte Carlo approximations to posterior quantities, such as

$$ E[g(θ)|y] ≈ \frac{1}{S} \sum_{ s=1}^S g(θ^{(s)} ). $$

If $p(θ^∗|y) > p(θ^{(s)}|y)$ then we want more $θ^∗$’s in the set than $θ(s)$’s. Since $θ(s)$ is already in the set, hen it seems we should include $θ^∗$ as well. On the other hand, if $p(θ^∗|y) < p(θ^{(s)}|y)$ then it seems we should not necessarily include $θ^∗$. So perhaps our decision to include $θ^∗$ or not should be based on a comparison of $p(θ^∗|y)$ to $p(θ^{(s)}|y)$.

$$ r = \frac{p(θ^∗|y)}{p(θ^{(s)}|y)}=\frac{p(y|θ^∗)p(θ^∗)}{p(y)}\frac{p(y)}{p(y|θ^{(s)})p(θ^{(s)})}=\frac{p(y|θ^∗)p(θ^∗)}{p(y|θ^{(s)})p(θ^{(s)})}. $$