# 变分推断

## 资料

http://blog.huajh7.com/2013/03/06/variational-bayes/

PRML Chapter 10

MLAPP Chapter 21

## 背景知识

$H[p] = - \int p(x) \ln{p(x)} dx$

$KL(q || p) = - \int q(x) \ln{\frac{q(x)}{p(x)}} dx$

## 问题

• 一是$p(Z|X)$可能非常难求,而且不好做后续的处理.
• 二是$p(Z,X)$可能会很好求.

## 解决方案

• 如何比较两个分布是否相近. 相对熵.
• 如何从分布家族中选取一个接近$p(Z|X)$. 取相对熵最小的分布,转化成最优解问题. $\min_{Z}{KL(q(Z)||p(Z|X))}$

## 最优化问题

$\begin{eqnarray} KL(q(Z)||p(Z|X)) &=& \int q(Z) \ln \frac{q(Z)}{p(Z|X)} dZ \\ &=& \int q(Z) \ln \frac{q(Z)p(X)}{p(Z,X)} dZ \\ &=& \int q(Z) \ln \frac{q(Z)}{p(Z,X)} dZ + \int q(Z) \ln p(X) dZ \\ &=& \int q(Z) \ln \frac{q(Z)}{p(Z,X)} dZ + \ln p(X) \\ \end{eqnarray}$

$\ln p(X) = KL(q(Z)||p(Z|X)) + L(q)$

## 分解q(Z)

$q(Z) = \prod_{i=1}^{M} q_i(z_i)$

$\begin{eqnarray} L(q) &=& \int q(Z) \ln \frac{p(Z,X)}{q(Z)} dZ \\ &=& \int \prod_{j=1}^{M} q_j \ln \frac{p(Z,X)}{\prod_{j=1}^{M} q_j} dZ \\ &=& \int \prod_{j=1}^{M} q_j ( \ln p(Z,X) - \sum_{k=1}^{M} \ln q_k) dZ \\ \end{eqnarray}$

$\begin{eqnarray} L(q) &=& \int \prod_{j=1}^{M} q_j ( \ln p(Z,X) - \sum_{k=1}^{M} \ln q_k) dZ \\ &=& \int q_j \{ \int \prod_{i \neq j} q_i ( \ln p(Z,X) - \sum_{k=1}^{M} \ln q_k ) dz_i \} dz_j\\ &=& \int q_j \{ \int \prod_{i \neq j} q_i \ln p(Z,X) dz_i \} dz_j - \int \prod_i q_i \sum_k \ln q_k dz_i \\ \end{eqnarray}$

$\begin{eqnarray} \ln p_j(z_j) &=& \int \prod_{i \neq j} q_i \ln p(Z,X) dz_i + const \\ &=& \mathrm{E}_{i \neq j} [\ln p(Z, X)] + const \end{eqnarray}$

$\int \prod_i q_i \ln q_j dZ = \int q_j \ln q_j dz_j$

$\begin{eqnarray} L(q) &=& \int q_j \ln p_j dz_j - \sum_i \int q_i \ln q_i dz_i \\ &=& \int q_j \ln p_j dz_j - \int q_j \ln q_j dz_j - \sum_{i \neq j} \int q_i \ln q_i dz_i \\ &=& \int q_j \ln p_j dz_j - \int q_j \ln q_j dz_j + const \\ &=& \int q_j (\ln p_j - \ln q_j) dz_j + const \\ &=& \int q_j \ln \frac{p_j}{q_j} dz_j + const \\ &=& -KL(q_j || p_j) + const \\ \end{eqnarray}$

## 最大化$L(q)$

$\ln q_j(z_j) = \mathrm{E}_{i \neq j} [\ln p(Z, X)] + const$