Machine Learning Quizzes

Quiz 1

Question

$y=Ax+v$, where $v$ is a Gaussian noise.

1. What is the optimal solution for $x$?
2. What is the optimal solution for $x$ if $v\sim \mathcal{N}(0,R)$?
3. What is the optimal solution for $x$ if $v\sim \mathcal{N}(0,R)$ and $X\sim N(0,aI)$?
4. $A$ and $X$ are unknown, what is the optimal solution for $x$?

Answer

1. $J(x)=\frac{1}{2}(y-Ax{)}^T(y-Ax)$, $\frac{\partial J}{\partial x}=0$, $x=(A^{T}A{)}^{-1}{A}^{T}y$
2. $J(x)=\frac{1}{2}(y-Ax{)}^T{R}^{-1}(y-Ax)$, $\frac{\partial J}{\partial x}=0$, $x=(A^T{R}^{-1}A{)}^{-1}{A}^T{R}^{-1}y$
3. $J(x)=\frac{1}{2}(y-Ax{)}^T{R}^{-1}(y-Ax)+\frac{1}{2}{x}^T(aI{)}^{-1}x$, $\frac{\partial J}{\partial x}=0$, $x=(A^T{R}^{-1}A+aI{)}^{-1}{A}^T{R}^{-1}y$
4. We can distinguish two cases:
   1. For x: $J(x)=\frac{1}{2}(y-Ax{)}^T{R}^{-1}(y-Ax)+\frac{1}{2}{x}^T(aI{)}^{-1}x$, $\frac{\partial J}{\partial x}=0$, $x=(A^T{R}^{-1}A+aI{)}^{-1}{A}^T{R}^{-1}y$
   2. For A: $Y^T=X^T{A}^T$ $J(A)=\frac{1}{2}(Y-XA{)}^T{R}^{-1}(Y-XA)$, $\frac{\partial J}{\partial A}=0$, $A^T=(X{R}^{-1}{X}^T{)}^{-1}X{R}^{-1}{Y}^T$

Quiz 2

Question

$Y=AX+\omega$, where $\omega \sim \mathcal{N}(0,Q)$ and $X\sim \mathcal{N}({\mu }_0,{\mathrm{\Sigma }}_0)$

1. What is $p(Y|X)$?
2. What is $p(Y)$?
3. What is $p(X|Y)$?
4. What is $p(Y^{\prime }|Y)$?

Answer

1. $p(Y|X)\sim \mathcal{N}(AX,Q)$ We regard $X$ as a constant under conditional probability.
2. $p(Y)\sim \int p(Y|X)p(X)dx\sim \mathcal{N}(A{\mu }_0,A{\mathrm{\Sigma }}_0{A}^T+Q)$.$$var[Y]=var[AX]+var[\omega ]=A{\mathrm{\Sigma }}_0{A}^T+Q$$
3. Assume that $p(X|Y)\sim \mathcal{N}(m,L)$, then we can use the equality of quadratic from to solve the problems.
   1. $p(X|Y)\sim p(Y|X)p(X)=\mathcal{N}(Y|AX,Q)\mathcal{N}(X|{\mu }_0,{\mathrm{\Sigma }}_9)$
   2. $-\frac{1}{2}(x-m{)}^T{L}^{-1}(x-m)\propto -\frac{1}{2}(y-Ax{)}^T{Q}^{-1}(y-Ax)-\frac{1}{2}(x-{\mu }_0{)}^T{\mathrm{\Sigma }}_0^{-1}(x-{\mu }_0)$
   3. We can get the result:
   $$\begin{array}{rl}{L}^{-1}& =A^T{Q}^{-1}A+{\mathrm{\Sigma }}_0^{-1}\\ L^{-1}m& =A^T{Q}^{-1}y+{\mathrm{\Sigma }}_0^{-1}{\mu }_0\end{array}$$
   4. By applying $[A+BCD{]}^{-1}=A^{-1}-A^{-1}B[C^{-1}+D{A}^{-1}B{]}^{-1}D{A}^{-1}$
   $$\begin{array}{rl}L& =(I-KA){\mathrm{\Sigma }}_0\\ m& ={\mu }_0+K(y-A{\mu }_0)\end{array}$$where $K={\mathrm{\Sigma }}_0{A}^T(A^T{\mathrm{\Sigma }}_{0}A+Q{)}^{-1}$
4. $p(Y^{\prime }|Y)\sim \int p(Y^{\prime }|X)p(X|Y)dx\sim \mathcal{N}(Am,AL{A}^T+Q)$. The same format as question 2.

Quiz 3

TIP

1. Learning: $p(\theta |\mathcal{D})\propto p(\mathcal{D}|\theta )p(\theta )$
2. Prediction: $p({\mathcal{D}}^{new}|\mathcal{D})=\int p({\mathcal{D}}^{new}|\theta )p(\theta |\mathcal{D})d\theta$
3. Evaluation: $p(\mathcal{D})=\int p(\mathcal{D}|\theta )p(\theta )d\theta$

Question

Given $t=\mathrm{\Phi }(x)\omega +v$ where $\mathrm{\Phi }(x)=[1,x,x...,x^M]$ and $v\sim \mathcal{N}(0,{\beta }^{-1})$, $\mathcal{D}=\{[x_1,...,x_N],[t_1,...,t_N]\}$

1. What is the solution of ${\omega }_{ML}$?
2. What is the solution of ${\omega }_{MAP}$ if $\omega \sim \mathcal{N}(0,{\alpha }^{-1}I)$?
3. What is the predictive distribution if ${\mathcal{D}}^{new}=\{x^{new},t^{new}\}$?
4. What is the model evaluation?

Answer

1. $J(\omega )=\frac{\beta }{2}(T-\mathrm{\Phi }\omega {)}^T(T-\mathrm{\Phi }\omega )\to {\omega }_{ML}=({\mathrm{\Phi }}^T\mathrm{\Phi }{)}^{-1}{\mathrm{\Phi }}^{T}T$
2. $J(\omega )=\frac{\beta }{2}(T-\mathrm{\Phi }\omega {)}^T(T-\mathrm{\Phi }\omega )+\frac{\alpha }{2}{\omega }^T\omega \to {\omega }_{MAP}=(\beta {\mathrm{\Phi }}^T\mathrm{\Phi }+\alpha I{)}^{-1}\beta {\mathrm{\Phi }}^{T}T$
3. $\mathcal{N}(\mathrm{\Phi }(x^{new}){\omega }_{MAP},\mathrm{\Phi }(x^{new}){\mathrm{\Sigma }}_{MAP}\mathrm{\Phi }(x^{new}{)}^T+\beta I)$
4. $\mathcal{N}(0,{\alpha }^{-1}\mathrm{\Phi }{\mathrm{\Phi }}^T+{\beta }^{-1}I)$

Quiz 4

Question

For $y=\sigma (\mathrm{\Phi }(x)w)$, and $\mathcal{D}=\{[x_1,...,x_N],[t_1,...,t_N]\}$, where $\sigma (x)=\frac{1}{1+e^{-x}}$.

1. What is the solution of $w_{ML}$?
2. What is the solution of $w_{MAP}$ if $w\sim \mathcal{N}(m_0,S_0)$?
3. What is the predictive distribution if ${\mathcal{D}}^{new}=\{x^{new},t^{new}=1\}$?
4. What is the model evaluation?

Answer

1. $J(w)=-\sum _{n=1}^N\{t_n\log y_n+(1-t_n)\log (1-y_n)\}b=▽J(w)=\sum _{n=1}^N{\varphi }^T(y_n-t_n)H=▽▽J(w)=\sum _{n=1}^N{y}_n(1-y_n){\varphi }_n^T\varphi$

   Because $\sigma$ is not a linear function, there are no explicit solution to find $w_{ML}$. We can use the gradient descent method to find the solution.

   $w^{+}\text{larr}w-H^{-1}b$

2. $J(w)=-\sum _{n=1}^N\{t_n\log y_n+(1-t_n)\log (1-y_n)\}+\frac{1}{2}(w-m_0{)}^T{S}_0^{-1}(w-m_0)$

   Therefore, $b = \triangledown J(w) = \sum_{n=1}^N \phi^T(y_n - t_n) + S_0^{-1}(w - m_0) $ and $H=▽▽J(w)=\sum _{n=1}^N{y}_n(1-y_n){\varphi }_n^T\varphi +S_0^{-1}$

3. $p(t^{new}=1|x^{new},\mathcal{D})=\int p(t^{new}=1|w)p(w|\mathcal{D})dw=\int \sigma ({\varphi }^{new}w)\mathcal{N}(w_{MAP},H^{-1})dw$

   $\sigma (\kappa ({\sigma }_a^2){\mu }_a)$

4. $\sum _{n=1}^N[t_n\ln y_n+(1-t_n)\ln (1-y_n){]}_{MAP}-\frac{1}{2}(w_{MAP}-m_0{)}^T{S}_0^{-1}(w_{MAP}-m_0)+\frac{M}{2}\ln 2\pi -\frac{1}{2}\ln |H{|}_{MAP}$