5707_10_auto-encoder.pptx

Editor's Notes

• #8: \subsection{Theory} %%eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee $x \rightarrow F \rightarrow x'$\\ $z=\sigma (Wx+b)------(*)$\\ $x'=\sigma'(W'z+b')---(**)$\\ Autoencoders are trained to minimize reconstruction errors (such as squared errors), often referred to as the "loss (L)":\\ By combining (*) and (**)\\ $Loss=L(x,x')=\| x-x' \|^2$\\ $\| x-\sigma' ( W' \sigma (W x+b)+b' \|^2$\\
• #28: subsection{Uni-variate and Multivariate Gaussian} %https://ttic.uchicago.edu/~shubhendu/Slides/Estimation.pdf $N_{univariate}(x)=\frac{1}{(2\pi \sigma^2)^{(1/2)}} exp\big( -\frac{1}{2}\frac{(x-\mu)^2}{\sigma^2} \big)$\\ Multivariate Gaussian, $x$=data sample, %$\mu$=mean, $\sum$=covariance \\ d-dimension\\ $N_{univariate}(x)= \frac{1}{(2\pi)^{(d/2)} | \Sigma |^{(1/2)}} exp\big( -\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu) \big)$
• #30: %%%%%%%%%%%% slide 54 %%%%%%%%%%%%%%%%%%%% %%% eeeeee eq:filtering_01A eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee %%% 1st order (A) Gaussian \begin{flalign} \label{eq:filtering_01a} \begin{aligned} G(x) = \frac{1} {\sqrt{2\pi{\sigma^2 }}} e^{- \frac{(x-\mu)^2} {2\sigma^2} } \end{aligned} \end{flalign} %%% eeeeee eq:filtering_01B eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee 1st order (B) Gaussian\\ \begin{flalign} \label{eq:filtering_01b} \begin{aligned} G(x) = \frac{1} {\sqrt{2\pi{\sigma^2 }}} exp\left({- \frac{(x-\mu)^2} {2\sigma^2} }\right) \end{aligned} \end{flalign} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%% eeeeee eq:filtering_02A eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2nd order (A) Gaussian\\ \begin{flalign} \label{eq:filtering_02A} \begin{aligned} G(x,y) =G(x)G(y)= \frac{1} {{2\pi{\sigma^2 }}} e^{- \frac{(x-\mu_x)^2+(y-\mu_y)^2} {2\sigma^2} } \end{aligned} \end{flalign} %%% 2nd order (B) Gaussian %%% eeeeee eq:filtering_02B eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee \begin{flalign} \label{eq:filtering_02B} \begin{aligned} G(x,y) =G(x)G(y)= \frac{1} {{2\pi{\sigma^2 }}} exp\left({- \frac{(x-\mu_x)^2+(y-\mu_y)^2} {2\sigma^2} }\right) \end{aligned} \end{flalign}
• #31: %%%%%%%%%%%%%slide 28 %%%%%%%%%%%%%%%%% 1-D Gaussian $x_j$= a sample,\\ $\mu_0=mean, \sigma_0=variance$\\ $N(x_j)=\frac{1}{(2 \pi \sigma_0^2)^{1/2}} exp \big (-\frac{1}{2}\frac{(x_j - \mu_0)^2}{\sigma_0^2} \big)$\\  %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2-D an isotropic (circular symmetric) Gaussian \\ assume mean is 0\\ $N(x_1,x_2)=\frac{1}{(2 \pi \sigma_0^2)} exp \big (-\frac{1}{2}\frac{(x_1^2+x_2^2 )^2}{\sigma_0^2} \big)$\\
• #32: %%%%%%%%%%slide 29 , 30%%%%%%%%%%%%%%%%%%%% ---------------slide 29,30 \\ 2-D an isotropic (circular symmetric) Gaussian for variable $(x,y)$ \\ assume 2-D mean is $(m_x,m_y)$\\ $G(x,y)=G(x)G(y)$\\ $N(x_1,x_2)=\frac{1}{(2 \pi \sigma_0^2)} exp \big (-\frac{1}{2}\frac{((x-m_x)^2+(y-m_y)^2 )^2}{\sigma_0^2} \big)$\\
• #33: %%%%%%%%%%slide 29 , 30%%%%%%%%%%%%%%%%%%%% ---------------slide 29,30 \\ 2-D an isotropic (circular symmetric) Gaussian for variable $(x,y)$ \\ assume 2-D mean is $(m_x,m_y)$\\ $G(x,y)=G(x)G(y)$\\ $N(x_1,x_2)=\frac{1}{(2 \pi \sigma_0^2)} exp \big (-\frac{1}{2}\frac{((x-m_x)^2+(y-m_y)^2 )^2}{\sigma_0^2} \big)$\\
• #49: %%%%%%%%%%slide 46%%%%%%%%%%%%%%%%%%%% ---------------slide 46 \\ \subsection{slide 46:The reconstruction loss $l()$} Given $x_i \epsilon X, z \epsilon Q, \text { and } E()$ is the expected value The idea is to train the Encoder/Decoder (Neural Network) to maximum the likelihood of the Mean squared error (MSE) between x and reconstructed $\hat x_i \epsilon \hat X$\\ % To maximize likelihood, we can minimize the “expected negative log-likelihood” $(l_i)$ of the $ i^{th}$ datapoint $x_i$. \\ % $ l_i(\theta, \phi | x_i)=-E_{x_i \epsilon X} \big[ E_{z \epsilon Q}[log P_{\phi (de)}(\hat x_i | z)] \big]$\\ $q_{\theta(en)}(z|x_i)$\\ $P_{\phi(de)}(\hat x_i |z)$\\ % $l_i(\theta,\phi)$\\
• #51: %%%%%%%%%%%%%%%%%%%%%%%% \subsection{slide 48: How to make sure the neural networks produce similar hidden data (means and standard deviations) from similar training images} Problem: Input that we regard as similar may end up very different in z space (hidden, means and standard deviations). That means some solutions may give small loss $l_i{(\theta, \phi)}$, even $q_{\theta(en)}$ and $p_{\phi(de)}$ are of very different distributions.\\ Solution: Use $p(z)=N(0,1)$, try to force $q_{\theta(en)}(z|x_i)$ (a neural network) to act similar to a standard normal probability density function. We can use Kullback-Leibler divergence $(D_{KL})$ to do the checking. % $l_i(\theta,\phi|x_i)$\\ $l_i()$
• #52: \subsection{Slide 49: Math background: Kullback–Leibler divergence} %https://ttic.uchicago.edu/~shubhendu/Slides/Estimation.pdf Math background: Kullback–Leibler divergence (also known as relative entropy) measures how one probability distribution is different from a second, reference probability distribution over the same variable X.  %1 Define:\\ $D_{KL}\big[ N(\mu_1,\sigma_1^2) || ( \mu_2,\sigma_2^2) \big]= \frac{1}{2} \Big[ tr \big ([\sigma_2^2]^T \cdot \sigma_1^2-I \big) +(\mu_1-\mu_2)^T[\sigma^2]^{-1}(\mu_1 - \mu_2) +log \big( \frac{det(\sigma_2^2)}{det(\sigma_1^2)} \big) \Big]$-----(I)\\ If $N(\mu_1,\sigma_1^2)=N(\mu(X),\sigma^2(X))$; \\Also $N(\mu_2,\sigma^2_2)=N(0,I)$\\ %2 % $D_{KL}\big[ N(\mu(X),\sigma^2(X)) || ( N(0,I) \big]= \frac{1}{2} \Big[ tr \big ([\sigma^2]^T \cdot \sigma^2-I \big) +(\mu(X))^T(\mu(X))) -log (det(\sigma^2(X))) \Big]$\\ %3 For $D[Q(z|x_i) || N(0,I)] \text{, where } Q(z|x_i) =N(\mu(X),\sigma^2(X))$\\ $D_{KL}\big[ N(\mu(X),\sigma^2(X)) || ( N(0,I) \big]= \frac{1}{2} \Big[ tr \big ([\sigma^2]^T \cdot \sigma^2-I \big) +(\mu(X))^T(\mu(X))) -log (det(\sigma^2(X))) \Big]$\\ Kullback–Leibler divergence $D_{KL} (D_1 || D_2)=0$ indicates the two distributions ${D_1,D_2}$ are identical %Tutorial on Variational Autoencoders by Carl Doersch & https://arxiv.org/abs/1606.05908  
• #53: \subsection{Slide50:Training (concept 1)} %See http://bjlkeng.github.io/posts/variational-autoencoders/ & https://arxiv.org/abs/1312.6114 Combining concept 1 and 2 to minimize Loss $l_i (X), \text{ of } X= {x_1,x_2,..,x_N} , E()$=expected value . For the whole $X$, the average loss is \\ Input to encoder = $x_i \epsilon X$\\ Output to encoder = $\hat{x_i} \epsilon \hat{X}$\\ $P(\hat{x_i} | z)$=Prob. distribution of $x_i$ generated by $z$ (decoder side)\\ % $P(z | x_i)$=Prob. distribution of $x_i$ generated by $z$ (decoder side)\\ % $P(z)$=Prob. distribution of the latent (hidden) variables,\\ % $E_{Z \epsilon Q}[log P(\hat x_i | z)]$=expected value (exp. val) of $\hat x_i$ generated at the decoder output\\ % $E_{x_i \epsilon X}[E_{Z \epsilon Q}[log(P(\hat {x_i | z)})]]$= exp. val. of $\hat{x_i}$ gen. at the decoder output when input = $x_i \epsilon X$ \\ % $\varepsilon = \text { random variable generated by a Gaussian function }\{ mean (\mu)=0, stdev( \sigma)=1\}, \varepsilon N(0,I) $\\ % At this stage $z$ can be any distribution, but we can assume $z$= Gaussian $N(\mu_{x_i|z}), \sigma_{x_i | z})$\\ % It can be formed by scaling $\varepsilon \epsilon N(),I)$, (\url{en. https://en.wikipedia.org/wiki/Normal_distribution})\\ The advantage is if $(\mu_{x_i | z}, \sigma_{x_i | z)})$ are found, then use a random gen. $N(\mu_{x_i | z}, \sigma_{x_i | z})$ to gen. $z$.\\ % Hence $log P(\hat{x_i} | z)=log P \big( \hat {x_i |z}=\mu_{x_i |z}(x_i)+\sigma_{x_i | z}(x_i) \ast \varepsilon \big)$ \\ % We want to maximize $E_{x_i \epsilon X}[ E_{z_i \epsilon Q}[log P (\hat x_i | z]]$ (to make input output similar)\\ % It is the same as to minimize $-\big( E_{x_i \epsilon X}[ E_{z_i \epsilon Q}[log P (\hat x_i | z]] \big)$\\ Objective function1=$-E_{x_i | X} \Big[ E_{z \epsilon Q}[ log P(\hat{x_i} | z=\mu_{x_i |z}(x_i)+\sigma_{x_i | z}(x_i) \ast \varepsilon )$ ] \Big]\\ Since P is Gaussian, we minimize Objective$\_$function1=$\frac{1}{N} \sum\limits_{x_i \epsilon X} \Big( \frac{1}{2 \sigma^2_{\hat {x_i}} |z} (x_i - \mu_{\hat{x_i}|z})^2 \Big)$ 
• #54: %%%%%%%%%%%%%%%%%%%%%%%% \subsection{slide 51, concept2} Training: Combining concept 1 and 2 to minimize Loss $l_i (X), of X= {x_1,x_2,..,x_N} , E()$=expected value . For the whole $X$, the average loss is Recall $q_{\theta(en)(z|x_i)}$=prob.distribution of $z$ generated by $x_i$ (encoder side)\\ % We mentioned earlier we want $q_{\theta(en)(z|x_i)}$ to be close to Gaussian, put $P(z)=N(0,I)$\\ % $D_{KL}\big[ q_{\theta(en)}(z|x_i) || ( N(0,I) \big]$=difference of $q_{\theta(en)(z|x_i)}$ and Gaussian, see previous discussion on $D_KL[]$\\ % objective$\_$func2= $D_{KL}\big[ q_{\theta(en)(z|x_i)} || ( N(0,I) \big]$, this is to be minimized\\ % Overall$\_$objective$\_$function =objective$\_$funct1+objective$\_$func2\\ $=\frac{1}{N} \sum\limits_{x_i \epsilon X} \Big( \frac{1}{2 \sigma^2_{\hat {x_i}} |z} (x_i - \mu_{\hat{x_i}|z})^2 \Big) +D_{KL}\big[ q_{\theta(en)(z|x_i)} || ( N(0,I) \big]$\\ We have shown earlier that\\ $D_{KL}\big[ q_{\theta(en)}(z|x_i) || ( N(0,I) \big]= % +\frac{1}{2} \big\{ tr (\sigma^2(X)-I) +\mu(X)^T \mu(X)-log(det(\sigma^2(X))) \big\}$ % % %% $l=\frac{1}{N} \sum\limits_{x_i \epsilon X} \Big( \frac{1}{2 \sigma^2_{\hat {x_i}} |z} (x_i - \mu_{\hat{x_i}|z})^2 \Big) +\frac{1}{2} \big\{ tr (\sigma^2(X)-I) +\mu(X)^T \mu(X)-log(det(\sigma^2(X))) \big\}$\\ The first term is for concept 1 and the second term is for concept 2\\ We will run an iterative algorithm to minimize $l$ See \url{ http://bjlkeng.github.io/posts/variational-autoencoders/ & https://arxiv.org/abs/1312.6114}
• #57: \subsection{slide 52:Use neural networks to implement system} $l=\frac{1}{N} \sum\limits_{x_i \epsilon X} \Big( \frac{1}{2 \sigma^2_{\hat {x_i}} |z} (x_i - \mu_{\hat{x_i}|z})^2 \Big) +\frac{1}{2} \big\{ tr (\sigma^2(X)-I) +\mu(X)^T \mu(X)-log(det(\sigma^2(X))) \big\}$\\
• #60: %%%%%%%%%%%%%%%%%%%% \subsection{slide 55: VAE generative model} In theory, we use a sample of z from $q_{\theta(en)} (z|x_i)$ as input to sample from $p_{\phi(de)}(\hat X_i| Z) $ to give an approximate reconstruction of $x_i$\\ Alternatively, if we sample any $z$ from $N(0,1)$ and use it as input to sample from $p_{\phi(de)}(\hat X_i | Z) $ then we can approximate the entire data distribution $p( )$. I.e., we can generate new samples that look like the input but aren’t in the input.
• #65: %%%%%%%%%%%%%%%%%%%% \subsection{slide 55: VAE generative model} In theory, we use a sample of z from $q_{\theta(en)} (z|x_i)$ as input to sample from $p_{\phi(de)}(\hat X_i| Z) $ to give an approximate reconstruction of $x_i$\\ Alternatively, if we sample any $z$ from $N(0,1)$ and use it as input to sample from $p_{\phi(de)}(\hat X_i | Z) $ then we can approximate the entire data distribution $p( )$. I.e., we can generate new samples that look like the input but aren’t in the input.
• #67: \subsection{slide 67: Summary: Forward pass}  $\epsilon$= the generated variable\\ $\mu$ = mean\\ $\sigma$= variance\\ $Z= \epsilon \sigma_x +\mu_x$
• #102: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{slide 64 :Derivation} We want to find $\min_\theta(E_q[X^2])$, thus we need to find $\nabla_\theta E_q[x^2]$\\ Since $\nabla_\theta q_\theta (x)=q_\theta(x)$, and $\frac{d(log(y))}{dx}=\frac{1}{log(y)}\frac{dy}{dx}$\\ Thus, $\nabla_\theta log(q_\theta (x)) =\frac{1}{q_\theta (x)} \nabla (q_\theta(x))$ -----(i)\\ % $\nabla\theta E_q[x^2]=\nabla_\theta \int q_\theta(x) x^2 dx$, by definition of exception\\ % $\nabla\theta E_q[x^2]= \int \nabla_\theta q_\theta(x) \frac{q_\theta(x)}{q_\theta(x)} \Big) x^2 dx$,\\ $=\int \Big( \frac{1}{q_\theta(x)} \nabla_\theta q_\theta(x) \Big) q_\theta(x) x^2 dx$---(ii), put (i) in (ii)\\ %%% $\nabla _\theta E_q[x^2]= \int q_\theta(x) \nabla_\theta log (q_\theta(x)) x^2 dx$\\ $=E_q[\nabla_\theta log(q_\theta(x))x^2]$, also by definition of expectation\\ % If $q_\theta = N(\theta, I) $ is a normal distribution of mean =$\theta$, variance =1\\ hence $\nabla_\theta log (q_\theta(x))=x-\theta$ (see appendix 1), therefore\\ $\nabla_\theta E_q[x^2]=E_q[x^2(x-\theta)]$, since $X=\theta + \epsilon, \epsilon \approx N(0,1)$\\ % then $E_q[x^2]=E_p[(\theta+\epsilon)^2]$, where p is the distribution of $\epsilon$,\\ i.e. $\epsilon \approx N(0,1)$, therefore derivative of $E_q[x^2]$ is \\ $\nabla_\theta E_q[x^2]=\nabla_\theta E_p [(\theta+\epsilon)^2]=E_p[(2\theta+\epsilon)]$\\ %%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%% Note by definition: $\frac{d}{dx}ln(x)=\frac{1}{x}, \text{ and also } \frac{log_b(x)}{dx}=\frac{1}{ln(b) \cdot x}$\\
• #105: %%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{slide 68:Summary: Backpropagation} % The gradient during backpropagation is\\ $\nabla E_q[x^2]=\nabla_\theta E_p [(\theta+\epsilon)^2]=E_p[2(\theta+\epsilon)]$\\ % $\epsilon$ is found the generated variable\\ $\theta$ is given\\ So the gradient can be found and used in backpropagation
• #106: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{slid 63:Gradient for back propagation} Derivation of $\nabla_{\theta}E_q[x^2]$\\ $X=\theta+\epsilon,\epsilon \approx N(0,I) $\\ Then $E_q [x^2]=E_p[(\theta+\epsilon)^2]$\\ $P= \text { distribution of } \epsilon \approx N(0,I)$\\ Thus, derivative of $E_q[x^2]$ $=\nabla_\theta E_q[x^2]=\nabla_\theta E_p[(\theta + \epsilon)^2]$ $=E_p[2(\theta+\epsilon)]$