Generative model Basic 1

$P(x)$를 학습한다 : input x에 대한 확률분포를 학습한다 

Suppose that we are given images of dogs we want to learn a probability distribution $p(x)$ such that

- Generation : If we sample $\tilde{x}p(x),\tilde{x}$ should look like a dog

- Density estimation : $p(x)$ shoud be high if x looks like a dog, and low otherwise.

- This is also known as explicit models.

 

- Then, how can we represent $p(x)$?

 

기본적인 확률분포는 알고있어야해요~

1. Bernoulli distribution : coin flip

2. Categorical distribution : m-side dice

 

Example

RGB image의 single pixel을 Modeling 한다면?

$(r,g,b)p(R,G,B)$

- Number of cases? => 256 x 256 x 256

- How many parameters do we need to specify? => 256 x 256 x 256 -1

 

$X_1,...,X_n,n$ 개의 binary pixel을 modeling 한다면?

- Number of cases => 2x2 x... x2 = $2^n$

- How many parameters do we need to specify? => $2^n-1$

 

-------> 어쨌든 전체경우의 수를 다 고려한 데이터를 확보하는 것은 현실적으로 불가능하다

 

What if $X_1,...,X_n$ are independent, then $P(X_1,...,X_n)=P(X_1)...P(X_n)$

- Number of cases => $2^n$

- How many parameters do we need to specify? =>

 

Q. n 왜 $2^n-1$ 에서 n으로 줄어들었나?

 

A.Conditional Independence

1. Chain rule

2. Baye's rule

3. Conditional independence

 

Autoregressive Model

- suppose we have 28 x 28 binary pixels

- Our goal is to learn $P(X)=P(X_1,...,x_{784})overX\in {0,1}^{784}$

- Then, how can we parametrize $P(x)$?

  - Let's use the chain rule to factor the joint distribution

  - In other words,

    - $P(X_{1:784}=P(X_1)P(X_2|X_1)P(X_3|X_2)...$

    - This is called an autoregressive model.

    - Note that we need an ordering of all random variables

 

 

첫 번째 nn model : NAEA(Neural Autoregressive Density Estimator)

: 단순히 생성할 수 있을 뿐만 아니라 새로운 입력에 대한 density를 구할 수 있음

- NADE is an explicit model that can compute the density of the given inputs

- BTW, how can we conpute the density of the given image? conditional probability

- continuous random variables에 대해서는 Mixture of Gaussian(MoG)와 같은 방법론을 사용할 수 있음