User:Hfgong/Manifold Learning

Draft page of Hfgong

Preface
Two main framework:
 * Global framework: MDS, Isomap
 * Local framework: LLE, Laplace Eigenmap, Hessian Eigenmap, etc.

Manifold
Summary. A manifold is a space that is locally like $$\reals^n$$, however lacking a preferred system of coordinates. Furthermore, a manifold can have global topological properties, such as non-contractible Curve, that distinguish it from the topologically trivial $$\reals^n$$.

Standard Definition. An $$n$$-dimensional topological manifold $$M$$ is a second countable, Hausdorff topological space (For connected manifolds, the assumption that $$M$$ is second-countable is logically equivalent to $$M$$ being paracompact, or equivalently to $$M$$ being metrizable. The topological hypotheses in the definition of a manifold are needed to exclude certain counter-intuitive pathologies. Standard illustrations of these pathologies are given by the long line (lack of paracompactness) and the forked line (points cannot be separated). These pathologies are fully described in Spivak.) that is locally homeomorphic to open subsets of $$\reals^n$$.

A differential manifold is a topological manifold with some additional structure information. A chart, also known as a system of coordinates, is a continuous injection from an open subset of $$M$$ to $$\reals^n$$. Let $$\alpha: U_\alpha \rightarrow \reals^n,$$ and $$\beta:U_\beta\rightarrow\reals^n$$ be two charts with overlapping domains. The continuous injection
 * $$\beta\circ\alpha^{-1}: \alpha(U_\alpha\cap

U_\beta)\rightarrow\reals^n$$ is called a \emph{transition function}, and also called a \emph{a change of coordinates}. An atlas $$\mathcal A$$ is a collection of charts $$\alpha:U_\alpha\rightarrow\reals^n$$ whose domains cover $$M$$, i.e.
 * $$M = \bigcup_{\alpha} U_\alpha.$$

Note that each transition function is really just $$n$$ real-valued functions of $$n$$ real variables, and so we can ask whether these are continuously differentiable. The atlas $$\mathcal A$$ defines a differential structure on $$M$$, if every transition functions corresponding to $$\mathcal A$$ is continuously differentiable.

More generally, for $$k=1,2,\ldots,\infty,\omega$$, the atlas $$\mathcal A$$ is said to define a $$\mathcal C^k$$ differential structure, and $$M$$ is said to be of class $$\mathcal C^k$$, if all the transition functions are $$k$$-times continuously differentiable, or real analytic in the case of $$\mathcal C^\omega$$. Two differential structures of class $$\mathcal{C^k}$$ on $$M$$ are said to be isomorphic if the union of the corresponding atlases is also a $$\mathcal C^k$$ atlas, i.e. if all the new transition functions arising from the merger of the two atlases remain of class $$\mathcal C^k$$. More generally, two $$\mathcal C^k$$ manifolds $$M$$ and $$N$$ are said to be diffeomorphic, i.e. have equivalent differential structure, if there exists a homeomorphism $$\phi:M\to N$$ such that the atlas of $$M$$ is equivalent to the atlas obtained as $$\phi$$-pullbacks of charts on $$N$$.

The atlas allows us to define differentiable mappings to and from a manifold. Let
 * $$f:U\rightarrow\reals,\quad U\subset M$$

be a continuous function. For each $$\alpha\in \mathcal A$$ we define
 * $$f_\alpha: V \rightarrow \reals,\quad V\subset\reals^n,$$

called the representation of $$f$$ relative to chart $$\alpha$$, as the suitably restricted composition
 * $$f_\alpha = f\circ \alpha^{-1}.$$

We judge $$f$$ to be differentiable if all the representations $$f_\alpha$$ are differentiable. A path
 * $$\gamma: I\rightarrow M,\quad I\subset\reals$$

is judged to be differentiable, if for all differentiable functions $$f$$, the suitably restricted composition $$f\circ\gamma$$ is a differentiable function from $$\reals$$ to $$\reals$$. Finally, given manifolds $$M, N$$, we judge a continuous mapping $$\phi:M\rightarrow N$$ between them to be differentiable if for all differentiable functions $$f$$ on $$N$$, the suitably restricted composition $$f\circ\phi$$ is a differentiable function on $$M$$.

Isomap
Basic idea: replace Euclidean distance with geodesic distance in MDS 1. Construct neighborhood graph 2. Compute shortest paths (geodesic distance) $$\delta_{ij}$$ This can be done using Dijkstra algorithm 3. Construct d-dimensional embedding This can be done using classical MDS algorithm Minimize the stress:
 * $$E(Y)=\sum_{i0, \sum_j{W_{ij}}=1$$

Construct d-dimensional embedding
Reconstruction Error:



\Phi (Y)=\sum_{i}\left\vert y_{i}-\sum_{j}W_{ij}Y_{j}\right\vert ^{2} $$

with constrain:

\sum_{i}Y_{i} = 0 $$
 * $$\frac{1}{N}\sum_{i}Y_{i}Y_{i}^{\top } = I

$$ LLE Website: []

Hessian Eigenmap
1. Identify neighbors 2. Obtain tangent coordinates 3. Develop quadratic form 4. Find approximate null space 5. Find basis for null space $$\mathcal{H}$$

Charting/co-ordination
Find low-dimensional local neighbourhoods on the manifold, then mix the neighbourhoods while trying to minimize distortion.

Optimization Issues in Manifold Learning

 * 1) Optimization on Stiefel and Grassmann Manifold
 * 2) Convert to Eigenvalue Problems

Kernel in Manifold Learning
Kernel View of Manifold Learning Kernel Matrix for Nonlinear Dimension Reduction