K-SVD

Machine learning and data mining
Problems Classification Clustering Regression Anomaly detection AutoML Association rules Reinforcement learning Structured prediction Feature engineering Feature learning Online learning Semi-supervised learning Unsupervised learning Learning to rank Grammar induction
Supervised learning (classification • regression) Decision trees Ensembles Bagging Boosting Random forest k-NN Linear regression Naive Bayes Artificial neural networks Logistic regression Perceptron Relevance vector machine (RVM) Support vector machine (SVM)
Clustering BIRCH CURE Hierarchical k-means Expectation–maximization (EM) DBSCAN OPTICS Mean-shift
Dimensionality reduction Factor analysis CCA ICA LDA NMF PCA t-SNE
Structured prediction Graphical models Bayes net Conditional random field Hidden Markov
Anomaly detection k-NN Local outlier factor
Artificial neural networks Autoencoder Deep learning DeepDream Multilayer perceptron RNN LSTM GRU) Restricted Boltzmann machine SOM Convolutional neural network U-Net
Reinforcement learning Q-learning SARSA Temporal difference (TD)
Theory Bias-variance dilemma Computational learning theory Empirical risk minimization Occam learning PAC learning Statistical learning VC theory
Machine-learning venues NIPS ICML ML JMLR ArXiv:cs.LG
Glossary of artificial intelligence Glossary of artificial intelligence
Related articles List of datasets for machine-learning research Outline of machine learning
Machine learning portal

In applied mathematics, K-SVD is a dictionary learning algorithm for creating a dictionary for sparse representations, via a singular value decomposition approach. K-SVD is a generalization of the k-means clustering method, and it works by iteratively alternating between sparse coding the input data based on the current dictionary, and updating the atoms in the dictionary to better fit the data. K-SVD can be found widely in use in applications such as image processing, audio processing, biology, and document analysis.

Problem description

Main article: Sparse approximation

The goal of dictionary learning is to learn an overcomplete dictionary matrix $D\in \mathbb {R} ^{n\times K}$ that contains $K$ signal-atoms (in this notation, columns of $D$). A signal vector $y\in \mathbb {R} ^{n}$ can be represented, sparsely, as a linear combination of these atoms; to represent $y$, the representation vector $x$ should satisfy the exact condition $y=Dx$, or the approximate condition $y\approx Dx$, made precise by requiring that $\|y-Dx\|_{p}\leq \epsilon$ for some small value ε and some L_p norm. The vector $x\in \mathbb {R} ^{K}$ contains the representation coefficients of the signal $y$. Typically, the norm $p$ is selected as L₁, L₂, or L_∞.

If $n<K$ and D is a full-rank matrix, an infinite number of solutions are available for the representation problem. Hence, constraints should be set on the solution. Also, to ensure sparsity, the solution with the fewest number of nonzero coefficients is preferred. Thus, the sparsity representation is the solution of either

$(P_{0})\quad \min \limits _{x}\|x\|_{0}\qquad {\text{subject to }}y=Dx$

or

$(P_{0,\epsilon })\quad \min \limits _{x}\|x\|_{0}\qquad {\text{subject to }}\|y-Dx\|_{2}\leq \epsilon$

where the $\|x\|_{0}$ counts the nonzero entries in the vector $x$. (See the zero "norm".)

K-SVD algorithm

K-SVD is a kind of generalization of K-means, as follows. The k-means clustering can be also regarded as a method of sparse representation. That is, finding the best possible codebook to represent the data samples $\{y_{i}\}_{i=1}^{M}$ by nearest neighbor, by solving

$\quad \min \limits _{D,X}\{\|Y-DX\|_{F}^{2}\}\qquad {\text{subject to }}\forall i,x_{i}=e_{k}{\text{ for some }}k.$

which is equivalent to

$\quad \min \limits _{D,X}\{\|Y-DX\|_{F}^{2}\}\qquad {\text{subject to }}\quad \forall i,\|x_{i}\|_{0}=1$.

The letter F denotes the Frobenius norm. The sparse representation term $x_{i}=e_{k}$ enforces K-means algorithm to use only one atom (column) in dictionary $D$. To relax this constraint, the target of the K-SVD algorithm is to represent signal as a linear combination of atoms in $D$.

The K-SVD algorithm follows the construction flow of the K-means algorithm. However, in contrary to K-means, in order to achieve a linear combination of atoms in $D$, the sparsity term of the constraint is relaxed so that the number of nonzero entries of each column $x_{i}$ can be more than 1, but less than a number $T_{0}$.

So, the objective function becomes

$\quad \min \limits _{D,X}\{\|Y-DX\|_{F}^{2}\}\qquad {\text{subject to }}\quad \forall i\;,\|x_{i}\|_{0}\leq T_{0}.$

or in another objective form

$\quad \min \limits _{D,X}\sum _{i}\|x_{i}\|_{0}\qquad {\text{subject to }}\quad \forall i\;,\|Y-DX\|_{F}^{2}\leq \epsilon .$

In the K-SVD algorithm, the $D$ is first fixed and the best coefficient matrix $X$ is found. As finding the truly optimal $X$ is impossible, we use an approximation pursuit method. Any algorithm such as OMP, the orthogonal matching pursuit can be used for the calculation of the coefficients, as long as it can supply a solution with a fixed and predetermined number of nonzero entries $T_{0}$.

After the sparse coding task, the next is to search for a better dictionary $D$. However, finding the whole dictionary all at a time is impossible, so the process is to update only one column of the dictionary $D$ each time, while fixing $X$. The update of the $k$-th column is done by rewriting the penalty term as

$\|Y-DX\|_{F}^{2}=\left\|Y-\sum _{j=1}^{K}d_{j}x_{T}^{j}\right\|_{F}^{2}=\left\|\left(Y-\sum _{j\neq k}d_{j}x_{T}^{j}\right)-d_{k}x_{T}^{k}\right\|_{F}^{2}=\|E_{k}-d_{k}x_{T}^{k}\|_{F}^{2}$

where $x_{T}^{k}$ denotes the k-th row of X.

By decomposing the multiplication $DX$ into sum of $K$ rank 1 matrices, we can assume the other $K-1$ terms are assumed fixed, and the $k$-th remains unknown. After this step, we can solve the minimization problem by approximate the $E_{k}$ term with a $rank-1$ matrix using singular value decomposition, then update $d_{k}$ with it. However, the new solution of vector $x_{T}^{k}$ is very likely to be filled, because the sparsity constraint is not enforced.

To cure this problem, Define $\omega _{k}$ as

$\omega _{k}=\{i\mid 1\leq i\leq N,x_{T}^{k}(i)\neq 0\},$

which points to examples $\{y_{i}\}_{i=1}^{N}$ that use atom $d_{k}$ (also the entries of $x_{i}$ that is nonzero). Then, define $\Omega _{k}$ as a matrix of size $N\times |\omega _{k}|$, with ones on the $(i,\omega _{k}(i)){\text{-th}},$ entries and zeros otherwise. When multiplying $x_{R}^{k}=x_{T}^{k}\Omega _{k}$, this shrinks the row vector $x_{T}^{k}$ by discarding the zero entries. Similarly, the multiplication $Y_{k}^{R}=Y\Omega _{k}$ is the subset of the examples that are current using the $d_{k}$ atom. The same effect can be seen on $E_{k}^{R}=E_{k}\Omega _{k}$.

So the minimization problem as mentioned before becomes

$\|E_{k}\Omega _{k}-d_{k}x_{T}^{k}\Omega _{k}\|_{F}^{2}=\|E_{k}^{R}-d_{k}x_{R}^{k}\|_{F}^{2}$

and can be done by directly using SVD. SVD decomposes $E_{k}^{R}$ into $U\Delta V^{T}$. The solution for $d_{k}$ is the first column of U, the coefficient vector $x_{R}^{k}$ as the first column of $V\times \Delta (1,1)$. After updating the whole dictionary, the process then turns to iteratively solve X, then iteratively solve D.

Limitations

Choosing an appropriate "dictionary" for a dataset is a non-convex problem, and K-SVD operates by an iterative update which does not guarantee to find the global optimum. However, this is common to other algorithms for this purpose, and K-SVD works fairly well in practice.

See also

• Sparse approximation
• Singular value decomposition
• Matrix norm
• K-means clustering
• Low-rank approximation