# Questions tagged [regularization]

Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, refers to a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting. (Def: http://en.wikipedia.org/wiki/Regularization_(mathematics))

211 questions
3answers
37 views

### Where does the identity matrix come from in the formula for ridge regression coefficients?

The formula for the ridge regression coefficients is $$\beta = {X^{\top}Y}({X^{\top}X+\lambda I})^{-1}$$ I have tried to derive it as follows: The loss is (I am omitting the sum before the square ...
1answer
16 views

### Bayesian LASSO: A step within the Gibbs sampler

I'm intending to implement a Bayesian LASSO inside the Gibbs sampler I use to estimate a multivariate time-series model, but I have a doubt about how to draw this step. The prior is a Double-...
1answer
22 views

### Matrix equality (related to Tikhonov regularization)

I am trying to prove this equality between matrices: $(A^T A + \mu I_n)A^T=A^T(AA^T+\mu I_m)$ where $A \in \mathbb{R}^{m\times n}, \mu \in \mathbb{R}, \mu > 0$. I was given a hint that I should use ...
1answer
28 views

### How to determine Tikhonov regularization parameter using standard deviation?

I have a linear ill posed problem with the form: $$Ax=b.$$ One approach to this problem is Tikhonov regularization, replacing it with $$\min_x( \|Ax-b\|^2+\alpha\|x\|^2 ).$$ In https://en.wikipedia....
1answer
31 views

### How is L2 regularization derived?

I just proved to myself why the regularization is added rather than multiplied to loss function. I did so by taking the MLE formula... $$argmax\sum log(P(x_{i}|\Theta ))$$ and since we know that ...
0answers
24 views

0answers
55 views

2answers
109 views

### Cost function with unique solution plus convex function has a unique solution?

I have an optimization problem with a cost function $J(X)$, $X$ is a matrix, the function is not convex but I can find the analitical solution and it solution is unique. I want to add a ...
1answer
82 views

### What is the idea behind zeta function regularization?

Trying to learn the path integral in Quantum Field Theory I came across some infinite products in Weinberg's book "The Quantum Theory of Fields". Heuristically, the author pretends that what can be ...
0answers
51 views

### SVD with Laplacian regularization and $L_{1,2}$ group-norm

I have a data matrix of the form $X \in \mathbb{R}^{n\times m}$ where the $n$ rows have spatial relationships and $m$ columns have temporal relationships. I am trying to model an objective function of ...
0answers
34 views

### Methods for filtering position signal

I'm working on a location detection algorithm. This is my outcome - postion: Here's first derivative after time - velocity: [ My goal is to apply some filtration to the signal for smoothing the ...
0answers
97 views

### Nuclear norm and Schatten norm in practice

I have a problem where the regularizer is the nuclear norm and the matrix being regularized is $n \times d$ with $d < n$. I was initially not getting low rank for the desired performance, the ...
1answer
468 views

### Derivation of Hard Thresholding Operator (Least Squares with Pseudo ${L}_{0}$ Norm)

The problem is given by: \begin{equation} \widetilde{f} = \arg \min_{f} \frac{1}{2} {\left\| f - x \right\|}_{2}^{2} + \lambda {\left\| f \right\|}_{0}. \end{equation} How do I find the closed form ...
0answers
33 views

### Proof in inverse scattering theory (regularization schemes)

I'm currently reading a book about inverse scattering theory and in this book there is a section about ill-posed problems and there's a proof I'm not completely sure I understand. There might be need ...
2answers
1k views

### How does L1 regularization present itself in gradient descent?

If we incorporated $L_1$ Loss in gradient descent, how would the update rule change? It's easy to write down the optimization objective. But I'm not sure what to put for the update rule.
0answers
23 views

1answer
44 views

### Uniform convergence of regularized inverse

I have a simple question but so far google has failed me. Suppose $A$ is a bounded compact and injective linear operator. Suppose $m$ is some function in a space $M$ and in the range of $A$ and so ...
1answer
95 views

### Zeta regularization vs Dirichlet series

Suppose you have a sequence of real numbers, denoted $a_n$. Then the sum of the sequence is $\sum_n a_n$ If this is divergent, we can use zeta regularization to get a sum. We can do this by defining ...
2answers
502 views

### Adding a diagonal matrix to a product of transpose of a matrix and itself is always invertible

I am asking this question in context to Regularization/Ridge Regression Let's say that there is a Matrix A of dimension n x d, where n is the number of rows and d is the number of columns ( n may or ...