Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [regularization]

Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, refers to a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting. (Def: http://en.wikipedia.org/wiki/Regularization_(mathematics))

0
votes
3answers
37 views

Where does the identity matrix come from in the formula for ridge regression coefficients?

The formula for the ridge regression coefficients is $$ \beta = {X^{\top}Y}({X^{\top}X+\lambda I})^{-1}$$ I have tried to derive it as follows: The loss is (I am omitting the sum before the square ...
0
votes
1answer
16 views

Bayesian LASSO: A step within the Gibbs sampler

I'm intending to implement a Bayesian LASSO inside the Gibbs sampler I use to estimate a multivariate time-series model, but I have a doubt about how to draw this step. The prior is a Double-...
0
votes
1answer
22 views

Matrix equality (related to Tikhonov regularization)

I am trying to prove this equality between matrices: $(A^T A + \mu I_n)A^T=A^T(AA^T+\mu I_m)$ where $A \in \mathbb{R}^{m\times n}, \mu \in \mathbb{R}, \mu > 0$. I was given a hint that I should use ...
2
votes
1answer
28 views

How to determine Tikhonov regularization parameter using standard deviation?

I have a linear ill posed problem with the form: $$Ax=b.$$ One approach to this problem is Tikhonov regularization, replacing it with $$\min_x( \|Ax-b\|^2+\alpha\|x\|^2 ).$$ In https://en.wikipedia....
0
votes
1answer
31 views

How is L2 regularization derived?

I just proved to myself why the regularization is added rather than multiplied to loss function. I did so by taking the MLE formula... $$argmax\sum log(P(x_{i}|\Theta ))$$ and since we know that ...
1
vote
0answers
24 views

Iterative least squares for TV-like regularizer

I am trying to implement a TV-like surface regularization in a least squares solver. The formulation is as follows: $E_{surface} = \sum |dA_\theta| $ Background $dA_{\theta} = \frac{\theta_1}{f_x ...
0
votes
0answers
30 views

how is regularization equivalent to constrained optimization

In machine learning, a common theme is to minimize a function of the form $$\sum(W \cdot x - y)^2 + \lambda \sum_i w_i^2$$ Most books simply state that this is equivalent to minimizing $$\sum(W \...
0
votes
0answers
33 views

What does it mean for a norm to be quadratic?

In Computational vision and regularization theory, the authors state The regularization of the ill-posed problem of finding z from the 'data' y $$Az = y$$ requires the choice of norms $|| ...
0
votes
2answers
61 views

What is the role of Tikhonov regularization in optimization?

Suppose I have the following objective function $$ L = \frac{1}{|X|}\sum_{x \in X} \| \hat{y} - y \|^2_2 + \lambda \|w\|_2^2 $$ where $X$ are my data, $\hat{y}$ the prediction, $y$ the target, $\...
2
votes
2answers
109 views

Stability of Tikhonov Regularization

I'm learning about Tikhonov regularization $$\underset{x\in X}{\arg\inf}\left\{||Ax-b||^2+\lambda ||x||^2\right\}$$ I have read that the solution keeps the residual $||Ax-b||^2$ small and is ...
0
votes
1answer
111 views

Why are these two expected values equal?

This question refers to section 13.2 in Chapter 13, "Regularization and Stability", from the book "Understanding Machine Learning: From Theory to Algorithms", which can be found in a PDF version here: ...
0
votes
1answer
35 views

The “units” of the terms of Tikhonov’s first order regularization functional

There’s one point about the order of regularization that I can’t understand. Let’s suppose that we have an ill-posed problem $Af=g$, where $g$ is the observation (with noise), $A$ is a linear operator,...
0
votes
1answer
91 views

Show that $\|.\|_{1/2}$ is not a norm. [duplicate]

Show that $\|.\|_{1/2}$ is not a norm. would anybody guide me that how can i prove or disprove? thanks
0
votes
1answer
25 views

Question regarding weak convergence

I'm doing my bachelor's thesis and I have stumbled upon a certain reoccurring thing in some of the proofs I have to make. The book that I use keeps on arguing that it is possible to assume weak ...
1
vote
0answers
35 views

Least-squares regularization matrix is not positive definite?

I am fitting data $(x_i,y_i)$ to the following model: $$ f(x) = \sum_j a_j g_j(x) = a^T g(x) $$ where $g_j(x)$ are well-conditioned basis functions (in my application they are B-splines, but I don't ...
0
votes
1answer
68 views

Sparse recovery with L1 shrinkage iteration for higher denominational image classification

For 2 months I have been studying sparse recovery and its applications for image classification and I have found that it's a broad area in mathematics which gives rise to a wide variety of ...
4
votes
1answer
48 views

Continuous Tikhonov Regularization for Deconvolution

I am trying to solve the following deconvolution problem where $g(s)$ is a known real valued function and has finite energy: $$g(s) = \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}f(t)e^{-(t-s)^2/2}...
0
votes
0answers
37 views

Preconditioning the Damped Normal Equations - Reference

For some people working with numerical approximations, it is very common to be dealing with a problem in the form $$\min_x \|Ax - b\|,$$ where $A$ is a matrix and $x,b$ are vectors. Consider that $A$ ...
0
votes
1answer
29 views

determine matrix and vector to fit regularized normal equation

I hope the title is not too unclear. I am given a Matrix $$A\in~\mathbb{R}^{K\times~N},~b\in~\mathbb{R}^{K}$$ and instead of solving the normal equation $min_{x\in~\mathbb{R}^N}|Ax-b|^2_2,~$ an $\...
0
votes
0answers
55 views

Nonparametric Bayesian estimation of several black-box functions of different variables from their noisy sums

In order to introduce my problem, let’s start with the nonparametric estimation of a single unknown/black-box function $f:{\Omega _f} \to \mathbb{R}$ of a discrete variable $x$ in a finite domain ${\...
3
votes
0answers
15 views

Interpolation with Regularized Linear Squares

I am fundamentally missing something and would appreciate some clarification. I am reading this text on Regularized Least Squares. This text has a problem where I have to find z, a set of unknown ...
1
vote
1answer
30 views

Question on derivation of probability proportion in Deep Learning Book

I am working through the Deep Learning Book, I am currently on the regularization chapter (https://www.deeplearningbook.org/contents/regularization.html). My question concerns the third step in the ...
3
votes
0answers
34 views

regularizing function under sqrt [closed]

Is it possible to find finite part of $\Sigma^\infty_n\sqrt{n^2 +a^2}$ using something like regularized zeta function?
1
vote
0answers
31 views

How can I incorporate this specific a-priori sparsity information in a regularization approach to guide my inverse problem?

Note: I will be using capital letters to denote matrices, lowercase letters to denote vectors, and the Greek alphabet to denote any scalar quantities. Note 2: I have tried my best to find a topic ...
3
votes
0answers
40 views

Dirac delta from poles of a function

Suppose we are given the simple expression $$ F(k) = \frac{1}{E^2-E(k)^2} $$ which has a pole when $E^2 = E(k)^2$ and where $E, E(k)$ are real numbers. When working with this expression (e.g. inside ...
0
votes
0answers
71 views

Laurent series of an integral with parameter

To find the Laurent series of function $f(a)$ at point $a=0$ $$ f(a)=\int^1_0 \frac{d x}{x^2+a^2} $$ one can first do the integral $$ f(a)=\frac{1}{a}\arctan(1/a) $$ then expand $\arctan(1/a)$ and ...
4
votes
1answer
134 views

Cesàro limit of a stochastic matrix

Let $A$ be a stochastic matrix. Then \begin{align*} \lim_{t \rightarrow\infty} A^t \end{align*} may not exist. For example: \begin{align*} A &= \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}...
3
votes
2answers
365 views

Understanding L2 Regularization Formula

I am currently following the Machine Learning Crash Course on Tensorflow and came across this formula: $$L_2\text{ regularization term} = \|\boldsymbol w\|_2^2 = {w_1^2 + w_2^2 + \cdots + w_n^2}$$ I ...
6
votes
3answers
581 views

Understanding the difference between Ridge and LASSO

I'll start by presenting my current understanding of regularized loss minimization. In the context of ML, generally we are trying to minimize directly an empirical loss, something of the form $$\...
0
votes
0answers
25 views

A ring to place numbers in, how to select numbers uniquely to minimize running variation?

Say I have the numbers in some list $\{1,2,2,3,4,3\}$ and I want to select them uniquely and place so that the average variation (for example running absolute difference, or running square difference) ...
4
votes
0answers
135 views

New/useful method for summation of divergent series?

Questions $$ S(n,x) = x+e^x + e^{e^x} + e^{e^{e^x}} + \dots \text{$n$ times}$$ Also obeys (see background for argument): $$ \frac{1}{2 \pi i} \oint e^{S(k,x)} \frac{\partial \ln(\frac{\int_0^\...
0
votes
0answers
159 views

Solution of L2 Norm and Total Variation (L1) regularizer combined

Is there any method for the direct solution of the following equation? $$ \min_{x} (||Hx - y||_2^2 + ||z - x||_2^2 + \lambda||\nabla x||_1) $$ I know the solution in case only the first two L2 norm ...
2
votes
0answers
105 views

Regularization of Exponentially exponential series?

Question What are the convergence properties of the last equation: $$K = e^{x} + x + \ln{x} + \ln\ln(x) + \dots $$ Can one artificially choose a value of $\ln (x)$ (since there is more than one ...
1
vote
0answers
58 views

How to derive a recursive version of a regularised cost function

I am to derive a recursive version of the following cost function and examine for which choice of D can we have a estimator windup $V(\theta) = \frac{1}{2}\sum_{t=1}^n(y(t)-\phi(t)^T\theta)^2 + \...
0
votes
2answers
109 views

Cost function with unique solution plus convex function has a unique solution?

I have an optimization problem with a cost function $J(X)$, $X$ is a matrix, the function is not convex but I can find the analitical solution and it solution is unique. I want to add a ...
2
votes
1answer
82 views

What is the idea behind zeta function regularization?

Trying to learn the path integral in Quantum Field Theory I came across some infinite products in Weinberg's book "The Quantum Theory of Fields". Heuristically, the author pretends that what can be ...
1
vote
0answers
51 views

SVD with Laplacian regularization and $L_{1,2}$ group-norm

I have a data matrix of the form $X \in \mathbb{R}^{n\times m}$ where the $n$ rows have spatial relationships and $m$ columns have temporal relationships. I am trying to model an objective function of ...
0
votes
0answers
34 views

Methods for filtering position signal

I'm working on a location detection algorithm. This is my outcome - postion: Here's first derivative after time - velocity: [ My goal is to apply some filtration to the signal for smoothing the ...
2
votes
0answers
97 views

Nuclear norm and Schatten norm in practice

I have a problem where the regularizer is the nuclear norm and the matrix being regularized is $n \times d$ with $d < n$. I was initially not getting low rank for the desired performance, the ...
1
vote
1answer
468 views

Derivation of Hard Thresholding Operator (Least Squares with Pseudo $ {L}_{0} $ Norm)

The problem is given by: \begin{equation} \widetilde{f} = \arg \min_{f} \frac{1}{2} {\left\| f - x \right\|}_{2}^{2} + \lambda {\left\| f \right\|}_{0}. \end{equation} How do I find the closed form ...
1
vote
0answers
33 views

Proof in inverse scattering theory (regularization schemes)

I'm currently reading a book about inverse scattering theory and in this book there is a section about ill-posed problems and there's a proof I'm not completely sure I understand. There might be need ...
2
votes
2answers
1k views

How does L1 regularization present itself in gradient descent?

If we incorporated $L_1$ Loss in gradient descent, how would the update rule change? It's easy to write down the optimization objective. But I'm not sure what to put for the update rule.
2
votes
0answers
23 views

How to utilize the right-hand side in inverse problems

Consider the inverse problem $A \, x = b$ with right-hand side $b$, using SVD: $\qquad A = \sum s_i \, U_i \otimes V_i \ $ — singular values $s_i, \ U_i$ and $V_i$ orthonormal bases $\qquad ...
0
votes
1answer
43 views

Solving minimization problem $L_2$ IRLS (Iteration derivation)

In the article ''' Chartrand, Rick, and Wotao Yin. "Iteratively reweighted algorithms for compressive sensing." Acoustics, speech and signal processing, 2008. ICASSP 2008. IEEE international ...
1
vote
0answers
40 views

SeDuMi form of $\min_x\left\{\|Ax-b\|_2^2 + \lambda\|x\|_2\right\}$

I have an application in which I need to minimize the following cost function. I made myself familiar with optimization up until very recently. Could someone kindly let me know what kind of ...
1
vote
1answer
144 views

Least-squares with $2$-norm penalty term

I want to minimize the following cost function $$J= \displaystyle\min_{\boldsymbol{\rm x}} \left\{\|A\boldsymbol{\rm x}_\lambda-b\|_2^2 + \lambda\|\boldsymbol{\rm x}_\lambda\|_2\right\}$$ It is a ...
0
votes
1answer
27 views

Composition of regularized inverse of linear operator on dense subspace converges on whole space?

Any help super appreciated on this. Consider a set of functions $\mathcal{F}$ that is some subset of $L_2$ and two bounded linear operators $A$ and $B$. The inverse $B^{-1}$ is well defined on $\...
0
votes
1answer
44 views

Uniform convergence of regularized inverse

I have a simple question but so far google has failed me. Suppose $A$ is a bounded compact and injective linear operator. Suppose $m$ is some function in a space $M$ and in the range of $A$ and so ...
5
votes
1answer
95 views

Zeta regularization vs Dirichlet series

Suppose you have a sequence of real numbers, denoted $a_n$. Then the sum of the sequence is $\sum_n a_n$ If this is divergent, we can use zeta regularization to get a sum. We can do this by defining ...
3
votes
2answers
502 views

Adding a diagonal matrix to a product of transpose of a matrix and itself is always invertible

I am asking this question in context to Regularization/Ridge Regression Let's say that there is a Matrix A of dimension n x d, where n is the number of rows and d is the number of columns ( n may or ...