Questions tagged [regularization]
Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, refers to a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting. (Def: http://en.wikipedia.org/wiki/Regularization_(mathematics))
369 questions
0
votes
0
answers
12
views
Bregman distance (divergence) in a $2$ convex and $p$ smooth Banach space.
I have been reading the paper Convex regularization in statistical inverse problem and I can't understand something which the author mentions as "obvious".
Let $X$ be a Banach space over the ...
- 56
1
vote
0
answers
43
views
Sinkhorn Knopp algorithm - Bregman projection in update rule
I don't understand the updating rule for $u^{l+1}$ in the Sinkhorn algorithm. The below images contain all necessary definitions of the projection operators $A_1$ and $A_2$, which project a discrete ...
- 153
0
votes
0
answers
23
views
Title: Calculating the First Derivative with Respect to $\xi_j$ in a Mixture Model
Title: Calculating the First Derivative with Respect to $\xi_j$ in a Mixture Model
I'm currently reading the section on soft parameter sharing in Chapter 9 of Deep Learning by Christopher M. Bishop, ...
- 416
8
votes
2
answers
270
views
Value of $\lim_{\epsilon \rightarrow 0^+} \left(\frac{1}{(x- i \epsilon)^a} - \frac{1}{(x+ i \epsilon)^a}\right)$?
I am trying to obtain a relation which generalises
$$
\lim_{\epsilon \rightarrow 0^+}\left(\frac{1}{x-i \epsilon} - \frac{1}{x+i \epsilon}\right) = 2 \pi i \delta(x)
$$
for some generic power $a$ of ...
- 71
1
vote
1
answer
38
views
Divergent Tail Sums of Approximations of Non-trace Class Compact Operators
I'm working on approximations of compact operators that are not trace class, and I'm looking for ways to provide meaningful approximation error estimates for truncated eigenfunction expansions. I ...
user1328882
0
votes
1
answer
86
views
convex optimization with L1 and L2 regularization
Can we solve the problem with the form $$min_{x}||Ax - b||^2_2+\lambda||x||_1+\mu||x^2-y||^2_2$$ where y is a given vector.
This problem is a combination of least-squares regression with L1 and L2 ...
1
vote
1
answer
74
views
Solve the Soft SVM Dual Problem with L1 Regularization
I'm considering a support vector regression model with a prediction
$$ \hat{y}(\mathbf{x}_\star)=\boldsymbol{\theta}^{\top} \boldsymbol{\phi}(\mathbf{x}_\star)$$
where $\boldsymbol{\theta}$ are the ...
- 239
2
votes
1
answer
118
views
Make a non-smooth function smooth [duplicate]
I am dealing with a piecewise affine function $f$ defined as follows: $f(x)=0$ if $x<1$, $f(x)=1-x$ if $x\in [1,2]$ and $f(x)=-1$ otherwise.
I want to make it smooth. I looked at sigmoid functions ...
- 169
2
votes
0
answers
126
views
Generalization error bound for Empirical Risk minimizer on Gaussian noisy data
I have datapoints that are sampled from a distribution $\mathbb{D}$. Each datapoint is a tuple $(t,y)$ of a time $t \in [0,T]$ that is sampled uniformly and a value $y(t) \sim u(t) + \mathcal{N}(0, \...
- 569
0
votes
0
answers
35
views
Is the function $\lVert A(A^TA+\lambda I)^{-1}A^Ty \rVert_2$ decreasing in $\lambda$?
Let $A$ be some matrix over $\mathbb R$.
Is the function $f(\lambda)=\lVert A(A^TA+\lambda I)^{-1}A^Ty \rVert_2$ decreasing for $\lambda > 0$? Here $y$ is an arbitrary real vector of the correct ...
- 929
0
votes
1
answer
53
views
Regularization of Interest Rate Inverse Problem failing
I'm going through this Inverse Problems textbook: https://www.rose-hulman.edu/~bryan/invprobs/inversefin1.pdf . In one exercise, I have to perform a regularization on an inverse problem, but I get ...
- 3
0
votes
1
answer
34
views
Can one use asymptotic behaviour to deduce the induced distribution from a function?
As an example, imagine I have $e^{i\sqrt{x^2+y^2}\cdot k}$ defined for $x$ on the reals. Then, in a distributional sense, I want to understand
$$\int e^{i\sqrt{x^2+y^2}\cdot k} dx$$
My naive approach ...
- 167
0
votes
0
answers
16
views
How this variational derivative is calculated?
In this paper https://arxiv.org/pdf/1907.09605.pdf \
let $\Omega \subset \mathbb{R}^n$ with $n \geq 1$ be a bounded Lipschitz domain with boundary $\partial \Omega$, $f: \Omega \rightarrow \mathbb{R}$ ...
- 1
1
vote
1
answer
69
views
Making a symmetric matrix positive (semi-)definite by adding a diagonal matrix
Say I have a matrix $A \in R^{p \times p}$ which is symmetric and with non-negative diagonal entries (i.e. $a_{ii} \geq 0 \forall i\in \{1, \ldots, p\}$). However, $A$ is not positive (semi-)definite.
...
- 25
1
vote
0
answers
64
views
Strong & weak solution of $xy' + y = 0$ [Friedlander ex. 2.3]
The ODE $xy' + y = 0$ has no strong solution over $\mathbb R$ but has solution $y(x) = \begin{cases} c_1/x & x<0 \\ c_2/x & x>0 \end{cases}$ over $\mathbb R^*$, which may equivalently be ...
- 687
0
votes
0
answers
52
views
Regularisation for a differential equation
When stuying a 1D differential equation, I tried to solve the problem by finding the Green's function and then solving with the inverse Fourier transform.
To make the problem well-behaved, I used a ...
- 1,281
1
vote
0
answers
82
views
Is there a Closed-Form Solution for L2 Regularization Raised to a Power?
Recently, I came across a modified L2 regularization term as stated in the equation below, where $\gamma$ is a positive number.
$$
\lambda'(w^Tw)^\gamma
$$
I'm curious if a closed-form solution ...
- 11
1
vote
1
answer
52
views
Is there any method that can optimize the problem whose regularizer is kurtosis term?
I recently worked on an optimization problem, whose regularizer $g(x)$ is kurtosis. The overall optimization formula is as follows.
$$\begin{align} \arg \min_x \frac12 \Vert Ax-b\Vert_2^2 + \lambda g(...
- 35
0
votes
1
answer
84
views
Iterative shrinkage algorithm for L1 optimization - A MATLAB Code testing
I am going to solve the problem:
$x =: \underset{x}{\text{argmin }}\frac{1}{2}||y-Ax||^2+||x||_1$
using iterative shrinkage algorithms (ISTA). There is a modified version know as FISTA which is as ...
- 381
3
votes
0
answers
46
views
Regularization involving Stieltjes constants: $\displaystyle\sum_{k=1}^{\infty}\frac{\ln(k)^n}{k}\overset{\mathcal{R}}{=}\gamma_n$
Notation
$\zeta(z)$ is the Riemann zeta function
$\operatorname{Li}_{\nu}(z)$ is the polylogarithm function
$\operatorname{Li}^{(n,0)}_{\nu}(z):=\frac{\partial^n}{\partial\nu^n}\operatorname{Li}_\nu(...
0
votes
0
answers
24
views
Penalty function $f(a, b)$ for $a, b \in \mathbb{R}^+$ which adds a high positive penalty only when both $a, b$ are very small
I'm doing Bayesian recruitment curve fitting where my curve has two parameters $a$ and $b$. Both $a, b \in \mathbb{R}^+$. I have put a Truncated Normal prior on $a$ and a Half Normal prior on $b$. I'm ...
- 47
0
votes
1
answer
87
views
Regularizing a divergent sum
I have a sum of an infinite series
$$ S = \frac{1}{3} - 4 + \frac{196}{15} - 21 + 27 - 33 + 39 - 45 + 51 - 57 + 63 + ... $$
which appears to diverge. This can be separated as such
$$ S = (\frac{1}{3} ...
1
vote
1
answer
35
views
How to solve matrix equations with inconsistent dimensions？
I'm confused by something in one of the papers I'm reading, but I'm not sure if there's any math in it that I'm not aware of.
$\mathbf{E}=\mathbf{R}\mathbf{e}$, where $\mathbf{E}\in\mathbb{C}^{n_1\...
- 21
0
votes
0
answers
61
views
$L^2$ norm regularization in linear regression
I'm reading Deep Learning (Ian Goodfellow e Yoshua Bengio) and I'm stuck in this section. The authors try to show how the $L^2$ norm regularization impact on a simple linear model.
Reducing the sum of ...
3
votes
2
answers
115
views
Can a diverging series have a finite convergent value?
This might be an odd question to ask,
$$a - a^2 + a^3 - a^4 + ... = \frac{a}{1+a}$$
I came across this realisation while trying to understand a feedback loop of a buffer circuit of an op-amp.
The ...
- 427
0
votes
1
answer
159
views
Least squares problem with second derivative regularization.
I want to minimise Least squares problem with second derivative regularisation, that is, minimizing the following w.r.t $\mathbf{g}(x)$.
$$min_{\mathbf{g}(x)} ||\mathbf{A} \cdot \mathbf{g}(x) - \...
0
votes
0
answers
51
views
regularization and punishment method in least square fitting
I have a least square problem:
$$
\min \sum_j^p| f(x_j)-y_j |^2 \\
\text{where } f(x) = a x - \sum_{i}^{n} b_i J_1(c_i x)
$$
where $J_1$ is the first order Bessel function. I have to find a set ...
- 133
2
votes
1
answer
99
views
Sobolev regularity required for $|\nabla u|^2 \in H^1(\Omega)$
We have $n=2$ and $\Omega=\{x \in \mathbb{R}^2\| ||x||_2<1 \}$. I need $|\nabla u|^2 \in H^1(\Omega)$. So i would choose $u \in W^{2,4}(\Omega)$. But since we have $n=2$ I thought that a lower ...
- 21
1
vote
0
answers
26
views
Morozov Discrepancy for PDE-Constrained Optimization With Bound Constraints
$\textbf{Background and context}$
Let $\beta:\Omega\rightarrow\mathbb{R}$, be an unknown spatially distributed parameter (over the spatial domain $\Omega$). Let $\mathcal{F}$ represent a parameter-to-...
- 2,115
2
votes
2
answers
48
views
Weighted sum of N images, which minimizes their TV norm
I have $K$ images $I_i, i\in{1 \ldots K}$ of the size $M \times N$. I wish to find weights $w_i$, s.t. $w_i \in [0,1]$ and $\sum_1^K
w_i = 1$ so that
$$|\sum_{i=1}^{K} w_i I_i|_{TV}$$
is minimal. I ...
- 370
1
vote
1
answer
90
views
Convergence of regularized optimization problems to constraint optimization problem
I have seen many times the following fact, which seems to be folklore knowledge. Assume that you have an optimization problem of the form
$$\min_{x \in \mathcal{X}} F(x) + \lambda G(x)$$
with $F$ and $...
- 822
0
votes
0
answers
109
views
About regularization of max function
I have a question in Demailly's book(Complex Analytic and Differential Geometry in Page 43):
Let $\theta \in C^{\infty}(R)$ be a nonnegative function with support in [-1,1] such that
$\int_{\mathbb{R}...
- 13
1
vote
0
answers
33
views
How do these definitions of regularization match?
I know regularization from the following point of view:
You add a term R(f) to a loss function, e.g:
$min_{f} \sum V(f(x_i),y_i) + \lambda R(f)$ where $\lambda$ is a parameter.
I recently read about ...
- 105
1
vote
1
answer
245
views
Regularized optimization : adding a vanishingly small penalty term does not change the solution set?
Say I am trying to minimize a differentiable function $R: \Theta\to\mathbb R^+$, where $\Theta\subseteq\mathbb R^p $ is a compact subset. Now, for $\lambda\ge0 $, I define the $ \lambda$-regularized ...
0
votes
1
answer
74
views
Definition of $a=0$ limit of Hurwitz Zeta function
The definition of Hurwitz zeta function is
$$
\zeta(s,a) =\sum_{k=0}^\infty \frac{1}{(k+a)^s}
$$
where $a=0$ limit is obvious singular.
But in functionsite: https://functions.wolfram.com/...
- 109
2
votes
1
answer
64
views
Rewriting the solution of $\ell_2$-reweighted least-squares
Consider the problem
$$ \underset{x \in \Bbb R^n}{\operatorname{argmin}} \|Ax-y\|_2^2 + \lambda\|Wx\|_2^2 $$
where $A \in \Bbb R^{m \times n}$ with $m<n$, $W \in \Bbb R^{n \times n}$ is an diagonal ...
- 21
1
vote
0
answers
73
views
Best inverse / minimization solution of ill conditioned matrix and underdetermined system
I have an ill-conditioned matrix G representing Green's function estimates of a strain model for a rock mass that has been instrumented with optical fiber measuring strain. My problem is seemingly the ...
- 11
1
vote
0
answers
84
views
Proximal operator of the $L^1$ norm, constrained to $x\ge0$
A standard variational problem (arising, e.g., in imaging) reads
$$ \operatorname{argmin}_x \frac{1}{2}\Vert Ax - y\Vert ^2 + \Vert x\Vert_1 $$
where the $1$-norm serves as a sparsifying regularizer. ...
- 905
0
votes
0
answers
59
views
L1 regularization success rates (MATLAB minL1lin function)
Following a study on Making do with less: An introduction to Compressed Sensing I am attempting to simulate the weighing of 100 coins. There are five sets of coins. In each set there are counterfeit ...
- 25
0
votes
1
answer
249
views
Arg Min in LASSO Regression
I am trying to wrap my head around Arg Min, and I think I may have grasped it, so I want to discuss the interpretation in a specific example from a LASSO regression model
$$
\hat{\beta_h} =
\arg \...
0
votes
0
answers
87
views
Least-squares optimization with $\ell_2$ and (negative) $\ell_1$ regularization
I want to solve the following optimization problem:
$$\mathop{\operatorname{arg\,min}}\limits_x \|y-Ax\|_2^2 + \lambda \|x\|_2^2 - \lambda \cdot 2c \cdot \|x\|_1$$
My idea was to use a proximal ...
- 123
1
vote
0
answers
201
views
Proximal operator of the addition of a quadractic term and Lasso penalty
I am trying to derive the proximal operator $\operatorname{prox}_{\lambda}f(v), v \in \mathbb R^n$ of the addition between a quadratic term and a lasso penalty
$f(x) = \lVert b - Ax \rVert_2^2 + \tau \...
- 11
2
votes
1
answer
166
views
Constant terms of asymptotic expansions of smoothed sums of all prime numbers
Consider the series
$$\sum_{n=1}^\infty n e^{-n \varepsilon}$$
For $\varepsilon \leq 0$, it diverges. For $\varepsilon > 0$, it converges and equals
$$\frac{e^\varepsilon}{(e^\varepsilon - 1)^2}$$
...
- 6,156
0
votes
1
answer
279
views
Difference in Tikhonov regularization for linear and non-linear case?
the Tikhonov regularization for a linear operator $T: X \rightarrow Y, x \mapsto y$ means minimizing the least square problem
$$\begin{align*}
\lVert Tx - y \rVert^2_Y + \alpha \lVert x\rVert_X \...
- 183
0
votes
1
answer
35
views
Does exchanging the role data fit term and regularization term make sens?
For example the total variation regularization formulation is given as
$$
\min_x \frac{1}{2} || Ax -y ||_2^2 + \alpha ||\nabla x||_1.
$$
Now I am wondering, would it make sense to ask for a solution ...
- 51
3
votes
0
answers
35
views
Finding the smoothest function bounded by two other functions
Suppose I have two polynomial functions $f(x)$ and $z(x)$ defined over some closed interval $[a,b]$ $\in$ $\mathbb{R}$. The functions are subject only to the restriction that $f(x) > z(x)$ $\forall$...
- 131
1
vote
1
answer
525
views
Explicit relation between the regularization parameters in Ivanov and Tikhonov regularization
Consider the class of functions $\mathcal H_\phi $ defined by
$$\mathcal H_\phi :=\left\{h:\mathbb R^d \to \mathbb R\mid h(x) = \langle w,\phi(x)\rangle+b,\ (w,b)\in\mathbb R^D\times\mathbb R\right\} $...
1
vote
1
answer
137
views
Looking to add L1 regularization to a quadratic minimization
I'm hoping to minimize $\| y - Ax \|^2$ subject to a total variation constraint on the derivative of $x$ where $y$ is given. I'm hoping to use CVXOPT. I think I can set this up as follows:
$$
P = \...
- 131
3
votes
0
answers
230
views
What are the properties of this new characteristic of mathematical objects?
I will call it "hypermodulus". In simple words, hypermodulus is the exponent of the scalar part of the finite part of the logarithm of the object: $H(A)=\exp (\operatorname{scal} \...
- 9,566
2
votes
0
answers
127
views
Why does the Lasso only select only $n$ variables if $p \gg n$
$L_1$ or lasso regularization in regression problems is defined as $$\min||X\beta - y||_2^2 + \lambda ||\beta||_1$$ Multiple resources point out that for $X\in \mathbb{C}^{n\times p}$ if $p \gg n$ ...
- 333