Questions tagged [regularization]
Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, refers to a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting. (Def: http://en.wikipedia.org/wiki/Regularization_(mathematics))
356
questions
0
votes
0
answers
19
views
Penalty function $f(a, b)$ for $a, b \in \mathbb{R}^+$ which adds a high positive penalty only when both $a, b$ are very small
I'm doing Bayesian recruitment curve fitting where my curve has two parameters $a$ and $b$. Both $a, b \in \mathbb{R}^+$. I have put a Truncated Normal prior on $a$ and a Half Normal prior on $b$. I'm ...
- 33
0
votes
0
answers
33
views
Regularizing a divergent sum
I have a sum of an infinite series
$$ S = \frac{1}{3} - 4 + \frac{196}{15} - 21 + 27 - 33 + 39 - 45 + 51 - 57 + 63 + ... $$
which appears to diverge. This can be separated as such
$$ S = (\frac{1}{3} ...
1
vote
1
answer
25
views
How to solve matrix equations with inconsistent dimensions？
I'm confused by something in one of the papers I'm reading, but I'm not sure if there's any math in it that I'm not aware of.
$\mathbf{E}=\mathbf{R}\mathbf{e}$, where $\mathbf{E}\in\mathbb{C}^{n_1\...
- 11
0
votes
0
answers
37
views
$L^2$ norm regularization in linear regression
I'm reading Deep Learning (Ian Goodfellow e Yoshua Bengio) and I'm stuck in this section. The authors try to show how the $L^2$ norm regularization impact on a simple linear model.
Reducing the sum of ...
2
votes
2
answers
94
views
Can a diverging series have a finite convergent value?
This might be an odd question to ask,
$$a - a^2 + a^3 - a^4 + ... = \frac{a}{1+a}$$
I came across this realisation while trying to understand a feedback loop of a buffer circuit of an op-amp.
The ...
- 417
0
votes
0
answers
18
views
Adding random noise to an ill-posed problem
Suppose I have found a (finite) solution $x$ to an ill-posed problem, e.g.
\begin{equation}
b = A x
\end{equation}
where $A$ is a $N\times N$ matrix with a large condition number, $b$ is a vector ...
0
votes
0
answers
18
views
How to show that the trace of a regularized Laplacian defined on two sphere with radius $h\geq 1$ is diverging logarithmically?
Let $h,m\in[1,\infty)$. I would like to verify that the following sum diverges logarithmically
\begin{equation}
\sum_{d=0}^{\infty} \frac{2d+1}{2h^2(1+\frac{d(d+1)}{h^2})(1+\frac{d(d+1)}{h^2m ^2})^{2}}...
- 11
0
votes
0
answers
34
views
Least squares problem with second derivative regularization.
I want to minimise Least squares problem with second derivative regularisation, that is, minimizing the following w.r.t $\mathbf{g}(x)$.
$$min_{\mathbf{g}(x)} ||\mathbf{A} \cdot \mathbf{g}(x) - \...
0
votes
0
answers
29
views
regularization and punishment method in least square fitting
I have a least square problem:
$$
\min \sum_j^p| f(x_j)-y_j |^2 \\
\text{where } f(x) = a x - \sum_{i}^{n} b_i J_1(c_i x)
$$
where $J_1$ is the first order Bessel function. I have to find a set ...
- 93
2
votes
1
answer
73
views
Sobolev regularity required for $|\nabla u|^2 \in H^1(\Omega)$
We have $n=2$ and $\Omega=\{x \in \mathbb{R}^2\| ||x||_2<1 \}$. I need $|\nabla u|^2 \in H^1(\Omega)$. So i would choose $u \in W^{2,4}(\Omega)$. But since we have $n=2$ I thought that a lower ...
- 21
1
vote
0
answers
14
views
Morozov Discrepancy for PDE-Constrained Optimization With Bound Constraints
$\textbf{Background and context}$
Let $\beta:\Omega\rightarrow\mathbb{R}$, be an unknown spatially distributed parameter (over the spatial domain $\Omega$). Let $\mathcal{F}$ represent a parameter-to-...
- 2,080
1
vote
1
answer
34
views
Weighted sum of N images, which minimizes their TV norm
I have $K$ images $I_i, i\in{1 \ldots K}$ of the size $M \times N$. I wish to find weights $w_i$, s.t. $w_i \in [0,1]$ and $\sum_1^K
w_i = 1$ so that
$$|\sum_{i=1}^{K} w_i I_i|_{TV}$$
is minimal. I ...
- 348
0
votes
0
answers
85
views
Sum regularization/ use of generalized functions in Ramanujan master theorem applied to inverse function of $\frac{\sin(x)}x$
$\def\Si{\operatorname{Si}}$
The converse Ramanujan master theorem finds Taylor series coefficients. The goal of our question is applying it on $h(x)=\cases{f^{-1}(x),x\ge1\\ 0,0\le x<1}$:
where $...
- 10.6k
1
vote
1
answer
47
views
Convergence of regularized optimization problems to constraint optimization problem
I have seen many times the following fact, which seems to be folklore knowledge. Assume that you have an optimization problem of the form
$$\min_{x \in \mathcal{X}} F(x) + \lambda G(x)$$
with $F$ and $...
- 792
0
votes
0
answers
50
views
About regularization of max function
I have a question in Demailly's book(Complex Analytic and Differential Geometry in Page 43):
Let $\theta \in C^{\infty}(R)$ be a nonnegative function with support in [-1,1] such that
$\int_{\mathbb{R}...
- 11
1
vote
0
answers
27
views
How do these definitions of regularization match?
I know regularization from the following point of view:
You add a term R(f) to a loss function, e.g:
$min_{f} \sum V(f(x_i),y_i) + \lambda R(f)$ where $\lambda$ is a parameter.
I recently read about ...
- 107
1
vote
1
answer
80
views
Regularized optimization : adding a vanishingly small penalty term does not change the solution set?
Say I am trying to minimize a differentiable function $R: \Theta\to\mathbb R^+$, where $\Theta\subseteq\mathbb R^p $ is a compact subset. Now, for $\lambda\ge0 $, I define the $ \lambda$-regularized ...
0
votes
0
answers
13
views
Random Forest and XG Boost underperforming logit and CART
Overview of my topic
I’m trying to predict sovereign debt crisis based on various econ. indicators. I have 3 dummy columns – crisis, in1y, in2y. Crisis indicates whether a country experienced a ...
- 1
0
votes
1
answer
47
views
Definition of $a=0$ limit of Hurwitz Zeta function
The definition of Hurwitz zeta function is
$$
\zeta(s,a) =\sum_{k=0}^\infty \frac{1}{(k+a)^s}
$$
where $a=0$ limit is obvious singular.
But in functionsite: https://functions.wolfram.com/...
- 109
2
votes
0
answers
52
views
Rewriting the solution of $\ell_2$-reweighted least-squares
Consider the problem
$$ \underset{x \in \Bbb R^n}{\operatorname{argmin}} \|Ax-y\|_2^2 + \lambda\|Wx\|_2^2 $$
where $A \in \Bbb R^{m \times n}$ with $m<n$, $W \in \Bbb R^{n \times n}$ is an diagonal ...
- 21
1
vote
0
answers
41
views
Best inverse / minimization solution of ill conditioned matrix and underdetermined system
I have an ill-conditioned matrix G representing Green's function estimates of a strain model for a rock mass that has been instrumented with optical fiber measuring strain. My problem is seemingly the ...
- 11
1
vote
0
answers
48
views
Proximal operator of the $L^1$ norm, constrained to $x\ge0$
A standard variational problem (arising, e.g., in imaging) reads
$$ \operatorname{argmin}_x \frac{1}{2}\Vert Ax - y\Vert ^2 + \Vert x\Vert_1 $$
where the $1$-norm serves as a sparsifying regularizer. ...
- 749
0
votes
0
answers
39
views
Can a function be constructed by Heuristic Manipulations on this toy series?
Background and Heuristic Manipulations
Consider
$$ M = {e^x }' + {e^{e^x}}' + {e^{e^{e^x}}}' + \dots \tag{0}$$
This is a variation of a toy series:
$$ M = e^x + e^{e^x} e^x + e^{e^{e^x}}e^{e^x} e^x + \...
- 1,195
0
votes
0
answers
33
views
Principal Value Integral convergence
I am using in my calculations the Principal Value Integral $$x^2\int_a^\infty dx' \frac{f(x')}{x'-x}.$$
I think that the result will always diverge for $x\to\infty$, no matter how strongly $f(x')$ ...
- 1
0
votes
0
answers
42
views
L1 regularization success rates (MATLAB minL1lin function)
Following a study on Making do with less: An introduction to Compressed Sensing I am attempting to simulate the weighing of 100 coins. There are five sets of coins. In each set there are counterfeit ...
- 25
0
votes
1
answer
132
views
Arg Min in LASSO Regression
I am trying to wrap my head around Arg Min, and I think I may have grasped it, so I want to discuss the interpretation in a specific example from a LASSO regression model
$$
\hat{\beta_h} =
\arg \...
0
votes
0
answers
74
views
Least-squares optimization with $\ell_2$ and (negative) $\ell_1$ regularization
I want to solve the following optimization problem:
$$\mathop{\operatorname{arg\,min}}\limits_x \|y-Ax\|_2^2 + \lambda \|x\|_2^2 - \lambda \cdot 2c \cdot \|x\|_1$$
My idea was to use a proximal ...
- 123
1
vote
0
answers
111
views
Proximal operator of the addition of a quadractic term and Lasso penalty
I am trying to derive the proximal operator $\operatorname{prox}_{\lambda}f(v), v \in \mathbb R^n$ of the addition between a quadratic term and a lasso penalty
$f(x) = \lVert b - Ax \rVert_2^2 + \tau \...
- 11
2
votes
1
answer
131
views
Constant terms of asymptotic expansions of smoothed sums of all prime numbers
Consider the series
$$\sum_{n=1}^\infty n e^{-n \varepsilon}$$
For $\varepsilon \leq 0$, it diverges. For $\varepsilon > 0$, it converges and equals
$$\frac{e^\varepsilon}{(e^\varepsilon - 1)^2}$$
...
- 5,907
0
votes
0
answers
45
views
Convergence of Tikohnov regularization for convex objective
I have the following question: Assume we want to minimize some convex (not strictly convex) and coercive ($f(x) \to \infty$ as $||x|| \to \infty$) function $f \in C^2(R^n,R)$, which has possibly ...
- 1
0
votes
1
answer
146
views
Difference in Tikhonov regularization for linear and non-linear case?
the Tikhonov regularization for a linear operator $T: X \rightarrow Y, x \mapsto y$ means minimizing the least square problem
$$\begin{align*}
\lVert Tx - y \rVert^2_Y + \alpha \lVert x\rVert_X \...
- 183
0
votes
1
answer
33
views
Does exchanging the role data fit term and regularization term make sens?
For example the total variation regularization formulation is given as
$$
\min_x \frac{1}{2} || Ax -y ||_2^2 + \alpha ||\nabla x||_1.
$$
Now I am wondering, would it make sense to ask for a solution ...
- 41
3
votes
0
answers
33
views
Finding the smoothest function bounded by two other functions
Suppose I have two polynomial functions $f(x)$ and $z(x)$ defined over some closed interval $[a,b]$ $\in$ $\mathbb{R}$. The functions are subject only to the restriction that $f(x) > z(x)$ $\forall$...
- 131
1
vote
1
answer
203
views
Explicit relation between the regularization parameters in Ivanov and Tikhonov regularization
Consider the class of functions $\mathcal H_\phi $ defined by
$$\mathcal H_\phi :=\left\{h:\mathbb R^d \to \mathbb R\mid h(x) = \langle w,\phi(x)\rangle+b,\ (w,b)\in\mathbb R^D\times\mathbb R\right\} $...
1
vote
1
answer
92
views
Looking to add L1 regularization to a quadratic minimization
I'm hoping to minimize $\| y - Ax \|^2$ subject to a total variation constraint on the derivative of $x$ where $y$ is given. I'm hoping to use CVXOPT. I think I can set this up as follows:
$$
P = \...
- 131
3
votes
0
answers
205
views
What are the properties of this new characteristic of mathematical objects?
I will call it "hypermodulus". In simple words, hypermodulus is the exponent of the scalar part of the finite part of the logarithm of the object: $H(A)=\exp (\operatorname{scal} \...
- 8,877
2
votes
0
answers
78
views
Why does the Lasso only select only $n$ variables if $p \gg n$
$L_1$ or lasso regularization in regression problems is defined as $$\min||X\beta - y||_2^2 + \lambda ||\beta||_1$$ Multiple resources point out that for $X\in \mathbb{C}^{n\times p}$ if $p \gg n$ ...
- 313
2
votes
0
answers
35
views
Is there a rigorous proof that $L_1$ regularization produces sparse results
I'm looking for a rigorous proof that the $L_1$ regularization really produces sparsity, there are multiple intuitive ones like these which I understand but all of these feel like they just explain ...
- 313
1
vote
1
answer
213
views
Why does least-squares need regularization?
If I understand regularization correctly, it helps if a least-squares problem is not well-posed thus...
the problem has no solution
the problem has multiple solutions
a small change in the input ...
- 313
6
votes
2
answers
276
views
Minimize trace of quadratic inverse + LASSO
Given a symmetric positive definite matrix $S \in \mathbb R^{d\times d}$ and $\lambda > 0$, I would like to find
$$X^\star := \underset{{X\in\mathbb R^{d\times d}}}{\operatorname{argmin}} \...
- 906
2
votes
0
answers
64
views
Derivatives of Matrices in Lagrangian and LASSO Optimization with Shrinkage Operators
The goal is to write a program that solves for x in
$$\textbf{min}\quad \|A\textbf{x}-\textbf{b}\|_1+\sigma \|\bf x \|_1$$
for $A\in \mathbb{R}^{m\times n}$. We assume that ${m}<{n}$ and ${\sigma}&...
- 201
1
vote
1
answer
112
views
Indefinite generalized Tikhonov regularization
Suppose we want to choose $x$ to minimise the following (generalized) Tikhonov regularized least squares objective:
$$(Ax-b)^\top (Ax-b) + \lambda [(x-c)^\top W (x-c)],$$
where $W$ is symmetric, but ...
- 715
1
vote
1
answer
132
views
Principal value integral with cosine
We are doing a practice set for an exam where we have to find the principal value of the integral
$$
I = P.V. \int^\infty_{-\infty} \frac{\cos(x)}{(x-1)(x^2+1)} dx
$$
Firstly, I am unsure if this ...
- 11
2
votes
1
answer
300
views
Optimal Transport and Entropic Regularization
We are working with discrete optimal transport.
Let $P$ be a matrix and let $H(P) =- \sum_{i,j} P_{i,j} (\log(P_{i,j})-1)$.
Let $C$ be the cost matrix. And $\langle C,P\rangle$ the Frobenius inner ...
- 1,588
0
votes
0
answers
92
views
Best optimization technique for solving overdetermined systems with a constraint
I am trying to make a prediction model based on a system of linear equations: $A\vec{x}=\vec{b}$, where $\vec{x}$ ($m\times1$) is my learning parameters, $A (m\times n)$ and $\vec{b}$ $(m\times1)$ are ...
- 43
1
vote
1
answer
138
views
For $u \in L^2([0,T])$ there exists $u_m \in C^\infty$ such that $u_m \to u$ in $L^2_{loc}(]0,T[)$.
I am reading Navier-Stokes Equations by Roger Temam and there is a point in a proof I do not understand. Let me explain:
We have a function $u: [0,T] \to H \in L^2([0,T]; H)$, for $H$ some Hilbert ...
- 3,923
2
votes
1
answer
49
views
Defining higher order (hypersingular) integrals in terms of derivatives of the Cauchy principal value?
In A Generalization of the Cauchy Principal Value, the author presents a way to assign values for hypersingular integrals of the form
$$
I=\int_a^b\frac{f(x)\,\mathrm dx}{(x-u)^n},\quad u\in(a,b)
$$
...
- 5,951
1
vote
1
answer
77
views
Interchanging order of integration in the Mellin transform
It is possible to understand (perhaps somewhat non-rigorously) the Fourier transform through interchanging order of integration and use of the delta function, like so:
$$\hat{f}(k)\equiv\int_{-\infty}^...
- 371
0
votes
1
answer
141
views
least squares with L1 regularization in selected entries
Say for $x \in \mathbb{R}^n$, I'm minimizing $\|Ax - b \|_2^2$ with L1 regularization on selected entries of $x$. i.e. instead of directly add a $\|x\|_1$ regularization term, it would be on $|x_i| + |...
- 255
0
votes
1
answer
261
views
Closed-form solution for quadratic optimization with L2 regularization?
Assume I have the following matrix equations, which I want to minimize over $x$
$$\min_{x}\left(\frac{1}{2}||Mx+b||^{2}_{2}+\lambda\left<x,x\right>\right)$$
where $x$ is a variable column vector,...
- 961