Questions tagged [regularization]

Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, refers to a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting. (Def: http://en.wikipedia.org/wiki/Regularization_(mathematics))

Filter by
Sorted by
Tagged with
0
votes
1answer
20 views

Choosing a regularization parameter to ensure positive definite-ness.

This is for use in a nonlinear optimization routine. If I have a Hessian (or Hessian approximation) $H$, that is not guaranteed to be positive semi-definite (PSD), I'd like to find a regularization ...
2
votes
1answer
18 views

Must a stronger regularization lead to a larger loss?

In many machine learning problems, the objective function we aim to solve has the form: $\min_w \mathcal{L}(w) + \lambda\mathcal{R}(w)$, where $\mathcal{L}(w)$ (e.g., squared loss) is a loss function, ...
0
votes
1answer
27 views

Why is regularization used in linear regression?

I already understand that the point of regularization is to penalize (drive down) higher-order parameters for a model thereby increasing its generality. Outside of polynomial regression, I do not ...
1
vote
0answers
14 views

Regularized polynomial regression

I observe a set of $n$ observations $(X_i,Y_i)$, $X_i, Y_i$ are both scalars. I wish to construct an optimization criterion which for low values of the tuning parameter $\lambda$ produces a cubic fit ...
-3
votes
0answers
21 views

How do I merge two Frobenius norm of matrix?

Consider a soft thresholding peoblem: $$\mathop{\arg \min}\limits_B \left\| B \right\|_1 + \frac{1}{2\mu } \left( \left\|Y - AB - Z - \Lambda \right\|_F^2 + \left\| L - B - \Gamma \right\|_F^2 \right) ...
0
votes
0answers
33 views

Solve underdetermined, highly multicollinear, poorly scaled, but sparse in Fourier domain, system to high accuracy

I wish to solve systems of the form $$ \mathbf{y} = \mathbf{A}\mathbf{x} $$ for $\mathbf{x}$, where $\mathbf{y} \in \mathbb{R}^n$ is known, $\mathbf{A} \in \mathbb{R}^{n\times m}$ is known, $\mathbf{x}...
2
votes
1answer
36 views

Derivative of L1 norm of Hadamard product

I am trying to find the derivative of $f(B)=\lambda\Vert W \bigodot B \Vert_1 + \frac{\rho}{2}\Vert A-B \Vert_F^2 + tr(\Delta^T(A-B))$ with respect to B. where B is (n×n)matrix, W is (n×n)constant ...
0
votes
2answers
62 views

What is this function? $f(x)=\frac12\int_{-\infty}^\infty \frac{{\rm sgn} (x-t)}t dt$?

What is $f(x)=\frac12\int_{-\infty}^\infty \frac{\operatorname{sgn} (x-t)}t dt$ in closed form? The integral should be understood as the Cauchy principal value and regularized values when diverges. ...
0
votes
1answer
27 views

Understanding regularization in Least squares

I have a quesiton form Boyd and Vanderberghe' convex optimization book. The picture is shown below. If you see the first term in the equation i.e. sum of squared differences, I can see that ...
1
vote
0answers
29 views

Unclear transition in proof of the equivalence between weight decay and Gaussian noise on the inputs

According to slide 16 of this lecture, in a simple net with a linear output unit directly connected to the inputs, if one adds Gaussian noise to the inputs ($x_i + \epsilon_i$ where $\epsilon_i \sim \...
2
votes
3answers
55 views

Find a Symmetric Matrix $ N $ to Minimize $ {\left\| N - M \right\|}_{F}^{2} $ with Constraint $ N d = g $

I have a similar problem to Linear Matrix Least Squares with Linear Equality Constraint - Minimize $ {\left\| A - B \right\|}_{F}^{2} $ Subject to $ B x = v $, where there is no symmetric constraint ...
0
votes
0answers
12 views

Derivation of a Hard Treshold Operator Version

Define $$H_{T,\lambda}(x) = \begin{cases} x_i & \text{if $T(x_i)>\lambda$,} \\ 0 & \text{else} \end{cases} $$ for some $\lambda>0$ and some function $T$. This is like a "generalization" ...
0
votes
1answer
27 views

Proof regarding Ridge and Lasso regularization

I've a problem with understanding this exercise. Would be very happy to receive a little help here. Thanks]1
0
votes
0answers
33 views

How to solve with Tikhonov regularization or some kind of regularization? Ax = b

This is a classical issue! It have its orgins from Observer Kalman Filter Identification I have tried to solve it with pseudo inverse, that is using Singular Value Decomposition(SVD). I have also ...
0
votes
0answers
18 views

Hard Threshold Estimator for Denosing a Matrix Signal

In order to denoise a signal $s$ I use the hard threshold estimator defined as $$\hat s = s\cdot\mathsf 1(|s|-\lambda>0)$$ for some $\lambda > 0$. But what happens if $s$ is no scalar? Like in ...
2
votes
2answers
47 views

Regularization of a Matrix Using a Diagonal Matrix

For a $n\times n$ matrix $A$ it is well known that $A + \lambda I$ for suffiecientely large $\lambda >0$ makes $A$ positive definite. The proof is straightforward by looking at the characteristic ...
0
votes
0answers
9 views

How to do regularization onto a vector?

Assume that we have a vector $x(k)$ that contains noise. We don't know the noise. Now we want to do regularization onto $x(k)$ so it will become more...clear. Is that possible? I assuming that it ...
1
vote
1answer
63 views

How does this L1 regularization derivation follow? (Proof it makes sparse models)

I'm reading the "Deep Learning"(Goodfellow et al, 2016) book and on pages 231-232(you can check them here) they show a very unique proof how L1 regularization makes model sparse. You can skip to the ...
0
votes
0answers
20 views

Why is higher model complexity associated with high slope for linear regression?

In machine learning, why is a steeper slope for linear regression associated with high complexity of the model. I'm aware of regularization techniques that attempt to reduce the complexity of the ...
0
votes
1answer
63 views

Product of all positive odd numbers

I am looking for simplest regularisation of infinite product: $$ \prod^{\infty}_{k≥0} (2k+1) = ?? $$
1
vote
1answer
43 views

SVD to transform regularization problem

Can anyone explain this transformation to me : $$ ||Ax-b||_2^2 + \delta ||x||_2^2 : A \in R^{m,n}, b \in R^m \rightarrow \\ \tilde{x} = (V^Tx, V_2^Tx), \tilde{b} = (U^Tb, U_2^Tb)\\ V_2 \in R^{n x (n-...
0
votes
1answer
46 views

regularization of $\hat{x} = \arg\min_x|Ax - b|^2 + \lambda|x|^2$ using SVD of $A$

Suppose that the following energy is provided $$ \hat{x} = \arg\min_x|Ax - b|^2 + \lambda|x|^2 $$ with a given matrix $A$, a vector $b$ and regularization paramter $\lambda$. Analyze how the solution ...
0
votes
0answers
47 views

How to calculate variance of coefficient in lasso regression?

I got the interview question of how I can calculate variance of beta in lasso regression. We can easily calculate a variance of coefficient in the simple linear regression by a formula. I believe ...
0
votes
0answers
27 views

Section 5.5.1: L2 Regularization in Neural Network. (Book: PRML by C. Bishop)

It is regarding the re-scaling of L2 regularization parameter $\lambda$: If input $x_i$ is transform as $ax_i$ then $w_{ji}$ is scaled as $\frac{1}{a}w_{ij}$. In PRML Section 5.5.1 before equation 5....
0
votes
0answers
27 views

Gradient descent and graphical lasso

I was looking into the graphical lasso algorithm and saw that the update is being carried out via block coordinate descent. I wanted to know whether or not gradient descent can be used for the same? ...
1
vote
1answer
25 views

Covariance shrinkage and L2 penalty

I was reading about covariance shrinkage on scikit learn guides and came across this line: Mathematically, this shrinkage consists in reducing the ratio between the smallest and the largest ...
0
votes
1answer
90 views

Sum of 1 by all integer numbers

I wanna to regularise sum, motivated by question Path integral for fermion on circle: $$ \sum_{n\in\mathbb{Z}} 1 $$ Can I say that $$ \sum_{n\in\mathbb{Z}} 1 = \zeta(0) = -\frac{1}{2}? $$ I think ...
0
votes
1answer
141 views

What's the proof that $\int_0^\infty \tan x\,dx=\ln 2$?

What's the proof that $\int_0^\infty \tan x\,dx=\ln 2$? https://mathoverflow.net/a/349627/10059
1
vote
0answers
108 views

How to solve the truncated nuclear norm minimization problem to global optima?

Definition: Given a matrix $X\in\mathbb{R}^{m\times n}$ and a positive integer $r<\min\{m,n\}$, the truncated nuclear norm $\|X\|_{r,*}$ is defined as the sum of $\min\{m,n\}-r$ minimum singular ...
1
vote
0answers
15 views

How to achieve fitting with arbitrary cost regularization?

Perhaps the mathematically easiest case of fitting is linear least squares $L_2$ ("sum of squares"). This is so easy we often learn it already in high school or maybe in our first linear algebra class ...
2
votes
0answers
33 views

How to numerically solve Fredholm equation of the first kind using Tikhonov regularization method?

I am tasked to write a program that solves Fredholm equation of the first kind using Tikhonov regularization method. So far, unfortunately, I've only found the general form of Fredholm equation of the ...
1
vote
1answer
84 views

Regularization of integrals of the type $\int^1_0dx\frac{\ln^n(1-x)}{1-x}$

In many books, articles and theses dealing with perturbative QCD it is claimed that integrals of the form $\int^1_0dx\frac{\ln^n(1-x)}{1-x}$ for $n\in\mathbb{N}$ become finite when multiplying the ...
2
votes
0answers
50 views

Proof KKT points are inside a ball and find the Lagrange multipliers

I am trying to learn continuous optimization and I need to solve the following exercise. Despite the fact that I solved some exercises about KKT conditions and Lagrange multipliers, I can't solve this ...
1
vote
1answer
45 views

least squares regularization

I am stuck at a numerical algebra question For $\lambda\geq0$ $\displaystyle y(λ) = \arg \min_{y\epsilon R_n} ||b-Xy||_2^2 + \lambda||Ly||^2_2$ Show that if $\lambda_1>\lambda_2\geq 0$ $||Ly(\...
0
votes
1answer
88 views

Why can't Simplex Method solve big equations? Have I forgot something?

I just wrote a Simplex Method in pure C-code and I have tested it. It works for the objective function: $$\max: c^T x$$ With subject to: $$Ax \le b \\ x \ge 0$$ Here is an example: ...
0
votes
0answers
10 views

Regularized inverse approximation by Kronecker products and families of iterative solvers?

Given we know that multiplication by a matrix $\bf B$ can be represented with the help of Kronecker products ($\otimes$) and vectorization ($\text{vec}(\cdot)$), we can formulate $$\min_{\bf B_i}\|{\...
0
votes
0answers
39 views

Least Squares Problem with Tikhonov Regularization - Compare for Different Regularizer Values

I have a classic Least Squares Problem with Tikhonov Regularization: $$ y \left( \lambda \right) = \arg \min_{y\in R^n} {\left\| X y - b \right\|}_{2}^{2} + \lambda {\left\| L y \right\|}_{2}^{2}, \;...
2
votes
1answer
48 views

Under what conditions is the solution to a Lasso problem the same as the solution to a corresponding $\ell_0$-penalized least squares problem?

Let $\lambda > 0$ and let $x^*$ be a minimizer for the optimization problem $$ \text{minimize} \quad \frac12 \| Ax - b \|_2^2 + \lambda \|x \|_1. $$ Here $A$ is an $m \times n$ matrix, $b \in \...
1
vote
2answers
120 views

Question on suitability of zeta-regularization for certain divergent series

In another question here in MSE I arrived at the idea to decompose a divergent series, which was not Abel-summable, termwise into combination of alternating and non-alternating zeta series-terms, and ...
0
votes
0answers
36 views

Min-sum-max norm optimization with orthogonality constraint and matrix regularization

Let $S = \{s_1,\cdots,s_N\}\subset\mathbb{R}^n$ be a finite set of points, and $A\in\mathbb{R}^{n\times n}$ an invertible matrix. Then I would like to solve the following optimization problem: $$ \...
1
vote
0answers
31 views

Subtleties about analytic regularization

I want to calculate the integral $$I_n(\omega)=\int_{-\infty}^{\infty} \log^n\left(it-\omega\right) \, {\rm d}t$$ by analytic regularization where $\omega >0$ and $n \in \mathbb{N}$. One ...
1
vote
1answer
40 views

Can $L1$-regularization be applied in general case?

I am not very clear about how far $L1$-regularization can work. For example, let $x\in \mathbb{R}^n$. \begin{equation}\label{eq:Lasse1} \begin{aligned} &\max_{\mathbf{x}} & & f(\mathbf{x}...
1
vote
0answers
57 views

Are there examples when Cesaro, Abel, Ramanujan regularizations are applicable but do not coincide?

Are there any cases when Abel, Cesaro, Borel, Ramanujan, Zeta regularizations are applicable for regularization of a divergent series or integral but give different results?
1
vote
1answer
175 views

L1-regularization for convex functions

I want to find sufficient conditions such that for a convex function $f \colon \mathbb{R}^n \to \mathbb{R}$, finding the best $x$ with $\lvert\lvert x \rvert\rvert_1 \leq \delta$ is equivalent to ...
1
vote
1answer
49 views

Open set is an increasing union of regular open sets?

Let $X$ be a topological space. An open set $U$ in $X$ is called regular open if it equals to the interior of its closure, namely $\mathrm{int}(\mathrm{cl}(U))=U$. $X$ is called semiregular if its ...
6
votes
3answers
152 views

Regularization of $\sum_{n=2}^\infty (-1)^n \log n$

I accidentally stumbled on the following regularization of this divergent series: $$\sum_{n=2}^\infty (-1)^n \log n "=" \frac{1}{2} \log \frac{\pi}{2}$$ I'm not familiar enough with regularization, ...
1
vote
0answers
23 views

Regularity of solutions to a quadratic program?

Consider the map $ A : K \subset \mathbb{R}^m \rightarrow \mathbb{R}^m$, with $K$ compact and convex, defined by $$ A(c) = \text{arg}\min_{y \succeq 0} y^TDy + c^T y $$ Where $D$ is a positive ...
3
votes
1answer
112 views

Regulator for the harmonic series

Assume that $f$ is a function such that $\lim_{\varepsilon \rightarrow 0^+} f(\varepsilon) = 1$. Is there an $f$ such that $$\lim_{\varepsilon \rightarrow 0^+} \lim_{m \rightarrow \infty} \sum_{n=1}^...
0
votes
0answers
59 views

How to convert the following optimization problem to SDP format?

I want to convert the optimization problem $x \in \mathbb R^n$ to SDP format to further solve it using the famous solvers like sdpt3 and sedumi. $$\text{minimize} \quad \| A(x) - B \|_* + \lambda \| ...
2
votes
1answer
208 views

When to use L2 regularization?

We know that L1 and L2 regularization are solutions to avoid overfitting. L1 regularization, can lead to sparsity and therefore avoiding fitting to the noise. However, L2 does not. So I wonder when ...

1
2 3 4 5 6