# Questions tagged [regularization]

Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, refers to a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting. (Def: http://en.wikipedia.org/wiki/Regularization_(mathematics))

253 questions
Filter by
Sorted by
Tagged with
20 views

### Choosing a regularization parameter to ensure positive definite-ness.

This is for use in a nonlinear optimization routine. If I have a Hessian (or Hessian approximation) $H$, that is not guaranteed to be positive semi-definite (PSD), I'd like to find a regularization ...
18 views

### Must a stronger regularization lead to a larger loss?

In many machine learning problems, the objective function we aim to solve has the form: $\min_w \mathcal{L}(w) + \lambda\mathcal{R}(w)$, where $\mathcal{L}(w)$ (e.g., squared loss) is a loss function, ...
27 views

### Why is regularization used in linear regression?

I already understand that the point of regularization is to penalize (drive down) higher-order parameters for a model thereby increasing its generality. Outside of polynomial regression, I do not ...
14 views

### Regularized polynomial regression

I observe a set of $n$ observations $(X_i,Y_i)$, $X_i, Y_i$ are both scalars. I wish to construct an optimization criterion which for low values of the tuning parameter $\lambda$ produces a cubic fit ...
21 views

46 views

### regularization of $\hat{x} = \arg\min_x|Ax - b|^2 + \lambda|x|^2$ using SVD of $A$

Suppose that the following energy is provided $$\hat{x} = \arg\min_x|Ax - b|^2 + \lambda|x|^2$$ with a given matrix $A$, a vector $b$ and regularization paramter $\lambda$. Analyze how the solution ...
47 views

### How to calculate variance of coefficient in lasso regression?

I got the interview question of how I can calculate variance of beta in lasso regression. We can easily calculate a variance of coefficient in the simple linear regression by a formula. I believe ...
27 views

### Section 5.5.1: L2 Regularization in Neural Network. (Book: PRML by C. Bishop)

It is regarding the re-scaling of L2 regularization parameter $\lambda$: If input $x_i$ is transform as $ax_i$ then $w_{ji}$ is scaled as $\frac{1}{a}w_{ij}$. In PRML Section 5.5.1 before equation 5....
27 views

### Gradient descent and graphical lasso

I was looking into the graphical lasso algorithm and saw that the update is being carried out via block coordinate descent. I wanted to know whether or not gradient descent can be used for the same? ...
25 views

### Covariance shrinkage and L2 penalty

I was reading about covariance shrinkage on scikit learn guides and came across this line: Mathematically, this shrinkage consists in reducing the ratio between the smallest and the largest ...
90 views

### Sum of 1 by all integer numbers

I wanna to regularise sum, motivated by question Path integral for fermion on circle: $$\sum_{n\in\mathbb{Z}} 1$$ Can I say that $$\sum_{n\in\mathbb{Z}} 1 = \zeta(0) = -\frac{1}{2}?$$ I think ...
141 views

### What's the proof that $\int_0^\infty \tan x\,dx=\ln 2$?

What's the proof that $\int_0^\infty \tan x\,dx=\ln 2$? https://mathoverflow.net/a/349627/10059
108 views

### How to solve the truncated nuclear norm minimization problem to global optima?

Definition: Given a matrix $X\in\mathbb{R}^{m\times n}$ and a positive integer $r<\min\{m,n\}$, the truncated nuclear norm $\|X\|_{r,*}$ is defined as the sum of $\min\{m,n\}-r$ minimum singular ...
15 views

### How to achieve fitting with arbitrary cost regularization?

Perhaps the mathematically easiest case of fitting is linear least squares $L_2$ ("sum of squares"). This is so easy we often learn it already in high school or maybe in our first linear algebra class ...
33 views

### How to numerically solve Fredholm equation of the first kind using Tikhonov regularization method?

I am tasked to write a program that solves Fredholm equation of the first kind using Tikhonov regularization method. So far, unfortunately, I've only found the general form of Fredholm equation of the ...
84 views

### Regularization of integrals of the type $\int^1_0dx\frac{\ln^n(1-x)}{1-x}$

In many books, articles and theses dealing with perturbative QCD it is claimed that integrals of the form $\int^1_0dx\frac{\ln^n(1-x)}{1-x}$ for $n\in\mathbb{N}$ become finite when multiplying the ...
50 views

### Proof KKT points are inside a ball and find the Lagrange multipliers

I am trying to learn continuous optimization and I need to solve the following exercise. Despite the fact that I solved some exercises about KKT conditions and Lagrange multipliers, I can't solve this ...
45 views

120 views

### Question on suitability of zeta-regularization for certain divergent series

In another question here in MSE I arrived at the idea to decompose a divergent series, which was not Abel-summable, termwise into combination of alternating and non-alternating zeta series-terms, and ...
36 views

Let $S = \{s_1,\cdots,s_N\}\subset\mathbb{R}^n$ be a finite set of points, and $A\in\mathbb{R}^{n\times n}$ an invertible matrix. Then I would like to solve the following optimization problem: $$\... 0answers 31 views ### Subtleties about analytic regularization I want to calculate the integral$$I_n(\omega)=\int_{-\infty}^{\infty} \log^n\left(it-\omega\right) \, {\rm d}tby analytic regularization where \omega >0 and n \in \mathbb{N}. One ... 1answer 40 views ### Can L1-regularization be applied in general case? I am not very clear about how far L1-regularization can work. For example, let x\in \mathbb{R}^n. \begin{equation}\label{eq:Lasse1} \begin{aligned} &\max_{\mathbf{x}} & & f(\mathbf{x}... 0answers 57 views ### Are there examples when Cesaro, Abel, Ramanujan regularizations are applicable but do not coincide? Are there any cases when Abel, Cesaro, Borel, Ramanujan, Zeta regularizations are applicable for regularization of a divergent series or integral but give different results? 1answer 175 views ### L1-regularization for convex functions I want to find sufficient conditions such that for a convex function f \colon \mathbb{R}^n \to \mathbb{R}, finding the best x with \lvert\lvert x \rvert\rvert_1 \leq \delta is equivalent to ... 1answer 49 views ### Open set is an increasing union of regular open sets? Let X be a topological space. An open set U in X is called regular open if it equals to the interior of its closure, namely \mathrm{int}(\mathrm{cl}(U))=U. X is called semiregular if its ... 3answers 152 views ### Regularization of \sum_{n=2}^\infty (-1)^n \log n I accidentally stumbled on the following regularization of this divergent series:\sum_{n=2}^\infty (-1)^n \log n "=" \frac{1}{2} \log \frac{\pi}{2}$$I'm not familiar enough with regularization, ... 0answers 23 views ### Regularity of solutions to a quadratic program? Consider the map  A : K \subset \mathbb{R}^m \rightarrow \mathbb{R}^m, with K compact and convex, defined by$$ A(c) = \text{arg}\min_{y \succeq 0} y^TDy + c^T y $$Where D is a positive ... 1answer 112 views ### Regulator for the harmonic series Assume that f is a function such that \lim_{\varepsilon \rightarrow 0^+} f(\varepsilon) = 1. Is there an f such that$$\lim_{\varepsilon \rightarrow 0^+} \lim_{m \rightarrow \infty} \sum_{n=1}^...
I want to convert the optimization problem $x \in \mathbb R^n$ to SDP format to further solve it using the famous solvers like sdpt3 and sedumi. \text{minimize} \quad \| A(x) - B \|_* + \lambda \| ...