Questions tagged [regularization]

Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, refers to a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting. (Def: http://en.wikipedia.org/wiki/Regularization_(mathematics))

Filter by
Sorted by
Tagged with
1 vote
0 answers
34 views

Can a function be constructed by Heuristic Manipulations on this toy series?

Background and Heuristic Manipulations Consider $$ M = {e^x }' + {e^{e^x}}' + {e^{e^{e^x}}}' + \dots \tag{0}$$ This is a variation of a toy series: $$ M = e^x + e^{e^x} e^x + e^{e^{e^x}}e^{e^x} e^x + \...
0 votes
0 answers
26 views

Principal Value Integral convergence

I am using in my calculations the Principal Value Integral $$x^2\int_a^\infty dx' \frac{f(x')}{x'-x}.$$ I think that the result will always diverge for $x\to\infty$, no matter how strongly $f(x')$ ...
  • 1
-1 votes
0 answers
16 views

Regularization not commutative

Is there any explicit examples that the regularized result of two diverging series is not equal to the result which is obtained by regularizing the series which is obtained by adding the terms under ...
  • 339
0 votes
0 answers
15 views

L1 regularization success rates (MATLAB minL1lin function)

Following a study on Making do with less: An introduction to Compressed Sensing I am attempting to simulate the weighing of 100 coins. There are five sets of coins. In each set there are counterfeit ...
  • 13
0 votes
1 answer
29 views

Arg Min in LASSO Regression

I am trying to wrap my head around Arg Min, and I think I may have grasped it, so I want to discuss the interpretation in a specific example from a LASSO regression model $$ \hat{\beta_h} = \arg \...
0 votes
0 answers
6 views

Equivalence of a l1-regularized regression problem and a specific robust optimization problem

The $l_1$-regularized $l_2$ regression problem is as follows, $$\min\limits_{x \in \mathbb{R}^n, t \in \mathbb{R}}\left\{\|b-A x-t e\|_2+\lambda\|x\|_1\right\},$$ where $A \in \mathbb{R}^{m \times n}, ...
0 votes
0 answers
55 views

Least-squares optimization with $\ell_2$ and (negative) $\ell_1$ regularization

I want to solve the following optimization problem: $$\mathop{\operatorname{arg\,min}}\limits_x \|y-Ax\|_2^2 + \lambda \|x\|_2^2 - \lambda \cdot 2c \cdot \|x\|_1$$ My idea was to use a proximal ...
  • 123
1 vote
0 answers
23 views

Proximal operator of the addition of a quadractic term and Lasso penalty

I am trying to derive the proximal operator $\operatorname{prox}_{\lambda}f(v), v \in \mathbb R^n$ of the addition between a quadratic term and a lasso penalty $f(x) = \lVert b - Ax \rVert_2^2 + \tau \...
0 votes
0 answers
9 views

Can I call a loss term 'regularizer' which is not related backpropagation?

I have a loss function for a supervised learning task, classification. Let's say $$ \min_{\theta} L = L_1 + L_2 $$ where $L_1$ is a BCE loss, and $L_2$ is a distance between the feature vector and a ...
  • 11
1 vote
0 answers
34 views

Why estimation of markov parameters are extreamly noise sensitive? [closed]

Assume that you have input $u(k)$ and output $y(k)$ and you want to find the impulse response $g(k)$, or even called markov parameters So let's say that I create some inputs $u(k)$ and outputs $y(k)$ ...
  • 2,700
2 votes
1 answer
115 views

Constant terms of asymptotic expansions of smoothed sums of all prime numbers

Consider the series $$\sum_{n=1}^\infty n e^{-n \varepsilon}$$ For $\varepsilon \leq 0$, it diverges. For $\varepsilon > 0$, it converges and equals $$\frac{e^\varepsilon}{(e^\varepsilon - 1)^2}$$ ...
  • 5,547
0 votes
0 answers
40 views

Convergence of Tikohnov regularization for convex objective

I have the following question: Assume we want to minimize some convex (not strictly convex) and coercive ($f(x) \to \infty$ as $||x|| \to \infty$) function $f \in C^2(R^n,R)$, which has possibly ...
  • 1
0 votes
1 answer
58 views

Difference in Tikhonov regularization for linear and non-linear case?

the Tikhonov regularization for a linear operator $T: X \rightarrow Y, x \mapsto y$ means minimizing the least square problem $$\begin{align*} \lVert Tx - y \rVert^2_Y + \alpha \lVert x\rVert_X \...
  • 121
0 votes
1 answer
25 views

Does exchanging the role data fit term and regularization term make sens?

For example the total variation regularization formulation is given as $$ \min_x \frac{1}{2} || Ax -y ||_2^2 + \alpha ||\nabla x||_1. $$ Now I am wondering, would it make sense to ask for a solution ...
  • 21
0 votes
0 answers
24 views

how to drive the right Moore–Penrose inverse

I am trying to prove that HTW=I where HT is NxM matrix and W is MxN matrix and I is NxN identity matrix the conditions are : Full Column Rank (or simply Column Rank): Rank of a matrix of order MxN is ...
3 votes
0 answers
30 views

Finding the smoothest function bounded by two other functions

Suppose I have two polynomial functions $f(x)$ and $z(x)$ defined over some closed interval $[a,b]$ $\in$ $\mathbb{R}$. The functions are subject only to the restriction that $f(x) > z(x)$ $\forall$...
1 vote
1 answer
127 views

Explicit relation between the regularization parameters in Ivanov and Tikhonov regularization

Consider the class of functions $\mathcal H_\phi $ defined by $$\mathcal H_\phi :=\left\{h:\mathbb R^d \to \mathbb R\mid h(x) = \langle w,\phi(x)\rangle+b,\ (w,b)\in\mathbb R^D\times\mathbb R\right\} $...
  • 2,322
1 vote
1 answer
48 views

Looking to add L1 regularization to a quadratic minimization

I'm hoping to minimize $\| y - Ax \|^2$ subject to a total variation constraint on the derivative of $x$ where $y$ is given. I'm hoping to use CVXOPT. I think I can set this up as follows: $$ P = \...
  • 121
0 votes
0 answers
15 views

Reproducing kernel Hilbert space norm as a smoothness functional

Let $K:X \times X \rightarrow \mathbb{R}$ be a Mercer kernel with an associated RKHS $H$ then the norm $|f|_H^2$ can be used as a way to ensure that $f$ is smooth in $H$. If i understand correctly, ...
  • 311
3 votes
0 answers
169 views

What are the properties of this new characteristic of mathematical objects?

I will call it "hypermodulus". In simple words, hypermodulus is the exponent of the scalar part of the finite part of the logarithm of the object: $H(A)=\exp (\operatorname{scal} \...
  • 8,320
1 vote
0 answers
41 views

Why does the Lasso only select only $n$ variables if $p \gg n$

$L_1$ or lasso regularization in regression problems is defined as $$\min||X\beta - y||_2^2 + \lambda ||\beta||_1$$ Multiple resources point out that for $X\in \mathbb{C}^{n\times p}$ if $p \gg n$ ...
0 votes
0 answers
9 views

Which $n$ values does lasso pick if $p \gg n$

It's known that lasso regression picks only $n$ variables in a matrix $X^{n\times p}$ matrix where $p \gg n$ but which variables exactly does lasso pick? Intuitively lasso should always pick the ...
2 votes
0 answers
28 views

Is there a rigorous proof that $L_1$ regularization produces sparse results

I'm looking for a rigorous proof that the $L_1$ regularization really produces sparsity, there are multiple intuitive ones like these which I understand but all of these feel like they just explain ...
1 vote
1 answer
95 views

Why does least-squares need regularization?

If I understand regularization correctly, it helps if a least-squares problem is not well-posed thus... the problem has no solution the problem has multiple solutions a small change in the input ...
6 votes
2 answers
213 views

Minimize trace of quadratic inverse + LASSO

Given a symmetric positive definite matrix $S \in \mathbb R^{d\times d}$ and $\lambda > 0$, I would like to find $$X^\star := \underset{{X\in\mathbb R^{d\times d}}}{\operatorname{argmin}} \...
1 vote
0 answers
53 views

Derivatives of Matrices in Lagrangian and LASSO Optimization with Shrinkage Operators

The goal is to write a program that solves for x in $$\textbf{min}\quad \|A\textbf{x}-\textbf{b}\|_1+\sigma \|\bf x \|_1$$ for $A\in \mathbb{R}^{m\times n}$. We assume that ${m}<{n}$ and ${\sigma}&...
0 votes
0 answers
40 views

derive the proximal operator for a squared error

To calculate the proximal operator of $f=\frac{1}{2} \Vert x \Vert_2^2$: $\mathbb{prox}_{\lambda f}(x)=\left(\frac{1}{1 + \lambda} \right) x$ following, what is the proximal operator of the pairwise ...
  • 101
1 vote
1 answer
77 views

Indefinite generalized Tikhonov regularization

Suppose we want to choose $x$ to minimise the following (generalized) Tikhonov regularized least squares objective: $$(Ax-b)^\top (Ax-b) + \lambda [(x-c)^\top W (x-c)],$$ where $W$ is symmetric, but ...
  • 623
1 vote
1 answer
91 views

Principal value integral with cosine

We are doing a practice set for an exam where we have to find the principal value of the integral $$ I = P.V. \int^\infty_{-\infty} \frac{\cos(x)}{(x-1)(x^2+1)} dx $$ Firstly, I am unsure if this ...
  • 11
2 votes
1 answer
109 views

Optimal Transport and Entropic Regularization

We are working with discrete optimal transport. Let $P$ be a matrix and let $H(P) =- \sum_{i,j} P_{i,j} (\log(P_{i,j})-1)$. Let $C$ be the cost matrix. And $\langle C,P\rangle$ the Frobenius inner ...
  • 1,436
0 votes
0 answers
63 views

Best optimization technique for solving overdetermined systems with a constraint

I am trying to make a prediction model based on a system of linear equations: $A\vec{x}=\vec{b}$, where $\vec{x}$ ($m\times1$) is my learning parameters, $A (m\times n)$ and $\vec{b}$ $(m\times1)$ are ...
1 vote
1 answer
101 views

For $u \in L^2([0,T])$ there exists $u_m \in C^\infty$ such that $u_m \to u$ in $L^2_{loc}(]0,T[)$.

I am reading Navier-Stokes Equations by Roger Temam and there is a point in a proof I do not understand. Let me explain: We have a function $u: [0,T] \to H \in L^2([0,T]; H)$, for $H$ some Hilbert ...
  • 3,375
2 votes
1 answer
37 views

Defining higher order (hypersingular) integrals in terms of derivatives of the Cauchy principal value?

In A Generalization of the Cauchy Principal Value, the author presents a way to assign values for hypersingular integrals of the form $$ I=\int_a^b\frac{f(x)\,\mathrm dx}{(x-u)^n},\quad u\in(a,b) $$ ...
1 vote
1 answer
61 views

Interchanging order of integration in the Mellin transform

It is possible to understand (perhaps somewhat non-rigorously) the Fourier transform through interchanging order of integration and use of the delta function, like so: $$\hat{f}(k)\equiv\int_{-\infty}^...
  • 361
0 votes
0 answers
28 views

Regularity results for ODEs weaker than Lipschitz in a single point

I have to solve the following minimal example for $t\geq 0 $: $$ \dot s(t)= \frac{1}{2s(t)}, \quad s(0)=0 $$ This is a minimal example, I know I could solve it easily using the separation of variables....
0 votes
0 answers
21 views

Regularizing Jacobian by altering the differentiated function?

I have a function $f$ such that I want to solve for a linear system with its Jacobian: $$ \left[\frac{\partial f}{\partial x}\right] a = b $$ I want to impose constraints to $x$ so that the space ...
0 votes
1 answer
63 views

least squares with L1 regularization in selected entries

Say for $x \in \mathbb{R}^n$, I'm minimizing $\|Ax - b \|_2^2$ with L1 regularization on selected entries of $x$. i.e. instead of directly add a $\|x\|_1$ regularization term, it would be on $|x_i| + |...
  • 255
0 votes
1 answer
126 views

Closed-form solution for quadratic optimization with L2 regularization?

Assume I have the following matrix equations, which I want to minimize over $x$ $$\min_{x}\left(\frac{1}{2}||Mx+b||^{2}_{2}+\lambda\left<x,x\right>\right)$$ where $x$ is a variable column vector,...
  • 895
-2 votes
1 answer
57 views

Minimizing a linear objective plus $\| \cdot \|_1$ [closed]

What is the best method for solving the following convex regularization problem? $$ \text{minimize} ~~ c^T x + \| A x \|_{1} \quad \text{subject to} \quad -1 \leq x_i \leq 1 $$ where $c$ is a vector ...
  • 6,644
0 votes
0 answers
38 views

Difference / similarities between Tikhonov's regularization and the least squares method?

What is the difference / similarities between Tikhonov's regularization and the least squares method? I have tried to find information on both but can´t find any clear answers to this question, so I ...
  • 359
0 votes
0 answers
35 views

Deconvolution experimental data by solving as Tikhonov regulation of Fredholm integral equation

From an experiment, I have data for time t and a function of time $f(t)$. Data can be described by a Fredholm integral equation such that: $$ F(t) = \int_{0}^{1}ke^{-kt}f(k) \,{\rm d} k $$ Here is the ...
0 votes
1 answer
72 views

Working out the derivative of the log-likelihood for group LASSO

I'm following the working of the sparse group LASSO in the paper 'A Sparse-Group LASSO' by Simon. For the linear case, we have the problem given as $$\text{min}_\beta \frac{1}{2}||y-\sum_{l=1}^m X^{(l)...
  • 3
2 votes
0 answers
47 views

How to minimize quartic function for regularizing commutativity?

My objective is regularizing a function over $X\in \mathbb{R}^{I\times J}$ such that its covariance is jointly diagonalizable with a positive semi-definite matrix $A \in \mathbb{R}^{I\times I}$. For ...
0 votes
0 answers
98 views

Transformation of Lasso to quadratic linear constrained form

The Lagrange formulation of the Lasso problem has the following form: $$\min_x ||Ax-b||_{2}^2+\lambda||x||_1$$ In the OSQP paper(A.5) by Stellato et al. there is the quadratic version $$\min_{x,t} y^\...
5 votes
0 answers
161 views

Weighted nuclear norm minimization

Crossposted on Operations Research Stack Exchange The problem. Let $X,A \in\mathbb{R}^{n\times m}$ and let $W\in\mathbb{R}^{nm\times nm}$ be a positive definite matrix. I want to know if there is a ...
0 votes
1 answer
22 views

Solve ridge regression gradient over line segment

I want to solve the equation for ridge regression: \begin{align} f(\beta)=&\frac{1}{2}||X \beta-y||_2^2+\lambda_{1} \frac{1}{2}||\beta||^2_2 \\ \end{align} Where $X\in \mathbb{R}^{n\times n}, \...
  • 47
1 vote
1 answer
22 views

Compute gradient of discretized energy function

Given the following discretized energy problem, I'd like to compute the gradient with respect to $u[i]$: $$ \min _{u} \sum_{i=1}^{N-1}|u[i+1]-u[i]|+\frac{\lambda}{2} \sum_{i=1}^{N}(u[i]-f[i])^{2} $$ I ...
3 votes
0 answers
50 views

Reference for $\ell_1$-$\ell_2$ regularization

Given matrices $A, W \in \mathbb{R}^{n \times n}$ and vector $y \in \mathbb{R}^n$, consider the following optimization problem $$ \min_{x \in \mathbb{R}^n} \| W x \|_1 + \| Ax - y \|_2^2,$$ Is there ...
1 vote
0 answers
20 views

Does the regularized optimization problem has solutions sufficiently close to those of the original problem?

Consider the following bilinear saddle point problem: \begin{align*} \min_x \max_y &~ f(x,y)\\ \text{where} &~ f(x,y) = x^\top A y + b^\top x + c^\top y, ~\text{and}~ x,y,b,c \in \...
1 vote
2 answers
84 views

Coercivness of a function involving $x$ and $\|x\|$

This problem often arises in machine learning. I'm looking at a regularized optimization problem of the form $$T(f) = L(f) + \lambda \|f\|^2 \tag{1}$$ where $f$ belongs to some reproducing kernel ...
  • 139

1
2 3 4 5
7