Questions tagged [regularization]

Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, refers to a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting. (Def: http://en.wikipedia.org/wiki/Regularization_(mathematics))

Filter by
Sorted by
Tagged with
6 votes
2 answers
144 views
+50

Minimize trace of quadratic inverse + LASSO

Given a symmetric positive definite matrix $S \in \mathbb R^{d\times d}$ and $\lambda > 0$, I would like to find $$X^\star := \underset{{X\in\mathbb R^{d\times d}}}{\operatorname{argmin}} \...
user avatar
1 vote
0 answers
33 views

Derivatives of Matrices in Lagrangian and LASSO Optimization with Shrinkage Operators

The goal is to write a program that solves for x in $$\textbf{min}\quad \|A\textbf{x}-\textbf{b}\|_1+\sigma \|\bf x \|_1$$ for $A\in \mathbb{R}^{m\times n}$. We assume that ${m}<{n}$ and ${\sigma}&...
user avatar
-1 votes
0 answers
17 views

Multiplying divergent integrals using Hardy fields approach

So, I wonder if the following makes sense. Suppose we want to multiply $\int_0^\infty e^x dx\cdot\int_0^\infty e^x dx$. The partial sums of these improper integrals are $\int_0^x e^x dx=e^x-1$. Now we ...
user avatar
  • 7,419
0 votes
0 answers
27 views

derive the proximal operator for a squared error

To calculate the proximal operator of $f=\frac{1}{2} \Vert x \Vert_2^2$: $\mathbb{prox}_{\lambda f}(x)=\left(\frac{1}{1 + \lambda} \right) x$ following, what is the proximal operator of the pairwise ...
user avatar
  • 101
1 vote
1 answer
56 views

Indefinite generalized Tikhonov regularization

Suppose we want to choose $x$ to minimise the following (generalized) Tikhonov regularized least squares objective: $$(Ax-b)^\top (Ax-b) + \lambda [(x-c)^\top W (x-c)],$$ where $W$ is symmetric, but ...
user avatar
  • 627
1 vote
1 answer
68 views

Principal value integral with cosine

We are doing a practice set for an exam where we have to find the principal value of the integral $$ I = P.V. \int^\infty_{-\infty} \frac{\cos(x)}{(x-1)(x^2+1)} dx $$ Firstly, I am unsure if this ...
user avatar
  • 11
2 votes
1 answer
37 views

Optimal Transport and Entropic Regularization

We are working with discrete optimal transport. Let $P$ be a matrix and let $H(P) =- \sum_{i,j} P_{i,j} (\log(P_{i,j})-1)$. Let $C$ be the cost matrix. And $\langle C,P\rangle$ the Frobenius inner ...
user avatar
  • 1,342
0 votes
0 answers
39 views

Best optimization technique for solving overdetermined systems with a constraint

I am trying to make a prediction model based on a system of linear equations: $A\vec{x}=\vec{b}$, where $\vec{x}$ ($m\times1$) is my learning parameters, $A (m\times n)$ and $\vec{b}$ $(m\times1)$ are ...
user avatar
1 vote
1 answer
89 views

For $u \in L^2([0,T])$ there exists $u_m \in C^\infty$ such that $u_m \to u$ in $L^2_{loc}(]0,T[)$.

I am reading Navier-Stokes Equations by Roger Temam and there is a point in a proof I do not understand. Let me explain: We have a function $u: [0,T] \to H \in L^2([0,T]; H)$, for $H$ some Hilbert ...
user avatar
  • 2,465
0 votes
0 answers
9 views

What is the difference between regularization and preconditioning for a matrix?

The literature says "Preconditioning is a technique for improving the condition number of a matrix. Suppose that $M$ is a symmetric, positive-definite matrix that approximates $A$, but is easier ...
user avatar
2 votes
1 answer
27 views

Defining higher order (hypersingular) integrals in terms of derivatives of the Cauchy principal value?

In A Generalization of the Cauchy Principal Value, the author presents a way to assign values for hypersingular integrals of the form $$ I=\int_a^b\frac{f(x)\,\mathrm dx}{(x-u)^n},\quad u\in(a,b) $$ ...
user avatar
1 vote
1 answer
54 views

Interchanging order of integration in the Mellin transform

It is possible to understand (perhaps somewhat non-rigorously) the Fourier transform through interchanging order of integration and use of the delta function, like so: $$\hat{f}(k)\equiv\int_{-\infty}^...
user avatar
  • 361
0 votes
0 answers
27 views

Regularity results for ODEs weaker than Lipschitz in a single point

I have to solve the following minimal example for $t\geq 0 $: $$ \dot s(t)= \frac{1}{2s(t)}, \quad s(0)=0 $$ This is a minimal example, I know I could solve it easily using the separation of variables....
user avatar
0 votes
0 answers
12 views

Regularizing Jacobian by altering the differentiated function?

I have a function $f$ such that I want to solve for a linear system with its Jacobian: $$ \left[\frac{\partial f}{\partial x}\right] a = b $$ I want to impose constraints to $x$ so that the space ...
user avatar
0 votes
1 answer
31 views

least squares with L1 regularization in selected entries

Say for $x \in \mathbb{R}^n$, I'm minimizing $\|Ax - b \|_2^2$ with L1 regularization on selected entries of $x$. i.e. instead of directly add a $\|x\|_1$ regularization term, it would be on $|x_i| + |...
user avatar
  • 155
0 votes
1 answer
48 views

Closed-form solution for quadratic optimization with L2 regularization?

Assume I have the following matrix equations, which I want to minimize over $x$ $$\min_{x}\left(\frac{1}{2}||Mx+b||^{2}_{2}+\lambda\left<x,x\right>\right)$$ where $x$ is a variable column vector,...
user avatar
  • 855
-2 votes
1 answer
53 views

Minimizing a linear objective plus $\| \cdot \|_1$ [closed]

What is the best method for solving the following convex regularization problem? $$ \text{minimize} ~~ c^T x + \| A x \|_{1} \quad \text{subject to} \quad -1 \leq x_i \leq 1 $$ where $c$ is a vector ...
user avatar
  • 6,423
0 votes
0 answers
20 views

Piecewise quadratic function

I'm reading this paper on the sparse-group lasso, and it states at a specific point that the function is a piecewise quadratic: (section 3.3, page 8) $$\left\|S(X^{(l)}y/n, \lambda \alpha) \right\|_2^...
user avatar
  • 13
0 votes
0 answers
25 views

Difference / similarities between Tikhonov's regularization and the least squares method?

What is the difference / similarities between Tikhonov's regularization and the least squares method? I have tried to find information on both but can´t find any clear answers to this question, so I ...
user avatar
  • 199
0 votes
0 answers
32 views

Deconvolution experimental data by solving as Tikhonov regulation of Fredholm integral equation

From an experiment, I have data for time t and a function of time $f(t)$. Data can be described by a Fredholm integral equation such that: $$ F(t) = \int_{0}^{1}ke^{-kt}f(k) \,{\rm d} k $$ Here is the ...
user avatar
0 votes
1 answer
38 views

Working out the derivative of the log-likelihood for group LASSO

I'm following the working of the sparse group LASSO in the paper 'A Sparse-Group LASSO' by Simon. For the linear case, we have the problem given as $$\text{min}_\beta \frac{1}{2}||y-\sum_{l=1}^m X^{(l)...
user avatar
  • 13
0 votes
0 answers
21 views

Modulus of the "pair" $(0,\infty)$ seems to be a finite value?

It is known that the split-complex numbers are isomorphic to the pairs of real numbers in the following way: $a + bj \leftrightarrow (a - b, a + b)$ with operations defined on the pairs element-wise. ...
user avatar
  • 7,419
2 votes
0 answers
36 views

How to minimize quartic function for regularizing commutativity?

My objective is regularizing a function over $X\in \mathbb{R}^{I\times J}$ such that its covariance is jointly diagonalizable with a positive semi-definite matrix $A \in \mathbb{R}^{I\times I}$. For ...
user avatar
0 votes
0 answers
41 views

When is Lagrangian form equal to constrained optimization form?

When can we say that $\min_x f(x)$ s.t. $g(x)<c$ and $\min_x f(x)+\lambda g(x)$,$\lambda>0$ are equivalent? To add more context here: I understood from class that these formulations are ...
user avatar
  • 7,897
0 votes
0 answers
41 views

Transformation of Lasso to quadratic linear constrained form

The Lagrange formulation of the Lasso problem has the following form: $$\min_x ||Ax-b||_{2}^2+\lambda||x||_1$$ In the OSQP paper(A.5) by Stellato et al. there is the quadratic version $$\min_{x,t} y^\...
user avatar
0 votes
0 answers
15 views

Do Lasso's optimal features form a linearly independent set?

The Lasso problem is $$\min_{x\in \mathbb R^n} \frac{1}{2}||y-Ax||_2^2 +\lambda ||x||_1$$ where $\lambda >0$, $y\in \mathbb R^n$ and $A=[a_1,...,a_n]\in \mathbb R^{m\times n}$ is a matrix with $n$ ...
user avatar
  • 1,208
5 votes
0 answers
126 views

Weighted nuclear norm minimization

Crossposted on Operations Research Stack Exchange The problem. Let $X,A \in\mathbb{R}^{n\times m}$ and let $W\in\mathbb{R}^{nm\times nm}$ be a positive definite matrix. I want to know if there is a ...
user avatar
0 votes
1 answer
15 views

Solve ridge regression gradient over line segment

I want to solve the equation for ridge regression: \begin{align} f(\beta)=&\frac{1}{2}||X \beta-y||_2^2+\lambda_{1} \frac{1}{2}||\beta||^2_2 \\ \end{align} Where $X\in \mathbb{R}^{n\times n}, \...
user avatar
  • 37
0 votes
1 answer
16 views

Compute gradient of discretized energy function

Given the following discretized energy problem, I'd like to compute the gradient with respect to $u[i]$: $$ \min _{u} \sum_{i=1}^{N-1}|u[i+1]-u[i]|+\frac{\lambda}{2} \sum_{i=1}^{N}(u[i]-f[i])^{2} $$ I ...
user avatar
3 votes
0 answers
45 views

Reference for $\ell_1$-$\ell_2$ regularization

Given matrices $A, W \in \mathbb{R}^{n \times n}$ and vector $y \in \mathbb{R}^n$, consider the following optimization problem $$ \min_{x \in \mathbb{R}^n} \| W x \|_1 + \| Ax - y \|_2^2,$$ Is there ...
user avatar
1 vote
0 answers
20 views

Does the regularized optimization problem has solutions sufficiently close to those of the original problem?

Consider the following bilinear saddle point problem: \begin{align*} \min_x \max_y &~ f(x,y)\\ \text{where} &~ f(x,y) = x^\top A y + b^\top x + c^\top y, ~\text{and}~ x,y,b,c \in \...
user avatar
0 votes
0 answers
10 views

Compatibility of a regularizer with linear transformations of inputs and outputs

I am currently reading section '5.5.1 Consistent Gaussian Priors' of 'Pattern Recognition and Machine Learning' by Bishop. In this section they define a 2-layer perceptron with linear output units. ...
user avatar
0 votes
0 answers
11 views

What kind of regularization for Linear regression with MSE encourages "duplication"?

Linear Regression with MSE(mean squared error) loss is one of the most fundamental regression model in statistic. Given dataset $(X, Y)$ where $X \in \mathbb{R}^{n\times d}, Y \in \mathbb{R}^n$, one ...
user avatar
  • 1,111
1 vote
2 answers
60 views

Coercivness of a function involving $x$ and $\|x\|$

This problem often arises in machine learning. I'm looking at a regularized optimization problem of the form $$T(f) = L(f) + \lambda \|f\|^2 \tag{1}$$ where $f$ belongs to some reproducing kernel ...
user avatar
  • 145
-1 votes
2 answers
110 views

How did the cutoff regulization expanded to $\sum_{n=1}^\infty n=-\frac{1}{12}$?

On the Wikipedia, the Cutoff regularization, or the asymptotic behavior of the smoothing, was described to be Smoothing is a conceptual bridge between zeta function regularization, with its reliance ...
user avatar
0 votes
0 answers
22 views

Question about ridge regression using lagrange.

While studying about ridge regression, there was something confusing between A and B. ($X:n\times p$ matrix) A) $\displaystyle\min_{\beta}$ $(y-X\beta)^T(y-X\beta)$ subject to $\beta^T \beta \leq C$ ...
user avatar
0 votes
1 answer
37 views

Matrix devision - Bias Variance Tradeoff

I am currently trying to prove that the ordinary least squares estimate doesn't have a bias with a given dataset with the bias given as Why does this identity hold in the following calculation $$(X^...
user avatar
3 votes
1 answer
49 views

Why is the $L_2$ regularization squared while the $L_1$ is not?

Best example to demonstate my question is the elastic net, which has the Risk (here for linear regression). For some $D=\{(x_i,y_i)\}_{i=1}^n$ with $ x_i\in \mathbb{R}^d, y_i\in\mathbb{R}$ and some $ \...
user avatar
  • 93
3 votes
1 answer
68 views

Distribution (or Expectation) of Number of Non-Zero Coefficients in L1-Regularized Regression

Related to this question, suppose we're performing $L1$-regularized linear regression. For a given regularization coefficient $\lambda$, what is the distribution over the number of non-zero parameters?...
user avatar
2 votes
0 answers
46 views

Is this function continuous? (Moreau-Yosida regularisation with additional function argument)

I'm considering a generalised form of Moreau-Yosida regularisation. Given a continuous function $f:X \times U \rightarrow \mathbb{R}$ where $f(x, \cdot)$ is convex for any $x \in X$ and $X$, $U$ ...
user avatar
0 votes
0 answers
34 views

Continuity of pointwise infimum (of a special function); Moreau-Yosida regularisation

I'm seeking how to prove that Moreau-Yosida regularisation provides a continuous function: Given a convex function $f:X \rightarrow \mathbb{R}$ where $X$ is a compact subspace of an Euclidean space (...
user avatar
2 votes
0 answers
94 views

Green’s function of poisson’s equation in general dimensions

Green’s function of the Helmholtz equation, in the general dimension D, can be calculated as follows: \begin{align} G(\mathbf{r})&=\int \frac{d^D{k}}{(2\pi)^D} \frac{e^{-i \mathbf{k}\mathbf{r}}}{\...
user avatar
  • 131
1 vote
0 answers
30 views

Relation between moments a measured PMF under matrix multiplication?

I'm working on a physics problem where we have a measured photon energy spectrum (I'm thinking of it as a probability mass function, PMF), which is created by an energy spectrum of electrons which ...
user avatar
6 votes
1 answer
329 views

Heat Kernel on a compact manifold without boundary [closed]

I was wondering if the conservation of mass which is obvious for the heat equation in $\mathbb{R}^n$ holds also for the heat kernel in a general compact manifold without boundary. I mean I want to ...
user avatar
  • 101
0 votes
0 answers
6 views

Minimum of regularized cost function for a movie rating system

I'm currently reading Rafael Irizarry's Machine Learning notes and I was a little confused with the approach towards regularization. To analyse a movie recommendation system, the following equation is ...
user avatar
0 votes
0 answers
114 views

Do these constants appear in other areas of mathematics?

I decided to consider divergent (to infinity) integrals as some new kind of number. Towards this end, I began by establishing certain rules defining equivalence of the integrals. It seems the usual ...
user avatar
  • 7,419
2 votes
0 answers
64 views

Dos the Euler-Mascheroni constant $\gamma$ correspond to infinite hyperbolic angle?

So, I am trying to find the regularized value of the divergent integral $I=\int_1^\infty \sqrt{x^2-1}dx$. Since the area of $\int_0^1 \sqrt{1-x^2}dx=\frac\pi4$, I wonder whether the area under ...
user avatar
  • 7,419
5 votes
1 answer
359 views

How can you simplify/verify this solution for $\int\limits_0^{.25991…} Q^{-1}(x,x,x)dx?$

As I do not know the complex behavior of this function, it would be even harder to integrate past the real domain. The upper bound for the domain is a constant I will denote β. $${{Q_2}=\int_0^βQ^{-1}(...
user avatar
  • 5,209
1 vote
0 answers
29 views

Why do we subtract $1$ in this definition of the shannon entropy?

In the context of entropic regularization, the (negative) entropy is involved in such way (last term): Why do we subtract $-1$? So far, I have only known of the entropy being $\sum x_{i,j}\log(x_{i,j}...
user avatar
  • 1,255
6 votes
1 answer
574 views

On completing the solution for $\int_0^1 Q^{-1}(x,x) dx$ and other constants.

$\Large{\text{Introduction:}}$ Here is a link to the Inverse Regularized Incomplete Gamma function used in this problem. For simplicity, let the unit interval be expressed as $I=[0,1]$: $$\mathfrak{Q}=...
user avatar
  • 5,209

1
2 3 4 5
7