Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [regularization]

Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, refers to a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting. (Def: http://en.wikipedia.org/wiki/Regularization_(mathematics))

0
votes
2answers
41 views

What is the role of Tikhonov regularization in optimization?

Suppose I have the following objective function $$ L = \frac{1}{|X|}\sum_{x \in X} \| \hat{y} - y \|^2_2 + \lambda \|w\|_2^2 $$ where $X$ are my data, $\hat{y}$ the prediction, $y$ the target, $\...
2
votes
1answer
41 views

Stability of Tikhonov Regularization

I'm learning about Tikhonov regularization $$\underset{x\in X}{\arg\inf}\left\{||Ax-b||^2+\lambda ||x||^2\right\}$$ I have read that the solution keeps the residual $||Ax-b||^2$ small and is ...
0
votes
1answer
110 views

Why are these two expected values equal?

This question refers to section 13.2 in Chapter 13, "Regularization and Stability", from the book "Understanding Machine Learning: From Theory to Algorithms", which can be found in a PDF version here: ...
0
votes
1answer
25 views

The “units” of the terms of Tikhonov’s first order regularization functional

There’s one point about the order of regularization that I can’t understand. Let’s suppose that we have an ill-posed problem $Af=g$, where $g$ is the observation (with noise), $A$ is a linear operator,...
0
votes
1answer
62 views

Show that $\|.\|_{1/2}$ is not a norm. [duplicate]

Show that $\|.\|_{1/2}$ is not a norm. would anybody guide me that how can i prove or disprove? thanks
0
votes
1answer
23 views

Question regarding weak convergence

I'm doing my bachelor's thesis and I have stumbled upon a certain reoccurring thing in some of the proofs I have to make. The book that I use keeps on arguing that it is possible to assume weak ...
1
vote
0answers
21 views

Least-squares regularization matrix is not positive definite?

I am fitting data $(x_i,y_i)$ to the following model: $$ f(x) = \sum_j a_j g_j(x) = a^T g(x) $$ where $g_j(x)$ are well-conditioned basis functions (in my application they are B-splines, but I don't ...
0
votes
1answer
58 views

Sparse recovery with L1 shrinkage iteration for higher denominational image classification

For 2 months I have been studying sparse recovery and its applications for image classification and I have found that it's a broad area in mathematics which gives rise to a wide variety of ...
4
votes
1answer
41 views

Continuous Tikhonov Regularization for Deconvolution

I am trying to solve the following deconvolution problem where $g(s)$ is a known real valued function and has finite energy: $$g(s) = \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}f(t)e^{-(t-s)^2/2}...
0
votes
0answers
29 views

Preconditioning the Damped Normal Equations - Reference

For some people working with numerical approximations, it is very common to be dealing with a problem in the form $$\min_x \|Ax - b\|,$$ where $A$ is a matrix and $x,b$ are vectors. Consider that $A$ ...
1
vote
0answers
32 views

How to prove this (alternative) solution of ridge regression?

The ridge regression optimization formula for $X\in\mathbb{R}^{n\times m}$, $y\in\mathbb{R}^n$ with $n>m$, $$ \arg \min_{w\in \mathbb{R}^n} ||y-Xw||_2^2 + \lambda ||w||_2^2 $$ I know that the ...
0
votes
1answer
27 views

determine matrix and vector to fit regularized normal equation

I hope the title is not too unclear. I am given a Matrix $$A\in~\mathbb{R}^{K\times~N},~b\in~\mathbb{R}^{K}$$ and instead of solving the normal equation $min_{x\in~\mathbb{R}^N}|Ax-b|^2_2,~$ an $\...
0
votes
0answers
50 views

Nonparametric Bayesian estimation of several black-box functions of different variables from their noisy sums

In order to introduce my problem, let’s start with the nonparametric estimation of a single unknown/black-box function $f:{\Omega _f} \to \mathbb{R}$ of a discrete variable $x$ in a finite domain ${\...
3
votes
0answers
12 views

Interpolation with Regularized Linear Squares

I am fundamentally missing something and would appreciate some clarification. I am reading this text on Regularized Least Squares. This text has a problem where I have to find z, a set of unknown ...
1
vote
1answer
29 views

Question on derivation of probability proportion in Deep Learning Book

I am working through the Deep Learning Book, I am currently on the regularization chapter (https://www.deeplearningbook.org/contents/regularization.html). My question concerns the third step in the ...
3
votes
0answers
31 views

regularizing function under sqrt [closed]

Is it possible to find finite part of $\Sigma^\infty_n\sqrt{n^2 +a^2}$ using something like regularized zeta function?
1
vote
0answers
30 views

How can I incorporate this specific a-priori sparsity information in a regularization approach to guide my inverse problem?

Note: I will be using capital letters to denote matrices, lowercase letters to denote vectors, and the Greek alphabet to denote any scalar quantities. Note 2: I have tried my best to find a topic ...
3
votes
0answers
33 views

Dirac delta from poles of a function

Suppose we are given the simple expression $$ F(k) = \frac{1}{E^2-E(k)^2} $$ which has a pole when $E^2 = E(k)^2$ and where $E, E(k)$ are real numbers. When working with this expression (e.g. inside ...
0
votes
0answers
65 views

Laurent series of an integral with parameter

To find the Laurent series of function $f(a)$ at point $a=0$ $$ f(a)=\int^1_0 \frac{d x}{x^2+a^2} $$ one can first do the integral $$ f(a)=\frac{1}{a}\arctan(1/a) $$ then expand $\arctan(1/a)$ and ...
4
votes
1answer
104 views

Cesàro limit of a stochastic matrix

Let $A$ be a stochastic matrix. Then \begin{align*} \lim_{t \rightarrow\infty} A^t \end{align*} may not exist. For example: \begin{align*} A &= \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}...
3
votes
2answers
192 views

Understanding L2 Regularization Formula

I am currently following the Machine Learning Crash Course on Tensorflow and came across this formula: $$L_2\text{ regularization term} = \|\boldsymbol w\|_2^2 = {w_1^2 + w_2^2 + \cdots + w_n^2}$$ I ...
6
votes
3answers
487 views

Understanding the difference between Ridge and LASSO

I'll start by presenting my current understanding of regularized loss minimization. In the context of ML, generally we are trying to minimize directly an empirical loss, something of the form $$\...
0
votes
0answers
25 views

A ring to place numbers in, how to select numbers uniquely to minimize running variation?

Say I have the numbers in some list $\{1,2,2,3,4,3\}$ and I want to select them uniquely and place so that the average variation (for example running absolute difference, or running square difference) ...
4
votes
0answers
131 views

New/useful method for summation of divergent series?

Questions $$ S(n,x) = x+e^x + e^{e^x} + e^{e^{e^x}} + \dots \text{$n$ times}$$ Also obeys (see background for argument): $$ \frac{1}{2 \pi i} \oint e^{S(k,x)} \frac{\partial \ln(\frac{\int_0^\...
0
votes
0answers
113 views

Solution of L2 Norm and Total Variation (L1) regularizer combined

Is there any method for the direct solution of the following equation? $$ \min_{x} (||Hx - y||_2^2 + ||z - x||_2^2 + \lambda||\nabla x||_1) $$ I know the solution in case only the first two L2 norm ...
2
votes
0answers
99 views

Regularization of Exponentially exponential series?

Question What are the convergence properties of the last equation: $$K = e^{x} + x + \ln{x} + \ln\ln(x) + \dots $$ Can one artificially choose a value of $\ln (x)$ (since there is more than one ...
1
vote
0answers
45 views

How to derive a recursive version of a regularised cost function

I am to derive a recursive version of the following cost function and examine for which choice of D can we have a estimator windup $V(\theta) = \frac{1}{2}\sum_{t=1}^n(y(t)-\phi(t)^T\theta)^2 + \...
0
votes
2answers
73 views

Cost function with unique solution plus convex function has a unique solution?

I have an optimization problem with a cost function $J(X)$, $X$ is a matrix, the function is not convex but I can find the analitical solution and it solution is unique. I want to add a ...
2
votes
1answer
72 views

What is the idea behind zeta function regularization?

Trying to learn the path integral in Quantum Field Theory I came across some infinite products in Weinberg's book "The Quantum Theory of Fields". Heuristically, the author pretends that what can be ...
1
vote
0answers
39 views

SVD with Laplacian regularization and $L_{1,2}$ group-norm

I have a data matrix of the form $X \in \mathbb{R}^{n\times m}$ where the $n$ rows have spatial relationships and $m$ columns have temporal relationships. I am trying to model an objective function of ...
0
votes
0answers
61 views

Derivation of Hard Thresholding Operator for penalized least squares

I am reading Fan and Li's 2001 paper. The hard thresholding solution for the penalized least squares problem $\frac{1}{2} (z - \theta)^2 + \lambda^2 - (|\theta| - \lambda)^2 I(|\theta|<\lambda)$ ...
0
votes
0answers
33 views

Methods for filtering position signal

I'm working on a location detection algorithm. This is my outcome - postion: Here's first derivative after time - velocity: [ My goal is to apply some filtration to the signal for smoothing the ...
2
votes
0answers
77 views

Nuclear norm and Schatten norm in practice

I have a problem where the regularizer is the nuclear norm and the matrix being regularized is $n \times d$ with $d < n$. I was initially not getting low rank for the desired performance, the ...
0
votes
0answers
56 views

Regularization of $\int_{0}^{\infty} dR \int_{0}^{\pi} d\lambda \int_{0}^{2 \pi} \frac{sin(\lambda) d\phi}{(1 - sin(\lambda) sin(R + \phi))^{3}}$

I would like to know if the following integral can be given a meaninful value: $$\int_{0}^{\infty} dR \int_{0}^{\pi} d\lambda \int_{0}^{2 \pi} \frac{sin(\lambda) d\phi}{(1 - sin(\lambda) sin(R + \phi)...
0
votes
0answers
19 views

Fourier Filtering

I have read TSVD used as regularization for ill-posed problems. But I also read something like fourier filtering. I want to know something about fourier filtering. Here is the explanation of TSVD ...
1
vote
1answer
295 views

Derivation of Hard Thresholding Operator (Least Squares with Pseudo $ {L}_{0} $ Norm)

The problem is given by: \begin{equation} \widetilde{f} = \arg \min_{f} \frac{1}{2} {\left\| f - x \right\|}_{2}^{2} + \lambda {\left\| f \right\|}_{0}. \end{equation} How do I find the closed form ...
1
vote
0answers
32 views

Proof in inverse scattering theory (regularization schemes)

I'm currently reading a book about inverse scattering theory and in this book there is a section about ill-posed problems and there's a proof I'm not completely sure I understand. There might be need ...
1
vote
2answers
415 views

How does L1 regularization present itself in gradient descent?

If we incorporated $L_1$ Loss in gradient descent, how would the update rule change? It's easy to write down the optimization objective. But I'm not sure what to put for the update rule.
2
votes
0answers
21 views

How to utilize the right-hand side in inverse problems

Consider the inverse problem $A \, x = b$ with right-hand side $b$, using SVD: $\qquad A = \sum s_i \, U_i \otimes V_i \ $ — singular values $s_i, \ U_i$ and $V_i$ orthonormal bases $\qquad ...
0
votes
1answer
40 views

Solving minimization problem $L_2$ IRLS (Iteration derivation)

In the article ''' Chartrand, Rick, and Wotao Yin. "Iteratively reweighted algorithms for compressive sensing." Acoustics, speech and signal processing, 2008. ICASSP 2008. IEEE international ...
1
vote
0answers
32 views

SeDuMi form of $\min_x\left\{\|Ax-b\|_2^2 + \lambda\|x\|_2\right\}$

I have an application in which I need to minimize the following cost function. I made myself familiar with optimization up until very recently. Could someone kindly let me know what kind of ...
1
vote
1answer
111 views

Least-squares with $2$-norm penalty term

I want to minimize the following cost function $$J= \displaystyle\min_{\boldsymbol{\rm x}} \left\{\|A\boldsymbol{\rm x}_\lambda-b\|_2^2 + \lambda\|\boldsymbol{\rm x}_\lambda\|_2\right\}$$ It is a ...
0
votes
1answer
27 views

Composition of regularized inverse of linear operator on dense subspace converges on whole space?

Any help super appreciated on this. Consider a set of functions $\mathcal{F}$ that is some subset of $L_2$ and two bounded linear operators $A$ and $B$. The inverse $B^{-1}$ is well defined on $\...
0
votes
1answer
43 views

Uniform convergence of regularized inverse

I have a simple question but so far google has failed me. Suppose $A$ is a bounded compact and injective linear operator. Suppose $m$ is some function in a space $M$ and in the range of $A$ and so ...
5
votes
1answer
84 views

Zeta regularization vs Dirichlet series

Suppose you have a sequence of real numbers, denoted $a_n$. Then the sum of the sequence is $\sum_n a_n$ If this is divergent, we can use zeta regularization to get a sum. We can do this by defining ...
3
votes
2answers
383 views

Adding a diagonal matrix to a product of transpose of a matrix and itself is always invertible

I am asking this question in context to Regularization/Ridge Regression Let's say that there is a Matrix A of dimension n x d, where n is the number of rows and d is the number of columns ( n may or ...
1
vote
0answers
58 views

How to solve L1 optimization of Low-Rank matrix?

I am trying to solve the following problem but have some difficulties. Any ideas? Any leads? $$ \min_{A,C} \|C\|_* +\alpha\|X-A\|_1 \text{ s.t. } A=AC $$ $\|*\|_1$ is the sum of absolute values of ...
0
votes
1answer
301 views

What does mean ‘regular function at the origin’? [closed]

Really I'm new here , and I seek for simple definition about regular function at the origin when the function is presented as a power series ?
0
votes
1answer
51 views

analysis of different L2 norms for regularization

In machine-learning, we often use the L2 norm to prevent the weight vector from being too "big" according to this norm and thus to try to generalize more from the trainig dataset. However it is also ...
2
votes
0answers
240 views

LASSO relationship between Lagrange multiplier and constraint and why it doesn't matter

My understanding of LASSO regression is that the regression coefficients are selected to solve the minimisation problem: $$\min_\beta \|y - X \beta\|_2^2 \ \\s.t. \|\beta\|_1 \leq t$$ In practice ...