Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [regularization]

Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, refers to a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting. (Def: http://en.wikipedia.org/wiki/Regularization_(...

1
vote
0answers
15 views

Least-squares regularization matrix is not positive definite?

I am fitting data $(x_i,y_i)$ to the following model: $$ f(x) = \sum_j a_j g_j(x) = a^T g(x) $$ where $g_j(x)$ are well-conditioned basis functions (in my application they are B-splines, but I don't ...
0
votes
1answer
49 views

Sparse recovery with L1 shrinkage iteration for higher denominational image classification

For 2 months I have been studying sparse recovery and its applications for image classification and I have found that it's a broad area in mathematics which gives rise to a wide variety of ...
4
votes
1answer
37 views

Continuous Tikhonov Regularization for Deconvolution

I am trying to solve the following deconvolution problem where $g(s)$ is a known real valued function and has finite energy: $$g(s) = \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}f(t)e^{-(t-s)^2/2}...
0
votes
0answers
26 views

Preconditioning the Damped Normal Equations - Reference

For some people working with numerical approximations, it is very common to be dealing with a problem in the form $$\min_x \|Ax - b\|,$$ where $A$ is a matrix and $x,b$ are vectors. Consider that $A$ ...
1
vote
0answers
28 views

How to prove this (alternative) solution of ridge regression?

The ridge regression optimization formula for $X\in\mathbb{R}^{n\times m}$, $y\in\mathbb{R}^n$ with $n>m$, $$ \arg \min_{w\in \mathbb{R}^n} ||y-Xw||_2^2 + \lambda ||w||_2^2 $$ I know that the ...
0
votes
1answer
26 views

determine matrix and vector to fit regularized normal equation

I hope the title is not too unclear. I am given a Matrix $$A\in~\mathbb{R}^{K\times~N},~b\in~\mathbb{R}^{K}$$ and instead of solving the normal equation $min_{x\in~\mathbb{R}^N}|Ax-b|^2_2,~$ an $\...
0
votes
0answers
50 views

Nonparametric Bayesian estimation of several black-box functions of different variables from their noisy sums

In order to introduce my problem, let’s start with the nonparametric estimation of a single unknown/black-box function $f:{\Omega _f} \to \mathbb{R}$ of a discrete variable $x$ in a finite domain ${\...
3
votes
0answers
12 views

Interpolation with Regularized Linear Squares

I am fundamentally missing something and would appreciate some clarification. I am reading this text on Regularized Least Squares. This text has a problem where I have to find z, a set of unknown ...
1
vote
1answer
26 views

Question on derivation of probability proportion in Deep Learning Book

I am working through the Deep Learning Book, I am currently on the regularization chapter (https://www.deeplearningbook.org/contents/regularization.html). My question concerns the third step in the ...
3
votes
0answers
31 views

regularizing function under sqrt [closed]

Is it possible to find finite part of $\Sigma^\infty_n\sqrt{n^2 +a^2}$ using something like regularized zeta function?
1
vote
0answers
28 views

How can I incorporate this specific a-priori sparsity information in a regularization approach to guide my inverse problem?

Note: I will be using capital letters to denote matrices, lowercase letters to denote vectors, and the Greek alphabet to denote any scalar quantities. Note 2: I have tried my best to find a topic ...
3
votes
0answers
30 views

Dirac delta from poles of a function

Suppose we are given the simple expression $$ F(k) = \frac{1}{E^2-E(k)^2} $$ which has a pole when $E^2 = E(k)^2$ and where $E, E(k)$ are real numbers. When working with this expression (e.g. inside ...
0
votes
0answers
58 views

Laurent series of an integral with parameter

To find the Laurent series of function $f(a)$ at point $a=0$ $$ f(a)=\int^1_0 \frac{d x}{x^2+a^2} $$ one can first do the integral $$ f(a)=\frac{1}{a}\arctan(1/a) $$ then expand $\arctan(1/a)$ and ...
4
votes
1answer
91 views

Cesàro limit of a stochastic matrix

Let $A$ be a stochastic matrix. Then \begin{align*} \lim_{t \rightarrow\infty} A^t \end{align*} may not exist. For example: \begin{align*} A &= \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}...
3
votes
2answers
124 views

Understanding L2 Regularization Formula

I am currently following the Machine Learning Crash Course on Tensorflow and came across this formula: $$L_2\text{ regularization term} = \|\boldsymbol w\|_2^2 = {w_1^2 + w_2^2 + \cdots + w_n^2}$$ I ...
6
votes
3answers
434 views

Understanding the difference between Ridge and LASSO

I'll start by presenting my current understanding of regularized loss minimization. In the context of ML, generally we are trying to minimize directly an empirical loss, something of the form $$\...
0
votes
0answers
25 views

A ring to place numbers in, how to select numbers uniquely to minimize running variation?

Say I have the numbers in some list $\{1,2,2,3,4,3\}$ and I want to select them uniquely and place so that the average variation (for example running absolute difference, or running square difference) ...
4
votes
0answers
130 views

New/useful method for summation of divergent series?

Questions $$ S(n,x) = x+e^x + e^{e^x} + e^{e^{e^x}} + \dots \text{$n$ times}$$ Also obeys (see background for argument): $$ \frac{1}{2 \pi i} \oint e^{S(k,x)} \frac{\partial \ln(\frac{\int_0^\...
0
votes
0answers
77 views

Solution of L2 Norm and Total Variation (L1) regularizer combined

Is there any method for the direct solution of the following equation? $$ \min_{x} (||Hx - y||_2^2 + ||z - x||_2^2 + \lambda||\nabla x||_1) $$ I know the solution in case only the first two L2 norm ...
2
votes
0answers
93 views

Regularization of Exponentially exponential series?

Question What are the convergence properties of the last equation: $$K = e^{x} + x + \ln{x} + \ln\ln(x) + \dots $$ Can one artificially choose a value of $\ln (x)$ (since there is more than one ...
1
vote
0answers
39 views

How to derive a recursive version of a regularised cost function

I am to derive a recursive version of the following cost function and examine for which choice of D can we have a estimator windup $V(\theta) = \frac{1}{2}\sum_{t=1}^n(y(t)-\phi(t)^T\theta)^2 + \...
0
votes
2answers
62 views

Cost function with unique solution plus convex function has a unique solution?

I have an optimization problem with a cost function $J(X)$, $X$ is a matrix, the function is not convex but I can find the analitical solution and it solution is unique. I want to add a ...
2
votes
1answer
64 views

What is the idea behind zeta function regularization?

Trying to learn the path integral in Quantum Field Theory I came across some infinite products in Weinberg's book "The Quantum Theory of Fields". Heuristically, the author pretends that what can be ...
1
vote
0answers
38 views

SVD with Laplacian regularization and $L_{1,2}$ group-norm

I have a data matrix of the form $X \in \mathbb{R}^{n\times m}$ where the $n$ rows have spatial relationships and $m$ columns have temporal relationships. I am trying to model an objective function of ...
0
votes
0answers
55 views

Derivation of Hard Thresholding Operator for penalized least squares

I am reading Fan and Li's 2001 paper. The hard thresholding solution for the penalized least squares problem $\frac{1}{2} (z - \theta)^2 + \lambda^2 - (|\theta| - \lambda)^2 I(|\theta|<\lambda)$ ...
0
votes
0answers
33 views

Methods for filtering position signal

I'm working on a location detection algorithm. This is my outcome - postion: Here's first derivative after time - velocity: [ My goal is to apply some filtration to the signal for smoothing the ...
2
votes
0answers
68 views

Nuclear norm and Schatten norm in practice

I have a problem where the regularizer is the nuclear norm and the matrix being regularized is $n \times d$ with $d < n$. I was initially not getting low rank for the desired performance, the ...
0
votes
0answers
49 views

Regularization of $\int_{0}^{\infty} dR \int_{0}^{\pi} d\lambda \int_{0}^{2 \pi} \frac{sin(\lambda) d\phi}{(1 - sin(\lambda) sin(R + \phi))^{3}}$

I would like to know if the following integral can be given a meaninful value: $$\int_{0}^{\infty} dR \int_{0}^{\pi} d\lambda \int_{0}^{2 \pi} \frac{sin(\lambda) d\phi}{(1 - sin(\lambda) sin(R + \phi)...
0
votes
0answers
18 views

Fourier Filtering

I have read TSVD used as regularization for ill-posed problems. But I also read something like fourier filtering. I want to know something about fourier filtering. Here is the explanation of TSVD ...
0
votes
1answer
258 views

Derivation of Hard Thresholding Operator (Least Squares with Pseudo $ {L}_{0} $ Norm)

The problem is given by: \begin{equation} \widetilde{f} = \arg \min_{f} \frac{1}{2} {\left\| f - x \right\|}_{2}^{2} + \lambda {\left\| f \right\|}_{0}. \end{equation} How do I find the closed form ...
1
vote
0answers
31 views

Proof in inverse scattering theory (regularization schemes)

I'm currently reading a book about inverse scattering theory and in this book there is a section about ill-posed problems and there's a proof I'm not completely sure I understand. There might be need ...
1
vote
2answers
201 views

How does L1 regularization present itself in gradient descent?

If we incorporated $L_1$ Loss in gradient descent, how would the update rule change? It's easy to write down the optimization objective. But I'm not sure what to put for the update rule.
2
votes
0answers
19 views

How to utilize the right-hand side in inverse problems

Consider the inverse problem $A \, x = b$ with right-hand side $b$, using SVD: $\qquad A = \sum s_i \, U_i \otimes V_i \ $ — singular values $s_i, \ U_i$ and $V_i$ orthonormal bases $\qquad ...
0
votes
1answer
39 views

Solving minimization problem $L_2$ IRLS (Iteration derivation)

In the article ''' Chartrand, Rick, and Wotao Yin. "Iteratively reweighted algorithms for compressive sensing." Acoustics, speech and signal processing, 2008. ICASSP 2008. IEEE international ...
1
vote
0answers
31 views

SeDuMi form of $\min_x\left\{\|Ax-b\|_2^2 + \lambda\|x\|_2\right\}$

I have an application in which I need to minimize the following cost function. I made myself familiar with optimization up until very recently. Could someone kindly let me know what kind of ...
1
vote
1answer
80 views

Least-squares with $2$-norm penalty term

I want to minimize the following cost function $$J= \displaystyle\min_{\boldsymbol{\rm x}} \left\{\|A\boldsymbol{\rm x}_\lambda-b\|_2^2 + \lambda\|\boldsymbol{\rm x}_\lambda\|_2\right\}$$ It is a ...
0
votes
1answer
25 views

Composition of regularized inverse of linear operator on dense subspace converges on whole space?

Any help super appreciated on this. Consider a set of functions $\mathcal{F}$ that is some subset of $L_2$ and two bounded linear operators $A$ and $B$. The inverse $B^{-1}$ is well defined on $\...
0
votes
1answer
41 views

Uniform convergence of regularized inverse

I have a simple question but so far google has failed me. Suppose $A$ is a bounded compact and injective linear operator. Suppose $m$ is some function in a space $M$ and in the range of $A$ and so ...
5
votes
1answer
80 views

Zeta regularization vs Dirichlet series

Suppose you have a sequence of real numbers, denoted $a_n$. Then the sum of the sequence is $\sum_n a_n$ If this is divergent, we can use zeta regularization to get a sum. We can do this by defining ...
0
votes
0answers
16 views

least square regression in matlab

I have an equation Ax=b (A is 500x5), where I know A,x,b. When I try to find x by x'=A\b. I found x' is not equal to x, actually, it differs a lot. If there a way to find x' that equal to x.
3
votes
2answers
314 views

Adding a diagonal matrix to a product of transpose of a matrix and itself is always invertible

I am asking this question in context to Regularization/Ridge Regression Let's say that there is a Matrix A of dimension n x d, where n is the number of rows and d is the number of columns ( n may or ...
1
vote
0answers
53 views

How to solve L1 optimization of Low-Rank matrix?

I am trying to solve the following problem but have some difficulties. Any ideas? Any leads? $$ \min_{A,C} \|C\|_* +\alpha\|X-A\|_1 \text{ s.t. } A=AC $$ $\|*\|_1$ is the sum of absolute values of ...
0
votes
1answer
214 views

What does mean ‘regular function at the origin’? [closed]

Really I'm new here , and I seek for simple definition about regular function at the origin when the function is presented as a power series ?
0
votes
1answer
38 views

analysis of different L2 norms for regularization

In machine-learning, we often use the L2 norm to prevent the weight vector from being too "big" according to this norm and thus to try to generalize more from the trainig dataset. However it is also ...
2
votes
0answers
214 views

LASSO relationship between Lagrange multiplier and constraint and why it doesn't matter

My understanding of LASSO regression is that the regression coefficients are selected to solve the minimisation problem: $$\min_\beta \|y - X \beta\|_2^2 \ \\s.t. \|\beta\|_1 \leq t$$ In practice ...
0
votes
0answers
23 views

Is every function that can be analytically continued asymptotically regular?

Consider a divergent series $S$ which can be regularized to a number of convergent ones $S^{n}_\epsilon$ which asymptote to the series $S$ when the regularizing parameter $\epsilon$ is removed. $$ \...
1
vote
1answer
50 views

How to regularize $\sum_{(a,b) \in \mathbb{Z}^2}' \frac{1}{\sqrt{a^2 + b^2}} $?

I would like to evaluate the Dedekind zeta function of $\mathbb{Z}[i]$ at $\frac{1}{2}$. Naively the Dekind zeta function should be $$ \zeta_{\mathbb{Z}[i]} (s) = \sum_{(a,b) \in \mathbb{Z}^2}' \...
2
votes
1answer
50 views

Existence or regularization of the limit: $\lim_{\beta\to\infty} \frac{\beta^2}{\sin(\beta^2)}$

I came across this limit and I really do think it is indeterminate or it doesn't exist: $$\lim_{\beta\to \infty} \frac{\beta^2}{\sin(\beta^2)}$$ The Sine function is bounded, and besides that, using ...
0
votes
1answer
169 views

Regularity of Cramer-Rao inequality for unbiased estimators

As I understand it, the use of Cramer-Rao inequality requires a few regularity conditions, including the parametric family being dominated and swapping of differentiation and integration in ...
1
vote
0answers
108 views

Smoothing vs Regularization

I am working on inverse problem - calibration of local volatility (financial application). This inverse problem is ill-posed. The optimization algorithm takes a $ \sigma $ surface as input and solves ...