Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, refers to a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting. (Def: http://en.m.wikipedia.org/wiki/Regularization_(...

learn more… | top users | synonyms

1
vote
2answers
29 views

LP: add extra costs in the objective function for every variable which is not equal to $0$

I am trying to optimise an LP problem but extra costs should be added if a variable is larger than $0$. For example, if we have the following objective function: $$\text{minimize} \qquad 2X_1 + 3X_2 ...
3
votes
0answers
119 views

Can we show that $1+2+3+\dotsb=-\frac{1}{12}$ using only stability or linearity, not both, and without regularizing or specifying a summation method?

Regarding the proof by Tony Padilla and Ed Copeland that $1+2+3+\dotsb=-\frac{1}{12}$ popularized by a recent Numberphile video, most seem to agree that the proof is incorrect, or at least, is not ...
0
votes
0answers
26 views

Image processing, optimization via regularization - efficient strategy

i would like to solve the following system: $J(x) = |Ax-b|_2^2+\gamma|\nabla x|_2^2$ subject to: $x \geq 0, \sum_i x_i = 1$ The underlying problem is to derive the PSF from a sharp and blurry ...
1
vote
1answer
84 views

How does zeta of zero equal to negative one half rather than to infinity?

$$\zeta(0)=(1/1^0)+(1/2^0)+(1/3^0)+(1/4^0)+(1/5^0)...$$ Am I right? Anything raised to the power of zero is one. One to the power of zero is one. One divided by one is one. $$1/1^0=1$$ Am I right? $$...
3
votes
0answers
63 views

Regularizing the sum of all factorials

Consider the series $$\sum_{n=0}^\infty n! = 0! + 1! + 2! + 3! + 4! + \ldots = 1 + 1 + 2 + 6 + 24 + \ldots$$ This series clearly diverges. Now, given that the Gamma function is defined by $$n! = \...
1
vote
1answer
29 views

If $f \in L^p(\Omega)$, then $(\rho_n *f) \to f $ in $L^p(\Omega)$, for a sequence $(\rho_n)$ of mollifiers.

I am reading Functional Analysis, Sobolev Spaces and Partial Differential Equations, by Haim Brezis. There it is shown that: If we have $f \in L^p(\mathbb{R}^n)$, then $(\rho_n *f) \to f $ in $L^p(...
0
votes
0answers
4 views

Regularization as an alternating objective or combined objective

I have a "primary" task loss function $L=L1$ which I want to minimize. Adding a regularization term via $L=L1+\lambda L2$ can be thought of as "forcing" the optimal solution to be meaningful for a "...
1
vote
0answers
12 views

regularized least squares (L1 norm)

My objective function that is to be minimized is as follows: $$\|y-Ax\|_2^2 + \alpha\|Lx\|_1$$ where $L$ is the gradient operator. Now this problem seems convex because the first term is quadratic ...
1
vote
1answer
15 views

How to derive the hard thresholding estimator?

The minimization problem is $$ \min_{\mu\in R^p} \sum_{i=1}^p (y_i-\mu_i)^2 + \lambda^2\sum_{i=1}^p \mathbb{1}(\mu_i\neq0) $$ for $y\in R^p$. This is also known as $l_0$ regularization. The solution ...
1
vote
0answers
8 views

LASSO variable selection for proportinal hazards

Can anybody recommend software that I could port into SAS to do LASSO (or related) variable selection on a proportional hazards model?
0
votes
1answer
22 views

Reverse economization of Chebyshev series

Suppose I have some function which is represented as converging series of Chebyshev polynomials of first kind in $[-1;1]$: $$ f(x)=\sum\limits_{n=1}^\infty a_n T_{2n}(x) $$ I need to transform this ...
0
votes
0answers
18 views

Upper estimate between an original function and its sup-convolution under a limitation

My setting maybe look rather special but I'm glad if you give some answers. Let $f:[0,1]\to\mathbb{R}$ be a bounded, upper semicontinuous function and $f^{\varepsilon}:[0,1]\to\mathbb{R}$ be $f$'s ...
0
votes
0answers
18 views

Want to factorize one matrix into three, with L1 regularization, which optimization algorithm to choose?

I need to factorize one matrix $R$ into three component: $ R = P^TAQ $, in which I want to apply L1 regularization on $A$ to encourage sparsity, and apply L2 regularization on $P$ and $Q$ to prevent ...
0
votes
1answer
31 views

Regularize the sum: $\sum_{n=0}^\infty (-1)^{n+1}({2+4n})$

$$\sum_{n=0}^\infty (-1)^{n+1}({2+4n})=2-6+10-14\cdots$$ is a clearly divergent sum, which I am trying to regularize. I need to figure out this sum because I have been trying to solve the sum integral ...
1
vote
1answer
29 views

l1 regularized minimization with equality constraint in ADMM

In section 6.3 of this note there is a method for minimizing a loss function with l1 regularization. i.e. minimize $l(\bf{x})+\lambda||x||_1$ How can I add the equality constraint $\sum\limits_{i} ...
0
votes
2answers
22 views

How to prove non-regularity of a language from the non-regularity of another language?

How can I prove that $L_1=\{a^nb^m\mid n\ne m\}$ is not regular based on the fact that the language $L_2=\{a^nb^n\mid n\in\Bbb N\}$ is not regular? Thank you
1
vote
0answers
13 views

Why divide regularization factor by size of dataset?

Suppose I'm trying to minimize a cost function: $$ J(\theta) = \frac {1} {2m} \sum _{i = 1}^ m (h_\theta (x^{(i)}) - y^{(i)})^2 $$ Adding regularization, as seen here, we get: $$ J(\theta) = \frac {...
2
votes
1answer
33 views

Convex Optimization - nuclear norm regularisation of symmetric matrix

I have a problem of the form $\min_{X\in \mathbb{R}^{n \times n}} g(X) - \lambda ||X||_{*}$ where $g$ is convex and differentiable. I would like to use proximal gradient descent to solve this. How ...
0
votes
0answers
76 views

Solve the problem $Ax = 0$ when $A$ has full rank.

Generally, the answers $x$ of this least square problem $$Ax = 0$$ where $A = []_{m\times n}$ and $x = []_{n\times 1}$ are in the null space of $A$. I know that people usually use the right-most ...
10
votes
1answer
284 views

Is $\prod_{n=1}^\infty P_{2n-1}$ regularizable?

Assume that $P_n$ denotes the $n$'th prime for this entire question. Inspriation: I was dumbfounded by the fact that: $$\hat\prod_\limits{n=1}^\infty P_{n}=4\pi^2$$ After further investigation, I ...
1
vote
0answers
17 views

Extension of $C^1$ function (regularity of mollification on boundary)

In my study I have faced the problem showing the property below: regularity of mollification on boundary. (However, I don't know whether this is true or not although I hope.) Let $f\in C^{1}((0,2]^...
5
votes
0answers
172 views

Prove the estimator $\hat{B}$ of ridge regression = mean of the posterior distribution under a Gaussian prior

I want to prove that the estimator of ridge regression is the mean of the posterior distribution under Gaussian prior. $$y \sim N(X\beta,\sigma^2I),\quad \text{prior }\beta \sim N(0,\gamma^2 I).$$ $...
0
votes
0answers
23 views

Penalty Term on Negative Matrix Elements for Gradient Descent Objective Function?

In fact, I'm doing a gradient ascent to estimate the elements of a matrix $X=f(A)$. Nevertheless, I've written "descent" into the title, because I think that this is the common term. Motivational ...
3
votes
4answers
254 views

regularization of sum $n \ln(n)$

I was testing out a few summation using my previous descriped methodes when i found an error in my reasoning. I'm really hoping someone could help me out. The function which i was evaluating was $\...
1
vote
0answers
164 views

Zeta regulated product, solving without the zeta function.

Earlier i've asked about how to calculate divergent products, i got some directions which made me curious. Now i'm wondering is this correctly done. Divergent products. The most commen divergent ...
7
votes
0answers
59 views

Does such divergent integral assume the same values for any regularization?

Consider the integral: $$\int_0^\infty\sin(x)dx.\tag1$$ It's clearly divergent, but if we regularize it as $$\int_0^\infty\sin(x)e^{-x/a}dx=\frac{a^2}{a^2+1},\tag2$$ we can take the limit of $a\to\...
1
vote
0answers
30 views

Estimating Markov transition matrix for regularization

Suppose that I have a sequence of discrete distributions: $$ p_j = (p_{1j},...,p_{Cj}), \: j=1...D,\\ p_{ij}>0 \:\: \forall i,j,\: \sum_{k=1}^Cp_{kj}=1\:\:\forall j. $$ I suppose that these ...
1
vote
0answers
29 views

About consistency in an inverse problem formulation

I'm a beginner with inverse problems and I was reading about regularization techniques. Consider the problem: $$d=Kf_{\text{true}}$$ $d$ is a data vector, $K$ is an linear operador, $d=\hat{d}+\eta$ ...
0
votes
1answer
44 views

Relation between Regularization and correlation

I was going through Chapter 3 (page 63 bottom) of Elements of Statistical Learning. While explaining regularization in ridge regression authors make the following statements. "When there are many ...
1
vote
1answer
60 views

Solve the following matrix equation $X'X=A$

I have square matrices $X$,$A$ and $X'X-A=0$. $A$ is given and is positive definite and I need to get matrix $X$. I know $X$ is not unique since $TX$ such that $T'T=I$ will satisfy. My problem is ...
1
vote
1answer
63 views

How do I set a lower bound to the solution's norm in a QP problem

I know that LASSO-regularization can be used to scale into an $L_1$ upper bound for a solution. But what if I want the norm to be within a specific range $[a,b]$? ie. I also want to set a lower bound? ...
2
votes
1answer
407 views

Solving L1 regularized Joint Least Squares and Logistic Regression

My objective function that is to be minimized is as follows: $f = -\sum_{n=1}^{N}log~p(y_{n}^{a}|x_{n},w) + \sum_{n=1}^{N}(y_{n}^{b}-w^{T}x_{n})^{2} +\lambda\|w\|_1$ The first term models the ...
2
votes
1answer
119 views

When do regularization methods for divergent series disagree?

Sometimes, it is possible to take a divergent series (in the sense of its sequence of partial sums failing to converge) and "regularize" it using one of a variety of methods to assign it a meaningful ...
4
votes
1answer
473 views

Tikhonov regularization vs truncated SVD

To find $\mathbf{x}$ such that $$A\mathbf{x}=\mathbf{b}$$ we can use least squares when the problem is not well posed. Further, we can use Tikhonov regularization when $A$ is ill-conditioned. In ...
1
vote
0answers
105 views

sum of divisors function $\sum \tau(n) = \frac{1}{4}$

These notes on multiplicative number theory mention the convolution $ 1 \ast 1 = \tau$ (where $\tau$ is the divisor function not Ramanujan tau function. Therefore $$ \bigg(\sum \frac{1}{n^s} \bigg)^...
1
vote
1answer
98 views

Zeta and Gamma function regularization with $\omega=1/0$

I have recently read about zeta function regularization, a way of ascribing values to functions having simple poles in a point and to divergent series. The values obtained are the same as those ...
5
votes
2answers
296 views

Is there a metric in which 1+2+3+4+… converges to -1/12?

It is well known that the sum $1+2+3+4+\ldots$, which tends to infinity in the regular sense, can be assigned the value $-\frac{1}{12}$ by different means, e.g., zeta regularization or different ...
0
votes
1answer
51 views

Optimization Problem (Linear Algebra)

I am not trying to cheat or anything, so any reference to online literature or MOOCs, that teach this stuff, will be highly appreciated. The problem is to prove that the following optimization ...
2
votes
0answers
43 views

Can this divergent integral transform be regularized?

The integral $$\int_0^{\infty} e^u \ K_{i t}(u) du$$ is the adjoint Kontorovich-Lebedev transform of the increasing exponential function, but unfortunately this integral is divergent because $$e^u \ ...
0
votes
1answer
25 views

zeta regularization separation of series

in the sense of infinite series and for an integer 'a' is then correct that $$ \sum_{n=1}^{\infty}n^{k} = \sum_{n=1}^{a}n^{k}+ \sum_{n=a+1}^{\infty}n^{k} $$ opther that works only when ยท$ re(k) > ...
2
votes
0answers
145 views

Fourier transform of an exponentially singular radial function

I am trying to compute the 3D Fourier transform of a spherically symmetric function of the form $$f(r) = e^{\frac{1}{r} e^{-r}} - 1\, ,$$ which entails the integral $$\begin{aligned}F(k) =& \int ...
0
votes
1answer
17 views

Support of vector $w$ in graph sparsity

I'm reading about graph sparsity and I have one problem in a paper I'm reading I don't understand, maybe someone can clarify: Graph Sparsity: In graph sparsity, we have a directed acyclic graph $...
1
vote
1answer
43 views

Differentiable L-1 Regularization

In machine learning we are often faced with optimization problems where we want to minimize some energy function using L1 regularization over some of the parameters, e.g.: $$ E(a,w) = [\text{sum of ...
0
votes
1answer
65 views

Regularization of underdetermined system to favour low frequency solutions?

Consider the ill-posed system $$ \mathbf A \mathbf x= \mathbf b.$$ One method to regularize the solution is the Tikhonov method which effectively minimizes $ ||\mathbf A \mathbf x - \mathbf b ||^2 + |...
1
vote
1answer
37 views

Integral in regularization involving exp of a parameter in the denominator

Can someone please point me a way to compute $$\int_0^\infty\frac1{s+t}\exp\left(-\alpha t+\frac{t^2\beta}{s+t}\right)dt$$ ? How about the following one? $$\int_0^\infty ds\int_0^\infty dt\,\frac1{s+...
4
votes
1answer
113 views

Is it possible to sum the divergent series with prime coefficients?

This is a follow-up of this question. It is known that the divergent series $$ P := \sum_{n=1}^\infty p_n \qquad \text{where } p_n \text{ is the $n$th prime} $$ cannot be summed by means of (prime) ...
0
votes
1answer
185 views

Why use regularization to reduce over-fitting

I'm having trouble understanding why should we use regularization for over-fitting when we can simply reduce the number of order to our polynomial function? Is it because it saves us time from having ...
7
votes
1answer
246 views

Connection between integral expression and the factorial of infinity

Does the fact that $$\int_{-\infty}^{\infty}\exp\left(-\frac{1}{2}x^2\right)\mathrm{d}x=\sqrt{2\pi}$$ Have something to do with the fact that the regularized factorial of infinity is also $\sqrt{2\...
1
vote
0answers
55 views

When does l1 regularisation give a sparse solution?

I was maximising a likelihood function, which is convex. I know that the system has a K-sparse solution. I wanted to know the conditions (or some sufficient conditions) on the likelihood function ...
1
vote
0answers
111 views

Regularity of semilinear heat equation

I'm facing following regularity issue and i wonder if anyone of you guys is able to help me. I'd like to show that the solution of a semilinear heat equation is classical, i.e. $C^2$ in space and $C^1$...