Skip to main content
3 votes

How does this L1 regularization derivation follow? (Proof it makes sparse models)

Below is a derivation using the method described in this video. As mentioned by @angryavian we want to minimize $𝑔(𝑤_𝑖):=\frac{1}{2}𝐻_{𝑖𝑖}(𝑤_𝑖−𝑤^∗_𝑖)^2+𝛼|𝑤_𝑖|$ We achieve this by taking ...
fsafe's user avatar
  • 41
2 votes

Why is the numerator-layout Jacobian transposed in backpropagation calculation?

Like the notation, $\def\qty#1{\left( #1 \right)}$ terminology is also a bit chaotic in machine learning. I assume that "sensitivities" refers to the partial derivatives of the loss function ...
Ted Black's user avatar
  • 802
1 vote

How to derive likelihood function

Here is a somewhat intuitive answer to hopefully help you understand the concept: the likelihood of an event is simply the probability of that event being observed: the likelihood of "Heads"...
plywood98's user avatar
1 vote
Accepted

What is so interesting about the Armijo-Rule or the Wolfe-Conditions for choosing the right step size?

Some communities focus on optimization methods where it is possible to prove some kind of theorem about them. Those communities generally don't work with the best optimization methods, because the ...
D.W.'s user avatar
  • 5,234
1 vote
Accepted

Min-max optimization and prediction of a parameter in a mathematical model

Let $x$ be a vector in $\mathbb{R}^{11}$ denoting the 11 parameters, i.e., the 10 parameters plus $R$. Let $f$ be some function (which you are trying to find) so that $f(x)$ is a reasonable estimate ...
D.W.'s user avatar
  • 5,234
1 vote
Accepted

Neural network that learns ODE's 'refuses to learn' initial conditions

You are using ReLU activation functions, which are nonsmooth, to represent the solution of an initial value problem, which is smooth. This can have the effect of the learned solution being highly ...
whpowell96's user avatar
  • 6,209
1 vote

Fitting a parabola through two points in which the derivatives are known with the aim of finding a minimum.

Consider the points $(w_0, L(w_0))$ and $(w_1, L(w_1))$, where $w_0 \neq w_1$. Let $P$ be the parabola $P(w) = a + bw + cw^2$ such that $P$ passes through both points. That is, $$P(w_0) = a + b w_0 + ...
Carlo Wood's user avatar
1 vote
Accepted

Is there any algorithm to calculate this ranking method quickly?

This problem goes by several names: rank aggregation, minimum cost feedback arc set, linear ordering problem, Kemeny-Young method, minimum violations ranking. You can solve the problem via integer ...
RobPratt's user avatar
  • 47k
1 vote
Accepted

Estimating the conditional entropy of a discrete variable conditioning on continuous variable

This is not trivial. Calling $X$ the outcome (discrete) variable, and $Y$ the continuous one, if $X$ is Bernoulli (two values), I would write $$I(X;Y) = h(Y) - h(Y|X) = h(Y) - (1-p) \cdot h( Y | X=0) -...
leonbloy's user avatar
  • 64k

Only top scored, non community-wiki answers of a minimum length are eligible