# Tag Info

### How does this L1 regularization derivation follow? (Proof it makes sparse models)

Below is a derivation using the method described in this video. As mentioned by @angryavian we want to minimize $𝑔(𝑤_𝑖):=\frac{1}{2}𝐻_{𝑖𝑖}(𝑤_𝑖−𝑤^∗_𝑖)^2+𝛼|𝑤_𝑖|$ We achieve this by taking ...
• 41

### Why is the numerator-layout Jacobian transposed in backpropagation calculation?

Like the notation, $\def\qty#1{\left( #1 \right)}$ terminology is also a bit chaotic in machine learning. I assume that "sensitivities" refers to the partial derivatives of the loss function ...
• 802
1 vote

### How to derive likelihood function

Here is a somewhat intuitive answer to hopefully help you understand the concept: the likelihood of an event is simply the probability of that event being observed: the likelihood of "Heads"...
1 vote
Accepted

### What is so interesting about the Armijo-Rule or the Wolfe-Conditions for choosing the right step size?

Some communities focus on optimization methods where it is possible to prove some kind of theorem about them. Those communities generally don't work with the best optimization methods, because the ...
• 5,234
1 vote
Accepted

### Min-max optimization and prediction of a parameter in a mathematical model

Let $x$ be a vector in $\mathbb{R}^{11}$ denoting the 11 parameters, i.e., the 10 parameters plus $R$. Let $f$ be some function (which you are trying to find) so that $f(x)$ is a reasonable estimate ...
• 5,234
1 vote
Accepted

### Neural network that learns ODE's 'refuses to learn' initial conditions

You are using ReLU activation functions, which are nonsmooth, to represent the solution of an initial value problem, which is smooth. This can have the effect of the learned solution being highly ...
• 6,209
1 vote

Consider the points $(w_0, L(w_0))$ and $(w_1, L(w_1))$, where $w_0 \neq w_1$. Let $P$ be the parabola $P(w) = a + bw + cw^2$ such that $P$ passes through both points. That is, $$P(w_0) = a + b w_0 + ... • 213 1 vote Accepted ### Is there any algorithm to calculate this ranking method quickly? This problem goes by several names: rank aggregation, minimum cost feedback arc set, linear ordering problem, Kemeny-Young method, minimum violations ranking. You can solve the problem via integer ... • 47k 1 vote Accepted ### Estimating the conditional entropy of a discrete variable conditioning on continuous variable This is not trivial. Calling X the outcome (discrete) variable, and Y the continuous one, if X is Bernoulli (two values), I would write$$I(X;Y) = h(Y) - h(Y|X) = h(Y) - (1-p) \cdot h( Y | X=0) -...
• 64k

Only top scored, non community-wiki answers of a minimum length are eligible