Questions tagged [machine-learning]
How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?
170
questions
196
votes
1
answer
192k
views
Derivative of Softmax loss function
I am trying to wrap my head around back-propagation in a neural network with a Softmax classifier, which uses the Softmax function:
\begin{equation}
p_j = \frac{e^{o_j}}{\sum_k e^{o_k}}
\end{equation}...
- 2,157
123
votes
8
answers
105k
views
derivative of cost function for Logistic Regression
I am going over the lectures on Machine Learning at Coursera.
I am struggling with the following. How can the partial derivative of
$$J(\theta)=-\frac{1}{m}\sum_{i=1}^{m}y^{i}\log(h_\theta(x^{i}))+(1-...
- 1,355
3
votes
1
answer
804
views
A matrix calculus problem in backpropagation encountered when studying Deep Learning
I am studying the Algorithm 6.4 in the textbook Deep Learning, which is about backpropagation.
I am confused by this line:
$$\nabla_{W^{(k)}}J = gh^{(k-1)T}+\lambda\nabla_{W^{(k)}}{\Omega(\theta)}$$
...
- 845
15
votes
5
answers
16k
views
Formal proof that mean minimize squared error function
On an important book of Machine Learning, I've found this proof.
We want to minimize the cost function $J_0(X_0)$ defined by the formula
$$J_0(x_0) = \sum_{k=1}^n \|x_0 - x_k \|^2.$$
The solution to ...
- 253
8
votes
3
answers
3k
views
$ {L}_{1} $ Regularized Unconstrained Optimization Problem
I am encountering an unconstrained minimization problem. The problem is of the form
$$\min_x \frac{\|x-a\|_2^2}{2}+\lambda\|x\|_1$$
where $x,a \in R^n$ and $x$ is the optimization variable. $\...
- 169
19
votes
4
answers
16k
views
Deriving cost function using MLE :Why use log function?
I am learning machine learning from Andrew Ng's open-class notes and coursera.org. I am trying to understand how the cost function for the logistic regression is derived. I will start with the cost ...
- 195
15
votes
4
answers
4k
views
What is a good book for math students to learn machine learning in depth?
I am a math master student and have done fundamental math courses like probability theory, measure theory, linear algebra and know a little bit about functional analysis. What is good way for me to ...
- 1,747
6
votes
3
answers
977
views
What all maths do I need to know to become good at machine learning.
I am a computer science engineer and I took a couple of maths classes in my first year they were on Fourier series(not transform) partial differential equations, vector calculus, infinite series ...
- 921
3
votes
3
answers
3k
views
Moore-Penrose pseudoinverse and the Euclidean norm
Section 2.9 The Moore-Penrose Pseudoinverse of the textbook Deep Learning by Goodfellow, Bengio, and Courville, says the following:
Matrix inversion is not defined for matrices that are not square. ...
- 4,470
3
votes
1
answer
227
views
Calling a space a set - abuse of terminology?
I am kind of confused on the terminology of "space". From https://en.wikipedia.org/wiki/Space_(mathematics) I am getting that
In mathematics, a space is a set (sometimes called a universe) ...
- 1,045
31
votes
2
answers
5k
views
Mathematical introduction to machine learning
At first glance, this is once again a reference request for "How to start machine learning".
However, my mathematical background is relatively strong and I am looking for an introduction to ...
- 2,089
16
votes
3
answers
7k
views
Gradients of marginal likelihood of Gaussian Process with squared exponential covariance, for learning hyper-parameters
The derivation of gradient of the marginal likelihood is given in http://www.gaussianprocess.org/gpml/chapters/RW5.pdf
But the gradient for the most commonly used covariance function, squared ...
- 517
15
votes
1
answer
3k
views
Transforming a distance function to a kernel
Fix a domain $X$: Let $d : X \times X \rightarrow \mathbb{R}$ be a distance function on $X$, with the properties
$d(x,y) = 0 \iff x = y$ for all $x,y$
$d(x,y) = d(y,x)$ for all $x,y$
Optionally, $d$ ...
- 1,302
12
votes
8
answers
3k
views
Where to start Machine Learning?
I've recently stumbled upon machine learning, generating user recommendations based on user data, generating text teaser based on an article. Although there are tons of framework that does this(Apache ...
- 229
11
votes
3
answers
15k
views
Understanding “the mean minimizes the mean squared error”
I am trying to understand the sentence
the mean minimizes the mean squared error.
from wikipedia https://en.wikipedia.org/wiki/Average_absolute_deviation.
From ...
- 213
11
votes
2
answers
2k
views
What is the motivation for using cross-entropy to compare two probability vectors?
Define a "probability vector" to be a vector $p = (p_1,\ldots, p_K) \in \mathbb R^K$ whose components are nonnegative and which satisfies $\sum_{k=1}^K p_k = 1$. We can think of a probability vector ...
- 50.5k
4
votes
5
answers
2k
views
Why use the kernel trick in an SVM as opposed to just transforming the data?
Why use the kernel trick in a support vector machine as opposed to just transforming the data and then using a linear classifier? Certainly, we'll approximately double the amount of memory required ...
- 2,472
3
votes
2
answers
3k
views
I am confused about the kernel of a matrix and the "kernel"
In linear algebra, the kernel of a matrix is its null space.
In machine learning and statistics, there are a bunch of matrices are called "kernel". For example,
I am totally confused. The second "...
- 5,466
2
votes
2
answers
1k
views
Proof: Rank of a Random (arbitrary size) Matrix is full rank with probability $1$?
I am wondering how to prove that a complex-valued random matrix say $\mathbf{A} \in \mathbb{C}^{M \times N}$ with size $M \times N$ has full-rank, i.e., $\textrm{rank} = \min\left\{M,N\right\}$ with ...
- 2,618
1
vote
1
answer
3k
views
The Sub Differential of a Matrix $ {L}_{1} $ Norm
Given a matrix $ A \in \mathbb{R}^{m \times n} $ what is the Sub Differential (Sub Gradient) of
$$ g \left( A \right) = \lambda {\left\| A \right\|}_{1} $$
- 255
0
votes
1
answer
129
views
Gradients of $ \sum_{i=1}^N \|W_3 g(W_2 f(W_1 x_i) ) - y_i \|_2^2$ w.r.t. $W_1$, $W_2$, and $W_3$?
How to obtain the gradient and optionally Hessian of
\begin{align}
L(W_1, W_2, W_3) := \sum_{i=1}^N \| W_3 \ g\left(W_2 \ f\left(W_1 x_i \right) \right) - y_i \|_2^2 \ ,
\end{align}
with respect to $...
- 589
66
votes
4
answers
53k
views
Why we consider log likelihood instead of Likelihood in Gaussian Distribution
I am reading Gaussian Distribution from a machine learning book. It states that -
We shall determine values for the unknown parameters $\mu$ and
$\sigma^2$ in the Gaussian by maximizing the ...
- 763
44
votes
2
answers
32k
views
How is logistic loss and cross-entropy related?
I found that Kullback-Leibler loss, log-loss or cross-entropy is the same loss function. Is the logistic-loss function used in logistic regression equivalent to the cross-entropy function? If yes, can ...
- 543
12
votes
1
answer
6k
views
What does it mean to "marginalise out" something?
Especially in machine learning one often reads the phrase "to marginalise out" something, and while I understand that this means to integrate over a property, I cannot quite grasp the larger ...
- 712
6
votes
3
answers
565
views
Application of the chain rule to $3$-layers neural network
Consider the differentiable functions $L^1(x,\theta^1),L^2(x^2,\theta^2),L^3(x^3,\theta^3)$, where every $x_k,\theta^k$ are real vectors, for $k=1,2,3$. Also define $\theta=(\theta^1,\theta^2,\theta^3)...
- 1,957
6
votes
2
answers
3k
views
why small L1 norm means sparsity?
I'm trying to understand regularization in machine learning. one way of regularization is adding a l1 norm to the error function. This is said to produce sparsity. But I can't understand.
sparsity is ...
- 233
4
votes
2
answers
719
views
High Dimensional Rotation Matrices As Product of In-Plane Rotations
Lately I've been thinking a lot about how to find high-dimensional rotation matrices. In particular, can any rotation in $n$-dimensional space be represented as the product of $2$D plane rotations? I'...
- 141
4
votes
1
answer
476
views
Is there an analogous Gibbs phenomena to approximating sinusoidal but with polynomial terms?
I was trying to approximate a sine curve with a finite number of polynomials terms using linear regression (or the pseudo-inverse). I construct a Vandermonde matrix (or a Kernel polynomial feature ...
- 6,476
4
votes
2
answers
2k
views
Mathematical Background Required for Advanced Machine Learning Concepts
What are the must-know concepts and best resources for preparing the mathematical background for advanced machine learning studies?
Currently, looking into the book What is Mathematics? by Richard ...
- 209
3
votes
1
answer
5k
views
Dependent Bernoulli trials
The probability of a sequence of n independent Bernoulli trials can be easily expressed as
$$p(x_1,...,x_n|p_1,...,p_n)=\prod_{i=1}^np_i^{x_i}(1-p_i)^{1-x_i}$$
but what if the trials are not ...
- 63
2
votes
1
answer
259
views
Why does k-fold cross validation generate an MSE estimator that has higher bias, but lower variance then leave-one-out cross-validation?
Looks like the rationale behind the accepted answer of this post is incorrect.
Under leave one out cross validation(LOOCV), the variance of its MSE estimator is
$$var [\frac{\Sigma_i x_i}{n}] = \...
- 1,747
2
votes
0
answers
106
views
Is this implementation use HBOS mathematic?
I'm experimenting with an unsupervised statistical-based outlier detection so-called XBOS on top of the KMeans clustering algorithm. It is claimed that XBOS generates outlier scores as HBOS does. I'm ...
- 73
2
votes
1
answer
276
views
Extension of binary classification to multi-class classification
Multi-class classification is a generalization of logistic regression wherein we are dealing with binary classification. The latter problem is a setting where a number should be mapped to either $0$ ...
user494522
2
votes
3
answers
1k
views
Derivative where the variable is a matrix
In the context of Neural Nets, I'm trying to derivate the objective function with respect to the weight-matrices. I'm stuck at the following point:
Let $N$ and $D$ two distinct integers, and $A\in\...
- 596
2
votes
1
answer
739
views
How to derive integrals with error function?
How to derive this integral
$\int_{-\infty}^{\infty}erf(\lambda x)\mathcal{N}(\mu, \sigma ^2)dx$
and this
$\int_{-\infty}^{\infty}(erf(\lambda x)-const)^2\mathcal{N}(\mu, \sigma ^2)dx$
where
$...
- 121
2
votes
1
answer
111
views
Averaged log-likelihood with a latent variable for mixture models
In class we've defined the following:
$$Q(\theta; \theta^t) = \sum_z P(Z=z\mid X=x; \theta^t) \log P(X=x; Z=z;\theta)$$
It's part of the EM algorithm. Here, $\theta^t$ are the assumed parameters at ...
- 159
2
votes
3
answers
1k
views
Derivative of Softmax loss function (with temperature T)
I am try to calculate the derivative of cross-entropy, when the softmax layer has the temperature T. That is:
\begin{equation}
p_j = \frac{e^{o_j/T}}{\sum_k e^{o_k/T}}
\end{equation}
This question ...
- 23
1
vote
2
answers
395
views
What mathematics should I study to understand Neural Nets / Machine Learning?
I am strongly fascinated by neural nets, and perhaps other forms of machine learning. There are so many (potential) applications: teaching a robot with shaft encoders to drive along different ...
- 883
1
vote
2
answers
484
views
Mathematics disciplines underpinning Machine Learning
I have an undergrad degree in computational mathematics (though that was about 10 years ago), and spent my professional career in software development.
If I wanted to understand what's happening ...
- 133
1
vote
2
answers
1k
views
Expected squared prediction error conditioned on training set
I'm reading Elements of Statistical Learning by Hastie and Tibshirani, and I am thoroughly confused by the way they conditioned expected squared prediction error in section 2.5 (p.26):
\begin{align*}
...
0
votes
1
answer
128
views
How can I find the rank and spark relation here?
Hi, I can not get rank/spark relation correctly, I know that rank is a number of linearly independent columns of a matrix and spark is linearly dependent ones. In this question I understand option a), ...
- 11
59
votes
5
answers
22k
views
Why divide by $2m$
I'm taking a machine learning course. The professor has a model for linear regression. Where $h_\theta$ is the hypothesis (proposed model. linear regression, in this case), $J(\theta_1)$ is the cost ...
- 1,305
26
votes
1
answer
21k
views
Log of Softmax function Derivative.
Could someone explain how that derivative was arrived at.
According to me, the derivative of $\log(\text{softmax})$ is
$$
\nabla\log(\text{softmax}) =
\begin{cases}
1-\text{softmax}, & \text{if $...
21
votes
4
answers
10k
views
Deriving the normal distance from the origin to the decision surface
While studying discriminant functions for linear classification, I encountered the following:
.. if $\textbf{x}$ is a point on the decision surface, then $y(\textbf{x}) = 0$, and so the normal ...
- 413
17
votes
2
answers
17k
views
Proof of nonnegativity of KL divergence using Jensen's inequality
I'm a bit confused by the proof:
$KL(p||q) = -\int p(x) \log\left\{\frac{q(x)}{p(x)}\right\}dx \ge -\log \int p(x) \frac{q(x)}{p(x)}dx = -\log \int q(x)dx = 0$
where the first inequality is the ...
- 725
17
votes
2
answers
2k
views
What is the relationship between the Boltzmann distribution and information theory?
I'm reading a paper on Boltzmann machines (a type of neural network in Machine Learning), and it mentions that "The Boltzmann distribution has some beautiful mathematical properties and it is ...
- 1,063
17
votes
3
answers
49k
views
Logistic regression - Prove That the Cost Function Is Convex
I'm reading about Hole House (HoleHouse) - Stanford Machine Learning Notes - Logistic Regression.
You can do a find on "convex" to see the part that relates to my question.
Background:
$h_\theta(X) =...
- 477
16
votes
3
answers
11k
views
The median distance from the origin to the closest data point and the curse of dimensionality
I'm reading The Elements of Statistical Learning. I have a question about the curse of dimensionality.
In section 2.5, p.22:
Consider $N$ data points uniformly distributed in a $p$-dimensional unit ...
- 493
16
votes
4
answers
13k
views
How to calculate Vapnik-Chervonenkis dimension
it's my first post here, so I apologize if I broke a rule!
I'm reading Introduction to Machine Learning and got stuck on VC dimension. Here's a quote from the book:
"...we see that an axis-aligned ...
- 325
14
votes
3
answers
35k
views
Derivative of Binary Cross Entropy - why are my signs not right?
I'm trying to derive formulas used in backpropagation for a neural network that uses a binary cross entropy loss function. When I perform the differentiation, however, my signs do not come out right:
...
- 241