Questions tagged [machine-learning]
How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?
3,217
questions
196
votes
1
answer
192k
views
Derivative of Softmax loss function
I am trying to wrap my head around back-propagation in a neural network with a Softmax classifier, which uses the Softmax function:
\begin{equation}
p_j = \frac{e^{o_j}}{\sum_k e^{o_k}}
\end{equation}...
- 2,157
140
votes
11
answers
206k
views
What is the difference between regression and classification?
What is the difference between regression and classification, when we try to generate output for a training data set $x$?
- 2,526
123
votes
8
answers
105k
views
derivative of cost function for Logistic Regression
I am going over the lectures on Machine Learning at Coursera.
I am struggling with the following. How can the partial derivative of
$$J(\theta)=-\frac{1}{m}\sum_{i=1}^{m}y^{i}\log(h_\theta(x^{i}))+(1-...
- 1,355
66
votes
4
answers
52k
views
Why we consider log likelihood instead of Likelihood in Gaussian Distribution
I am reading Gaussian Distribution from a machine learning book. It states that -
We shall determine values for the unknown parameters $\mu$ and
$\sigma^2$ in the Gaussian by maximizing the ...
- 763
59
votes
5
answers
22k
views
Why divide by $2m$
I'm taking a machine learning course. The professor has a model for linear regression. Where $h_\theta$ is the hypothesis (proposed model. linear regression, in this case), $J(\theta_1)$ is the cost ...
- 1,303
44
votes
2
answers
32k
views
How is logistic loss and cross-entropy related?
I found that Kullback-Leibler loss, log-loss or cross-entropy is the same loss function. Is the logistic-loss function used in logistic regression equivalent to the cross-entropy function? If yes, can ...
- 543
34
votes
3
answers
3k
views
Mathematical preparation for postgraduate studies in Linguistics
I am an undergraduate student in Mathematics and I would like to continue my postgraduate studies in the harder, more mathematical aspects of Linguistics. What exactly would that include is unknown ...
- 1,049
31
votes
2
answers
5k
views
Mathematical introduction to machine learning
At first glance, this is once again a reference request for "How to start machine learning".
However, my mathematical background is relatively strong and I am looking for an introduction to ...
- 2,089
29
votes
7
answers
5k
views
What are the best books to study Neural Networks from a purely mathematical perspective?
I am looking for a book that goes through the mathematical aspects of neural networks, from simple forward passage of multilayer perceptron in matrix form or differentiation of activation functions, ...
- 571
27
votes
2
answers
21k
views
Invert the softmax function
Is it possible to revert the softmax function in order to obtain the original values $x_i$?
$$S_i=\frac{e^{x_i}}{\sum e^{x_i}} $$
In case of 3 input variables this problem boils down to finding $a$, ...
- 1,253
26
votes
1
answer
21k
views
Log of Softmax function Derivative.
Could someone explain how that derivative was arrived at.
According to me, the derivative of $\log(\text{softmax})$ is
$$
\nabla\log(\text{softmax}) =
\begin{cases}
1-\text{softmax}, & \text{if $...
21
votes
4
answers
10k
views
Deriving the normal distance from the origin to the decision surface
While studying discriminant functions for linear classification, I encountered the following:
.. if $\textbf{x}$ is a point on the decision surface, then $y(\textbf{x}) = 0$, and so the normal ...
- 413
20
votes
2
answers
50k
views
What is divergence in image processing?
What is the difference between gradient and divergence?
I understood that gradient points in the direction of steepest ascent and divergence measures source strength. I couldn't relate this to the ...
- 411
19
votes
4
answers
16k
views
Deriving cost function using MLE :Why use log function?
I am learning machine learning from Andrew Ng's open-class notes and coursera.org. I am trying to understand how the cost function for the logistic regression is derived. I will start with the cost ...
- 195
18
votes
1
answer
14k
views
Why does a radial basis function kernel imply an infinite dimension map?
I understand that each kernel implies a particular feature map. For instance for $x,z \in R^2$ the kernel $K(x,z)=(\textrm{dot}(x,z))^2$ implies a feature map $$\langle\phi(x_1), \phi(x_2)\rangle=\...
- 333
17
votes
3
answers
48k
views
Logistic regression - Prove That the Cost Function Is Convex
I'm reading about Hole House (HoleHouse) - Stanford Machine Learning Notes - Logistic Regression.
You can do a find on "convex" to see the part that relates to my question.
Background:
$h_\theta(X) =...
- 477
17
votes
2
answers
16k
views
Proof of nonnegativity of KL divergence using Jensen's inequality
I'm a bit confused by the proof:
$KL(p||q) = -\int p(x) \log\left\{\frac{q(x)}{p(x)}\right\}dx \ge -\log \int p(x) \frac{q(x)}{p(x)}dx = -\log \int q(x)dx = 0$
where the first inequality is the ...
- 725
17
votes
2
answers
2k
views
What is the relationship between the Boltzmann distribution and information theory?
I'm reading a paper on Boltzmann machines (a type of neural network in Machine Learning), and it mentions that "The Boltzmann distribution has some beautiful mathematical properties and it is ...
- 1,063
16
votes
3
answers
7k
views
Gradients of marginal likelihood of Gaussian Process with squared exponential covariance, for learning hyper-parameters
The derivation of gradient of the marginal likelihood is given in http://www.gaussianprocess.org/gpml/chapters/RW5.pdf
But the gradient for the most commonly used covariance function, squared ...
- 517
16
votes
3
answers
11k
views
The median distance from the origin to the closest data point and the curse of dimensionality
I'm reading The Elements of Statistical Learning. I have a question about the curse of dimensionality.
In section 2.5, p.22:
Consider $N$ data points uniformly distributed in a $p$-dimensional unit ...
- 493
16
votes
2
answers
5k
views
Tricky proof of a result of Michael Nielsen's book "Neural Networks and Deep Learning".
In his free online book, "Neural Networks and Deep Learning", Michael Nielsen proposes to prove the next result:
If $C$ is a cost function which depends on $v_{1}, v_{2}, ..., v_{n}$, he states that ...
- 796
16
votes
4
answers
13k
views
How to calculate Vapnik-Chervonenkis dimension
it's my first post here, so I apologize if I broke a rule!
I'm reading Introduction to Machine Learning and got stuck on VC dimension. Here's a quote from the book:
"...we see that an axis-aligned ...
- 325
15
votes
5
answers
16k
views
Formal proof that mean minimize squared error function
On an important book of Machine Learning, I've found this proof.
We want to minimize the cost function $J_0(X_0)$ defined by the formula
$$J_0(x_0) = \sum_{k=1}^n \|x_0 - x_k \|^2.$$
The solution to ...
- 253
15
votes
1
answer
3k
views
Transforming a distance function to a kernel
Fix a domain $X$: Let $d : X \times X \rightarrow \mathbb{R}$ be a distance function on $X$, with the properties
$d(x,y) = 0 \iff x = y$ for all $x,y$
$d(x,y) = d(y,x)$ for all $x,y$
Optionally, $d$ ...
- 1,302
15
votes
4
answers
4k
views
What is a good book for math students to learn machine learning in depth?
I am a math master student and have done fundamental math courses like probability theory, measure theory, linear algebra and know a little bit about functional analysis. What is good way for me to ...
- 1,747
15
votes
1
answer
1k
views
Stochastic gradient descent for convex optimization
What happens if a convex objective is optimized by stochastic gradient descent?
Is a global solution achieved?
- 3,456
14
votes
3
answers
35k
views
Derivative of Binary Cross Entropy - why are my signs not right?
I'm trying to derive formulas used in backpropagation for a neural network that uses a binary cross entropy loss function. When I perform the differentiation, however, my signs do not come out right:
...
- 241
14
votes
3
answers
13k
views
When does a maximum likelihood estimate fail to exist?
I have been told that a maximum likelihood estimate (MLE) does not always actually exist. Why is this the case? It is clear that the MLE may not be unique, but there should always be a maximum, no?
- 141
14
votes
1
answer
3k
views
How can I derive the back propagation formula in a more elegant way?
When you compute the gradient of the cost function of a neural network with respect to its weights, as I currently understand it, you can only do it by computing the partial derivative of the cost ...
- 1,069
14
votes
4
answers
7k
views
Lagrange multipliers and KKT conditions - what do we gain?
I'm working through an optimization problem that reformulates the problem in terms of KKT conditions. Can someone please have a go at explaining the following in simple terms?
What do we gain by ...
- 689
14
votes
3
answers
1k
views
What is the most general formalism for machine learning?
Most of the literature I can find in the field of machine learning is extremely practical, listing many techniques you can use like neural networks, SVMs, random forests, and so on. There are lots of ...
- 495
13
votes
2
answers
20k
views
what argmax means?
my main problem is that, i don't understand what argmax means in this equation (page 134., figure 4., the output part)
I want to write a code, but i don't understand this equation. Is there any ...
- 141
13
votes
3
answers
46k
views
How many parameters does the neural network have?
We have a neural network with an input layer of ℎ0 nodes, hidden layers of ℎ1 , ℎ2 , ℎ3 , ..., ℎ𝑙−1 nodes respectively and an output layer of ℎ𝑙 nodes.
How many parameters does the network ...
- 139
13
votes
2
answers
5k
views
Size of the vocabulary in Laplace smoothing for a trigram language model
Let's say we have a text document with $N$ unique words making up a vocabulary $V$, $|V| = N$. For a bigram language model with add-one smoothing, we define a conditional probability of any word $w_{i}...
- 231
12
votes
8
answers
3k
views
Where to start Machine Learning?
I've recently stumbled upon machine learning, generating user recommendations based on user data, generating text teaser based on an article. Although there are tons of framework that does this(Apache ...
- 229
12
votes
3
answers
6k
views
Could somebody elaborate "dimensional space" and "hyperplane"?
I am reading a text related to SVM, and the mathematical language is giving me a little hard time.
Here training vectors xi
are mapped into a higher (maybe
infinite) dimensional space by the
...
- 875
12
votes
5
answers
8k
views
Why eigenvectors with the highest eigenvalues maximize the variance in PCA?
I'm learning Principal Component Analysis (PCA) and came to know that eigenvectors of the covariance matrix of the data are the principal components, which maximizes the variance of the projected data....
- 675
12
votes
1
answer
6k
views
What does it mean to "marginalise out" something?
Especially in machine learning one often reads the phrase "to marginalise out" something, and while I understand that this means to integrate over a property, I cannot quite grasp the larger ...
- 712
11
votes
3
answers
15k
views
Understanding “the mean minimizes the mean squared error”
I am trying to understand the sentence
the mean minimizes the mean squared error.
from wikipedia https://en.wikipedia.org/wiki/Average_absolute_deviation.
From ...
- 213
11
votes
1
answer
15k
views
Why are the eigenvalues of a covariance matrix equal to the variance of its eigenvectors?
This assertion came up in a Deep Learning course I am taking. I understand intuitively that the eigenvector with the largest eigenvalue will be the direction in which the most variance occurs. I ...
- 248
11
votes
2
answers
2k
views
What is the motivation for using cross-entropy to compare two probability vectors?
Define a "probability vector" to be a vector $p = (p_1,\ldots, p_K) \in \mathbb R^K$ whose components are nonnegative and which satisfies $\sum_{k=1}^K p_k = 1$. We can think of a probability vector ...
- 50.5k
11
votes
2
answers
8k
views
why is the least square cost function for linear regression convex
I was looking at Andrew Ng's machine learning course and for linear regression he defined a hypothesis function to be $h(x) = \theta_0 + \theta_1x_1 + ... + \theta_nx_n$, where $x$ is a vector of ...
- 371
11
votes
1
answer
3k
views
Category Theory & Artificial Intelligence (AI)
Category theory turns out to be useful in more and more areas.
(see e.g. MSE - Category Theory & Biology)
Question. Does anyeone know of some connection of category theory to (convolutional) ...
- 1,777
11
votes
1
answer
13k
views
What is the difference between Curve Fitting and Regression(Machine Learning)?
I know that Machine Learning regression algorithms try to find the function of the data. That is, if we have 1000 data points (x,y), to find a general continuous function that follows the trends of ...
- 657
10
votes
3
answers
19k
views
How do you minimize "hinge-loss"?
A lot of material on the web regarding Loss functions talk about "minimizing the Hinge Loss".
However, nobody actually explains it, or at least gives some example.
The best material I found is here ...
- 1,632
10
votes
2
answers
8k
views
What is a sampling density? Why is the sampling density proportional to $N^{1/p}$?
I'm reading a book named The Elements of Statistical Learning by Hastie, in section 2.5, Local Methods in High Dimensions, it says that the sampling density is proportional to $N^{\frac{1}{p}}$, where ...
- 197
10
votes
1
answer
2k
views
What connections between machine learning and dynamical systems?
I have a background of ("pure") dynamical systems and ergodic theory, but I am switching to machine learning.
Can some machine learning questions be treated from a dynamical systems/ergodic theory ...
- 133
10
votes
1
answer
3k
views
Inner Product Space vs. Vector Space
I had no trouble understanding what a vector space is: a constraint on the type of vectors you can create, such that certain operations could be performed with them.
For example, a vector space of
$$...
- 1,632
10
votes
3
answers
4k
views
What is the significance of theoretical linear algebra in machine learning/computer vision research?
I am a computer science research student working in application of Machine Learning to solve Computer Vision problems.
Since, lot of linear algebra(eigenvalues, SVD etc.) comes up when reading ...
- 267
10
votes
1
answer
742
views
Category theory for sensorimotor learning?
Robotics, machine learning, inference, control, decision theory, system identification. There are many different views on how the information would flow from the environment into a robot, and the ...
- 781