Questions tagged [machine-learning]
How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?
3,378
questions
1
vote
0
answers
19
views
Why the left matrix of single value decomposition obtained by scipy.linalg.svd() and eigenvectors obtained from scipy.linalg.eig() are different?
I tried to get eigen vectors by single value decomposition using scipy.linalg.svd() which gives me three outputs the left matrix ...
1
vote
2
answers
80
views
Mathematics for Machine Learning by Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong. Theorem 2.20 (basis change) questions
I am struggling to understand this proof, and have some questions which I will list. Here is a link to the textbook
https://mml-book.github.io/book/mml-book.pdf
and the theorem and proof I am ...
- 10.7k
0
votes
1
answer
31
views
Why is the base of the exponential function in the softmax formula chosen as e, rather than 2 or other numbers?
I have been studying the softmax function, and I am curious why the natural constant $e$ is used as the base of the exponential function in the softmax formula. Specifically, I would like to ...
- 26.1k
0
votes
0
answers
9
views
Expected vs empirical loss with data augmentation
In a typical supervised learning setup, we assume our data $X$ with labels $Y$ comes from a data distribution $(X,Y) \sim P(X,Y)$.
The expected loss for some loss function, $L$, is:
$R_{L,P,f} = \...
- 121
1
vote
0
answers
8
views
Centered Kernel Alignment and eigenspaces
In the paper I found, I read a paragraph that Centered Kernel Alignment shows the similarity between the top eigenspaces of kernels, before centering.
Centered Kernel Alignment is
$$\frac{HSIC(K, K')}{...
- 11
-1
votes
0
answers
40
views
Why does ReLU work? [closed]
In our machine learning course about NN's, we learned about non-linearity functions and had couple of examples like sigmoid, hyperbolic and ReLU. Intuitively I do understand, why sigmoid for example ...
- 1
0
votes
1
answer
2k
views
Scaling Cumulative Probability Distribution function values
We have a cumulative probability distribution function (cdf), we want to scale it down for using it in anomaly detection. The mapping should look like this.
CDF value: 0.1 ... 0.5 ... 0.9 ... ...
CommunityBot
- 1
0
votes
0
answers
15
views
The Laplacian matrix in GNNs [closed]
I’m a bit confused about the role of the Laplacian matrix in GNNs. In classic GNN papers, such as GCN and GraphSAGE, we only need the normalized adjacency matrix for message passing. Why do many ...
- 43
0
votes
0
answers
43
views
Bishop gradient calculation
In section 3.1.1 of Pattern Recognition and Machine Learning by Christopher Bishop, it is written that
$$\ln p(\mathbf{t} | \mathbf{w}, \beta) = \frac{N}{2} \ln \beta - \frac{N}{2} \ln (2 \pi) - \beta ...
- 37.7k
0
votes
2
answers
2k
views
Similarity metric between two sets of points with varying densities
How can I create a similarity metric that describes the top left set of points as more similar to the bottom left set of points than the top right set of points? Clearly least-squares distance doesn't ...
CommunityBot
- 1
0
votes
1
answer
2k
views
Understanding this Graph: What is a PetaFlop?
I was looking at this paper (https://arxiv.org/pdf/2005.14165.pdf) and came across this graph:
I am trying to understand the following two things about this graph:
What is PetaFLOP/s-days? I read ...
CommunityBot
- 1
1
vote
0
answers
71
views
Approximating the gradient of $f(x)g(x)$ when $f(x)$ is non-differentiable but has a smooth approximate $h(x)$?
Consider optimizing $\sum_{t=1}^n f_t(x)g_t(x)$ using online gradient descent. However $f_t(x)$ is non-differentiable and has a smooth approximate $h_t(x)$ . The online gradient descent is as follows:...
- 123
2
votes
0
answers
24
views
Connection between general loss functions and the MLE
first up a big disclaimer: This question might already be answered somewhere, so if I have missed it, please just redirect me.
Let $X,Y$ be two (real-valued) random variables. I will assume that their ...
- 189
0
votes
1
answer
24
views
Is the separability of $m+1>d$ vectors in $\mathbb{R}^d$ preserved if we use a weight vector orthogonal to one of the vectors?
$\def\bx{\mathbf{x}}$
$\def\bw{\mathbf{w}}$
$\def\R{\mathbb{R}}$
Vectors $\bx_1,\ldots, \bx_{m},\bx_{m+1} \in \R^d$ ($m>d$) are linearly separable by a hyperplane $\bw \cdot \bx =0$:
\begin{align}
\...
- 167
8
votes
1
answer
5k
views
How to weight Jaccard Similarity
I'd like to calculate the similarity between two sets using Jaccard similarity but temper the results using the relative frequency of each item within a corpus.
Jaccard similarity is defined as the ...
- 84.6k
0
votes
0
answers
25
views
Weighted Jaccard with positive and negative weights
Problem
I am trying to use weighted jaccard to compare two weighted sets, S and T, with weights that range from -1 to +1 – for example:
...
- 399k
12
votes
1
answer
22k
views
Derivative of Mean Squared Error
I'm studying with a book and I'm at the Linear Regression part. The author is showing that we have to calculate the derivative of each part of the equation that leads to the loss.
But he's using the ...
- 111
0
votes
0
answers
28
views
Does the Sum of Weighted Inverse Roots of Cumulative Samples Scale with Total Samples?
I have a sampling process defined over a sample space $\{1,2,\dots, M\}$, and $M$ is the number of items.
The sampling is done in $B$ batches, and each batch is repeated $\ell_i$ times. In each repeat,...
- 49
2
votes
1
answer
167
views
Representing Machine Learning in Mathematical Language
Consider a generic (univariate) time series model:
$$ \hat{y}(x_n, \ldots, x_{n+h-1} \mid x_0, \ldots, x_{n-1}; \boldsymbol{\theta}) $$
Where:
$$ \hat{y}_{t+h} \text{ is the forecast for the next } h \...
- 776
0
votes
0
answers
24
views
Does the loss function necessarily converge to zero as we get more samples?
Background: I'm reading quite a long paper(link) about quantile regression which uses the transfer learning method in this field. A crucial part of this method is to avoid negative transfer --when you ...
- 362
7
votes
4
answers
457
views
Rigorous Mathematical foundations of Machine Learning / Deep Learning / Neural Networks
I am an Engineering Graduate (with a strong background in Probability/Measure Theory, Linear Algebra and Calculus) wanting to dig deep into Deep Learning and Neural Networks, and I'm looking for ...
- 411
1
vote
2
answers
31
views
Why does Least Absolute Deviation regression exactly fit n measureemnts for a linear system with n independent variables?
I am applying LAD regression to conduct some research. I have the following questions regarding LAD:
I know LAD exactly fits n measurements for a linear system with n variables. But I cannot easily ...
- 1,910
0
votes
0
answers
35
views
Finding Weight Differentials in Differential Neural Network
I am implementing a Neural Network from scratch that trains on standard Labels $(Y)$ and Differential Labels $(\frac{\partial Y}{\partial X} = \bar{X})$. This involves using a cost function
$C = (\...
1
vote
1
answer
39
views
Transformation Matrix Definition
The textbook I am referencing is here
https://mml-book.github.io/book/mml-book.pdf
and the definition I am asking about is on page 51.
In definition 2.19, the mapping $\Phi$ takes us from $V \...
- 1,878
2
votes
1
answer
1k
views
Is there a concrete proof showing that hinge loss is an upper bound on the 0-1 loss?
While it is stated in several places that the Hinge loss is a convex upper bound on the 0-1 loss, is there a proof behind it? From what I have seen, most resources just show the plots of hinge loss ...
CommunityBot
- 1
2
votes
4
answers
875
views
Gaussian complexity bound
I am reading Foundations of Machine Learning (1st edition). It seems that most generalization bounds in the literature are based on Rademacher complexity, rather than Gaussian complexity. So, I was ...
- 11
0
votes
2
answers
2k
views
VC-dimension of parity functions
Consider the boolean hypercube $\{0,1\}^N$. For a set I $\subseteq$ {1,2,...N}, we define the parity function $h_I$ as follows. For a binary vector x = $(x_1, x_2, ...,x_N) \in \{0,1\}^N$,
$h_I(x) = \...
CommunityBot
- 1
0
votes
1
answer
64
views
Intuition for Why Resampling Should Create New Information
If one plots a bootstrap distribution of "resample means" obtained by resampling a single, real sample from a population distribution, the mean of that plot gives no more information than ...
- 5,525
0
votes
0
answers
12
views
Distance in Grassmann Manifolds between 2 samples of different datasets
I have a question about the paper 'Domain Adaptation as Optimal Transport on
Grassmann Manifolds' by Long et al.
In the paper, they use distance on Grassmann Manifolds to define a cost function for an ...
- 1,040
0
votes
0
answers
19
views
Rate of the variance decrease in the elbow plot of K-means clustering
Say there are $k$ points $\{x_1,...,x_k\}$, let $f(n)$ be the minimum within-cluster variance with $n$ cluster centroids, that is
$$
f(n)=\min_{\{\mu_1,...\mu_n\}}\sum_{i=1}^k\min_{j\in[n]}\|x_i-\mu_j\...
- 1
2
votes
1
answer
108
views
Number of ways of separating a finite set with linear hyperplanes.
I have the following question (3.3 (b)) from Foundations of Machine Learning by Mehryar Mohri (second edition).
3.3 (b). Let $\mathcal{X}=\{\mathbf{x}_1,\dots, \mathbf{x}_m\}$ be a subset of
$\mathbb{...
- 1,165
0
votes
1
answer
41
views
Is Marginalizing in the Condition Legal?
In this video, the professor writes...$$\Pr(y\ |\ x\ \cap\ \vec D) = \int_\vec w \Pr(y\ |\ x\ \cap\ \vec D\ \cap\ \vec w)\ d\vec w = \int_\vec w \Pr(y\ |\ x\ \cap\ \vec w)\Pr(\vec w\ |\ \vec D)\ d\vec ...
- 2,482
0
votes
0
answers
38
views
Definition of the Bayes Optimal Predictor with a conditional probability
I am reading the book "Understanding Machine Learning" by Shalev-Shwartz and Ben-David and I have a problem about how a certain conditional probability is define.
Here is the context :
For ...
- 4,819
0
votes
0
answers
35
views
Intuition on the norm induced by a positive semi-definite matrix
I am reading a paper about machine learning. The notations and problem setting can be simplified as follows:
We want to learn the real parameter $\theta^{*}$. Given data point $\{(x_{1}, y_{1}) ..., (...
- 434
0
votes
0
answers
21
views
Minimization of KL divergence is equivalent to minimization of NLL using Dirac
I am reading "Probabilistic Machine Learning: An Introduction", In chapter 4 Statistics page 109:
A empirical distribution
$$p_D(y) = \frac{1}{N}\sum_{n=1}^N\delta(y-y_n)$$
The KL divergence
...
- 2,482
0
votes
0
answers
52
views
Can a simple neural network model(MLP) predict or fit the square root of x?
According to the description in the Can AI Predict What Will Happen? section of the article Can AI Solve Science? on stephenwolfram.com,
... there are "no model-less models": different ...
- 151
2
votes
0
answers
36
views
Is building the universes for (∞-)toposes (of subsets of data points) a categorical generalization of theory induction in ML and algorithm?
I am reading https://arxiv.org/abs/1904.07004 "All (∞,1)-toposes have strict univalent universes" and I am thinking about its applications to the machine learning.
Can we recast the ...
- 1,363
0
votes
0
answers
26
views
What am I missing in deriving the dual of this soft margin support vector machine?
I am trying to derive the dual form of a support vector machine from its primal form (see top of section "Kernel SVM").
While missing from the formulation linked above, I've proceeded thus ...
- 1,964
2
votes
0
answers
35
views
Change of variable to find PDF of Y = U(X) given PDF of X
I am referring to the section on change of variable in 'Mathematics for Machine Learning' book. Essentially, given a random variable X defined in the range $[a, b]$, the ask is to find the PDF of ...
- 43
1
vote
0
answers
17
views
Recommendations for Learning About Hypergraphs
I'm looking to deepen my understanding of hypergraphs and their applications, particularly in the context of neural networks. I have a background in mathematics from my university studies, although I ...
- 11
0
votes
1
answer
26
views
Convergence of log-density ratio of KL divergence(discriminator) when generator converges
In the paper "A Deep Generative Approach to Conditional Sampling", the author writes in the proof of Theorem 4.1:
Since
$$\Vert G^* - \bar{G}_\theta \Vert_{L^\infty(E_1)}\to 0, \quad \text{...
- 25
1
vote
2
answers
59
views
Barto and Sutton's Reinforcement Learning Exercise 3.11: Don't understand solution because of knowledge gaps
I'm going through Barto and Sutton's Reinforcement Learning book with only knowledge of high-school Pre-calc and previous programming experience. I've been able to go through Chapters 1 and 2 ...
- 11
0
votes
0
answers
12
views
Lipschitz Continuous Function in the Embedding Space
Given three metric spaces $(X,d_X),(Y,d_Y),(Z,d_Z)$ and an embedding function $g: X \rightarrow Z$, I am interested in finding more information about functions $f: X \rightarrow Y$ in which there ...
- 48.2k
0
votes
0
answers
37
views
Approximating inverse of an ill-conditioned matrix
I am not sure whether this is the right place to ask this question or not, it might be not really a mathematical question but more a deep learning question. But the base of question is mathematics, so ...
- 71
0
votes
0
answers
55
views
How to estimate the inverse of a non-invertible matrix?
So I'm working on a machine learning problem where my solution requires taking the inverse of a matrix at some point. The problem is that this matrix is sometimes non-invertible. In theory the the ...
- 21
0
votes
0
answers
37
views
Formulating a solution ansatz for the 1D heat equation in polar coordinates to learn the PDE in a PINN setting
Hello Math Stack Exchange Community,
I am working on solving a partial differential equation (PDE) with a neural network in a PINN-like fashion, and I am seeking advice on identifying an appropriate ...
- 1
7
votes
1
answer
87
views
What is the collection of functions that a given finite neural network can approximate with ease?
To my understanding, one version of the universal approximation theorem runs as follows: Let $\Phi$ be the family of (trained) feedforward neural networks of bounded width, arbitrary depth, and mild ...
- 269
5
votes
2
answers
208
views
An upper bound of the sum of factorial in Vapnik and Chervonenkis, 1971
How to prove the following inequality:
\begin{align*}
\Gamma=\sum_{k\in[0,m]\wedge|\frac{2k}{l}-\frac{m}{l}|\ge\frac{\epsilon}{2}}\frac{{m\choose k}{2l-m\choose l-k}}{{2l\choose l}}\le2e^{-\frac{\...
- 129
0
votes
1
answer
23
views
Question about likelhood function of discriminative models
Im a little confused with the likelihood function. For discriminative models, we have a hypothesis function $h_{\theta}(x) = p(y \mid x ; \theta)$. Using the principles of maximim likelihood we want ...
- 161k
0
votes
1
answer
14k
views
machine learning octave code gradient descent question
I'm taking Coursera Machine learning course. so who take this courses will able to help this problem.
this is the octave code to find the delta for gradient descent.
...
CommunityBot
- 1