Questions tagged [machine-learning]
How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?
1,457
questions with no upvoted or accepted answers
10
votes
0
answers
458
views
Algebra & Artificial Intelligence (AI)
Artificial intelligence, especially deep learning & neural networks for image processing and classfication, are related to statistics and physics e.g. as decribed in below papers.
Statistics and ...
- 1,847
8
votes
0
answers
837
views
Optimization / personalization within clusters
I have the following optimization problem:
I have a (random and very noisy) objective function $f(A, P)$, where $A$ is a vector of "observable" parameters of the input and $P$ is the parameters that ...
- 261
7
votes
0
answers
2k
views
How are category theory and probability theory related?
How are category theory and probability theory related ? Category theory seems very useful for understanding objects with definite relationships, whereas probability theory (particular Bayesian ...
- 1,659
7
votes
0
answers
361
views
Absolute sum of partitioned Rademacher variables
$
\newcommand{\E}{\mathop{\mathbb{E}}}
$
Hi, this is the first time I post a question here, so I'd be glad to have comments to make it better. So here it goes.
The problem
I am looking to ...
- 71
7
votes
0
answers
534
views
Perturbation theory for least squares for very different A, b
Consider the least squares problem
$f(x;A,b) = \|Ax-b\|_2^2$
and define $x^*$ the minimizer of $f(x;\hat A,\hat b)$, and $\hat x$ the minimizer of $f(x; A_2, b_2)$.
I want to put some bound on $\|...
- 1,846
6
votes
0
answers
196
views
Necessary and sufficient conditions of linear separability of labelled cube $\{0,1\}^n$
This is a question I came up with when studying machine learning. A simple example revealing the limit of linear classifiers would be the four vertices of a square, one diagonal labelled $+1$ and the ...
- 103
6
votes
0
answers
175
views
Langevin dynamics
From this paper, if the deterministic dynamics of
$x_t$ is $dx_t=v_tdt$ where $v_t=\nabla\log\pi-\nabla\log\mu_t$ with $\mu_t$ denotes the law of $x_t$ and $\pi$ is a distribution depending on $x_t$, ...
- 61
6
votes
0
answers
476
views
Distance between point and convex hull in high dimensions
I am trying to develop an intuition for the properties of the convex hull of a set of points in high ($d>20$) dimensions.
Consider a set of $n$ data points which are iid distributed according to ...
- 232
6
votes
0
answers
1k
views
Euclidean distance (cosine) between two random positive unit vectors in high dimensional space
I found out that the largest possible euclidean distance (which is the cosine) between two random positive unit vectors decreases as the dimension of vector increases and approximates 0.71. This was ...
- 61
6
votes
2
answers
745
views
Bishop - Pattern Recognition & Machine Learning, Exercise 1.4
I'm working on exercise 1.4 in Bishop's Pattern Recognition & Machine Learning book.
This exercise is about probability densities. I've two questions about this exercise.
First, I don't understand ...
- 123
5
votes
0
answers
80
views
Conditional Bias Variance Decomposition
The standard bias variance decomposition says that:
$$
E |f(X) - Y|^2 = \int_{\mathbb{R}^d} |f(x) - m(x)|^2 \mu(dx) + E|m(X) - Y|^2,
$$
where $\mu$ is some distribution over $X$. I am trying to ...
- 6,126
5
votes
0
answers
562
views
Approximation of positive definite functions by neural networks
Bochner's theorem shows that probability measures $\mu$ are linked with positive definite functions via Fourier transform:
$f(k) = \int_{\mathbb{R}^n} e^{-2 \pi i k x} \,d\mu(x)$
Currently, ...
- 341
5
votes
0
answers
194
views
Is there any rigorous treatment of Markov Decision Process?
I am trying to find a mathematically rigorous introduction to MDP.
There are tons of resouces online but all of them are ... frankly terrible (and not even properly typesetted).
Just picking a few: ...
- 5,493
5
votes
0
answers
105
views
Bayes Estimator under $L_{\eta}$
I am wondering if the following loss function is well known and if it is, does it have a standard name:
$$
L_{\eta} (\theta, a) = (\theta-a) (\eta - \mathbb{I}_{(-\infty, a)} (\theta) ), \quad \eta \...
- 6,126
5
votes
0
answers
987
views
Roadmap to Differential Geometry for Machine Learning
Recently within machine learning, there are a lot of works on non-convex optimization and natural gradients methods etc which are based on differential geometry, it gives rise to increased need to ...
- 2,121
5
votes
0
answers
115
views
Why does minimizing $H[f] =\sum^{N}_{i=1}(y_i-f(x_i))^2+\lambda \| Pf \|^2 $ leads to solution of the form $ f(x) =\sum^N_{i=1}c_iG(x; x_i)+p(x)$?
I was reading the following paper of dimensionality reduction (1) and also one on theory of networks for approximations and learning (2) and was trying to understand how the regularization problem ...
- 6,644
5
votes
0
answers
444
views
VC-Dimension of Balls intersected with half-spaces
In $d$ dimensional Euclidean space, the VC-dimension of both the set of balls and the set of half-spaces is $d+1$. It follows that the VC-dimension of balls intersected with half-spaces is $O(d \log d)...
- 51
5
votes
1
answer
335
views
Quantitative Analysis of Structure of Gaussian Mixture Model
I am fitting a Gaussian Mixture Model to high-dimensional data (40 dimensions).
I have trained the model using EM, learned the parameters and now I want to know quantitatively:
What is most ...
- 51
4
votes
0
answers
61
views
Second order derivative of matrix functions
Let $A, B : \text{Sym}(d) \rightarrow \text{Sym}(d) $, where $\text{Sym}(d)$ is the set of $d \times d$ symmetric matrices. I am interested in computing the second order differential of $F(X) = A(X) B(...
4
votes
0
answers
125
views
Book suggestion on "Banach space geometry for machine learning"
Is there any book for a Mathematics student who can learn Machine learning in the aspect of Banach space geometry? Or, one can understand the connection between Geometry of Banach spaces and Machine ...
- 178
4
votes
0
answers
113
views
Similarity between mathematical expressions
I'm currently working on a neural network evaluating algebraic expressions. To validate the model we need a metric $D: X \times Y \to \mathbb{R^+}$, where $X$ and $Y$ are the predicted and correct ...
- 56
4
votes
0
answers
112
views
Consequence of Dvoretzky Stochastic Approximation Theorem
I am having some problems trying to apply Dvoretzky Stochastic Approximation Theorem to one Lemma used in a paper I found about the proof of convergence of some reinforcement learning temporal ...
- 41
4
votes
0
answers
426
views
Multiclass Linear Discriminant Analysis
This question is based on the Multiclass Linear Discriminant Analysis (MLDA) describe in Lectures slides by Olga Veksler, which is a generalization of Fisher's Linear Discriminant. My use in MLDA is ...
- 332
4
votes
0
answers
40
views
Kullback–Leibler divergence between a quasi-arithmetic mean of Normal distributions and a standard Normal distribution
Is there a way to compute the Kullback–Leibler divergence between a quasi arithmetic mean of normal distributions and a standard normal distribution in closed form?
$$D_{KL} = D_{KL}\left(f^{-1}\left(...
- 171
4
votes
0
answers
376
views
Question about the definition of feature map arising in machine learning
I'm working through the following paper of learning a non-negative function in a reproducing kernel hilbert space setting (RKHS). In particular, section 2.2 on page 3 is a bit confusing to me in terms ...
- 4,447
4
votes
0
answers
91
views
What might be good textbooks for the mathematics behind Machine Learning?
I am currently studying Linear Algebra and I have taken Calculus 1, 2, and 3, as well as Differential Equations. Can someone please provide guidance on what material I should study next in order to ...
- 2,651
4
votes
0
answers
126
views
Can we sequentially optimize a function that is convex but not jointly convex?
Suppose I have a function $f(x,y)$ where $f$ is convex in $x$ and $y$ but not in $(x,y)$. Suppose that for a fixed $y$, the minimization over $x$ lands on a solution that doesn't depend on $y$, e.g.
...
- 1,846
4
votes
0
answers
153
views
Why does Logistic Regression need Normalized data
I am trying to implement logistic regression in some problem, but while using normal data gives me some nan results. When I normalize the data I get correct results, so why does Logistic Regression ...
4
votes
0
answers
577
views
Looking for density estimator with time complexity $< \mathcal O(n^2)$
I am doing univariate non-parametric density estimation on a dataset $D$, and I want to
1) Train a density estimator on $D$
2) Compute the estimated density at each point in $D$
These two ...
- 326
4
votes
0
answers
85
views
Advanced Math for Reinfrocement Learning - state space and state sequences (policies)
Reinforcement learning has two important notions and I am interested in advanced math that can investigate those notions:
State space - set of states. Apparently, deep structures should exist in this ...
- 1,353
4
votes
0
answers
181
views
Multivariate Conditional Entropy as a test of correlation between random variables
I use the word columns to mean the data from which a random variable can be estimated. It is a sample of a random variable.
I am working with $N$ columns of weakly correlated data. Furthermore, I ...
- 1,291
4
votes
0
answers
207
views
James–Stein estimator
Consider a FIR model of the form $y= Ug_0+e$ with $e$ white noise with variance $\sigma^2$. We assume that we have collected N input-output measurements $y$ and $U$.
The James–Stein estimator is ...
- 484
4
votes
2
answers
175
views
Estimate $P(A_1|A_2 \cup A_3 \cup A_4...)$, given $P(A_i|A_j)$
This question is related to some undergraduate research on summary generation of documents of which I am a part of. I am trying to estimate $P(A_1|A_2 \cup A_3 \cup A_4...A_k)$, where I know the ...
- 1,067
4
votes
0
answers
210
views
Genetic algorithm - find max of minimized subsets
I have a combinatorial optimization problem for which I have a genetic algorithm to approximate the global minima.
Given X elements find: min f(X)
Now I want to expand the search over all possible ...
- 41
4
votes
1
answer
296
views
Is Multiple Integration Over Random Variables NP HARD
In a seminar, in a passing moment a presenter mentioned that multiple integrals over random variables is NP-Hard. This means there is no efficient algorithm to compute the integral.
So
Theorem: ...
- 111
4
votes
0
answers
349
views
Illustration of Von Neumann's Minimax theorem in games?
The Von Neumann's Minimax theorem gives the conditions that make the max-min inequality an equality.
I understand the max-min inequality, basically $\min(\max(f))\ge \max(\min(f))$.
The Von Neumann'...
- 496
4
votes
0
answers
448
views
Backpropagation in a Convolutional Neural Network
Consider a Convolutional Neural Network with the following architecture:
\begin{align}
Input---C_1 P_1 --- C_2 P_2 ---Softmax
\end{align}
Here $C_i$ refers to the $i^{th}$ convolutional layer ...
4
votes
0
answers
291
views
Derivation of back-propagation equation $\frac{\partial E(\theta)}{\partial W^k}=x*\delta h^k+\tilde{h}^k*\delta y$ for convolutional autoencoders
I was reading the following paper on convolution stacked auto-encoders and they had the following convolution neural network (for auto-encoders, notice I didn't write the offset term [to avoid ...
- 6,644
4
votes
0
answers
196
views
Machine Learning and Probability/Stochastics
Main question: What connections are there between machine learning and stochastics (Probability theory, analysis, processes, SDEs)?
Background: I've just been accepted into a master's programme for ...
- 5,710
4
votes
0
answers
1k
views
Is information entropy $H(X)$ a sub modular function?
I was trying to learn more about sub modular functions and wanted to see an example of proving that some function is sub modular. Wikipedia said that Entropy was an example so I decided to try it out ...
- 6,644
4
votes
1
answer
445
views
Relation between RKHS and space of continuous functions
Consider a Mercer Kernel $K\colon \mathcal{X}\times \mathcal{X}\to \mathbb{R}$, $\mathcal{X}$ being a compact subset of $\mathbb{R}^m$, and its (unique) associated Reproducing Kernel HIlbert Space $...
- 2,279
4
votes
0
answers
857
views
Modern Mathematical Theory for Neural Networks, Cellular Automata, Neuroscience
Is it possible for someone to do research on subjects like neural networks, cellular automata, or neuroscience as an applied mathematician?
I have in mind the theoretical development of these fields,...
- 81
4
votes
1
answer
517
views
Relation between factor graph and conditional probability distribution
First, I'm from computer science. I don't know how to say this problem in a mathematical way. So please bear with me.
The question
Let say I have a factor graph illustrated in the figure.
The ...
- 135
4
votes
0
answers
175
views
Boltzmann machines - motivation for the energy function
I've been studying Boltzmann machines lately and was wondering if anyone could give me a "high-level" explanation or motivation for the energy function used:
$$E = -\sum_{i<j} w_{ij} \, s_i \, s_j ...
- 175
4
votes
1
answer
765
views
$i,j,k$ Values of the $\Theta$ Matrix in Neural Networks
SO I'm looking at these two neural networks and walking through how the $ijk$ values of $\Theta$ correspond to the layer, the node number.
Either there are redundant values or I'm missing how the ...
- 141
4
votes
0
answers
584
views
Theoretical proof of convergence of sequential weight update procedure (Neural Networks and Machine Learning)
My question is at the bottom.
(Most of the descriptive words come from Chris. Bishop's Neural Networks for Pattern Recognition)
Let $w$ be the weight vector of the neural network and $E$ the error ...
- 571
3
votes
0
answers
121
views
Kernel Regression, Similarity-Based Modeling & Weights Normalization
Let be $D=[x_j(t_i)]_{i,j} \ M_{n,p}(\mathbb{R})$ a state matrix of $n$ $x(t_i) \in M_{1,p}(\mathbb{R})$ observations of $p$ sensors $X_j$ representing the normal conditions of a system for some ...
- 101
3
votes
0
answers
68
views
A self-proof of Vapnik - Chervonenkis theorem
Theorem: For every $\varepsilon >0$, with the probability greater than $1-\varepsilon$
\begin{align*}
R_p(\hat{g}_{n,\mathcal{G}}) - R_{p}(g^*_{p,\mathcal{G}}) \le 2 \sqrt{\dfrac{2V_{\mathcal{G}...
- 57
3
votes
0
answers
45
views
Minimizaiton of $f\mapsto\int\frac12\|f\|^2+\nabla\cdot f\:{\rm d}\mu$, when $\mu$ is only given by i.i.d. samples
I know this question is quite vague, but I need some indication. I have a problem where I have a probability distribution $\mu$ on $\mathbb R^d$ and I want to find a differentiable function $f:\mathbb ...
- 13.9k
3
votes
1
answer
133
views
How do I understand this aspect of Shapley Values?
I have read a few of the posts on here regarding Shapley values and have started to form an intuition surrounding it, especially in connection with explainability of ML models. However, I am still ...
- 6,670