Questions tagged [machine-learning]

How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

1,429 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
9 votes
0 answers
357 views

Algebra & Artificial Intelligence (AI)

Artificial intelligence, especially deep learning & neural networks for image processing and classfication, are related to statistics and physics e.g. as decribed in below papers. Statistics and ...
FWE's user avatar
  • 1,777
8 votes
0 answers
832 views

Optimization / personalization within clusters

I have the following optimization problem: I have a (random and very noisy) objective function $f(A, P)$, where $A$ is a vector of "observable" parameters of the input and $P$ is the parameters that ...
jkff's user avatar
  • 261
7 votes
0 answers
862 views

How are category theory and probability theory related?

How are category theory and probability theory related ? Category theory seems very useful for understanding objects with definite relationships, whereas probability theory (particular Bayesian ...
Richard Southwell's user avatar
7 votes
0 answers
318 views

Absolute sum of partitioned Rademacher variables

$ \newcommand{\E}{\mathop{\mathbb{E}}} $ Hi, this is the first time I post a question here, so I'd be glad to have comments to make it better. So here it goes. The problem I am looking to ...
jsleb333's user avatar
7 votes
0 answers
509 views

Perturbation theory for least squares for very different A, b

Consider the least squares problem $f(x;A,b) = \|Ax-b\|_2^2$ and define $x^*$ the minimizer of $f(x;\hat A,\hat b)$, and $\hat x$ the minimizer of $f(x; A_2, b_2)$. I want to put some bound on $\|...
Y. S.'s user avatar
  • 1,756
6 votes
0 answers
371 views

Distance between point and convex hull in high dimensions

I am trying to develop an intuition for the properties of the convex hull of a set of points in high ($d>20$) dimensions. Consider a set of $n$ data points which are iid distributed according to ...
Seraf Fej's user avatar
  • 232
6 votes
0 answers
960 views

Euclidean distance (cosine) between two random positive unit vectors in high dimensional space

I found out that the largest possible euclidean distance (which is the cosine) between two random positive unit vectors decreases as the dimension of vector increases and approximates 0.71. This was ...
siminsimisim's user avatar
6 votes
0 answers
1k views

Gradient Descent vs Lagrange Multipliers

I'm bit confused between Gradient descent and convex optimization using Lagrange Multipliers. I know that we use Lagrange multipliers when we have an optimization problem with one or more constraints. ...
Kaushal28's user avatar
  • 675
6 votes
2 answers
634 views

Bishop - Pattern Recognition & Machine Learning, Exercise 1.4

I'm working on exercise 1.4 in Bishop's Pattern Recognition & Machine Learning book. This exercise is about probability densities. I've two questions about this exercise. First, I don't understand ...
Naryxus's user avatar
  • 113
5 votes
0 answers
74 views

Conditional Bias Variance Decomposition

The standard bias variance decomposition says that: $$ E |f(X) - Y|^2 = \int_{\mathbb{R}^d} |f(x) - m(x)|^2 \mu(dx) + E|m(X) - Y|^2, $$ where $\mu$ is some distribution over $X$. I am trying to ...
WeakLearner's user avatar
  • 5,889
5 votes
0 answers
512 views

Approximation of positive definite functions by neural networks

Bochner's theorem shows that probability measures $\mu$ are linked with positive definite functions via Fourier transform: $f(k) = \int_{\mathbb{R}^n} e^{-2 \pi i k x} \,d\mu(x)$ Currently, ...
qwerty43's user avatar
  • 341
5 votes
0 answers
132 views

Is there any rigorous treatment of Markov Decision Process?

I am trying to find a mathematically rigorous introduction to MDP. There are tons of resouces online but all of them are ... frankly terrible (and not even properly typesetted). Just picking a few: ...
Olórin's user avatar
  • 5,386
5 votes
0 answers
101 views

Bayes Estimator under $L_{\eta}$

I am wondering if the following loss function is well known and if it is, does it have a standard name: $$ L_{\eta} (\theta, a) = (\theta-a) (\eta - \mathbb{I}_{(-\infty, a)} (\theta) ), \quad \eta \...
WeakLearner's user avatar
  • 5,889
5 votes
0 answers
934 views

Roadmap to Differential Geometry for Machine Learning

Recently within machine learning, there are a lot of works on non-convex optimization and natural gradients methods etc which are based on differential geometry, it gives rise to increased need to ...
Xingdong's user avatar
  • 2,063
5 votes
0 answers
114 views

Why does minimizing $H[f] =\sum^{N}_{i=1}(y_i-f(x_i))^2+\lambda \| Pf \|^2 $ leads to solution of the form $ f(x) =\sum^N_{i=1}c_iG(x; x_i)+p(x)$?

I was reading the following paper of dimensionality reduction (1) and also one on theory of networks for approximations and learning (2) and was trying to understand how the regularization problem ...
Charlie Parker's user avatar
5 votes
0 answers
308 views

VC-Dimension of Balls intersected with half-spaces

In $d$ dimensional Euclidean space, the VC-dimension of both the set of balls and the set of half-spaces is $d+1$. It follows that the VC-dimension of balls intersected with half-spaces is $O(d \log d)...
Travis's user avatar
  • 51
5 votes
1 answer
332 views

Quantitative Analysis of Structure of Gaussian Mixture Model

I am fitting a Gaussian Mixture Model to high-dimensional data (40 dimensions). I have trained the model using EM, learned the parameters and now I want to know quantitatively: What is most ...
Jorche's user avatar
  • 51
4 votes
0 answers
76 views

Book suggestion on "Banach space geometry for machine learning"

Is there any book for a Mathematics student who can learn Machine learning in the aspect of Banach space geometry? Or, one can understand the connection between Geometry of Banach spaces and Machine ...
Tutun's user avatar
  • 129
4 votes
0 answers
100 views

Similarity between mathematical expressions

I'm currently working on a neural network evaluating algebraic expressions. To validate the model we need a metric $D: X \times Y \to \mathbb{R^+}$, where $X$ and $Y$ are the predicted and correct ...
ElMiho's user avatar
  • 56
4 votes
0 answers
76 views

Consequence of Dvoretzky Stochastic Approximation Theorem

I am having some problems trying to apply Dvoretzky Stochastic Approximation Theorem to one Lemma used in a paper I found about the proof of convergence of some reinforcement learning temporal ...
Kareit's user avatar
  • 41
4 votes
0 answers
37 views

Kullback–Leibler divergence between a quasi-arithmetic mean of Normal distributions and a standard Normal distribution

Is there a way to compute the Kullback–Leibler divergence between a quasi arithmetic mean of normal distributions and a standard normal distribution in closed form? $$D_{KL} = D_{KL}\left(f^{-1}\left(...
Jimmy2027's user avatar
  • 161
4 votes
0 answers
264 views

Question about the definition of feature map arising in machine learning

I'm working through the following paper of learning a non-negative function in a reproducing kernel hilbert space setting (RKHS). In particular, section 2.2 on page 3 is a bit confusing to me in terms ...
math's user avatar
  • 4,197
4 votes
0 answers
86 views

What might be good textbooks for the mathematics behind Machine Learning?

I am currently studying Linear Algebra and I have taken Calculus 1, 2, and 3, as well as Differential Equations. Can someone please provide guidance on what material I should study next in order to ...
Jose M Serra's user avatar
  • 2,553
4 votes
0 answers
96 views

Can we sequentially optimize a function that is convex but not jointly convex?

Suppose I have a function $f(x,y)$ where $f$ is convex in $x$ and $y$ but not in $(x,y)$. Suppose that for a fixed $y$, the minimization over $x$ lands on a solution that doesn't depend on $y$, e.g. ...
Y. S.'s user avatar
  • 1,756
4 votes
0 answers
126 views

Why does Logistic Regression need Normalized data

I am trying to implement logistic regression in some problem, but while using normal data gives me some nan results. When I normalize the data I get correct results, so why does Logistic Regression ...
Felippe Trigueiro 's user avatar
4 votes
0 answers
371 views

Looking for density estimator with time complexity $< \mathcal O(n^2)$

I am doing univariate non-parametric density estimation on a dataset $D$, and I want to 1) Train a density estimator on $D$ 2) Compute the estimated density at each point in $D$ These two ...
Tyler Collins's user avatar
4 votes
0 answers
82 views

Advanced Math for Reinfrocement Learning - state space and state sequences (policies)

Reinforcement learning has two important notions and I am interested in advanced math that can investigate those notions: State space - set of states. Apparently, deep structures should exist in this ...
TomR's user avatar
  • 1,255
4 votes
0 answers
169 views

Multivariate Conditional Entropy as a test of correlation between random variables

I use the word columns to mean the data from which a random variable can be estimated. It is a sample of a random variable. I am working with $N$ columns of weakly correlated data. Furthermore, I ...
Ben Sprott's user avatar
  • 1,233
4 votes
0 answers
184 views

James–Stein estimator

Consider a FIR model of the form $y= Ug_0+e$ with $e$ white noise with variance $\sigma^2$. We assume that we have collected N input-output measurements $y$ and $U$. The James–Stein estimator is ...
Betelgeuse's user avatar
4 votes
2 answers
167 views

Estimate $P(A_1|A_2 \cup A_3 \cup A_4...)$, given $P(A_i|A_j)$

This question is related to some undergraduate research on summary generation of documents of which I am a part of. I am trying to estimate $P(A_1|A_2 \cup A_3 \cup A_4...A_k)$, where I know the ...
IamThat's user avatar
  • 1,067
4 votes
0 answers
210 views

Genetic algorithm - find max of minimized subsets

I have a combinatorial optimization problem for which I have a genetic algorithm to approximate the global minima. Given X elements find: min f(X) Now I want to expand the search over all possible ...
Wafram's user avatar
  • 41
4 votes
1 answer
260 views

Is Multiple Integration Over Random Variables NP HARD

In a seminar, in a passing moment a presenter mentioned that multiple integrals over random variables is NP-Hard. This means there is no efficient algorithm to compute the integral. So Theorem: ...
david nadal's user avatar
4 votes
0 answers
281 views

Illustration of Von Neumann's Minimax theorem in games?

The Von Neumann's Minimax theorem gives the conditions that make the max-min inequality an equality. I understand the max-min inequality, basically $\min(\max(f))\ge \max(\min(f))$. The Von Neumann'...
dontloo's user avatar
  • 476
4 votes
0 answers
445 views

Backpropagation in a Convolutional Neural Network

Consider a Convolutional Neural Network with the following architecture: \begin{align} Input---C_1 P_1 --- C_2 P_2 ---Softmax \end{align} Here $C_i$ refers to the $i^{th}$ convolutional layer ...
Shubham Gupta's user avatar
4 votes
0 answers
284 views

Derivation of back-propagation equation $\frac{\partial E(\theta)}{\partial W^k}=x*\delta h^k+\tilde{h}^k*\delta y$ for convolutional autoencoders

I was reading the following paper on convolution stacked auto-encoders and they had the following convolution neural network (for auto-encoders, notice I didn't write the offset term [to avoid ...
Charlie Parker's user avatar
4 votes
0 answers
190 views

Machine Learning and Probability/Stochastics

Main question: What connections are there between machine learning and stochastics (Probability theory, analysis, processes, SDEs)? Background: I've just been accepted into a master's programme for ...
Dahn's user avatar
  • 5,414
4 votes
0 answers
1k views

Is information entropy $H(X)$ a sub modular function?

I was trying to learn more about sub modular functions and wanted to see an example of proving that some function is sub modular. Wikipedia said that Entropy was an example so I decided to try it out ...
Charlie Parker's user avatar
4 votes
0 answers
367 views

Relation between RKHS and space of continuous functions

Consider a Mercer Kernel $K\colon \mathcal{X}\times \mathcal{X}\to \mathbb{R}$, $\mathcal{X}$ being a compact subset of $\mathbb{R}^m$, and its (unique) associated Reproducing Kernel HIlbert Space $...
Ludwig's user avatar
  • 2,229
4 votes
0 answers
843 views

Modern Mathematical Theory for Neural Networks, Cellular Automata, Neuroscience

Is it possible for someone to do research on subjects like neural networks, cellular automata, or neuroscience as an applied mathematician? I have in mind the theoretical development of these fields,...
ask's user avatar
  • 81
4 votes
1 answer
482 views

Relation between factor graph and conditional probability distribution

First, I'm from computer science. I don't know how to say this problem in a mathematical way. So please bear with me. The question Let say I have a factor graph illustrated in the figure. The ...
v4r's user avatar
  • 135
4 votes
0 answers
173 views

Boltzmann machines - motivation for the energy function

I've been studying Boltzmann machines lately and was wondering if anyone could give me a "high-level" explanation or motivation for the energy function used: $$E = -\sum_{i<j} w_{ij} \, s_i \, s_j ...
John Manak's user avatar
4 votes
1 answer
762 views

$i,j,k$ Values of the $\Theta$ Matrix in Neural Networks

SO I'm looking at these two neural networks and walking through how the $ijk$ values of $\Theta$ correspond to the layer, the node number. Either there are redundant values or I'm missing how the ...
carl crott's user avatar
4 votes
0 answers
577 views

Theoretical proof of convergence of sequential weight update procedure (Neural Networks and Machine Learning)

My question is at the bottom. (Most of the descriptive words come from Chris. Bishop's Neural Networks for Pattern Recognition) Let $w$ be the weight vector of the neural network and $E$ the error ...
Chen's user avatar
  • 561
4 votes
2 answers
1k views

How does the classification using the 0-1 loss matrix method work?

In this machine learning lecture the professor says: Suppose $\mathbf{X}\in\Bbb R^p$ and $g\in G$ where $G$ is a discrete space. We have a joint probability distribution $\Pr(\mathbf{X},g)$. ...
user avatar
3 votes
0 answers
40 views

Path Signatures and Picard iterations

Recently, I've started studying path signatures and, currently, I'm reading a standard reference, namely "A Primer on the Signature Method in Machine Learning" by Ilya Chevyrev and Andrey ...
Oscar's user avatar
  • 415
3 votes
0 answers
75 views

When can a neural network simultaneously approximate a function and its derivative?

Let $L^2\equiv L^2(\mathcal X,\mu)$ be the space of square integrable functions over the bounded domain $\mathcal X\subset \mathbb R^d$ with Lebesgue measure $\mu$, equipped with its usual norm and ...
Stratos supports the strike's user avatar
3 votes
0 answers
93 views

Normalizing Flow Penalization

I am looking to fit a normalizing flow, specifically a Masked Autoregressive Flow model. However, this model leads to high variance on lower dimensional, less complex data. I am using a neural network ...
user2793618's user avatar
3 votes
0 answers
148 views

Multiclass Linear Discriminant Analysis

This question is based on the Multiclass Linear Discriminant Analysis (MLDA) describe in Lectures slides by Olga Veksler, which is a generalization of Fisher's Linear Discriminant. My use in MLDA is ...
Triceratops's user avatar
3 votes
0 answers
92 views

Why can't a single layer neural network not solve the AND problem?

I am playing around with simple neural networks using Pytorch and I am confused about something. I think it should be able to solve the AND problem, since it is linearly separable, but after ...
bep's user avatar
  • 31
3 votes
1 answer
188 views

Comparing the Training Costs of Machine Learning Algorithm: A Mathematical Perspective

Recently, I was looking at the optimization functions required in training Kernel Based Methods compared to Neural Networks. 1) Kernel Methods: For instance, I was looking at the optimization in ...
stats_noob's user avatar
  • 1,788

1
2 3 4 5
29