How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

learn more… | top users | synonyms

1
vote
1answer
29 views

Is there any difference between statistical learning and machine learning?

Straight to the point, I'm a math student and I have a course this year called Statistical Learning. From the description, the course contains: Large datasets analysis, regression, principal ...
0
votes
0answers
10 views

Gaussian Process with explicit basis functions

I am considering the Gaussian process with explicit basis functions as discussed in the book (section 2.7): http://www.gaussianprocess.org/gpml/chapters/RW2.pdf Has anyone tried to derive formulas ...
1
vote
1answer
17 views

How to calculate a posterior probability with a given Gaussian Mixture Model?

I'm building a GMM-based classifier in speech processing and I'm using GMM as a probabilistic scoring mechanism (therefore I don't intrinsically care about the underlying mixture components). For ...
0
votes
0answers
17 views

Struggling to understand multi-class logistic regression

It is well defined that given a data set of $N$ $i.i.d$ observations $\mathbf{X} = \{\vec{\mathbf{x}}_1, \dots, \vec{\mathbf{x}}_n\}$, along with corresponding target values $\vec{\mathbf{t}} = {t_1, ...
0
votes
0answers
18 views

Testing independency using random projection

I need to demonstrate testing independency of high dimensional two random vectors with projecting them to a very low-dimensional sub-space, say 1,2,3. I.E. I have to find a high dimensional ...
2
votes
0answers
21 views

Why does minimizing $H[f] =\sum^{N}_{i=1}(y_i-f(x_i))^2+\lambda \| Pf \|^2 $ leads to solution of the form $ f(x) =\sum^N_{i=1}c_iG(x; x_i)+p(x)$?

I was reading the following paper of dimensionality reduction (1) and also one on theory of networks for approximations and learning (2) and was trying to understand how the regularization problem ...
0
votes
0answers
9 views

ARD Kernel - explanation

The following text discusses that the ARD kernel is a regular gaussian kernel but one where $\Sigma$ is diagnonal and one where the $\sigma$'s go to infinity. It seems that the $\kappa$(x,x') would ...
1
vote
0answers
10 views

How close is $\operatorname{argmax}_p E[\log(f(p,\alpha)]$ to $\operatorname{argmax}_p \log(E[f(p,\alpha)])$?

Here $\alpha$ is a random variable and the expectation is taken with respect to that variable. I am wondering if it's the same in any case or there's a theorem quantifying how close both things are. ...
0
votes
0answers
24 views

What type of self-adjoint operator does $\hat{P}$ has to be for Green's function to result in a radial exponetial $e^{-\| x-t \|^2}$

I was reading the following paper on hyper basis function (HBF) (similar to radial basis function RBF network) and was trying to understand when is it the case that the network has radial basis ...
0
votes
0answers
24 views

Optimize to Find the Mahalanobis Distance to Minimize the Term

I have an optimization problem defined as following: Assuming we have a data set $ { \left\{ \left( {x}_{i}, {y}_{i} \right) \right\}}_{i = 1}^{N} $ where $ {x}_{i} \in {\mathbb{R}}^{d} $ and $ ...
0
votes
0answers
16 views

How do you compute the weighted sum of data points for learning the centers of a hyper basis function network (HBF)?

I was reading the following paper on hyper basis function (HBF) (similar to radial basis function RBF network) and was trying to figure out how one learns the movable centers of the hyper basis ...
1
vote
0answers
29 views

how to test if Linear Discriminant Analysis (LDA) I implemented works?

I have implemented Linear Discriminant Analysis (LDA) in C by referring various sources. Now, I would like to test the system with a simple configuration. How can I do that? I work on a speech ...
1
vote
1answer
29 views

How do the rows of a change of basis matrix form a basis for expressing columns?

I am reading this article on Principal Component Analysis (PCA) and in section III-B (page 3) it has strange definition I don't understand. In the toy example $\mathbf{X}$ is an $m \times n$ ...
2
votes
0answers
17 views

EM algorithm with constrained equation

I am reading a paper where author uses EM for the following equation to find the parameters $\theta$(and $\beta$) : $$ J=\sum_m \alpha_{m}\sum_i\sum_j w_{mij}\log\sum_k \theta_{ik}\beta_{mjk} $$ ...
0
votes
0answers
15 views

Is total least square solution only valid for isotropic error

Let $\mathbf{y} = \mathbf{Ax}$ represent a system of equation where, $\mathbf{y}\in\mathbb{R}^n, \mathbf{A}\in\mathbb{R}^{n\times m}$. However due to error in sensor, what we observe is the following ...
0
votes
1answer
33 views

Mathematical Intuition behind the tf-idf formula in Statistics

I was reading: https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Definition But I cannot seem to understand exactly why the formula was constructed the way it is. What I do Understand: iDF should at ...
1
vote
0answers
54 views

Derivation of back-propagation equation $\frac{\partial E(\theta)}{\partial W^k}=x*\delta h^k+\tilde{h}^k*\delta y$ for convolutional autoencoders

I was reading the following paper on convolution stacked auto-encoders and they had the following convolution neural network (for auto-encoders, notice I didn't write the offset term [to avoid ...
0
votes
0answers
16 views

Why use an un-squared L2-norm regularizer?

Presumably it would be useful if you want to penalize small values more and large values less, but L1-norm shouldn't work too bad in those cases either and you get the benefit of sparsity. The only ...
3
votes
3answers
75 views

How does kernel work work

all, I have been learning kernel method for a long time. But I am still not very sure how it works. In my opinion, it works as follows: say $f(x) = \sum_i\alpha_ik(x_i, x)$. First we need to decide ...
3
votes
1answer
44 views

What's Being Returned Here?

I'm working my way through this paper, and I'm having a bit of trouble understanding what it's telling me to do. Here's the specific excerpt that's tripping me up: A (finite) one-shot game is a tuple ...
3
votes
0answers
24 views

What does “the activation of a basis” mean?

In the paper Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, Andrew Y. Ng, Self-taught learning: transfer learning from unlabeled data, ICML '07 Proceedings of the 24th international ...
2
votes
1answer
52 views

What does it mean to convolve matrices of finite dimension?

If one is given two matrices $I$ and $K$ what does the notation: $$ I * K $$ mean rigorously/precisely? I do know the definition of convolution: $$ s[i, j] = (I * K)[i, j] = \sum_m \sum_n I[m,n] ...
0
votes
0answers
10 views

Softmax Regression Gradient Derivation

I'm implementing softmax regression and am deriving the max-log-likelihood update for gradient descent by hand first. Coming from the Stanford UFLDL site, they show the gradient of the cost function ...
0
votes
4answers
60 views

What maths courses are needed for Machine Learning

I may sound dump. But I really like to know what maths courses are needed for Machine Learning. I am not computer science graduate but seriously interested in AI, ML, Neural network etc, and I know ...
0
votes
1answer
18 views

Need some help understanding the notation for Online Machine Learning algorithms

I'm reading the Wikipedia article on Online Machine Learning and some of the algorithms mentioned there seem to be missing some context: ...
0
votes
0answers
35 views

What does $\frac{d^k h}{dx^k}$ mean in the context of vectors and regularization in machine learning?

I was watching a machine learning videos from the caltech course CS 156 and they have a slide where they talk about how radial basis functions (RBFs) can be derived from the following variational ...
0
votes
0answers
15 views

Pattern recognition and machine learning - Bishop: Difficulty deriving (2.111) and (2.112)

I'm having a hard time deriving (2.111) and (2.112) in this book (it's on page 110/703). Here is what it says: Finally, we seek an expression for the conditionalp(x|y). Recall that the results for ...
1
vote
1answer
53 views

Studying machine learning

I'm currently studying machine learning using Bishop's book "Pattern Recognition and Machine Learning". The main disadvantage of this book (for me) is a lack of practical applications. Also it seems ...
0
votes
0answers
32 views

What is topological learning?

I am getting this term topological learning in few places for example a reference is below at section 1.1.2: http://virenjain.org/thesis/VirenJainThesis_official.pdf Can anyone point out what ...
1
vote
1answer
33 views

In an ODE dynamic system, is there a convient way or algorithms for estimating the parameters which make the ODE solution satisfing some constraint?

I have construct a ODE dynamic system like this $$molA(t)==sa$$ $$molB'(t)=sb-db\;molB(t)+\frac{kab\;molA(t)\;molB(t)}{molB(t)+Jab}-\frac{kgb\;molG(t)\;molB(t)}{molB(t)+Jgb} $$ $ molC'(t)=sc-dc\ ...
1
vote
1answer
45 views

Problem with understanding the induction when proving Sauer Lemma.

I will replicate the proof here which is from the book "Learning from Data" B(N, k) is the maximum number of dichotomies on N points such that no subset of size k of the N points can be shattered by ...
0
votes
0answers
15 views

PCA when SVD is a skinny SVD

A = m * n matrix. When $m \ge n$, it is easy to see that the V matrix in the full SVD ($A = U*S*V^T$, where U and V are both orthonormal square matrix) and V in a skinny SVD are the same. When $m \lt ...
0
votes
0answers
96 views

Homogeneous polynomials on sphere - need an example that is used in machine learning.

My question is about an example of use of homogeneous polynomials on sphere as a hypothesis space in learning problem. In order to ask a question I need to make a quick introduction: I'm reading an ...
0
votes
0answers
24 views

periodic radial basis function

A have a point cloud ,described in spherical coordinates, which I need to fit with a smooth surface. I'm trying to do this with a bivariate radial basis function network, which operates on a spherical ...
0
votes
0answers
16 views

Chernoff Bound: Prove that $P[u \geq \alpha] \leq (e^{-s\alpha} U(s))^N$.

Let $u_1, ..., u_N$ be random variables, and let $u = \frac{1}{N} \sum_{n=1}^N u_n$. If $U(s) = E_{u_n}(e^{su_n})$ (for any $n$), prove that $P[u \geq \alpha] \leq (e^{-s\alpha} U(s))^N$. $s > 0$ ...
0
votes
0answers
16 views

Having trouble creating my Neural Network inputs

I'm currently working on a neural network that should have $N$ parameters in input. Each parameters can have $M$ different values (discrete values), let's say $\{A,B,C,\dotsc,M\}$. It also has a ...
2
votes
0answers
28 views

How to use Hoeffding Inequality?

I am new to Hoeffding Inequality and can someone kindly explain to me how to use it? I need to solve the following problem. If $\mu = 0.9$, use Hoeffding Inequality to bound the probability that a ...
1
vote
0answers
21 views

How to use 2D Translation and Rotational error to get offset value for new point?

Here I am trying to detect FIDUCIAL points on PCB in real time using camera. After googling for Two days and reading many post and blog. I found that I have to do something called translational error ...
0
votes
0answers
8 views

Is there any way to compute the probability density value for a given power series function, as implied by a Gaussian Process distribution?

So a Gaussian Process defined over the real line is specified by a mean function $\mu(x)$ and a "kernel" function $k(x_1,x_2)$ both defined for all real inputs. The mean function can be anything ...
0
votes
1answer
17 views

Fisher's linear discriminant

Followed by this book I faced with lack of understanding of4.1.4 section. An author obtained $$\mathbf w\propto \mathbf S_W^{-1}(\mathbf m_2 - \mathbf m_1)$$ and suggested to find a threshold $y_0$ ...
1
vote
1answer
23 views

Logistic regression coefficients problem

I'm using logistic regression model to do a multi-class classification (4 classes). I want to look at the logistic regression coefficients to see the importance of different features. I got model ...
2
votes
0answers
23 views

VC-Dimension of Balls intersected with half-spaces

In $d$ dimensional Euclidean space, the VC-dimension of both the set of balls and the set of half-spaces is $d+1$. It follows that the VC-dimension of balls intersected with half-spaces is $O(d \log ...
0
votes
0answers
26 views

Hölder's inequality/Cauchy-Schwarz for Bregman Divergence?

Consider the Bregman divergence. $$ D_F(p, q) = F(p)-F(q)-\langle \nabla F(q), p-q\rangle. $$ And its dual norm: $D_{F*}(p, q) $ where $ F^*(y) = \arg\min_x \left\{ \langle x, y\rangle - F(x) ...
2
votes
0answers
38 views

Hoeffding’s inequality extension

In Hoeffding’s inequality we assume that the random variables $X_i$ ,$i=1,..,n$ are i.i.d. and bounded . Is there any extension to Hoeffding’s inequality for the case that $X_i$ are identically ...
0
votes
0answers
16 views

How to solve this optimization problem (Interpolation on trigram, bigram, and unigram for language model)?

I am a newbie in optimization and learn about the language model in NLP. I am studying the basic interpolation method to estimate the probability of the current word given the last 2 words, $P(w_i | ...
0
votes
0answers
23 views

Size of remaining search space for Vehicle Routing Problem given a partial solution

The vehicle routing problem is a NP-hard problem that, in its most basic form, involves scheduling routes for v vehicles that have to make n deliveries in total. So a solution (schedule) has the form ...
0
votes
0answers
23 views

Maximizing Autoencoder Hidden Unit Function

Given \begin{align} a = f\left(\sum_{j=1}^{100} W_j x_j \right). \end{align} where $f$ is the sigmoid function, $W$ and $x$ are $100 \times 1$ matrices with the constrain \begin{align} ||x||^2 = ...
6
votes
1answer
679 views

How to quantify the differencen between 2/4 and 20/40?

Assume I have two methods to do prediction. The first method makes 4 predictions and 2 out of 4 are correct. The second method makes 40 predictions and 20 out of 40 are correct. The prediction ...
1
vote
1answer
25 views

Kernel Principal Component Analysis (PCA)

I learn kernel PCA from wikipedia. In this article, the eigen equation is \begin{equation} N \lambda \vec{\alpha} = \boldsymbol{K} \vec{\alpha} \end{equation} where $\lambda$ is the eigen value, ...
1
vote
1answer
29 views

I would like to know how to do log transformation of hyperparameters in Gaussian Process Classification.

I am using Gaussian Process classification and I want to do log transform of the hyperparameters so that they are all positive. From this www.lce.hut.fi/research/mm/gpstuff/GPstuffDoc.pdf document, I ...