How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

learn more… | top users | synonyms

1
vote
1answer
7 views

Understanding PAC (probably approximately correct) bounds on the realizable case (and finite hypothesis class)

I was trying to understand PAC bounds on the realizable case (i.e. when there is some perfect $h^* \in \mathcal{H}$ and its generalization error is zero). Notation: Training data: $$S_n$$ Training ...
1
vote
1answer
20 views

Is there a distance metric for dot product similarity that preserves the ordering of nearest neighbors?

The dot product and cosine similarity measures on vector space are frequently used in machine learning methods. However, efficient data structures and algorithms often require a metric space distance ...
0
votes
0answers
6 views

Shapley value regression documentation

Can anyone point me to comprehensive description on shapley value regression? I've tried to google it, but i didn't found any book or papers on this regression algorithm, only scraps of disjointed ...
0
votes
0answers
7 views

Computing Object Classification with bayesian statistics

Say I want to know if there is a zebra $\theta$, in an image $x$. According to Bayes statistics applies to image recognition, I should be computing: ...
0
votes
0answers
10 views

Fisher Linear Discriminant Analysis

I was programming an LDA function and wanted to validate my results so I used SPSS but I was unable to generate similar coefficients. the code for weights is weights = covmatr *(groupMeans1 - ...
0
votes
0answers
16 views

Combine SVM kernels

I have a question regarding SVM (Support Vector Machines) and support vectors regression. Is it possible to develope a kernel that combines the linear and the rbf ...
0
votes
0answers
10 views

Is Expectation Propagation (EP) affected by the prior?

I understand EP by reading Minka's thesis: http://research.microsoft.com/en-us/um/people/minka/papers/ep/minka-ep-uai.pdf I'm trying to apply it to solve a Bayesian inference problem. However, I'm ...
2
votes
3answers
51 views

Why can we use entropy to measure the quality of a language model?

I am reading the < Foundations of Statistical Natural Language Processing >. It has the following statement about the relationship between information entropy and language model: ...The ...
0
votes
0answers
5 views

Substitution of generalized perceptron equation with a two classifier architechture.

In http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf it gives the generalised form for the perceprtron loss function in equation (7) and the substitution values for the two energy function terms as ...
3
votes
1answer
31 views

Motivation behind steps in proof of Hoeffding Inequality

The lemma that is proved for proving Hoeffding's inequality is: If $a\leq X\leq b$ and $E[X]=0$, $E[e^{tX}] \leq e^{\frac{t^2(b-a)^2}{8}}$ Here's a link to the proof: ...
-1
votes
0answers
15 views

How to Build a Foresight System? [migrated]

For a research project, I'm asked to find ways to build an economic foresight system. For example, for the production of cheese. We will have data about the market indicators, like price, demand etc. ...
1
vote
0answers
21 views

Name of decision method in which probability of taking an action is exactly past successes / past attempts, while alternative actions normalize

The probability of choosing among options $X_1$, $X_2$, $X_3$, $...X_n$ is initially uniform, i.e. $P(X_j)=1/n$. On choosing $X_j$, either success or failure will occur (with unknown probabilities, ...
0
votes
0answers
17 views

Expectation Maximization (EM)

I was wondering what "hard" EM is ? How is it different from the normal EM and what are the E and M steps?
0
votes
1answer
22 views

Probability Distribution of z/x given x

It may seem a simple question for you, but it's driving me crazy. Given the regression model $z = wx + \epsilon$, where $ \epsilon \sim \mathcal{N} (0, (\sigma x)^{2} $, $ z \sim \mathcal{N}(wx, ...
0
votes
0answers
17 views

Deriving the optimal value for the intercept term in SVM

I was reading andrew ng's machine learning lecture notes on SVM. I came across the following equation (finding the optimal value for the intercept term $b$ in the SVM problem): However, I have no ...
2
votes
0answers
39 views

Simplify huge optimization task (updated)

Shorten and simplified explanation of a problem: We have a sequence of variables from R of length S : $SEQ = [x_1^{(1)} \ x_2^{(2)} \ x_4^{(3)} \ x_2^{(4)} \ x_{12}^{(5)} \ x_6^{(6)} \ x_{15}^{(7)} ...
2
votes
1answer
26 views

Why do polynomial sequences have the coefficient $\sqrt2$ in front of them as in $\phi(x) = [x_1, x_2, \sqrt2x_1x_2, x_1^2, x_2^2]$ [closed]

Why do polynomial sequences have the coefficient $\sqrt2$ in front of them as in $\phi(x) = [x_1, x_2, x_1x_2, x_1^2, x_2^2]$? For example if our original feature space is: $$x = [x_1, x_2]$$ Then ...
1
vote
0answers
16 views

How do you compute the offset parameter for an SVM from the dual solution?

If the plane equation for an SVM is: $$\theta \cdot x^{(i)} + \theta_0$$ How do you compute $\theta_0$ from the dual solution? What I have so far is, for every support vector (SV) $x^{(t)}$ we ...
1
vote
0answers
5 views

Why is the constraint for SVM with offset the way it is? $\sum^{n}_{t=1}\alpha_ty^{(t)} = 0$

When doing the dual formulation of the SVM we get the lagrangian: $$L(\theta, \alpha) = \frac{1}{2}||\theta||^2 + \sum_{t=1}^n{ \alpha_t(1-y^{(t)}(\theta \cdot x^{(t)} + \theta_0) )}$$ and then by ...
1
vote
1answer
25 views

Given a set of data points, how to use gradient descent to find the minimum in the function that passes from those data points?

I have a function with n parameters. I don't know the formula of the function but I can generate as many data points as I want using the function that I have. My question is, how can I find the set of ...
2
votes
0answers
25 views

Normalizing multiple different features from unknown distributions

I'm doing some "exploratory" data analysis over a large set of classes/proteins, with a few hundred different features (I.E. Continuous variables) extracted from the data. The features are calculated ...
1
vote
1answer
12 views

SVM primal formulation, does the constants constraints matter/

When finding the maximum margin separator in the primal form we have the quadratic program: $$min\frac{1}{2}||\theta||^2$$ $$\text{ subject to: } y^{(t)}(\theta \cdot x^{(t)} + \theta_0) \geq 1, \ ...
1
vote
0answers
13 views

Understanding the regularization parameter in polynomial regression

Suppose I have the following $k$ degree polynomial regression model with a data set of size $n$ which includes a $k$-dimensional feature vector $x$ and an outcome denoted $t_i$ for each vector in the ...
5
votes
0answers
70 views

What is the most general formalism for machine learning?

Most of the literature I can find in the field of machine learning is extremely practical, listing many techniques you can use like neural networks, SVMs, random forests, and so on. There are lots of ...
1
vote
0answers
9 views

What is the rationale behind ROC curves?

I am not sure how ROC curves work. I see that the X-Axis is the false positive rate while the Y axis is the true positive rate. 1) I don't understand how for a given statistical learning model, you ...
0
votes
1answer
24 views

Physical Meaning Behind Matrix Factorization

As we all know, Matrix Factorization is an effective method to do rating prediction jobs in recommender systems. Thanks to the work of Yahuda Koren. My question is why MF can do this job ? What's the ...
0
votes
1answer
26 views

Diversity of a Set & Hypercube Vertices related to Statistical Machine Learning

I am reading a paper on Statistical Machine Learning and am having a hard time visualizing/understanding some terminology (my background is EE). The part that I am having a hard time with is this ...
0
votes
0answers
29 views

Deep Neural Network - Lower Dimension Problems

Deep Neural Networks have been used mostly for high dimensional problems such as image processing and speech recognition, as it has shown to be superior to neural networks trained with the ...
1
vote
0answers
26 views

SVM - Variable Input Dimension

Is it possible for a trained support vector machine (SVM) to take an input of a different length (say during the testing phase) than the length used when it was trained? e.g. training data input: ...
1
vote
1answer
35 views

What is the difference between Curve Fitting and Machine Learning?

I know that Machine Learning regression algorithms try to find the function of the data. That is, if we have 1000 data points (x,y), to find a general continuous function that follows the trends of ...
2
votes
1answer
33 views

Given $N$ coins, find a coin with minimal bias based on $N$ samples

General description: Given $N$ coins $Z_1,...,Z_N$ (Bernoulli RVs), where the $i$-th coin has probability $p_i$ for "Head", I'm trying to find $\min\limits _{i\in[N]}p_{i}$. I'm interested in a ...
3
votes
0answers
37 views

Mathematical analysis of e-shop

I'm ukrainian student, studying applied mathematics in Kiev. I have an online store and some statistics data on it's work. Also I've learned a bit about optimization problems and operation reasearch. ...
0
votes
0answers
36 views

SVM - Functional Margin

In Andrew Ng's notes on SVM, the functional margin is defined as: $\hat{\gamma}^{i} = y^{i}(wx+b) $ He also mentions that one property of the functional margin makes it not a very good measure of ...
1
vote
0answers
12 views

The cost of an optimal clustering for a special case of small radius and large distance between clusters, is it independent of the number of clusters?

Say we have N points in 4 clusters. Each cluster has a small radius $\delta$ and a large distance between the clusters say B. What would be the cost of the optimal clustering? Apparently the ...
0
votes
1answer
45 views

Is my implementation of FastICA right?

I have written the following code to perform ICA on the mixed signals that were generated using a random mixing matrix. But the performance of the algorithm is not satisfactory. Did I do something ...
1
vote
0answers
18 views

How do you choose the learning rate such that stochastic gradient descent provably converges?

Recall stochastic gradient descent in the case of regression for n training pointes: $\text{randomly select }\, t \in [1,n],\{\\ \quad \theta^{(k+1)} = \theta^{k} + \eta_k(y^{(t)} - \theta \cdot ...
1
vote
1answer
177 views

How to derive integrals with error function?

How to derive this integral $\int_{-\infty}^{\infty}erf(\lambda x)\mathcal{N}(\mu, \sigma ^2)dx$ and this $\int_{-\infty}^{\infty}(erf(\lambda x)-const)^2\mathcal{N}(\mu, \sigma ^2)dx$ where ...
1
vote
0answers
23 views

VC dimension problem

Let $\mathcal{F}_1$, $\mathcal{F}_2$, and $\mathcal{F}_3$ denote three Boolean classes of functions on a space $\mathcal{X}$ each having VC dimension at most $D$. Define $$ \mathcal{F} = \{ \max(f_1 ...
0
votes
0answers
38 views

Efficient approximation of derivatives of an integral

Suppose $ \phi(z) $ is the probit function (http://en.wikipedia.org/wiki/Probit). And $$ Z = \int \phi(\mathbf{w}^\top \mathbf{x}) \mathcal{N}(\mathbf{w}; \mathbf{\mu}, \mathbf{\Sigma}) d\mathbf{w} ...
5
votes
2answers
139 views

What all maths do I need to know to become good at machine learning.

I am a computer science engineer and I took a couple of maths classes in my first year they were on Fourier series(not transform) partial differential equations, vector calculus, infinite series ...
1
vote
1answer
27 views

Combine linear models of different sets of data.

I'm working on a large data set D that can be partitioned into some disjoint subsets D1, D2, ..., Dn. For each subset Di, I have a linear model Mi that minimizes the residual error for data in Di. ...
1
vote
0answers
22 views

Estimate distance between approximated posterior and true posterior

I'm working on a paper about using graphical models to do some prediction tasks with known observations. Since the model is complicated, finding the maximum a posteriori on the true posterior ...
1
vote
1answer
32 views

Perceptron find weight exercise

I have some difficulties with the following exercise. There are three different diagrams. If possible, find the perceptron-weights $w_0, w_1,$ and $w_2$ for each of them (the decision surface is ...
1
vote
1answer
27 views

How do you express the maximization step of EM algorithm in matrix formulation?

Specifically, I am interested in how the covariance matrix is calculated. In terms of dimensions of factors involved, let's say I am given some data set X of dimension m x d, covariance matrix S of ...
2
votes
2answers
33 views

Regression model when under-estimations costs us more than over-estimations

We have a factory and we are planning how many items produce in 2014. During the learning process we minimize the mean squared error. But, under-estimations costs us more than over-estimations. Let's ...
0
votes
0answers
21 views

VC dimension for Rotatable Rectangles

It can be shown that VC dimension of rotatable rectangles is 7. The problem is I cannot understand how to approach the solution. So far I used bruteforce to solve this kind of problem, I was drawing ...
1
vote
0answers
33 views

K-means convergence to local maxima

I study K-means clustering algorithm. It's known that K-means algorithm converges to the local maximum, the problem is I cannot come up with the examples that shows this. If you know the simple ...
0
votes
0answers
10 views

Loss specific inference in graphical models

As far as I have seen, in graphical models, the inference (for training or parameter estimation) is done via maximizing likelihood. While in many applications people need loss specific optimization of ...
1
vote
0answers
18 views

Continuity of covariance kernels

Let $I$ be a locally compact Hausdorff (LCH) topological space. Let $c : I \times I \to \mathbb R$ be a covariance kernel, that is, a symmetric, nonnegative-definite function. Does it follow that $c$ ...
0
votes
1answer
66 views

Perceptron exercise

I wonder how to find a solution to the following questions: Q: Design a two-input perceptron that implements the boolean function A ∧¬ B? Q: Design a two-layer network of perceptrons that ...