How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

learn more… | top users | synonyms

0
votes
0answers
3 views

Non convex objective in SVM

In the formulation of svm.. The line underline says the norm of the vector w is a non convex constraint.. But how is this so.. Isn't norm a convex function.. Also aren't the other objectives ...
1
vote
1answer
15 views

Proof of the PAC generalization error bound using VC dimension

There is a theorem in PAC ("Probably Approximately Correct" model, in computational learning theory) that reads as follows: To guarantee that any hypothesis that perfectly fits the training data ...
0
votes
0answers
14 views

Efficiency of quasiconvex optimization

Summary: Could we minimize quasiconvex objectives in polynomial time? Whenever an objective function of an optimization problem can be formed as a convex function, this is considered as victory. ...
0
votes
0answers
15 views

convex optimization with multiple nonsmooth terms

Is there a general algorithm for solving $$ \min f(x) + g(x) + h(x) $$ where all three functions are convex and proximable, $f(x)$ is smooth, and $g(x)$ and $h(x)$ are both nonsmooth? Note that if ...
0
votes
0answers
11 views

How many degrees of freedom exist in an agglomerative hierarchical clustering?

The computational complexity of generating an agglomerative hierarchical clustering from n vectors is $O(n^2)$ (calculating the pairwise distance matrix) dendrogram example However, the total number ...
0
votes
1answer
10 views

Softmax Regression Derivative

This website, http://deeplearning.stanford.edu/wiki/index.php/Softmax_Regression, claims the derivative of a multinomial regression: $$ J(\theta) = -\frac{1}{m}\sum_{i=1}^m \sum_{j=1}^k 1\{y^i =j\} ...
0
votes
1answer
11 views

Why is $\frac{1}{n}\sum_{i=1}^{n}(\langle\vec{x}_i,\vec{w}\rangle-y_i)^2 = \frac{1}{n}(X\vec{w}-\vec{y}_i)^T (X\vec{w}-\vec{y}_i)$?

I am reading about Ridge Regression in Machine Learning (in particular, the calculation of the empirical risk w.r.t. the square loss function) and do not understand the following step: ...
0
votes
0answers
20 views

What is the Bernoulli class conditional distribution?

What is the Bernoulli class conditional distribution? I am trying to implement a procedure for computing a naive Bayes classifier for binary features with a Bernoulli class conditional distribution. ...
2
votes
2answers
31 views

chain rule conditional entropy

I have to prove the chain-rule for conditional entropy. I kept getting stuck on one step, so I looked up a proof and found this: \begin{align}H(Y\mid X)&= \sum_{x\in\mathcal X, y\in\mathcal ...
0
votes
0answers
8 views

trie based computation of the string spectrum kernel [closed]

I have found a trie based computation algorithm for computing the string spectrum kernel in the book "kernel methods for pattern Analysis" i still dont understand the algorithm, can anyone help me. ...
3
votes
1answer
63 views

Interpolation and mapping between scattered vectors in two unequally dimensioned spaces

Imagine two spaces: An ‘input’ space with dimension $m$. An ‘output’ space with dimension $n$. $m \geq n$ There are points in each of these spaces defined such that some characteristic is ...
0
votes
1answer
27 views

An explanation of how this solution is derived

I am having difficulty understanding the solution to this problem. Since the solution is in the form of Bayes theorem I expected something along the lines that looked similar to Bayes theorem. ...
0
votes
0answers
15 views

Use of activation derivative in back propagation algorithm

I'm a little confused how the activation derivative in back propagation work. Firstly, when I remove the activation derivative from the back propagation algorithm and replace it with a constant the ...
0
votes
0answers
20 views

Nando de Freitas' Machine Learning Homework 2 Questions 1 & 2 Solutions

I've been following Nando de Freitas' Machine Learning course from UBC. While I have been enjoying the course I thought it would be good to see if I could do the homework along with it. So I'm on ...
0
votes
0answers
4 views

Difference between Nearest Neighbour and Nearest Centroid

I'm trying to understand the difference between Nearest neighbour classifiers and Nearest centroid classifier. Using the nearest neighbour, one selects a data point ...
0
votes
0answers
6 views

How to use Support Vector Machines with Mixed data?

I have a dataset regarding student records with a mix of continuous, discrete & categorical data - the categorical data takes both nominal and ordinal forms. Ex: Continuous - GPA Ex: Discrete - ...
1
vote
0answers
15 views

Outer product approximation of Hessian for least squares

On p251 of Bishop's machine learning book, the Hessian for least squares is derived (as a preliminary step to the outer product approximation): $ E = \frac{1}{2} \sum_{n=1}^{N} (y_n - t_n)^2$ $H = ...
0
votes
0answers
19 views

Lower bound of Nearest Neighbour Rule

It is stated often as a matter of fact that the lower bound for Nearest Neighbour rule is the Baye's rate. However when I tried to mathematically prove it,I hit a dead end. For reference : Error for ...
1
vote
0answers
12 views

Supported Vector Machine constraint condition.

I think I can follow the following statement, but I can't reach the conclusion "subject to $\sum_{i=1}^n c_i y_0 = 0$". Could someone give me a pointer from which to reach this constraint? Thank you! ...
1
vote
1answer
18 views

Bernoulli Naive Bayes Classification

I am having trouble understanding the following text regarding Bernoulli Naive Bayes. Specifically, the author mentions that $i$ is a feature. However, what is the difference between $x_i$ and $i$? ...
0
votes
1answer
22 views

Gradient of cost function

I have tried to calculate the gradient of the LMS cost function as follows but have a problem. $$J(\theta) = \frac12(y - X'\theta)^2$$ where $y$ is a scalar, theta and $X$ is a $n$ dimensional vector ...
1
vote
0answers
41 views

Lower bound on dimension for nearest neighbor classifier to fail at k=1 and pass at k=3

What is the minimum dimensionality of a dataset of a finite number of points where 1-NN has an accuracy of 0% but 3-NN has an accuracy of 100%. This is certainly possible in 3 dimensions and my ...
0
votes
0answers
32 views

Two different ways to compute PCA?

I am working on the PCA. On the internet I found two different ways to compute it - but they produce different results. First there is this solution: http://stanford.io/2060AxA . They are taking the ...
2
votes
0answers
18 views

Comparing two textbooks for machine learning

I am a Ph.D student in Electrical Engineering. I am going to study the field of machine learning and I found some textbooks to study this field. 1) Probabilistic Graphical Models: Principles and ...
1
vote
0answers
114 views

What does this probability notation denote: $P( Y=y | X =x)$?

What does this probability notation denote: $P( Y=y | X =x)$? I came across this while looking at my notes for optimal prediction rule. Optimal prediction rule: for each $x ∈ X$ , chose $y ...
0
votes
0answers
8 views

When does the UCB algorithm find best arm in Multi-armed Bandits problem?

Some article or paper said "UCB(or a derivative algorithm) can find the best arm among multiple arms in Multi-armed Bandits problem". But I don't remember which one said so, and even worse, I'm not ...
0
votes
0answers
13 views

Equivalence between Label propagation and Likelihood estimation over a Markov Random Field

On page 13/14 of label propagation the near equivalence is set up between what they call 33/34 (reproduced below). This near equivalence is not obvious to me. $$P_{F'}(Y) = \frac{1}{Z} \exp \left( ...
1
vote
2answers
80 views

Neural network cost function - why squared error?

Question: Why is the squared error most often used for training neural networks? Context: Neural networks are trained by adjusting the link weights. The key factor that informs these adjustments is ...
0
votes
0answers
8 views

How to evaluate a point in a weighted Mixture of Gaussians model?

Example: I have a MoG comprised of 2 1-d gaussians. The first gaussian has a weight of 0.8 the other 0.2. I have a sample point which I can easily evaluate on each individual gaussian. The ...
1
vote
0answers
22 views

Expectation Maximization - question about the 2 different formulations of E-Step

I saw the following 2 formulations for the E-Step: $Q(θ,θ_t )=E_{θ_t} ( log(p_θ (X,Z))│X=x)$ $Q(θ,θ_t )= E_{Z|X,θ_t } [log(Pr_θ(X=x,Z))]$ I can't understand why they are equal.
0
votes
1answer
25 views

Solving SVM classifier with two weight vectors

I am trying to implement a paper that basically proposes the following way to train two classifiers on some data with two types of labels. I do not know how to tweak existing solvers for SVM to do the ...
0
votes
1answer
37 views

Meaning of “linear in the xi variables”

I'm afraid my maths is so feeble I was not sure how to entitle the question. Essentially I was hoping to ask what the below passage means in the context of the passage that follows? (If that is ...
0
votes
0answers
14 views

Variational inference on a Normal distribution: is my choice of priors passable?

I am trying to understand the basics of Variational Inference. In order to do so I designed a very simple problem: using the free-form mean field method to approximate the posteriori distribution of ...
0
votes
0answers
10 views

Efficient exploration of the “least important” nodes in a large graph

We call Markov Chain Crawler (MCC) a graph learner that is given query access to a Markov Chain Teacher (MCT) which itself is given a specific Markov Chain. At the beginning, the MCC is given some ...
0
votes
1answer
15 views

Does the Markov Blanket of a node include the node itself?

The definition states the Markov Blanket includes the parents of the children of the node, so does this include the node itself too?
0
votes
0answers
18 views

Neural network for regression

The way I understand regression for neural networks is weights being added to each x-input from the dataset. I want something slightly different. I want weights ...
4
votes
1answer
220 views

What is an example of a SVM kernel, where one implicitly uses an infinity-dimensional space?

Reading the Wikipedia article about SVMs, I noticed More formally, a support vector machine constructs a hyperplane or set of hyperplanes in a high- or infinite-dimensional space, which can be ...
0
votes
0answers
46 views

Gradient Boosting Loss Function Derivation

http://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf XGBoost, a popular machine learning library, refers to the above slides in its documentation. I am working through the derivation of the ...
0
votes
0answers
48 views

What are some methods for prototyping polynomials

I work for an eCommerce company that sells office supplies, and each day, the number of orders we take follows a specific shape/curve (we collect order counts at one minute intervals, so we end up ...
0
votes
0answers
15 views

Where do I need to use regularization parameter lambda for better results?

In polynomial curve fitting problem as below, if $y(x,w)$ is the output when $x$ is the input vector, $w$ are the coefficients and and $M$ is the order of polynomial.. And if $t_n$ is the target ...
0
votes
0answers
45 views

Kernel-induced metric

Given a kernel k on input space X defining RKHS (Reproducing kernel Hilbert space) H. Let Φ : X → H denote the corresponding feature map (think of Φ(x) = k(x, .)). Let x, z ∈ X . How can I show that ...
0
votes
0answers
8 views

Bayesian Networks: simple example when to use discrete network and when to use linear Gaussian network

So I am not sure when to use which. Is there a simple example that a non maths pro would understand when to use which? I use libpgm and the pgmlearner provides different functions to train on data. I ...
1
vote
0answers
18 views

How does Restricted Boltzmann Machine (RBM) try to model the distribution of data?

An RBM is a non-directed graphical model that defines the distribution over some input vector X. I know it's going to model the distribution of that those vectors in my training data X using a layer ...
2
votes
0answers
37 views

Does linear regression form a subspace?

The author writes Given a vector of inputs $X^T = (X_1, \dots ,X_p)$, we can predict an output $Y$ via $$ \hat{Y} = \beta_0 + \sum_{j = 1}^p X_j \beta_j$$ He goes on to note that if we include a 1 in ...
1
vote
1answer
48 views

Logistic regression: Prove that the cost function is convex

I'm reading this You can do a find on "convex" to see the part that relates to my question. Background: $h_\theta(X) = sigmoid(\theta^T X)$ --- hypothesis/prediction function $y \in \{0,1\}$ ...
0
votes
0answers
16 views

How to decide a binary hidden variable's state in RBM using CD-Algorithm or Gibbs Sampling?

I'm a 3rd-grade college student and recently I'm reading tutorial materials about Restricted Boltzmann Machines for better understanding the paper Hilton published in ICML 2007. I'm little confused ...
0
votes
0answers
38 views

Is it legal to use gradient descend method in neural networks?

"The usual" neural network for me is system which was cascading via usage of functions from this class: Linear functions Heaviside step and it's approximations I have some troubles with ...
2
votes
0answers
34 views

What temperature of Softmax layer should I use during neural network training?

I've written GRU (gated recurrent unit) implementation in C#, it works fine. But my Softmax layer has no temperature parameter (T=1). I want to implement "softmax with temperature": $$ P_{i} = ...
0
votes
0answers
7 views

The Elements of Statistical Learning: How does this nature cubic spline have K basis functions with K knots with the given solution? See Details.

I'm referring to this found in chap 5: Picture from ELS If K=2 (as in Sec 5.2), then we have N1, N2, N3 and N4 basis functions. So 4 basis functions and 2 knots. I know that a natural cubic spline ...
0
votes
0answers
7 views

Can I view the problem of principal component analysis as an problem of rank minimization?

Can I view the PCA problem as a following optimization problem? $$\min \text{rank}(X)$$ $$s.t. \quad ||M-L||_2 \le \epsilon$$ where M is the observation matrix and L is the low-rank principle ...