How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

learn more… | top users | synonyms

0
votes
0answers
8 views

naive bayes theorem, what fruit is it?

i have a little problem and i want it to solve it using a niave bayes classifier. Lets say that i got a basket full of fruit, and i take 1000 fruits where; 500 of them are bananas 300 of them are ...
0
votes
0answers
18 views

How to Derive Softmax Function

Can someone explain step by step how to to find the derivative of this softmax loss function/equation. \begin{equation} L_i=-log(\frac{e^{f_{y_{i}}}}{\sum_j e^{f_j}}) = -f_{y_i} + log(\sum_j e^{f_j}) ...
1
vote
1answer
33 views

Is this a correct interpretation of maximum likelihood estimation?

Here is an excerpt from Pattern Recognition and Machine Learning by Christopher Bishop: This seems to be not quite right—"the probability of the data set", when the data set is drawn from a ...
0
votes
1answer
26 views

Why are most Lagrange multipliers zero in the SVM solution?

I read everywhere that a non-zero Lagrange multiplier $\lambda_i$ signifies that the corresponding point $x_i$ is a support vector, but I can't see how a support vector and a non-support vector have a ...
0
votes
0answers
13 views

partial derivative of neural network

I am doing Oxford's deep learning course and i have a question in the lecture slides. In computing the derivative of cost function wrt theta1- (please see the links for image, i couldn't embed image ...
1
vote
0answers
20 views

Machine Learning: are there other functions similar to the softmax?

Recall in probability and machine learning softmax is defined as: $\sigma(\mathbf{z})_j = \dfrac{e^{z_j}}{\sum_{k=1}^K e^{z_k}}$ for $j = 1, ..., K.$ where $\sigma: \mathbb{R}^k \to (0,1)$ ...
0
votes
0answers
31 views

SVM optimality criterion in Bottou, Lin (2006)

My question relates to an alternative optimality criterion for an SVM dual solution derived in Bottou, Lin (2006) in pages 8 and 9. Let: $\alpha^* = (\alpha_1^*,\dots,\alpha_n^*)$ be a dual ...
0
votes
0answers
11 views

machine learning octave code gradient descent question

I'm taking Coursera Machine learning course. so who take this courses will able to help this problem. this is the octave code to find the delta for gradient descent. ...
0
votes
0answers
4 views

choosing variable for loss function in regression and classification

I am new in machine learning, so please bear with me. I am trying to gain intuition about the loss functions used in regression and classification. Right now, I am reading this paper. I don't ...
0
votes
1answer
15 views

Geometric interpretation of support vector values in primal space

The Linear Support Vector Machine classification ($y_{k} = -1\ \mathrm{or}\ +1$) with misclassification tolerance loss function in primal weight space looks like this: $$\min\limits_{w,b,\xi} ...
-1
votes
0answers
24 views

how to minimize huber loss? (SVM related) [closed]

I don't know how to minimize huber loss which is convex and continuous, but how can I minimize that? Can I express it in only one ojective function and obtain its dual problem? My object is to ...
0
votes
1answer
12 views

SVM / QP result for impossible to satisfy conditions

The theory behind Linear Support Vector Machines with tolerance of misclassifications states that we are trying to minimise in the primal weight space the following function: $$\min\limits_{w,b,\xi} ...
0
votes
0answers
27 views

Query about hyperplane in SVM

I am a beginner in Machine Learning. I was reading through basics of SVM and read this definition: The goal of a support vector machine is to find the optimal separating hyperplane which ...
0
votes
1answer
22 views

Support Vector machine & Support Vector

I had gone through several example of SVM and I see one starts explaining SVM by picking up the support vectors upfront (like this https://www.youtube.com/watch?v=1NxnPkZM9bc). Basically those vectors ...
0
votes
0answers
14 views

Stochastic gradient descent in neural network with logistic activation function

I am trying to derive the update rules for a unit of a neural network. To simplify, let's assume that need to perform a binary classification task on a dataset $\mathbf{X} = \{\mathbf{x}_i\mid ...
1
vote
1answer
20 views

Valid approach to generate new training data out of some existing training data

Is there any valid approach to generate new training data out of some existing training data. I ask this question only in regard of my learning problem not in a general context. My learning problem ...
0
votes
0answers
14 views

What is the purpose of the 1/2 factor in SVM minimalisation equations?

The objective function for Support Vector Machines is in most sources formulated as: $\min\limits_{w,w_{0}} \frac{1}{2}||w||^2 + C\sum\limits_{i=1}^{N}\xi_{i}$ What is the signifance of the ...
1
vote
2answers
36 views

Least Square Method(LSM) and Partial differential equation

Hello people, I was looking at the machine learning book and try to understand the Least square method using partial differential equation. $$ s = \sum( y_i ...
1
vote
0answers
24 views

What should I be studying if I want to calculate correlation between data sets?

I'm building an app that brings in data from multiple API's (Stripe, Google Analytics, Github). I'd want to be able to analyze the different sets of data against each other if at all possible and draw ...
0
votes
0answers
14 views

Adaboost weights adding up to $1$

In the Adaboost algorithm, I understand that for a given $m$th iteration, the weights all add up to $1$. Based on Patrick Winston's lecture, it seems like this is a constraint. Is there a way to prove ...
0
votes
0answers
24 views

Bound the vc dimension of hypothesis class

Given some set $V$ of size $n$, define the domain $X = V \times V$. In addition, define the hypotheses class $H$ to be all the equivalence relations over $V$ with at most $k$ equivalent classes. I am ...
0
votes
0answers
10 views

Minimizing log-likelihood function

Below is a problem I'm currently working on. I am having trouble seeing how I can obtain the wk and wko values for equation (1). I cannot see how one would solve the negative log-likelihood function ...
4
votes
2answers
48 views

L1 regularized unconstrained optimization problem

I am encountering an unconstrained minimization problem. The problem is of the form $$\min_x \frac{\|x-a\|_2^2}{2}+\lambda\|x\|_1$$ where $x,a \in R^n$ and $x$ is the optimization variable. ...
-1
votes
0answers
27 views

How to explain polynomial coefficients by minimezed Error function?

We wish to predict ${\bf{t}}$ from an observed $\bf{x}$.We shall fit the data using a polynomial function of the form$$y({\bf{x}},{\bf{w}})=w_0+w_1x+w_2x^2+...+w_Mx^M=\sum_{j=0}^{M}w_jx^j$$ where $M$ ...
0
votes
0answers
19 views

Neural Net Matrix Multiplication

I'm trying to figure out the matrix multiplications for the implementation of a single hidden layer neural net for MNIST digit recognition. Like the following: ...
0
votes
1answer
32 views

Why is there a factor of 1.7159 with the tanh function used in neural network activation?

I was reading about neural networks when I came across the line : Recommended f (x) = 1.7519 tanh (2/3 * x). How do we arrive at these values (we can fix the other once the other is obtained using ...
0
votes
0answers
22 views

Shatter coefficient and VC dimension of a grid in $R^d$

Given $\epsilon>0$, partition the cube $[0, 1]^d$ with square of side length $\epsilon$. The total number of square in the partition is $$ N = \left(\frac{1}{\epsilon}\right)^d. $$ What is the ...
0
votes
0answers
7 views

PCA: MSE of projecting a set of points to a subspace

Let $\{x_1,\dots,x_n\}$ be a dataset of n vectors in $\mathbb{R}^d$ s.t. $\sum_{i=1}^n x_i = 0$. Let $p_1, \dots, p_k$ be a set of k orthonormal unit vectors and V the subspace that they span. Given ...
0
votes
0answers
27 views

understanding a proof for uniform convergence of Fourier features

See part of the proof. In the proof, $\tilde{f} := \tilde{z}(x)'\tilde{z}(y) - k(x,y)$, the vector product is the approximation of a shift-invariant kernek $k(x,y)$. What is not clear to me is the ...
0
votes
0answers
24 views

The proof for the value of growth function for convex sets

I reinforce my education through self-study of Machine Learning. When I come across the problem of Growth Function generated by Convex Set I see only the result and skimming over a proof. I have to ...
2
votes
2answers
33 views

Question about the constraint in Laplacian eigenmaps

When calculating Laplacian Eigenmaps, the original paper mentions about the constraint $$y^TDy=1$$ as "removes an arbitrary scaling factor in the embedding". My understanding is that it prevents $y$ ...
0
votes
1answer
32 views

Inequality of the expectation vs monotone function

I'm reading understanding machine learning and several of the latest lemmas I've studied involved this inequality which I've searched for but found no justification of whatsoever. Could anyone point ...
0
votes
1answer
12 views

Support vector machine, dimension reduction of hyper plane

Question I have this question as an assigned task. In 2D the points are {2,4},{1,1},{5,25}{5,25}. I don't know how to officially calculate the optimal separation hyperplane between them, but from the ...
1
vote
0answers
20 views

Backpropagation in a Convolutional Neural Network

Consider a Convolutional Neural Network with the following architecture: \begin{align} Input---C_1 P_1 --- C_2 P_2 ---Softmax \end{align} Here $C_i$ refers to the $i^{th}$ convolutional layer ...
5
votes
1answer
93 views

Complexity of Gaussian Process algorithms is $\mathcal{O}(n^3)$

It is often quoted that the complexity of Gaussian Process algorithms is $\mathcal{O}(n^3)$ due to the need to invert an $n \times n$ matrix, where $n$ is the number of data points. But as far as I ...
0
votes
0answers
16 views

Gradient of softmax composed with cross-entropy

I am unsure if this is more appropriate for here or CV, but since it is mostly a question about calculus, I figured posting it here would be a reasonable idea. More specifically, I am interested in ...
0
votes
1answer
22 views

Why can't I use sum of probabilities as my loss function for machine learning?

I'd like to understand what is the major reason that we are using loss function of the following form in machine learning (I know it is obtained by taking a logarithm of the likelihood of the ...
0
votes
0answers
7 views

Error derivative of a stochastic binary neurons

I've some perceptrons which perform a stochastic sampling of the input. One of these units calculates it's output by: x = inputVector p(1;x) = 1.0/(1.0+exp(-sum_over_i(x_i*w_i))) Now I want to ...
1
vote
0answers
13 views

Implementing a Least Squares Kernel classifier

I am trying to find the equation I would need to use in order to implement a Least Squares Kernel classifier for a dataset with $N$ samples of feature length $d$. I have the kernel equation $k(x_i, ...
0
votes
0answers
29 views

Vector Euclidean norm upper bound by his coordinates average.

I'm trying to extend the Rademacher complexity and have the following question: For $ (v_1,..,v_m) = V \in {\mathbb{R}}^{m} $ , I will be glad to find an upper abound for the Euclidean norm: $$ ...
0
votes
0answers
14 views

Getting a feel for the Normal-Inverse-Wishart conjugate prior to multivariate normal distribution

I am trying to get a feel for the Normal-Inverse-Wishart conjugate prior, which I have started to use, sparingly, in my work, where I am trying to cluster multivariate normal data. As Wikipedia ...
1
vote
3answers
42 views

Solving vectors such that the dot product = 0

I'm doing some machine learning problems (namely logistic regression), and something I'm trying to do is calculate the decision boundary given a weight vector $\mathbf{w}$. The decision boundary lies ...
0
votes
0answers
10 views

Evidence Approximation

Derivation for Bayesian linear regression Can someone explain how 3.80 is obtained from 3.79? What does completing the square mean in this case.
2
votes
0answers
23 views

Regression With Additive Gaussian Noise

I am studying Machine Learning through Prof. Ng's Stanford lectures. In lecture 3 (lecture notes) he models the output variable $Y$ and input variable $X$ with parameters $\theta$ as $$Y = \theta^T X ...
2
votes
0answers
10 views

Understanding linear separability

Having read the wikipedia article and a similar question on the topic of linear separability, I still lack the understanding of this concept to explain any more than the most rudimentary euclidian ...
2
votes
0answers
23 views

Machine Learning - global minima and convexity in gradient descent

In linear regression we use gradient descent to find the global minimum provided that the cost function is a convex. J = x0 + x1*theta1 + x2*theta2 But if we ...
2
votes
0answers
49 views

What would be a good book to study machine learning from? [closed]

I'm looking for a up to date, readable book that assumes strong mathematical background. Thanks!
3
votes
2answers
46 views

How to show that the signal $x_n = A\cos(\omega n)$ can be fully predicted by a system with two weights $w_1,w_2$

I am trying to solve the following exercise: Show that the signal $x_n = A\cos(\omega n)$ can be fully predicted by a system with two weights $w_1,w_2$ (i.e. $x_n = w_1 x_{n-1} + w_2 x_{n-2}$). ...
0
votes
0answers
15 views

Error propagation: add errors in quadrature, or use a weighted standard deviation?

I have a measurement $x$ with a known uncertainty $\sigma_m$. I have a black box that can take an error-free measurement $x$ and produce a value $y$ with a known uncertainty $\sigma_{b}$ (which is ...
1
vote
2answers
25 views

Intuitively, why does squaring a loss function change optimal values?

In many optimization problems, it is clear that by performing a non-linear operation we change the outcome of any potential optimal values. For example in machine learning: summing over errors ...