How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

learn more… | top users | synonyms

-2
votes
2answers
29 views

The derivative of the absolute value |x|

I read about the derivative of the absolute value |x|, but why the absolute value is not differentiable at point zero, and when it becomes 1 or -1 {geometrically}? Thanks
0
votes
0answers
12 views

Solving $I^* = \arg\min_{I'} \left( \|\phi_\ell(I) - \phi_\ell(I')\|_2^2 + R(I') \right)$ with gradient descent

I am trying to create the results from this a paper that is trying to understand the types of features a convolutional neural network is learning to recognize. I don't think understanding ...
0
votes
0answers
21 views

Roadmap to Differential Geometry for Machine Learning

Recently within machine learning, there are a lot of works on non-convex optimization and natural gradients methods etc which are based on differential geometry, it gives rise to increased need to ...
-3
votes
0answers
15 views

how to prove that a gaussian kernel function is a kernel function?

How to show that the gaussian kernel $$K(x,y)=\exp\left(\frac{||x-y||^2}{2}\right)$$ is a kernel function? (meaning it is a Gram matrix of some transformation of $x$ and $y$) I know that to prove ...
0
votes
0answers
16 views

The importance of lambda in a regularization function, with respect to the hypothesis. [migrated]

I'm working through some parts of Russel & Norvig's Artificial Intelligence, and this is the cost function they give: Cost(h) = EmpericalLoss(h) + λComplexity(h) How does choosing a value for ...
0
votes
0answers
7 views

Precision-Recall Graph: F1 Score v.s. Break-Even Point

To evaluate two classifiers from the aspects of Precision-Recall, two measures are often used: F1 score and Break Even Point (BEP for short. I failed to find any document about it from wiki, and it is ...
0
votes
0answers
21 views

formulate the nearest-neighbour classifier for a general nonlinear kernel

I have an input vector x and the nearest input vector $x_n$ from the training set. The distance is defined as $||x-x_n||^2$. How can I express it in terms of scalar products and then make use of ...
0
votes
1answer
15 views

Is the EM-algorithm the same thing that variational inference in LDA?

I am new in the probabilistic topic modeling, and I need to understand deeply the LDA process, I understand what want to do the inference process in LDA, and I understand too that there is 2 "types" ...
0
votes
0answers
16 views

Gradient descent rule for a particular matrix in a parse tree RNN

Consider the following structure of a recursive neural network. My input is a parse tree, which is a sentence parsed into a binary tree such that an entry is a leaf if and only if it is a word, else ...
1
vote
2answers
24 views

Support Vector Machines: Hype or Hallelujah? - what is alfa? [closed]

I at the moment trying to understand how SVM works with the help of this paper The paper itself explains things pretty well, but there is an alfa term, which doesn't seem to be documented anywhere? ...
0
votes
1answer
37 views

Undestanding SVM

I am the moment trying to understand how SVM works.. I understand the concept of finding a seperating hyperplane with the highest margin, but i do not understand how it works in mathmatically. Mor ...
2
votes
1answer
47 views

Gaussian process for machine learnig

Here is my question in the equation 2.11 A is N by N matrix, so there is not feasible if N is large the textbook say in the euqation 2.12, we only need to invert size n by n. But I think $K$ is 1 ...
1
vote
0answers
33 views

Size of the vocabulary in Laplace smoothing for a trigram language model

Let's say we have a text document with $N$ unique words making up a vocabulary $V$, $|V| = N$. For a bigram language model with add-one smoothing, we define a conditional probability of any word $w_{i}...
0
votes
0answers
30 views

naive bayes theorem, what fruit is it?

i have a little problem and i want it to solve it using a niave bayes classifier. Lets say that i got a basket full of fruit, and i take 1000 fruits where; 500 of them are bananas 300 of them are ...
0
votes
0answers
46 views

How to Derive Softmax Function

Can someone explain step by step how to to find the derivative of this softmax loss function/equation. \begin{equation} L_i=-log(\frac{e^{f_{y_{i}}}}{\sum_j e^{f_j}}) = -f_{y_i} + log(\sum_j e^{f_j}) ...
1
vote
1answer
35 views

Is this a correct interpretation of maximum likelihood estimation?

Here is an excerpt from Pattern Recognition and Machine Learning by Christopher Bishop: This seems to be not quite right—"the probability of the data set", when the data set is drawn from a ...
0
votes
1answer
43 views

Why are most Lagrange multipliers zero in the SVM solution?

I read everywhere that a non-zero Lagrange multiplier $\lambda_i$ signifies that the corresponding point $x_i$ is a support vector, but I can't see how a support vector and a non-support vector have a ...
1
vote
0answers
23 views

Machine Learning: are there other functions similar to the softmax?

Recall in probability and machine learning softmax is defined as: $\sigma(\mathbf{z})_j = \dfrac{e^{z_j}}{\sum_{k=1}^K e^{z_k}}$ for $j = 1, ..., K.$ where $\sigma: \mathbb{R}^k \to (0,1)$ ...
0
votes
0answers
33 views

SVM optimality criterion in Bottou, Lin (2006)

My question relates to an alternative optimality criterion for an SVM dual solution derived in Bottou, Lin (2006) in pages 8 and 9. Let: $\alpha^* = (\alpha_1^*,\dots,\alpha_n^*)$ be a dual ...
0
votes
0answers
18 views

machine learning octave code gradient descent question

I'm taking Coursera Machine learning course. so who take this courses will able to help this problem. this is the octave code to find the delta for gradient descent. ...
0
votes
0answers
6 views

choosing variable for loss function in regression and classification

I am new in machine learning, so please bear with me. I am trying to gain intuition about the loss functions used in regression and classification. Right now, I am reading this paper. I don't ...
0
votes
1answer
17 views

Geometric interpretation of support vector values in primal space

The Linear Support Vector Machine classification ($y_{k} = -1\ \mathrm{or}\ +1$) with misclassification tolerance loss function in primal weight space looks like this: $$\min\limits_{w,b,\xi} J_{P}(w,...
0
votes
1answer
12 views

SVM / QP result for impossible to satisfy conditions

The theory behind Linear Support Vector Machines with tolerance of misclassifications states that we are trying to minimise in the primal weight space the following function: $$\min\limits_{w,b,\xi} ...
0
votes
0answers
31 views

Query about hyperplane in SVM

I am a beginner in Machine Learning. I was reading through basics of SVM and read this definition: The goal of a support vector machine is to find the optimal separating hyperplane which ...
0
votes
1answer
24 views

Support Vector machine & Support Vector

I had gone through several example of SVM and I see one starts explaining SVM by picking up the support vectors upfront (like this https://www.youtube.com/watch?v=1NxnPkZM9bc). Basically those vectors ...
0
votes
0answers
15 views

Stochastic gradient descent in neural network with logistic activation function

I am trying to derive the update rules for a unit of a neural network. To simplify, let's assume that need to perform a binary classification task on a dataset $\mathbf{X} = \{\mathbf{x}_i\mid i=1,\...
1
vote
1answer
21 views

Valid approach to generate new training data out of some existing training data

Is there any valid approach to generate new training data out of some existing training data. I ask this question only in regard of my learning problem not in a general context. My learning problem ...
0
votes
0answers
15 views

What is the purpose of the 1/2 factor in SVM minimalisation equations?

The objective function for Support Vector Machines is in most sources formulated as: $\min\limits_{w,w_{0}} \frac{1}{2}||w||^2 + C\sum\limits_{i=1}^{N}\xi_{i}$ What is the signifance of the $\frac{1}...
1
vote
2answers
41 views

Least Square Method(LSM) and Partial differential equation

Hello people, I was looking at the machine learning book and try to understand the Least square method using partial differential equation. $$ s = \sum( y_i -...
1
vote
0answers
24 views

What should I be studying if I want to calculate correlation between data sets?

I'm building an app that brings in data from multiple API's (Stripe, Google Analytics, Github). I'd want to be able to analyze the different sets of data against each other if at all possible and draw ...
0
votes
0answers
14 views

Adaboost weights adding up to $1$

In the Adaboost algorithm, I understand that for a given $m$th iteration, the weights all add up to $1$. Based on Patrick Winston's lecture, it seems like this is a constraint. Is there a way to prove ...
0
votes
0answers
24 views

Bound the vc dimension of hypothesis class

Given some set $V$ of size $n$, define the domain $X = V \times V$. In addition, define the hypotheses class $H$ to be all the equivalence relations over $V$ with at most $k$ equivalent classes. I am ...
0
votes
0answers
10 views

Minimizing log-likelihood function

Below is a problem I'm currently working on. I am having trouble seeing how I can obtain the wk and wko values for equation (1). I cannot see how one would solve the negative log-likelihood function ...
4
votes
2answers
48 views

L1 regularized unconstrained optimization problem

I am encountering an unconstrained minimization problem. The problem is of the form $$\min_x \frac{\|x-a\|_2^2}{2}+\lambda\|x\|_1$$ where $x,a \in R^n$ and $x$ is the optimization variable. $\...
0
votes
0answers
25 views

Neural Net Matrix Multiplication

I'm trying to figure out the matrix multiplications for the implementation of a single hidden layer neural net for MNIST digit recognition. Like the following: ...
0
votes
1answer
33 views

Why is there a factor of 1.7159 with the tanh function used in neural network activation?

I was reading about neural networks when I came across the line : Recommended f (x) = 1.7519 tanh (2/3 * x). How do we arrive at these values (we can fix the other once the other is obtained using ...
0
votes
0answers
23 views

Shatter coefficient and VC dimension of a grid in $R^d$

Given $\epsilon>0$, partition the cube $[0, 1]^d$ with square of side length $\epsilon$. The total number of square in the partition is $$ N = \left(\frac{1}{\epsilon}\right)^d. $$ What is the ...
0
votes
0answers
10 views

PCA: MSE of projecting a set of points to a subspace

Let $\{x_1,\dots,x_n\}$ be a dataset of n vectors in $\mathbb{R}^d$ s.t. $\sum_{i=1}^n x_i = 0$. Let $p_1, \dots, p_k$ be a set of k orthonormal unit vectors and V the subspace that they span. Given ...
0
votes
0answers
29 views

understanding a proof for uniform convergence of Fourier features

See part of the proof. In the proof, $\tilde{f} := \tilde{z}(x)'\tilde{z}(y) - k(x,y)$, the vector product is the approximation of a shift-invariant kernek $k(x,y)$. What is not clear to me is the ...
0
votes
0answers
24 views

The proof for the value of growth function for convex sets

I reinforce my education through self-study of Machine Learning. When I come across the problem of Growth Function generated by Convex Set I see only the result and skimming over a proof. I have to ...
2
votes
2answers
39 views

Question about the constraint in Laplacian eigenmaps

When calculating Laplacian Eigenmaps, the original paper mentions about the constraint $$y^TDy=1$$ as "removes an arbitrary scaling factor in the embedding". My understanding is that it prevents $y$ ...
0
votes
1answer
34 views

Inequality of the expectation vs monotone function

I'm reading understanding machine learning and several of the latest lemmas I've studied involved this inequality which I've searched for but found no justification of whatsoever. Could anyone point ...
0
votes
1answer
12 views

Support vector machine, dimension reduction of hyper plane

Question I have this question as an assigned task. In 2D the points are {2,4},{1,1},{5,25}{5,25}. I don't know how to officially calculate the optimal separation hyperplane between them, but from the ...
1
vote
0answers
28 views

Backpropagation in a Convolutional Neural Network

Consider a Convolutional Neural Network with the following architecture: \begin{align} Input---C_1 P_1 --- C_2 P_2 ---Softmax \end{align} Here $C_i$ refers to the $i^{th}$ convolutional layer ...
5
votes
1answer
95 views

Complexity of Gaussian Process algorithms is $\mathcal{O}(n^3)$

It is often quoted that the complexity of Gaussian Process algorithms is $\mathcal{O}(n^3)$ due to the need to invert an $n \times n$ matrix, where $n$ is the number of data points. But as far as I ...
0
votes
0answers
18 views

Gradient of softmax composed with cross-entropy

I am unsure if this is more appropriate for here or CV, but since it is mostly a question about calculus, I figured posting it here would be a reasonable idea. More specifically, I am interested in ...
0
votes
1answer
23 views

Why can't I use sum of probabilities as my loss function for machine learning?

I'd like to understand what is the major reason that we are using loss function of the following form in machine learning (I know it is obtained by taking a logarithm of the likelihood of the ...
0
votes
0answers
7 views

Error derivative of a stochastic binary neurons

I've some perceptrons which perform a stochastic sampling of the input. One of these units calculates it's output by: x = inputVector p(1;x) = 1.0/(1.0+exp(-sum_over_i(x_i*w_i))) Now I want to ...
1
vote
0answers
13 views

Implementing a Least Squares Kernel classifier

I am trying to find the equation I would need to use in order to implement a Least Squares Kernel classifier for a dataset with $N$ samples of feature length $d$. I have the kernel equation $k(x_i, ...
0
votes
0answers
30 views

Vector Euclidean norm upper bound by his coordinates average.

I'm trying to extend the Rademacher complexity and have the following question: For $ (v_1,..,v_m) = V \in {\mathbb{R}}^{m} $ , I will be glad to find an upper abound for the Euclidean norm: $$ ||V|...