How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

learn more… | top users | synonyms

0
votes
0answers
3 views

effect of number of samples on performance of unsupervised clustering algorithms

I want to apply a k means classification algorithm on a set of data. How does the performance of the clustering algorithm depend on the number of samples available to it? Specifically, instead of ...
0
votes
0answers
22 views

A heuristic explanation of the Curse of Dimensionality

From Principles and Theory for Data Mining and Machine Learning, Clarke et al. (2009): This phrase [the "Curse of Dimensionality"] was first used by Bellman (1961)... The result is that ...
1
vote
1answer
55 views

Implementing gradient descent based on formula

The gradient descent algorithm is given as : repeat { $$\displaystyle \theta_j := \theta_j - \frac{1}{m} \alpha \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)}) x^{(i)}_j $$ } Given these values : <...
0
votes
0answers
7 views

Combining datasets based on their prior distributions in classification

Given $k$ datasets $D_1,...,D_k$, each dataset consists of a a collection of features $X_i$ and the corresponding labels $y_i$. The seperation of datasets is based on a preprocessing of the data. ...
0
votes
0answers
13 views

Problem OOB in random forest

I have a 6 levels of one group and i have to do a random forest classification. My problem is that OOB in test set is too low and cv give me a almost 0 error so i don't understand if this can be a big ...
3
votes
2answers
76 views

What is a good book for math students to learn machine learning in depth?

I am a math master student and have done fundamental math courses like probability theory, measure theory, linear algebra and know a little bit about functional analysis. What is good way for me to ...
0
votes
0answers
26 views

Number of neighbors as a function of dimension

I apologize in advance for perhaps an imprecise formulation of the question. If I have a point in 1D, it has precisely 2 nearest neighbors independent of choices. In 2D, if I allow arbitrary ...
1
vote
1answer
49 views

Mathematical notation for neural network

there are so many "styles" to express neural net in mathematical notation, for example Michael Nielson defines $w_{jk}$ as weight from $k$-th neuron to $j$-th, Andrew Ng defines it otherwise, some ...
0
votes
0answers
26 views

Does this calculus argument involving rank-4 tensors make sense?

Edit: Completely rewritten to be shorter and easier to digest. Background and the Actual Question: I'm trying to derive a gradient formula (back propagation) for a machine learning application. The ...
3
votes
1answer
29 views

Convergence sequence of mean implies convergence in mean / weakly consistence of subsequence of regression function estimates

Let $(X_n)$ be a sequence of positive random variables. Suppose that the limit of expectation of this sequence $\lim_{n\rightarrow\infty}\mathbb{E}[X_n]=0$. This imply that $(X_n)$ converges to zero ...
0
votes
1answer
18 views

Alternatives to Kruskal-Wallis or one way Anova test for small size samples

I have a group of measurements 'grouped by year' and on each year I have only one recorded measure, such as the example below: ...
0
votes
1answer
14 views

KKT condition of linearly inseparable Support Vector Machine (SVM)

In the paper Sequential Minimal Optimization:A Fast Algorithm for Training Support Vector Machines, the optimization problem for linearly inseparable SVM is \begin{align} \min\limits_{\boldsymbol{w},...
0
votes
0answers
25 views

Softmax Derivation Help

I've been reading a paper that derives logistic regression from a few assumptions . Here is the link. If you go to page 5 and look at equation 18 the author claims that this essentially says the ...
0
votes
0answers
14 views

What is KL-Divergence? Why Do I need it? How do I use it?

I am currently studying KL Divergence. But It seems very confusing that I don't maybe understand why do I ever need it and what is that for? As I have been reading stuff about Mutual Information, it ...
-3
votes
2answers
40 views

The derivative of the absolute value |x| [duplicate]

I read about the derivative of the absolute value |x|, but why the absolute value is not differentiable at point zero, and when it becomes 1 or -1 {geometrically}? Thanks
0
votes
0answers
17 views

Solving $I^* = \arg\min_{I'} \left( \|\phi_\ell(I) - \phi_\ell(I')\|_2^2 + R(I') \right)$ with gradient descent

I am trying to create the results from this a paper that is trying to understand the types of features a convolutional neural network is learning to recognize. I don't think understanding ...
-1
votes
1answer
78 views

Does any technical definition of embedding accept a “non-injective” function as opposed to only “injective”?

Embedding is defined to be a one-to-one structure preserving mapping. My question is if the one-to-one condition is really critical. Like if linear mappings from high-dimensional space to low-...
0
votes
0answers
24 views

Roadmap to Differential Geometry for Machine Learning

Recently within machine learning, there are a lot of works on non-convex optimization and natural gradients methods etc which are based on differential geometry, it gives rise to increased need to ...
0
votes
0answers
12 views

Precision-Recall Graph: F1 Score v.s. Break-Even Point

To evaluate two classifiers from the aspects of Precision-Recall, two measures are often used: F1 score and Break Even Point (BEP for short. I failed to find any document about it from wiki, and it is ...
0
votes
0answers
21 views

formulate the nearest-neighbour classifier for a general nonlinear kernel

I have an input vector x and the nearest input vector $x_n$ from the training set. The distance is defined as $||x-x_n||^2$. How can I express it in terms of scalar products and then make use of ...
0
votes
1answer
16 views

Is the EM-algorithm the same thing that variational inference in LDA?

I am new in the probabilistic topic modeling, and I need to understand deeply the LDA process, I understand what want to do the inference process in LDA, and I understand too that there is 2 "types" ...
0
votes
0answers
16 views

Gradient descent rule for a particular matrix in a parse tree RNN

Consider the following structure of a recursive neural network. My input is a parse tree, which is a sentence parsed into a binary tree such that an entry is a leaf if and only if it is a word, else ...
1
vote
2answers
24 views

Support Vector Machines: Hype or Hallelujah? - what is alfa? [closed]

I at the moment trying to understand how SVM works with the help of this paper The paper itself explains things pretty well, but there is an alfa term, which doesn't seem to be documented anywhere? ...
0
votes
1answer
37 views

Undestanding SVM

I am the moment trying to understand how SVM works.. I understand the concept of finding a seperating hyperplane with the highest margin, but i do not understand how it works in mathmatically. Mor ...
2
votes
1answer
47 views

Gaussian process for machine learnig

Here is my question in the equation 2.11 A is N by N matrix, so there is not feasible if N is large the textbook say in the euqation 2.12, we only need to invert size n by n. But I think $K$ is 1 ...
1
vote
0answers
33 views

Size of the vocabulary in Laplace smoothing for a trigram language model

Let's say we have a text document with $N$ unique words making up a vocabulary $V$, $|V| = N$. For a bigram language model with add-one smoothing, we define a conditional probability of any word $w_{i}...
0
votes
0answers
33 views

naive bayes theorem, what fruit is it?

i have a little problem and i want it to solve it using a niave bayes classifier. Lets say that i got a basket full of fruit, and i take 1000 fruits where; 500 of them are bananas 300 of them are ...
0
votes
0answers
58 views

How to Derive Softmax Function

Can someone explain step by step how to to find the derivative of this softmax loss function/equation. \begin{equation} L_i=-log(\frac{e^{f_{y_{i}}}}{\sum_j e^{f_j}}) = -f_{y_i} + log(\sum_j e^{f_j}) ...
1
vote
1answer
38 views

Is this a correct interpretation of maximum likelihood estimation?

Here is an excerpt from Pattern Recognition and Machine Learning by Christopher Bishop: This seems to be not quite right—"the probability of the data set", when the data set is drawn from a ...
0
votes
1answer
44 views

Why are most Lagrange multipliers zero in the SVM solution?

I read everywhere that a non-zero Lagrange multiplier $\lambda_i$ signifies that the corresponding point $x_i$ is a support vector, but I can't see how a support vector and a non-support vector have a ...
1
vote
0answers
23 views

Machine Learning: are there other functions similar to the softmax?

Recall in probability and machine learning softmax is defined as: $\sigma(\mathbf{z})_j = \dfrac{e^{z_j}}{\sum_{k=1}^K e^{z_k}}$ for $j = 1, ..., K.$ where $\sigma: \mathbb{R}^k \to (0,1)$ ...
0
votes
0answers
33 views

SVM optimality criterion in Bottou, Lin (2006)

My question relates to an alternative optimality criterion for an SVM dual solution derived in Bottou, Lin (2006) in pages 8 and 9. Let: $\alpha^* = (\alpha_1^*,\dots,\alpha_n^*)$ be a dual ...
0
votes
0answers
27 views

machine learning octave code gradient descent question

I'm taking Coursera Machine learning course. so who take this courses will able to help this problem. this is the octave code to find the delta for gradient descent. ...
0
votes
0answers
7 views

choosing variable for loss function in regression and classification

I am new in machine learning, so please bear with me. I am trying to gain intuition about the loss functions used in regression and classification. Right now, I am reading this paper. I don't ...
0
votes
1answer
17 views

Geometric interpretation of support vector values in primal space

The Linear Support Vector Machine classification ($y_{k} = -1\ \mathrm{or}\ +1$) with misclassification tolerance loss function in primal weight space looks like this: $$\min\limits_{w,b,\xi} J_{P}(w,...
0
votes
1answer
12 views

SVM / QP result for impossible to satisfy conditions

The theory behind Linear Support Vector Machines with tolerance of misclassifications states that we are trying to minimise in the primal weight space the following function: $$\min\limits_{w,b,\xi} ...
0
votes
0answers
33 views

Query about hyperplane in SVM

I am a beginner in Machine Learning. I was reading through basics of SVM and read this definition: The goal of a support vector machine is to find the optimal separating hyperplane which ...
0
votes
1answer
24 views

Support Vector machine & Support Vector

I had gone through several example of SVM and I see one starts explaining SVM by picking up the support vectors upfront (like this https://www.youtube.com/watch?v=1NxnPkZM9bc). Basically those vectors ...
0
votes
0answers
16 views

Stochastic gradient descent in neural network with logistic activation function

I am trying to derive the update rules for a unit of a neural network. To simplify, let's assume that need to perform a binary classification task on a dataset $\mathbf{X} = \{\mathbf{x}_i\mid i=1,\...
1
vote
1answer
23 views

Valid approach to generate new training data out of some existing training data

Is there any valid approach to generate new training data out of some existing training data. I ask this question only in regard of my learning problem not in a general context. My learning problem ...
0
votes
0answers
15 views

What is the purpose of the 1/2 factor in SVM minimalisation equations?

The objective function for Support Vector Machines is in most sources formulated as: $\min\limits_{w,w_{0}} \frac{1}{2}||w||^2 + C\sum\limits_{i=1}^{N}\xi_{i}$ What is the signifance of the $\frac{1}...
1
vote
2answers
43 views

Least Square Method(LSM) and Partial differential equation

Hello people, I was looking at the machine learning book and try to understand the Least square method using partial differential equation. $$ s = \sum( y_i -...
1
vote
0answers
25 views

What should I be studying if I want to calculate correlation between data sets?

I'm building an app that brings in data from multiple API's (Stripe, Google Analytics, Github). I'd want to be able to analyze the different sets of data against each other if at all possible and draw ...
0
votes
0answers
14 views

Adaboost weights adding up to $1$

In the Adaboost algorithm, I understand that for a given $m$th iteration, the weights all add up to $1$. Based on Patrick Winston's lecture, it seems like this is a constraint. Is there a way to prove ...
0
votes
0answers
24 views

Bound the vc dimension of hypothesis class

Given some set $V$ of size $n$, define the domain $X = V \times V$. In addition, define the hypotheses class $H$ to be all the equivalence relations over $V$ with at most $k$ equivalent classes. I am ...
0
votes
0answers
10 views

Minimizing log-likelihood function

Below is a problem I'm currently working on. I am having trouble seeing how I can obtain the wk and wko values for equation (1). I cannot see how one would solve the negative log-likelihood function ...
4
votes
2answers
51 views

L1 regularized unconstrained optimization problem

I am encountering an unconstrained minimization problem. The problem is of the form $$\min_x \frac{\|x-a\|_2^2}{2}+\lambda\|x\|_1$$ where $x,a \in R^n$ and $x$ is the optimization variable. $\...
0
votes
0answers
32 views

Neural Net Matrix Multiplication

I'm trying to figure out the matrix multiplications for the implementation of a single hidden layer neural net for MNIST digit recognition. Like the following: ...
0
votes
1answer
34 views

Why is there a factor of 1.7159 with the tanh function used in neural network activation?

I was reading about neural networks when I came across the line : Recommended f (x) = 1.7519 tanh (2/3 * x). How do we arrive at these values (we can fix the other once the other is obtained using ...
0
votes
0answers
23 views

Shatter coefficient and VC dimension of a grid in $R^d$

Given $\epsilon>0$, partition the cube $[0, 1]^d$ with square of side length $\epsilon$. The total number of square in the partition is $$ N = \left(\frac{1}{\epsilon}\right)^d. $$ What is the ...