How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

learn more… | top users | synonyms

1
vote
0answers
22 views

What should I be studying if I want to calculate correlation between data sets?

I'm building an app that brings in data from multiple API's (Stripe, Google Analytics, Github). I'd want to be able to analyze the different sets of data against each other if at all possible and draw ...
0
votes
0answers
13 views

Adaboost weights adding up to $1$

In the Adaboost algorithm, I understand that for a given $m$th iteration, the weights all add up to $1$. Based on Patrick Winston's lecture, it seems like this is a constraint. Is there a way to prove ...
0
votes
0answers
24 views

Bound the vc dimension of hypothesis class

Given some set $V$ of size $n$, define the domain $X = V \times V$. In addition, define the hypotheses class $H$ to be all the equivalence relations over $V$ with at most $k$ equivalent classes. I am ...
0
votes
0answers
9 views

Minimizing log-likelihood function

Below is a problem I'm currently working on. I am having trouble seeing how I can obtain the wk and wko values for equation (1). I cannot see how one would solve the negative log-likelihood function ...
4
votes
2answers
43 views

L1 regularized unconstrained optimization problem

I am encountering an unconstrained minimization problem. The problem is of the form $$\min_x \frac{\|x-a\|_2^2}{2}+\lambda\|x\|_1$$ where $x,a \in R^n$ and $x$ is the optimization variable. ...
-1
votes
0answers
25 views

How to explain polynomial coefficients by minimezed Error function?

We wish to predict ${\bf{t}}$ from an observed $\bf{x}$.We shall fit the data using a polynomial function of the form$$y({\bf{x}},{\bf{w}})=w_0+w_1x+w_2x^2+...+w_Mx^M=\sum_{j=0}^{M}w_jx^j$$ where $M$ ...
0
votes
0answers
8 views

Transforming sigmoid function to concave function [closed]

Can someone please tell me of a function that transforms a sigmoid function into a pure concave function?
-1
votes
0answers
29 views

Gradient of a vector [closed]

Matrix $V$ is 200 by 785. Matrix $X$ is 785 by 1. Matrix $W$ is 10 by 201. Matrix $y$ is 10 by 1. First, I do: $ V * X$ Then, I apply $tanh()$ to every element of that resulting matrix. The result ...
0
votes
0answers
17 views

Neural Net Matrix Multiplication

I'm trying to figure out the matrix multiplications for the implementation of a single hidden layer neural net for MNIST digit recognition. Like the following: ...
0
votes
0answers
11 views

machine learning feature vector for periodic point distributions in 3D [closed]

Is there any standard procedure (building feature vectors) that could be used to build machine learning models based on discrete and infinite point distributions on a hyperplane (practically 2D 3D), ...
0
votes
1answer
32 views

Why is there a factor of 1.7159 with the tanh function used in neural network activation?

I was reading about neural networks when I came across the line : Recommended f (x) = 1.7519 tanh (2/3 * x). How do we arrive at these values (we can fix the other once the other is obtained using ...
0
votes
0answers
20 views

Shatter coefficient and VC dimension of a grid in $R^d$

Given $\epsilon>0$, partition the cube $[0, 1]^d$ with square of side length $\epsilon$. The total number of square in the partition is $$ N = \left(\frac{1}{\epsilon}\right)^d. $$ What is the ...
0
votes
0answers
14 views

feature vector for periodic physical system for machine learning application [closed]

The question is motivated from a physics problem: Let's first discuss the 1D infinitely long discrete system on a lattice, a system can look like: system 1: ...(ABAC)(ABAC)(ABAC)... this leads to ...
0
votes
0answers
6 views

PCA: MSE of projecting a set of points to a subspace

Let $\{x_1,\dots,x_n\}$ be a dataset of n vectors in $\mathbb{R}^d$ s.t. $\sum_{i=1}^n x_i = 0$. Let $p_1, \dots, p_k$ be a set of k orthonormal unit vectors and V the subspace that they span. Given ...
0
votes
0answers
26 views

understanding a proof for uniform convergence of Fourier features

See part of the proof. In the proof, $\tilde{f} := \tilde{z}(x)'\tilde{z}(y) - k(x,y)$, the vector product is the approximation of a shift-invariant kernek $k(x,y)$. What is not clear to me is the ...
0
votes
0answers
24 views

The proof for the value of growth function for convex sets

I reinforce my education through self-study of Machine Learning. When I come across the problem of Growth Function generated by Convex Set I see only the result and skimming over a proof. I have to ...
2
votes
2answers
24 views

Question about the constraint in Laplacian eigenmaps

When calculating Laplacian Eigenmaps, the original paper mentions about the constraint $$y^TDy=1$$ as "removes an arbitrary scaling factor in the embedding". My understanding is that it prevents $y$ ...
0
votes
1answer
28 views

Inequality of the expectation vs monotone function

I'm reading understanding machine learning and several of the latest lemmas I've studied involved this inequality which I've searched for but found no justification of whatsoever. Could anyone point ...
0
votes
1answer
10 views

Support vector machine, dimension reduction of hyper plane

Question I have this question as an assigned task. In 2D the points are {2,4},{1,1},{5,25}{5,25}. I don't know how to officially calculate the optimal separation hyperplane between them, but from the ...
1
vote
0answers
17 views

Backpropagation in a Convolutional Neural Network

Consider a Convolutional Neural Network with the following architecture: \begin{align} Input---C_1 P_1 --- C_2 P_2 ---Softmax \end{align} Here $C_i$ refers to the $i^{th}$ convolutional layer ...
5
votes
1answer
93 views

Complexity of Gaussian Process algorithms is $\mathcal{O}(n^3)$

It is often quoted that the complexity of Gaussian Process algorithms is $\mathcal{O}(n^3)$ due to the need to invert an $n \times n$ matrix, where $n$ is the number of data points. But as far as I ...
0
votes
0answers
11 views

Gradient of softmax composed with cross-entropy

I am unsure if this is more appropriate for here or CV, but since it is mostly a question about calculus, I figured posting it here would be a reasonable idea. More specifically, I am interested in ...
0
votes
1answer
21 views

Why can't I use sum of probabilities as my loss function for machine learning?

I'd like to understand what is the major reason that we are using loss function of the following form in machine learning (I know it is obtained by taking a logarithm of the likelihood of the ...
0
votes
0answers
6 views

Error derivative of a stochastic binary neurons

I've some perceptrons which perform a stochastic sampling of the input. One of these units calculates it's output by: x = inputVector p(1;x) = 1.0/(1.0+exp(-sum_over_i(x_i*w_i))) Now I want to ...
1
vote
0answers
13 views

Implementing a Least Squares Kernel classifier

I am trying to find the equation I would need to use in order to implement a Least Squares Kernel classifier for a dataset with $N$ samples of feature length $d$. I have the kernel equation $k(x_i, ...
0
votes
0answers
29 views

Vector Euclidean norm upper bound by his coordinates average.

I'm trying to extend the Rademacher complexity and have the following question: For $ (v_1,..,v_m) = V \in {\mathbb{R}}^{m} $ , I will be glad to find an upper abound for the Euclidean norm: $$ ...
0
votes
0answers
10 views

Getting a feel for the Normal-Inverse-Wishart conjugate prior to multivariate normal distribution

I am trying to get a feel for the Normal-Inverse-Wishart conjugate prior, which I have started to use, sparingly, in my work, where I am trying to cluster multivariate normal data. As Wikipedia ...
1
vote
3answers
41 views

Solving vectors such that the dot product = 0

I'm doing some machine learning problems (namely logistic regression), and something I'm trying to do is calculate the decision boundary given a weight vector $\mathbf{w}$. The decision boundary lies ...
0
votes
0answers
32 views

Show that the null space of a full row rank N × l matrix X is a subspace of dimensionality N, for N < l.

please help me with this question. I could not able to figure out the approach from 1st step itself. Provide detailed explanation for the question
0
votes
0answers
10 views

Evidence Approximation

Derivation for Bayesian linear regression Can someone explain how 3.80 is obtained from 3.79? What does completing the square mean in this case.
2
votes
0answers
20 views

Regression With Additive Gaussian Noise

I am studying Machine Learning through Prof. Ng's Stanford lectures. In lecture 3 (lecture notes) he models the output variable $Y$ and input variable $X$ with parameters $\theta$ as $$Y = \theta^T X ...
2
votes
0answers
8 views

Understanding linear separability

Having read the wikipedia article and a similar question on the topic of linear separability, I still lack the understanding of this concept to explain any more than the most rudimentary euclidian ...
2
votes
0answers
19 views

Machine Learning - global minima and convexity in gradient descent

In linear regression we use gradient descent to find the global minimum provided that the cost function is a convex. J = x0 + x1*theta1 + x2*theta2 But if we ...
2
votes
0answers
49 views

What would be a good book to study machine learning from? [closed]

I'm looking for a up to date, readable book that assumes strong mathematical background. Thanks!
3
votes
2answers
46 views

How to show that the signal $x_n = A\cos(\omega n)$ can be fully predicted by a system with two weights $w_1,w_2$

I am trying to solve the following exercise: Show that the signal $x_n = A\cos(\omega n)$ can be fully predicted by a system with two weights $w_1,w_2$ (i.e. $x_n = w_1 x_{n-1} + w_2 x_{n-2}$). ...
0
votes
0answers
13 views

Error propagation: add errors in quadrature, or use a weighted standard deviation?

I have a measurement $x$ with a known uncertainty $\sigma_m$. I have a black box that can take an error-free measurement $x$ and produce a value $y$ with a known uncertainty $\sigma_{b}$ (which is ...
1
vote
2answers
24 views

Intuitively, why does squaring a loss function change optimal values?

In many optimization problems, it is clear that by performing a non-linear operation we change the outcome of any potential optimal values. For example in machine learning: summing over errors ...
4
votes
2answers
78 views

Why is it so hard to translate some proves into machine-readable form?

I have just read a topic on mathoverflow about man vs. machine in mathematics. The topic was inspired by the recent victory of Alpha Go over the World Go Champion, Lee Sedol. It reminded me of an ...
1
vote
1answer
214 views

The mathematics behind AlphaGo AI and Google Deepmind [closed]

in case you are following the tech news, Google's AlphaGo beat the world Go champion Lee Sedol, not once but twice. Here is the link of the game 2 review: ...
0
votes
0answers
20 views

Inference on a factor graph (Sum-product Algorithm)

I was going through the sum-product algorithm which can be used to find marginal distribution efficiently(and exactly) when the factor graph is a tree. I found it difficult to understand the way they ...
0
votes
0answers
24 views

Geometric Probability Function on real Dataset

Can anyone explain how to apply geometric distribution on a real dataset? I know there's a formula but I need to estimate the prob of success first. My problem is I want to build a model using such ...
0
votes
0answers
8 views

Computing Positive definite covariance matrix in Variational Bayes GMM

This question is about a part of variational Bayes problem for GMM. (more in Bishop, Pattern recognition and Machine learning, part $10.2.1$). We are looking for $q(\mu_k,\Lambda_k)$, so we have: $$ ...
2
votes
0answers
76 views

Tricky proof of a result of Michael Nielsen's book “Neural Networks and Deep Learning”.

In his free online book, "Neural Networks and Deep Learning", Michael Nielsen proposes to prove the next result: If $C$ is a cost function which depends on $v_{1}, v_{2}, ..., v_{n}$, he states that ...
2
votes
2answers
18 views

Proof of Variance of the Irreducible Error

In Introduction to Statistical Learning, given the general form of a quantitative response between a set of predictor variables and a target variable $$Y=f(X)+\epsilon$$ and the general form for a ...
-1
votes
1answer
16 views

Does the Information Gain algorithm favor a high-entropy attribute or a low-entropy one?

This might not be mutual to mathematics but it does relate to Information-Theory. My question is: Does the InformationGain algorithm, in Decision-Tree machine-learning, favor a high-entropy ...
0
votes
1answer
47 views

Proximity operator for logistic function

I am reading the ADMM paper by S. Boyd et al: http://web.stanford.edu/~boyd/papers/pdf/admm_distr_stats.pdf I'm interested in implementing a L1-regularized feature-wise distributed multinomial ...
1
vote
0answers
6 views

Kernel functions

Shown is the kernel function $k(x,x')$ for $x'=0.$ Their point is to show the localization of kernel func. But if $x'=0$, how is it varying over space. Shouldn't the dot product be zero? Can ...
2
votes
2answers
55 views

I am confused about the kernel of a matrix and the “kernel”

In linear algebra, the kernel of a matrix is its null space. In machine learning and statistics, there are a bunch of matrices are called "kernel". For example, I am totally confused. The second ...
2
votes
1answer
34 views

PCA Interpretation

The problem formulation is to show that PCA involves choosing a vector u, so as to minimize the sum of the squares of the projection errors(of the training examples x onto u),subject to u'*u=1 x(i) = ...
0
votes
0answers
35 views

Decision boundary for a poisson distribution

I'm working on deriving the decision boundary for a poisson distribution of the form: $$P(\lambda_{k}) = \frac{e^{-\lambda_{k}} \lambda_{k}^{x}}{x!}$$ Taking the argmax over k, I've ended up at ...