How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?
2
votes
1answer
21 views
Basic understanding of sampling from a continuous distribution.
For continuous distribution (on R) the probability of a single point is $0$.
So I'm not sure what does it mean to sample $M$ elements from a continuous distribution.
Let say there is a continuous ...
0
votes
0answers
15 views
Naive Bayesian Classifier for Object with Variable attributes
Let say our objects are connected graphs. They are to be classified into two categories, say A and B. However, for our purpose attributes for each graph is equal to the number of vertex of the graph ...
-1
votes
0answers
20 views
Which Regression methods are suitable for binary valued features and continuous output?
I want to build a machine learning model to regression on continuous output given binary valued features(0,1). the dimension of my problem is around 200. which of the flowing methods seems suitable ...
1
vote
1answer
54 views
Lagrange multipliers and KKT conditions - what do we gain?
I'm working through an optimization problem that reformulates the problem in terms of KKT conditions. Can someone please have a go at explaining the following in simple terms?
What do we gain by ...
2
votes
1answer
47 views
How gaussian mixture models work?
I am given an example:
Suppose 1000 observations are drawn from $N(0,1)$ and $N(5,2)$ with mixing parameters $\pi_{1}=0.2$ and $\pi_{2}=0.8$ respectively. Suppose we only know $\sigma$ and want to ...
3
votes
0answers
59 views
Relation between factor graph and conditional probability distribution
First, I'm from computer science. I don't know how to say this problem in a mathematical way. So please bear with me.
The question
Let say I have a factor graph illustrated in the figure.
The ...
0
votes
0answers
18 views
What does “Boundary Fragment” mean
I am reading a paper that uses the terms "Boundary Fragment Class", "Boundary Fragment Structure" and "Boundary Fragment" without first defining it. When I google these terms, I see papers that use ...
0
votes
0answers
19 views
Probability of error for Nearest Neighbors algorithm given K neighbors
Let me explain K nearest neighbors algorithm:
I have a field of points on a plane, each with a label(0 or 1 let's say). I am then given a point x randomly placed on the plane. We do not know the ...
0
votes
0answers
12 views
Confusion related to solving the optimization in linear svm using dual coordinate descent
I have this confusion related to L1 and L2 svm. I was reading this paper
I am attaching the screenshot and the part I didn't understand
The part that I didn't understand how it was derived
I ...
0
votes
0answers
33 views
How do I use Q-Learning to update values?
I am aware that the table initially has values of 3 but don't understand how to update the values. Any help would be appreciated.
1
vote
0answers
16 views
Is there any complete mathematical model for demand prediction?
Assuming that I have the history of the demand as a function of price $p \in [p_{min},p_{max}]$. I need a model better than the linear regression to predict the demand and its elasticity. The model ...
2
votes
0answers
48 views
Boltzmann machines - motivation for the energy function
I've been studying Boltzmann machines lately and was wondering if anyone could give me a "high-level" explanation or motivation for the energy function used:
$$E = -\sum_{i<j} w_{ij} \, s_i \, s_j ...
0
votes
0answers
43 views
Stochastic gradient descent for nonconvex functions
I am trying to optimize a nonconvex function of the form
$$f(x) = \sum_i g_i(x) - h_i(x)$$
where x is a vector of variables, and $g_i$ and $h_i$ are both convex. While I am aware that such a ...
2
votes
1answer
34 views
Why are log-probabilities of global states of Boltzmann machines linear in their energies?
While going through the Wikipedia article about Boltzmann machines, I read the following on the probabilities of global states of Boltzmann machines in the "Equilibrium state" section ...
3
votes
1answer
41 views
How do I find the size of this set?
For homework, I need to show that the size of a certain set is $\le 2^{(3n)^k}$ but I'm not getting this (I think I may just misunderstand how the set is defined).
So the set is defined as follows:
...
2
votes
1answer
35 views
Amount of information a hidden state can convey (HMM)
In this paper (Products of Hidden Markov Models, http://www.cs.toronto.edu/~hinton/absps/aistats_2001.pdf), the authors say that:
The hidden state of a single HMM can only convey log K bits of ...
2
votes
0answers
31 views
Interpretation of MATLAB's NaiveBayses 'posterior' function
After we created a Naive Bayes classifier object nb (say, with multivariate multinomial (mvmn) distribution), we can call ...
0
votes
0answers
36 views
Connection between Vapnik-Chervonenkis dimension and regularization
I think this is my first research-level question, so I'm going to ask it here first before going to Math Overflow.
In most tutorial papers like Burges, the Vapnik-Chervonenkis dimension is introduced ...
0
votes
0answers
41 views
Estimating co-occurrence probabilities of pairs of sample points from different clusters
I have a problem as follows:
I have a collection of samples which I wish to cluster. The sample points are acquired in "groups/batches", and we know that samples from the same "group" cannot be ...
1
vote
1answer
40 views
Is Wishart Matrix?
Analyzing a system, I have faced a problem which is related to Random Matrices and in particular Wishart matrix. The problem is as follows:
Lets assume $\boldsymbol{H}$ is an $m\times n$ random ...
1
vote
0answers
16 views
Confusion related to convexity and concavity of a problem
I was reading this paper http://www.ist.temple.edu/~vucetic/documents/wang11kdd.pdf related to adaptive multi-hyperplane machine for non linear classification
In that paper, they have mentioned about ...
2
votes
0answers
9 views
Confusion related to convexity of a problem
I was reading this paper related to Multiclass Classification with
Multi-Prototype Support Vector Machines - paper
However, I am having difficulty in understanding why they have mentioned the ...
1
vote
1answer
48 views
Why “hinge” loss is equivalent to 0-1 loss in SVM?
I'm reading a book named The Elements of Statistical Learning by Hastie et al. In $\S 12.3.2$ it introduced the SVM as a penalization method:
With $f(x)=h(x)^T \beta+\beta_0 $, the solution of the ...
2
votes
1answer
139 views
Reproducing Kernel Hilbert Spaces for Dummies
I am in the middle of some machine learning paper that states that for function $f$, imposing the norm constraint, $\|f \|=1$, corresponds to an orthogonal projection onto the direction selected in ...
1
vote
0answers
37 views
How to understand the task about bootstrap?
I have the following task. Let $X_1,\ldots,X_n$ be a sample (i.i.d.), $T_n = \overline{X}_n^2=\left(\sum\limits_{i=1}^nX_i\right)^2,$ and $\hat\alpha_k ...
1
vote
0answers
70 views
How to weight Jaccard Similarity
I'd like to calculate the similarity between two sets using Jaccard but temper the results using the relative frequency of each item within a corpus.
Jaccard is defined as the magnitude of the ...
0
votes
0answers
24 views
Lagrangian Duality Complementary Slackness solution
If $$\alpha^*,\mu^* $$ is the solution of optimalization $$\max_{\alpha, \mu}\mathcal{L}(\omega, \xi, b, \alpha, \mu)$$ How I can show that "complementary slackness condition" $$\alpha_i^* = 0$$ ...
2
votes
0answers
32 views
Kmeans on “symmetric” data
A set is said to be fully-symmetric if for every $x$ in it, negating one of its components results in $y$ such that $y$ is in the set as well.
A set is said to be semi-symmetric if for every $x$ in ...
0
votes
0answers
54 views
Derivatives on hidden layers in backpropagation (ANNs)
I'm working on understanding all the math used in artificial neural networks. I have gotten stuck at calculating the error function derivatives for hidden layers when performing backpropagation.
On ...
2
votes
2answers
54 views
What is the exact nature of messages in belief propagation on a factor graph?
The recursive nature of the definition of messages in belief propagation on a factor graph makes it quite confusing for me what the messages exactly correspond to: ...
14
votes
3answers
202 views
Mathematical preparation for postgraduate studies in Linguistics
I am an undergraduate student in Mathematics and I would like to continue my postgraduate studies in the harder, more mathematical aspects of Linguistics. What exactly would that include is unknown ...
0
votes
0answers
32 views
Is HMM discriminative or generative?
Wikipedia
"An HMM can be considered as the simplest dynamic Bayesian network." Here.
"In probability and statistics, a generative model is a model for randomly generating observable data, ...
0
votes
0answers
53 views
Why Local Minimum is calculated for a derivative function instead of actual function?
In Machine learning regression problem, why the local minimum is computed for a derivative function instead of the actual function?
Example:
http://en.wikipedia.org/wiki/Gradient_descent
The ...
1
vote
0answers
59 views
Multivariate Gaussian equivalent for a Gaussian integration identity.
For a one-dimensional x,
$$\int_{-\infty}^{\infty}x^{2}e^{-x^{2}}dx=\frac{1}{2}\int_{-\infty}^{\infty}e^{-x^{2}}dx$$
This can be shown through integration by parts. There is a good derivation of ...
0
votes
0answers
58 views
What similarity measures would you recommend for product vectors given only 1, 0, and -1?
I am currently building a product recommendation engine using item based filtering techniques.
Each user on our site can leave the following feedback: Like (represented by 1), Not seen (v = 0) and ...
2
votes
1answer
74 views
What is a sampling density? Why is the sampling density proportional to $N^{\frac{1}{p}}$?
I'm reading a book named The Elements of Statistical Learning by Hastie, in section 2.5, Local Methods in High Dimensions, it says that the sampling density is proportional to $N^{\frac{1}{p}}$, where ...
1
vote
0answers
136 views
One cluster Davies–Bouldin Index
I'm trying to evaluate what is the right number of cluster needed for clusterize some data.
I know that this is possible using Davies–Bouldin Index (DBI).
To using DBI you have to compute it for any ...
2
votes
1answer
89 views
Why does a radial basis function kernel imply an infinite dimension map?
I understand that each kernel implies a particular feature map. For instance for $x,z \in R^2$ the kernel $K(x,z)=(dot(x,z))^2$ implies a feature map $φ(<x_1, x_2>)=<x_1^2 , x_1 x_2 , x_1 ...
1
vote
0answers
40 views
Classification: Why k-Nearest Neighbor method is more appropriate for a Mixture of Gaussians?
I'm reading a book named "The Elements of Statistical Learning" in which it states 2 scenarios when we are trying to predict the class label:
Scenario 1: The training data in each class were ...
1
vote
0answers
30 views
How do I measure the similarity of two bivariate time series?
Suppose I have two bivariate time series:
$$ ts1 = [<a_1, b_1>, <a_2, b_2>, \cdots, <a_N, b_N>] $$
$$ ts2 = [<c_1, d_2>, <c_2, d_2>, \cdots, <c_N, d_N>] $$
Which ...
1
vote
0answers
38 views
Gaussian prior from feature to input space
if I have Gaussian prior ($\exp\left(\dfrac{-\sum_i w_i}{2\gamma^2}\right)$) on my weights in a linear classifier, how can I transform this so I can apply it for my kernel parameters $\alpha$? I have ...
0
votes
1answer
160 views
Dependent Bernoulli trials
The probability of a sequence of n independent Bernoulli trials can be easily expressed as
$$p(x_1,...,x_n|p_1,...,p_n)=\prod_{i=1}^np_i^{x_i}(1-p_i)^{1-x_i}$$
but what if the trials are not ...
2
votes
0answers
68 views
What is the computational complexity of the EM algorithm?
In general, and more specifically for Bernoulli mixture model (aka Latent Class Analysis).
1
vote
0answers
117 views
Gradient Descent for Primal Kernel SVM with Soft-Margin(Hinge) Loss
Given the primal objective
$$F({\bf a})=L\sum_{i,j}a_{i}a_{j}k(x_i,x_j) + \sum_{i}max(0, 1-y_i \sum_{j}a_jk(x_i,x_j)$$
for the soft margin SVM, where ${\bf a}=(a_1,...,a_N)$, N being the number of ...
0
votes
0answers
32 views
R to $R^{20}$ linear transformation for classification?
I try to do some type of classification.I have 2 timeseries signals to distinguish from eachother.
Here is the way i was told to do it:
I take into account one of these signals and call it MySignal ...
0
votes
1answer
58 views
Gradient of log softmax in matrix form
Suppose $J(\mathbf{A})$ is defined as follows
$$J=\text{tr}(\log \mathbf{P})$$
$$\mathbf{P}=\frac{e^\mathbf{A}}{\mathbf{1} \mathbf{1}' e^\mathbf{A}}$$
where division, exp and log are taken pointwise, ...
0
votes
0answers
44 views
Translation invariant kernel and symmetric kernel
Let $x,y \in \mathbb{R}$ where $y=x-t$. Translation-invariant (or shift-invariant) kernel $\kappa(\cdot,\cdot)$ is defined as
$\kappa(x,y) = \kappa(x,x-t) = \kappa(t)$.
Can I say that the function ...
0
votes
1answer
67 views
Can I say that if a kernel is translation-invariant (or shift-invariant), then the kernel is symmetric?
Let $x,y \in \mathbb{R}$ where $y=x-t$. Translation-invariant (or shift-invariant) kernel $\kappa(\cdot,\cdot)$ is defined as
$\kappa(x,y) = \kappa(x,x-t) = \kappa(t)$.
Can I say that the function ...
1
vote
0answers
39 views
i,j,k values of the theta matrix in neural networks
SO I'm looking at these two neural networks and walking through how the ijk values of Theta correspond to the layer, the node number, and either there are redundant values or I'm missing how the ...
0
votes
0answers
25 views
is it possible to reduce the weight of a best fit line (least squares) given new data points?
I have a simple best-fit-line algorithm similar to this description.
Without memorizing the points history, it is easy to calculate a rolling best fit line as long as we remember (store) the ...
