0
votes
0answers
23 views

Deriving equation in vector notation

I had some trouble deriving an equation from the book 'Elements of statistical Learning' p. 108 equation 4.9. This heavily relies on linear algebra, so I was wondering how the author came to his final ...
0
votes
0answers
15 views

Expectation Maximization Algorithm for Gaussian Mixture Model

Can we use the Expectation Maximization algorithm for estimation of Gaussian Mixture Model with full covariance matrices? If yes then can you please give me a reference paper? So far all the machine ...
1
vote
2answers
34 views

Why we consider log likelihood instead of Likelihood in Gaussian Distribution

I am reading Gaussian Distribution from a machine learning book. It states that - We shall determine values for the unknown parameters mu and sigma^2 in the gaussian by maximizing the ...
0
votes
1answer
57 views

Why use regularization to reduce over-fitting

I'm having trouble understanding why should we use regularization for over-fitting when we can simply reduce the number of order to our polynomial function? Is it because it saves us time from having ...
0
votes
2answers
38 views

Creating a polynomial function with no x-intercept

I have an understanding of polynomials and how to create a function based on the leading coefficient, degrees, x-intercepts, etc. My question is how do i go about creating a polynomial function that ...
1
vote
0answers
20 views

Deriving cost function using MLE :Why use log function?

I am learning machine learning from Andrew Ng's open-class notes and coursera.org. I am trying to understand how the cost function for the logistic regression is derived. I will start with the cost ...
0
votes
0answers
14 views

Sparse data and covariance matrix computation

Background I am trying to apply Gaussian Discriminant Analysis (GDA) on the MNIST dataset of hand-written digits, with 10 classes for 10 digits. In this dataset, each point is a vector of 784 ...
3
votes
1answer
39 views

Math formulas on Clustering

I am currently studying Clustering in Machine Learning. I have found a document regarding guessing the right number of clusters. I am reading the first part of it, having difficulties in understanding ...
1
vote
0answers
20 views

How to represent the parameters in logistic function

I want to find the parameters in logistic function. I read the guide at here. It very clear to explain. But it did not has final solution that I need. Now, we will consider a basis logistic function ...
1
vote
0answers
21 views

feature selection for continuous variables

I wonder how exactly "feature selection" should be performed in case of continuous feature values. When feature values are discrete it is very straitforward to apply feature selection, but what to do ...
0
votes
1answer
20 views

Understand the English paragraph on association rule.

I am currently studying Association Rule Pattern Mining. I am reading the explanation on wikipedia about it. Somehow, I feel like I have a problem in understanding the paragraph below. Can somebody ...
0
votes
0answers
18 views

How to evaluate the difference between two classes of data which are highly overlapped

I’m trying to implement a classifier based on a dataset comprising two classes of high dimensional time-series observations (the values of the two classes of observations are highly similar). I ...
0
votes
0answers
25 views

Expected in-sample error of linear regression with respect to a dataset D

In my textbook, there is a statement mentioned on the topic of linear regression/machine learning, and a question, which is simply quoted as, Consider a noisy target, $ y = (w^{*})^T \textbf{x} + ...
0
votes
0answers
24 views

What method should I use for this optimization / feature selection project

I'm going to describe a problem and I'm not sure how to best solve it. I will describe the situation. When answering please recommend a method and maybe a software library. I'm using Python for my ...
0
votes
1answer
22 views

Expected error of best possible linear fit?

I asked the following question on stat SE, but I could not get a mathematically rigorous answer, and I have decided to ask here again. In my textbook, there is a statement mentioned on the topic of ...
0
votes
1answer
22 views

Is it a wrong expression for the local log-likelihood of logistic regression?

In page 206 of the book 'Elements of statistical learning', the author wrote: The local log-likelihood for this $J$ class model can be written $\sum_{i=1}^NK_\lambda (x_0, ...
0
votes
0answers
9 views

Invalid Kernels with $a<0 \;\rm{and}\; b<0$

How can we prove for $a<0$ and $b<0$, $k(x,y)=(x^Ty+a)^b$ is not a valid kernel? For $b<0$, can we write $k(x,y)$ cannot be represented as an inner product?
1
vote
2answers
39 views

Curse of Dimensionality … as illustrated by Christopher Bishop

I'm reading Christopher Bishop's book "Neural Networks for Pattern Recognition". I'm on pg 7 about curse of dimensionality. Here is the relevant part: For simplicity assume the dimensionality we ...
0
votes
0answers
14 views

Why do the two conditions listed in this paper imply the third?

I am working my way through this paper. On page 4, the author says: Why does A(λ', λ) have to be greater than 0? If the likelyhood with λ' is higher than the likelyhood with λ and A(λ', λ)<0 ...
0
votes
0answers
11 views

Is Expectation Propagation (EP) affected by the prior?

I understand EP by reading Minka's thesis: http://research.microsoft.com/en-us/um/people/minka/papers/ep/minka-ep-uai.pdf I'm trying to apply it to solve a Bayesian inference problem. However, I'm ...
0
votes
0answers
23 views

Deriving the optimal value for the intercept term in SVM

I was reading andrew ng's machine learning lecture notes on SVM. I came across the following equation (finding the optimal value for the intercept term $b$ in the SVM problem): However, I have no ...
2
votes
0answers
31 views

Normalizing multiple different features from unknown distributions

I'm doing some "exploratory" data analysis over a large set of classes/proteins, with a few hundred different features (I.E. Continuous variables) extracted from the data. The features are calculated ...
1
vote
0answers
15 views

What is the rationale behind ROC curves?

I am not sure how ROC curves work. I see that the X-Axis is the false positive rate while the Y axis is the true positive rate. 1) I don't understand how for a given statistical learning model, you ...
2
votes
1answer
36 views

Given $N$ coins, find a coin with minimal bias based on $N$ samples

General description: Given $N$ coins $Z_1,...,Z_N$ (Bernoulli RVs), where the $i$-th coin has probability $p_i$ for "Head", I'm trying to find $\min\limits _{i\in[N]}p_{i}$. I'm interested in a ...
1
vote
0answers
26 views

VC dimension problem

Let $\mathcal{F}_1$, $\mathcal{F}_2$, and $\mathcal{F}_3$ denote three Boolean classes of functions on a space $\mathcal{X}$ each having VC dimension at most $D$. Define $$ \mathcal{F} = \{ \max(f_1 ...
1
vote
1answer
34 views

Combine linear models of different sets of data.

I'm working on a large data set D that can be partitioned into some disjoint subsets D1, D2, ..., Dn. For each subset Di, I have a linear model Mi that minimizes the residual error for data in Di. ...
1
vote
0answers
27 views

Estimate distance between approximated posterior and true posterior

I'm working on a paper about using graphical models to do some prediction tasks with known observations. Since the model is complicated, finding the maximum a posteriori on the true posterior ...
1
vote
1answer
42 views

How do you express the maximization step of EM algorithm in matrix formulation?

Specifically, I am interested in how the covariance matrix is calculated. In terms of dimensions of factors involved, let's say I am given some data set X of dimension m x d, covariance matrix S of ...
0
votes
0answers
43 views

VC dimension for Rotatable Rectangles

It can be shown that VC dimension of rotatable rectangles is 7. The problem is I cannot understand how to approach the solution. So far I used bruteforce to solve this kind of problem, I was drawing ...
0
votes
2answers
63 views

What is -2loglikelihood?

I don't understand the term "loglikelihood"? I'd like to have a practical understanding of this word, and of why this is important. Besides this whenever we calculate some statistic like chisquare or ...
1
vote
1answer
683 views

Preventing underflow, log sum exp trick

I have some difficulties with understanding the schema to prevent underflow, which is very often mentioned as The log-sum-exp trick, the partial decription is The log-sum-exp trick. In short, I ...
0
votes
1answer
22 views

Modeling 2 conditionally dependent variables

I am working on image processing and my probability theory knowledge is low. My question here is I am working with 2 variables X and Y which is dependent on each other. That is we can compute P(X|Y) ...
0
votes
0answers
26 views

empty clusters during k-means iterations

We are given points $ S = \{ x_1,\dots,x_n \}$ in $R^m$ and $ k < n $ and we have to partition $S$ into k disjoint subsets. Consider the following variant of the k-means algorithm to create such a ...
1
vote
2answers
45 views

Scaling data into $[-1,1]$

I have a data in the matrix for: \begin{bmatrix} 1 & 2 & 3 & 9 & 6\\ 8 & 2 & 7 & 4 & 6 \\ 1 & 2 & 8 & 7 & 4 \end{bmatrix} Each row corresponds ...
1
vote
0answers
30 views

Is the Support Vector Classifier in some sense optimal?

My question is, is the original hard-margin support vector classifier optimal in some sense? If you have an answer that refers to the soft-margin SVC instead, I'd also be interested. I know that the ...
0
votes
1answer
18 views

Making sense of 1-dimensional data

I am collecting data of individual people. Each data is one dimensional, indicating a time difference of them doing certain things. Each person requires a different time value measured in miliseconds ...
0
votes
0answers
61 views

How to find “approximate most common” value from a list of RGB values

I have about 50 equally sized photos of magazine covers, which I'm attempting to blend into one composite image that shows the "average" cover. Each of the covers has a single face on it, so the ...
0
votes
1answer
131 views

Gaussian with a linear combination random variable mean

A very simple (looks like...) statistical problem, however I don't even know how to name it in a formal way... Suppose in a Bayesian framework I have random variables $y, x_1,$ and $x_2$, $$f(x) = ...
2
votes
1answer
109 views

expectation of norm of orthogonal projector

The question has to do with calculating the expected squared norm of a random projection. We have a 2D subspace $T := span\{U1, U2\}$ where $U1$ is a random vector uniformly distributed over unit ...
0
votes
1answer
30 views

Finding Kernel function for the data set

For a set of data points how to find an appropriate kernel function to map it to higher dimension so that it will be linearly seperable
0
votes
2answers
40 views

Testing using training data

I've been trying to prove that estimates of a classifier's performance using training data is a bad thing. Does "bad" mean it is biased? This is part of a larger proof. If somebody knows of previous ...
2
votes
0answers
103 views

Building Bayesian Networks, Causality and Cyclic Reasoning

I am studying Bayesian Statistics and I am trying to get a good understanding on Bayesian Networks, which seems to be vital in order to make something useful in Machine Learning. Most of the texts I ...
2
votes
0answers
101 views

L2-Regularized\Penalized Logistic Regression

Suppose you have an $n$ dimensional data vector $x = (x_1, \ldots, x_n)$ and two classes $y = 0$ or $y = 1$. Assuming the dimensions of $x$ are conditionally independent given $y$, and that the ...
0
votes
1answer
52 views

How to compare training and test errors in statistics?

I have a data set and I need to compare the performance of various statistical models: Least Squares, LASSO, Ridge Regression, to name a few of the key ones. What are standard techniques for ...
0
votes
0answers
191 views

Comparing k nearest neighbors (knn) and least squares bias and variance

I'm reading the textbook The Elements of Statistical Learning. In section 2.3.3, it says "The linear decision boundary from least squares is very smooth, and apparently stable to fit. [...] In ...
0
votes
1answer
63 views

K Nearest Neighbors classification Special Case with Identical Points

The question is about KNN algorithm for classification - the class labels of training samples are discrete. Suppose that the training set has n points that are ...
7
votes
2answers
2k views

derivative of cost function for Logistic Regression

I am going over the lectures on Machine Learning at Coursera. I am struggling with the following. How can the partial derivative of ...
0
votes
1answer
68 views

online learning to maximize profit

I have a software which takes input as investment and gives the output as return on a particular stock. Now profit metric $x_i$ is defined as the ratio of return $g_i$ to maximum possible return ...
0
votes
0answers
43 views

Trouble reading multinomial naive bayes notation

$C_m$: m = most likely class (wanted to write C subscript MAP for "maximum a posteriori" but couldn't do MAP with MathJax) ...
0
votes
1answer
133 views

Expression for Sum of Multivariate Gaussian Random Vectors

Consider two multidimensional random vectors $x$ and $z$ having Gaussian distributions $P(x)=N(x\mid\mu_x,\Sigma_x)$ and $P(z)=N(z\mid\mu_z,\Sigma_z)$, respectively, together with their sum $y=x+z$. ...