-1
votes
1answer
14 views

Usefulness of Laplacians for directed graphs

Are laplacians for directed graphs used in any algorithms ? For example laplacians for the undirected graphs are used in algorithms such as spectral clustering.
4
votes
1answer
101 views

Clarification about solution of linear SVM problem

I'm reading this tutorial about SVMs. I'd like to have two clarifications: at page 4 (bottom), why is that, after using (1.10) the summation is extended to only $m \in S$? In (1.10) the summation ...
0
votes
1answer
33 views

different Interpolation techniques

what are the differences between spline and Lagrange interpolation, and are there any other kinds that might be similar that perform well ?
0
votes
1answer
18 views

Curve fitting to connect certain points

Well the image says everything, anyone has any idea how to, or where should i look to be able to draw the BLACK curve ? in fact i need a function that would connect the summits of these red dotted ...
2
votes
1answer
52 views

Inner Product Space vs. Vector Space

I had no trouble understanding what a vector space is: a constraint on the type of vectors you can create, such that certain operations could be performed with them. For example, a vector space of ...
1
vote
1answer
33 views

support vector machine - vector algebra

I am trying to understand the basics of SVM algebra, but fail to understand per below: Let us formalize an SVM with algebra. A decision hyperplane can be defined by an intercept term $b$ and a ...
0
votes
1answer
40 views

How to represent a separating hyperplane in two dimensions

I am reading a book about support vector machine, and I don't understand some of the math in it. Consider the training sample ${(x_{i}, d_{i})}^{N}_{i=1}$ where $x_{i}$ is the input pattern for ...
7
votes
1answer
378 views

Gradient Descent on Non-Convex Function Works But How?

For Netflix Prize competition on recommendations one method used a stochastic gradient descent, popularized by Simon Funk who used it to solve an SVD approximately. The math is better explained here ...
1
vote
1answer
95 views

Linear discriminant analysis - Properties of weight vectors and decision surfaces

I'm learning about "linear discriminant analysis" on "Statistical Pattern Recognition" of A.R. Webb and K.D. Copsey (chapter 5 of 3rd edition). The general idea is introduced where we suppose to have ...
3
votes
1answer
135 views

Why the largest eigenvalue is the bound?

I am new to Linear Algebra, say introduction level (know the definition and some basic properties, theories about it but not really familiar with doing math on it). Please explain to me: If A is an ...
1
vote
1answer
156 views

rank one update for Cholesky factor

I have covariance matrix known to be $$K = \sum_{i=1}^Nx_ix_i^T$$ where the dimension of $x$ is big (like 50000) so I don't want to really compute any outer-product to expand it as a full matrix. ...
3
votes
1answer
44 views

Path connectedness of the set of separating hyperplanes?

Let $X \subset \mathbb{R}^m$ be a finite set of points, classified by a function $f:X \to \{-1, 1\}$. We say that a hyperplane $H = \{x \in \mathbb{R}^m : \langle x, v \rangle + c = 0\}$ is a ...
2
votes
1answer
108 views

expectation of norm of orthogonal projector

The question has to do with calculating the expected squared norm of a random projection. We have a 2D subspace $T := span\{U1, U2\}$ where $U1$ is a random vector uniformly distributed over unit ...
2
votes
1answer
110 views

how is the gradient derived here?

I'm taking an online machine learning class and in lecture 9 which covers gradient descent, I can't quite follow how he derives the direction vector of the descent (around the 57:15 mark). He's ...
0
votes
0answers
191 views

Comparing k nearest neighbors (knn) and least squares bias and variance

I'm reading the textbook The Elements of Statistical Learning. In section 2.3.3, it says "The linear decision boundary from least squares is very smooth, and apparently stable to fit. [...] In ...
0
votes
0answers
39 views

Finding an optimal set of weights for combining correlated classifiers

In order to combine classifiers that are correlated with one another, I would need to solve the following optimization problem: Find a vector $\mathbf{w}$ that minimizes $\mathbf{w}^T M \mathbf{w}$ ...
0
votes
1answer
70 views

Problem with equation derivation

I'm studying support vector machines and the book I'm using states in one particular part the following: We know that: $$\vec{y} = \vec{x} + v\vec{w}$$ and that $$|\;\vec{y}-\vec{x}\;| = 2M$$ and so ...
2
votes
1answer
138 views

derivation of Support Vector Machine

I was watching Andrew Ng's machine learning lecture on SVM. There is one line that puzzles me. $$x^{(i)} - \gamma^{(i)} \frac{w}{||w||}$$ I dont understand how can the line above give the ...
4
votes
1answer
319 views

Reproducing Kernel Hilbert Spaces for Dummies

I am in the middle of some machine learning paper that states that for function $f$, imposing the norm constraint, $\|f \|=1$, corresponds to an orthogonal projection onto the direction selected in ...
1
vote
1answer
303 views

How to find Basis vectors of a matrix $X$, given basis vectors of its kernel matrix $XX^T$?

If we know basis vectors for $K=XX^T$ (e.g. will be eigenvectors here since $K$ is symmetric), how can we find base vectors for $X$?
1
vote
1answer
115 views

Linear algebra: finding a Tikhonov regularizer matrix

A more general soft constraint is the Tikhonov regularization constraint $$ \mathbf{w}^\text{T}\Gamma^\text{T}\Gamma\mathbf{w} \leq C $$ which can capture relationships among the $w_i$ (the matrix ...
0
votes
2answers
997 views

Uniqueness of symmetric positive definite matrix decomposition

We know that any symmetric positive semi-definite matrix $K$ can be written as $K= AA^T$, where $A$ has real components. One way to get to $A$ is to compute eigen value decomposition of $K= P^T DP$ ...
8
votes
3answers
223 views

Minimize $||Ax-b||$ but for $A$, not $x$

I have a machine learning regression problem. I need to minimize $$ \sum_i||Ax_i-b_i||_2^2 $$ However I am trying to find matrix $A$, not the usual $x$, and I have lots of example data for $x_i$ and ...
2
votes
1answer
273 views

Concise foundational math books

I am tackling some topics in Probability Theory and Machine Learning and while I have plenty of resources dedicated to those disciplines I am lacking in a good basic math foundation. Does anyone know ...
0
votes
1answer
161 views

linear kernel pca get corresponding dimension

I am implementing my own version of linear kernel principal component analysis for better understanding the algorithm. I faced a problem which seems to be specific ...
0
votes
1answer
442 views

Math problem in Pattern Recognition and Machine Learning

I had been carefully following Bishop's Pattern Recognition book and came across some of difficulties due to my merely basic math ground.First begin by the Gaussian section on page 80. The ...
1
vote
2answers
356 views

Simple Least Squares Regression?

I have a vector X of 50 real numbers and a vector Y of 50 real numbers. I want to model them as y = ax + b How do I determine a and b such that it minimizes the ...
0
votes
2answers
153 views

Solving $X^{T}Xw = X^{T}y$ for $w$ in Octave / MATLAB

I have a matrix $X$ and a vector $y$, how do I solve the following equation for $w$: $$ X^{T}Xw = X^{T}y $$ in Octave and/or MATLAB?
0
votes
1answer
84 views

Regression on Linear Model?

I have 50 or so training examples involving a set of 200 or so real numbers (x1,x2,...,x200) (normalized to a 0 mean and std dev 1), and a single output real (y) in the range 0.0..1.0. I want to fit ...
3
votes
3answers
928 views

What is the significance of theoretical linear algebra in machine learning/computer vision research?

I am a computer science research student working in application of Machine Learning to solve Computer Vision problems. Since, lot of linear algebra(eigenvalues, SVD etc.) comes up when reading ...
4
votes
3answers
364 views

First Course in Linear algebra books that start with basic algebra?

I'm 30 years old, and the only math I can remember from college is basic algebra and some probabilities. Next month, I have a machine learning project I'd like to work on, but I'll need a solid ...
0
votes
1answer
178 views

Rewriting Linear Regression Error Function in Matrix Form

I am having trouble try to show that this linear regression summation: $J(w) = \sum^m_{i=1} u_i (w^T x_i - y_i)^2$ can be rewritten in the following matrix form: $J(w) = (Xw - y)^T U(Xw - y)$ ...