How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

learn more… | top users | synonyms

0
votes
0answers
10 views

How can I scale the covariance matrix which represent a gaussian distribution ? [on hold]

I have a model genrated by using GMM the output is the mean and covariance matrix .I need to scale the cov matrix .for example I want to double the elipse that represent this gaussian .
0
votes
0answers
6 views

Decision Boundary of A Single Perception with Logistic Function

I am currently studying neural networks and have been trying to reason about this for a while to no avail. I understand that given a perceptron(such as above) with f as a step function, any ...
0
votes
1answer
19 views

How to show that SVM is convex problem

It's well-known fact that SVM is convex problem $min \frac{1}{2} \left \| w \right \|^2$ s.t. $(wx_i+b)y_i \geq 1$ I don't understand how given the LP formulation of SVM I can coclude that it's ...
0
votes
0answers
19 views

What is the meaning of sub-constant error?

The error is defined as $E \geq \frac{1}{2ab(1+ab)}$, where $a$ and $b$ are both positive . The claim is: if we fix the value $a$, then to get a sub-constant error $E$, we must ensure that ...
0
votes
0answers
9 views

What is the proper name of a model that takes as input the output of another model?

Thanks in advance for the help. I am writing a paper and for the life of me can't remember the proper term for a model that works as follows. rawData -> model1 -> outputModel1 -> model2 -> ...
0
votes
0answers
5 views

What is a basic description of how diffusion mapping works?

I have been trying to get a basic understanding of diffusion mapping, and I think I understand the concept, but I am having trouble understanding the math behind it (I have knowledge of advanced math ...
0
votes
0answers
6 views

LDA with fixed topics?

Suppose I have a collection of "topic" probability distributions $\{\phi^{z}\}$ for LDA (Latent Dirichlet Allocation) that I have found via alternate methods; is there a closed form MLE for the ...
0
votes
0answers
11 views

Assessing the “Quality ” of a solution sobtained by using lagrangian multipliers

I have an ill-defined question. I work in machine learning and am trying to learn the parameters of a model, such that my problem amounts to constrained optimization. That is, I have some training ...
0
votes
1answer
11 views

Why is it that large linear SVM coefficients denote the most important features?

I'm looking for an intuitive explanation, preferably geometric. Why is it that I sort the coefficients of my linear SVM I get the most indicative features as the ones with the large coefficients? ...
1
vote
1answer
19 views

Effects of feature scaling on weight vectors for linear regression

Given that linear regression or polynomial regression can be represented as: $\textbf{w} = (X^{T}X)^{-1}X^{T}Y$ It is standard practice in machine learning to scale each column in their training ...
0
votes
0answers
14 views

Specific utility (error) function for machine learning

I need a differentiable analog of following piecewise-defined function for machine learning application: $E=E(x,y)$ when $y=1$, $E=1/(x+1)$ when $y=-1$, $E=-1/(x-1)$ $y\in \{-1,1\}$ (two values, ...
0
votes
1answer
27 views

strong convexity of loss function in multi-dimensional (high-dimensional) space

My question is based on this paper (see the last 10 rows in page 7). It seems this is a general claim: In machine learning or statistic, the loss function $l(W^TX, y)$ (a linear predictor) can never ...
0
votes
1answer
16 views

How to derive this solution to this minimization problem in vector form?

We want to minimize the mean squared error $$ \sum_{t=1}^n (y_t - \theta^T x_t - \theta_0)^2. $$ Letting $X = [x_t, 1]$, we can rewrite the above problem in vector form as $$ \sum_{t=1}^n (y_t - ...
1
vote
1answer
24 views

What is the meaning ‘uniformly converge’?

Assuming that, we randomly sample $n$ data following a distribution, then if someone claims that the average of these $n$ data uniformly converge to its expectation with rate $O(\sqrt{1/n})$. Here, ...
2
votes
1answer
28 views

Machine Learning: Linear Regression models

I'm currently in a course learning about neural networks and machine learning, and I came across these two formulas in this textbook page on linear regression: 1) $y(x) = a + bx$ and 2) $y(x) = ...
0
votes
0answers
19 views

Please explain to me derivation of backpropagation algorithm

I cant understand derivation for update of inner weight of the network. I tried to do it for network of size 1x1x2x1 (last unit - output), but even there got lost in crowd of partial derivatives and ...
2
votes
0answers
33 views

Why this two SVM problems are equivalent?

Consider the two classification problem, our classifier is a hyperplane $w^T x+b=0$, where $w$ is the weight vector, $x$ is the input vector and $b$ is the bias. we have input $x_i\in ...
1
vote
3answers
104 views

Probability that the fish that set off metal detector is the one true fish

I have exams in Machine Learning coming up and I need help answering this question. There are a million identical fish in a lake, one of which has swallowed the One True Ring. You must get it ...
0
votes
0answers
9 views

Concept Learning algorithm in Artificial Intelligence

Im going through Machine Learning in Artificial intelligence(Artificial Intelligence by George F Luger-Page 398) . Im going through the candidate elimination algorithm. There is this specific to ...
0
votes
0answers
4 views

Meaning of coherence measure $\frac{n}{s}\max_{1 \le i \le n}\sum_{j=1}^n U_{ij}^2$

In the paper Extracting Certainty from Uncertainty: Transductive Pairwise Classification from Pairwise Similarities, the authors use a coherence measure defined as $$ \mu_s=\frac{n}{s}\max_{1 \le i ...
2
votes
0answers
46 views

Rigorously, what is the goal of (machine/statistical) Learning and why is that the goal?

After some time doing machine learning and statistical learning theory, I decided to return to my foundations and make sure that the goal of what I am doing makes sense. First let me define $I(f)$ as ...
0
votes
0answers
23 views

ANN learning algorythm for desired Inputs

I want to build a System like this one to determine if its possible to adjust the given system according to the "wished" input parameters: So the goal is to "prove" that a ANN is able to get ...
0
votes
0answers
28 views

In kernel rules, why is a regular kernel necessarily bounded?

Request from a dabbler in measure theory. Here a kernel merely designates a real-valued function. Quoted from "A Probabilistic Theory of Pattern Recognition", Luc Devroye, Laszlo Gyorfi, Gabor ...
1
vote
0answers
30 views

Understanding a step while calculating the gradient of the log-likelihood of Restricted Boltzmann Machines

I'm reading this paper about Restricted Boltzmann Machines. However there are two steps that I don't understand when they compute the gradient of the log-likelihood (section 4.1). Here is a screenshot ...
2
votes
0answers
21 views

What makes the Gaussian kernel so magical for PCA, and in general? [migrated]

I was reading about kernel PCA (1 2 3) with Gaussian and polynomial kernels. How does the Gaussian kernel separate seemingly any sort of nonlinear data exceptionally well? Please give an intuitive ...
0
votes
0answers
16 views

unique children of a point in a boolean lattice

I am working with two-element boolean algebra, e.g. points composed of strings of $0$s and $1$s and bit-wise $AND$ and $OR$ to find maxima and minima. In the domain I'm working in, I need to assign ...
1
vote
1answer
30 views

Difference among convergence almost surely, in probability, and in distribution. And advantages.

I know the definition of the three convergences. My question is, in statistical, machine learning and/or engineering applications, do the stronger convergences have any advantage over the weaker ...
1
vote
0answers
21 views

How to prove this specific kernel is not in RKHS?

Consider $\mathcal{X}=\mathbb{R}$, and $k(x,y)=xy=[\frac{x}{\sqrt{2}},\frac{x}{\sqrt{2}}]\cdot [\frac{y}{\sqrt{2}},\frac{y}{\sqrt{2}}]^T$, where we thus can define two kinds of feature maps ...
2
votes
0answers
31 views

How to prove $\Phi_w(x)=\cos (w^T x+b)$ is outside the RKHS associated with the Gaussian kernel function$K(x,y)=\exp(-\frac{||x-y||^2}{2\sigma})$?

How to prove $\Phi_w (x)=\cos (w^T x+b)$ is outside the RKHS ( Reproducing Kernel Hilbert Space) associated with the Gaussian kernel function$K(x,y)=\exp(-\frac{||x-y||^2}{2\sigma})$?
1
vote
0answers
37 views

How to understand ' Let $\mathcal{H}$ be a Hilbert space of functions $f$ : $ \mathcal{X} \rightarrow R$, denoted on a non-empty set $\mathcal{X}$.'

I am a beginner. By asking this question, I means that, to construct a Hilbert space, should $\mathcal{X}$ satisfy some properties? Furthermore, in some papers especially on machine learning, ...
8
votes
1answer
487 views

How can I derive the back propagation formula in a more elegant way?

When you compute the gradient of the cost function of a neural network with respect to its weights, as I currently understand it, you can only do it by computing the partial derivative of the cost ...
1
vote
0answers
22 views

The optimization problem of soft margin Support Vector Machine: How to interpret?

I try to understand what exactly we are trying to optimize in the case of Support Vector Machine problem, which supports soft margins. The original problem is posed first as, without soft margins ...
0
votes
1answer
45 views

How is logistic loss and cross-entropy related?

I found that Kullback-Leibler loss, log-loss or cross-entropy is the same loss function. Is the logistic-loss function used in logistic regression equivalent to the cross-entropy function? If yes, can ...
2
votes
1answer
32 views

Analytic solution for matrix factorization using alternating least squares

The standard form for ridge regression aims to minimize the following cost function. $$ \min\ \ \sum_i(y_i-x_i^T\beta)^2 + \lambda\sum_j\beta^2_j $$ As described here, it's possible to differentiate ...
1
vote
1answer
50 views

Gaussian Process Regression

Observations: $$ X= \begin{pmatrix} x_1 \\ x_2 \\ \end{pmatrix} = \begin{pmatrix} 0 & 1 \\ 0.5 & 2 \\ \end{pmatrix} $$ $$ y= ...
0
votes
1answer
24 views

proof that a density proportional to Gaussian is Gaussian

I try to develop bayesian estimation for one dimensional Gaussian with unknown $\mu$ and known $\sigma$. I got $$p(x\mid D) = \int p(x\mid\mu)p(\mu\mid D) \, d\mu =\int \frac{1}{\sigma ...
3
votes
2answers
54 views

Intuition about gradient

https://en.wikipedia.org/wiki/Gradient Gradient is a vector which we can obtain from any differentable function taking its partial derivatives. From Wiki: "...the gradient points in the direction of ...
1
vote
0answers
44 views

Gram matrix of Gaussian kernel is not positive definite

I am developing a machine learning software, where I am trying to apply kernel methods. I have N uniformly sampled scalar values, $\{x_1,\dots,x_N\}$ from a given interval $[a,b]$. My aim is to ...
0
votes
1answer
32 views

How we come up with: $\mathcal{w}^T\mathcal{w} = \sum_{i,j} y^{(i)}y^{(j)} \alpha_{j}\alpha_{j} (\mathbf{x}^{(i)})^T {\mathbf{x}^{(i)}}$?

The primal optimization problem for finding the optimal margin classifier is: \begin{align} \arg\min_{\mathbf{w}}\frac{1}{2}\|\mathbf{w}\|^2_2 \\ \text{subject to } \quad ...
-1
votes
1answer
28 views

simTuring Machines

Is a player piano a Turing Machine reading data on the piano roll when perforations in paper control mechanical stops that intonate sounds according to the player algorithm? And by analogy then is a ...
0
votes
1answer
25 views

How to get a valid distance metric?

I have got a problem to devise a distance metric to get the similarity measurement of vectors. Someone suggested me to use dot product, which seems to me the same as the Cosine similarity metric; ...
1
vote
0answers
29 views

Extracting $G$ from $K^{-1}=G^TG$, knowing $K$

In a machine learning project I am working on, I have came across the following problem. I have a symmetric, $d\times d$ matrix $K_{oij}=k(x_i,x_j)$ where $k$ is a Gaussian kernel function. $x_1,\dots ...
0
votes
0answers
16 views

Determining a good genome for a genetic algorithm to alter neural network characteristics?

I am developing an application to run a genetic algorithm over the input characteristics of a neural network. I am currently looking for help finding a good "genome" to use along with good example ...
0
votes
0answers
12 views

Application of Gaussian Width in “ a subspace missing a set ”

As in Gaussian Mean Width, it says the concept of Gaussian width of a set $K$, which is a subset of a unit sphere $S^{n-1} \in R^n$. As a popular application, we use it to estimate if there ...
0
votes
0answers
31 views

Why does $\int f(x)(y-r(x))\;dP(y,x) = 0$?

My question is, why does: $$\int f(x)(y-r(x))\;dP(y,x) = 0,$$ where $r(x) = \int y \;dP(y|x)$ and $P$ is a probability distribution function. It was also given (in my book) that: $$\int ...
2
votes
1answer
54 views

Reproducing kernel Hilbert space, why?

Let $K: X \times X \rightarrow \mathbb{C}$ be a positive definite kernel on a set $X$, i.e. for any $x_1, \cdots, x_n \in X$, the matrix $$ [K(x_i, x_j)]_{ij} \in \mathbb{C}^{n \times n} $$ is ...
0
votes
0answers
25 views

What is the Gini impurity index of an empty set?

Now, this may be a silly question because in practice you would never calculate the gini impurity on an empty set of observations. However, I did notice that while the shannon entropy is 1.0 for an ...
2
votes
0answers
17 views

Non parametric estimators for noisy funcions

Suppose there is a function $f(a,b,c,\ldots)$ of $M$ variables (fixed numbers, not random variables). Add some Gaussian noise to this function: $$ g(a,b,c,\ldots) = f(a,b,c,\ldots) + ...
0
votes
0answers
24 views

An example shows the difference between inference in Bayesian network and Junction Tree

Why inference in Junction tree is more efficient? There are directed graph BN and the corresponded undirected graph transformed by Junction tree algorithm. The literature describes that inference in ...
1
vote
0answers
15 views

Relation between RKHS and space of continuous functions

Consider a Mercer Kernel $K\colon \mathcal{X}\times \mathcal{X}\to \mathbb{R}$, $\mathcal{X}$ being a compact subset of $\mathbb{R}^m$, and its (unique) associated Reproducing Kernel HIlbert Space ...