Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [machine-learning]

How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

0
votes
2answers
14 views

Proving concavity of a function in multiple variables

How do I prove that $f(\vec{x}) = \vec{b}.\vec{x} - \log \left (1+e^{\vec{a}.\vec{x}} \right )$ is concave? where $\vec{a}$ and $\vec{b}$ are constant vectors. My steps are as follows: The first ...
-4
votes
0answers
24 views

How to solve this Matrix by using Particular solution and general solution

I am new to mathematics concepts so i need to solve this step by step. This book solve the problem by just taking the answer without explaining anything. I just want to know how they have solve that ...
1
vote
2answers
20 views

In Support-Vector-Machine, why is the hyperplane given by $(p-1)$ dimensions?

In SVM, each feature vector is viewed as a data point in a $p$-dimensional plane and different labels are separated by a $(p-1)$ dimensional hyperplane. Please see the wikipedia entry: https://en....
0
votes
0answers
8 views

Possible Growth Function for a hypothesis set

I am new to machine learning and I am trying to resolve a homework problem. How do I determine the possible growth function $mH(N)$ for some hypothesis set? My choices are $1,2^N,2^\sqrt{N},N^2-N+2$ ...
1
vote
0answers
14 views

Approximate monotonicity of $\epsilon$-covering number

This is from Exercise 4.2.10 in Roman Vershynin's book, High-Dimensional Probability: An Introduction with Applications in Data Science. Let $(T;d)$ be a metric space and $N(A,d,\epsilon)$ be the $\...
0
votes
1answer
36 views

VC dimension of half planes

Let $\mathcal H$ be the set of all half spaces in the two-dimensional plane ($\mathbf{R}^2$). Two questions. 1) How can we formally show that the VC dimension of our half spaces is 3? That is, how ...
0
votes
1answer
17 views

Why does a $\sum_k$ appear when using the chain rule to derive $\delta^L_j?$

I'm following along this book on machine learning. At the moment, the author is proving that \begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}...
0
votes
1answer
20 views

Why do first and second moments insure stability?

In Pattern Recognition and Machine Learning, Bishop states "Let us now consider the maximum entropy configuration for a continuous variable. In order for this maximum to be well defined, it will ...
2
votes
1answer
18 views

Universal Approximation with Fixed Layer

Fix an activation function $\sigma$, and denote the class of all Neural-networks from $\mathbb{R}\rightarrow\mathbb{R}$ defined by this activation function by $NN^{\sigma}$. The classical universal ...
-1
votes
0answers
27 views

How to induct the Nonnegative Matrix Factorization on Orthogonal Subspace

I am studying the paper Nonnegative Matrix Factorization on Orthogonal Subspace I am sorry my reputation is too low to post the pic. The objective function is F and rewritten as J, I could not ...
0
votes
1answer
31 views

Derivative of Least Squares with L2 Norm

I'm new to matrix calculus, and I've never really taken derivatives of summations before. Could someone show me how I would get the first order derivative of this? $J(w)=\frac{1}{2}[\sum_{i=1}^{m}(w^...
0
votes
1answer
16 views

Definition of a Domain in Pan and Yang's article: “A survey of transfer learning”

In the article "A survey of transfer learning": https://ieeexplore.ieee.org/abstract/document/5288526 Pan and Yang give a good mathematical definition of transfer learning which seems to be widely ...
1
vote
0answers
16 views

Derivation of partial derivative of cost function with respect to weights in backpropagation algorithm

I am studying Machine Learning from Andrew Ng's Machine Learning course on coursera. I am stuck at understanding math behind back propagation. Here is an image of backpropagation algorithm from his ...
0
votes
0answers
17 views

Gradient computation with shared weights

In many applications we encounter weight sharing. I want to understand how the gradient with be computed in such a case. Is it feasible to train such nets? how easy or difficult? Consider the ...
0
votes
1answer
14 views

Calculating the integrity of the result of a weighted voting system

I have an ensemble model, which votes over many regression systems. I give my observation to all the models and record their output. Now I have knowledge of models accuracies as follows: I know the ...
0
votes
0answers
23 views

What are the joint and marginals in this neural network paper?

This paper has an algorithm which draws from the joint and then from the marginal. What does this actually mean in the code? Is the joint an (x,y) training pair? Then what is the marginal?
2
votes
1answer
82 views
+100

Shattering with sinusoids

Let $d \geq 2$ and $K$ some positive integer. Consider distinct points $\theta_1, \ldots, \theta_K\in \mathbb{T}^d$ and (not necessarily distinct) $z_1, \ldots, z_K \in \{-1,1\}$, does there exist an ...
2
votes
1answer
53 views

Bounding in Glivenko-Cantelli theorem

Problem Let $X_1, X_2, \cdots, X_n$ be iid random variables. The cdf and empirical cdf are $F(t)=P[X\leq t]$ and $\hat{F}_n(t)=\frac{1}{n} \sum_{i=1}^n 1(X_i\leq t)$. The Glivenko-Cantelli theorem ...
2
votes
0answers
15 views

Single-sided and double-sided concentration inequality

Problem In statistical learning theory, people are interested to bound something like $P[\vert X-\mathbb{E}[X]\vert\geq t]$ and this is done by $$ P[\vert X-\mathbb{E}[X]\vert\geq t]\leq P[X-\mathbb{...
1
vote
1answer
24 views

What is the interpretation of the expectation notation in the GAN formulation?

I'm confused about the expectation notation in the context of GAN loss functions. The GAN loss for the discriminator is binary cross-entropy. ie: is this real or not. real = $D(x)$ (ie: give ...
4
votes
0answers
60 views

Machine learning book with robust linear algebra approach

I am looking for machine learning book - neural network, deep learning etc etc - that use linear algebra in a robust manner. I found satisfactory the old book of Simon Haykin : Neural Networks : A ...
1
vote
1answer
21 views

VC Dimension of the support of a function

Problem Let $\mathcal{X}$ be a finite set, the support of a binary function $f: \mathcal{X} \rightarrow \{0,1\}$ is defined as $supp(f)=\{x\in\mathcal{X}: f(x)=1\}$. For any $k\leq \vert \mathcal{X}\...
1
vote
0answers
53 views

Deriving the error in activation nodes in back propagation algorithm

I am trying to understand back propagation algorithm from Andrew Ng's Machine learning course. Here is a pitcure of the slide on which I am stuck. I know that error in a function $f(x)$ is calculated ...
0
votes
0answers
20 views

Matrix representation of features in Linear regression [closed]

I am able to solve the equation assuming this is the correct notation, but I'm unable to get the intuition behind this. Since we denote a single input row as a single example in linear regression, I ...
0
votes
1answer
51 views

Machine learning: What method should I use for classification?

I post this on Math Stack Exchange instead of Data Science Stack Exchange because I want to have the theory, not Pyton import. Assume that we have a vector who contains decimal values, sorted. $$V =...
0
votes
1answer
16 views

derivation of softmax

This is the derivation of softmax in Bishop's PRML: $$ln(\frac{u_k}{1-\sum_ju_j}) = n_k$$ "Which we can solve for $u_k$ by first summing both sides over k and then rearranging and back-substituting ...
0
votes
1answer
38 views

Why can an element of a Hilbert space be written as this sum?

From Understanding Machine Learning: Let $w^*$ be an optimal solution of this equation: $$\min_w (f(\langle w, \psi(x_1)\rangle, \dots , \langle w, \psi(x_m) \rangle) + R(||w||))$$ where $...
1
vote
0answers
24 views

A confusion on the variance of weight matrix of Kaiming He's initialization

When trainning neural networks with ReLU function, Kaiming He's initialization is a good way to initialize parameters. In the paper "Delving Deep into Rectifiers", the author gave detailed derivation ...
0
votes
0answers
18 views

Why must every element of this set be of the form $\alpha x$ where $|a| \le 1$?

From Understading Machine Learning: Why is it the case that any subgradient of the hinge function $\text{max}\{0, 1-y\langle w,x \rangle\}$ at $w$ is of the form $\alpha x$, where $|a| \le 1$? I can'...
0
votes
0answers
22 views

What does $Pa(u^{(i)})$ notation mean?

I'm reading the deep-learning book and this expression comes up, and I'm pretty sure it hasn't been described earlier, it's not in the glossary and I can't find it used in google. This is the ...
1
vote
0answers
12 views

Redefining BC within heterogeneous BVP

I am trying to solve the following boundary value problem using an Artificial Neural Network, over the dimensions $x$ and $t$ for $x \in [0,1]$ and $t \in [0,1]$. It can be thought of a representing ...
0
votes
1answer
30 views

Why can't we just have $\xi = 0$ as the optimal value?

From Understanding Machine Learning: In the proof below, why does the choice of $\xi$ being $0$ or $1-(y_1\langle w,x_i\rangle + b )$ matter? Why can't we just have $\xi = 0$ as the optimal value as ...
0
votes
0answers
37 views

Expected loss for regression tree with absolute loss function in case of choosing labels randomly

Suppose we have a regression tree (binary). The absolute loss function $L(y,\hat{y})=|y-\hat{y}|$. I know that the optimal prediction that minimizes $\frac{1}{n}\sum_{i=1}^{n}|y_{i}-\hat{y}|$ is the ...
0
votes
1answer
20 views

How does this prove that the solution is optimal?

From Understanding Machine Learning: In the red box below: How does this show that $(\hat w, \hat b)$ is an optimal solution? This seems to just show that for each $i$, it's larger than the ...
0
votes
1answer
42 views

Gradients of $ \sum_{i=1}^N \|W_3 g(W_2 f(W_1 x_i) ) - y_i \|_2^2$ w.r.t. $W_1$, $W_2$, and $W_3$?

How to obtain the gradient and optionally Hessian of \begin{align} L(W_1, W_2, W_3) := \sum_{i=1}^N \| W_3 \ g\left(W_2 \ f\left(W_1 x_i \right) \right) - y_i \|_2^2 \ , \end{align} with respect to $...
0
votes
0answers
55 views

logistic regression without $L_2$-regularization does not have optimum?

I have a question related to machine learning. Consider the case when in the problem of binary classification the training set is linearly separable. How to show that in this case the optimization ...
0
votes
1answer
63 views

$K(x,x')=\exp(-\|x-x'\|^{2})$ is positive definite kernel [closed]

How to show that the function $K(x,x')=\exp(-\|x-x'\|^{2})$ is positive definite kernel?
0
votes
0answers
15 views

eigenfunction representation with spline: show that coefficients fall faster than order of eigenvalues

I'm trying to understand the Proof of Theorem 3 in Bühlmann & Yu 2003 (Boosting with the $L_2$-Loss). The paper considers some projection matrix $S$ corresponding to a smoothing spline of degree $...
2
votes
1answer
28 views

A class of sets “picking out” a set

The idea of sets being shattered comes up a lot in statistical learning theory with applications to VC dimension. Before learning about this, I am trying to understand the following definition. ...
0
votes
0answers
45 views

Perpendicular distance from a hyperplane

Let the hyperplane equation be $\theta^Tx + \theta_0 = 0.$ Let p be any point. Find the signed perpendicular distance between the point and the hyperplane. (Answer in terms of $\theta^Tx$, $\theta_0$, ...
0
votes
0answers
17 views

Matrix Derivatives In Back-propagation: Matrix times vector

I am currently performing back-propagation on a neural network by hand and am perform derivatives on matrices. How would I calculate $\frac{d}{dh_a}(W_xX + W_hh_a + h_bh_a)$ where $W_x$ is 3x3 matrix, ...
1
vote
0answers
22 views

Strange vector valued integral notation

Let $f:\mathbb R^d\to\mathbb R$ be Lebesgue integrable. How would we define the integral $$I=\int_\mathbb{R^d}xf(x)dx$$ when we require that the integral is itself an element of $\mathbb R^d$? I came ...
0
votes
0answers
23 views

what is fuzzy svm?

I have to solve this question for my homework but I don't get how to formulate svm to FSVM. can someone please guide me? What is your idea to have a model of SVM classifier in which instances can ...
0
votes
1answer
23 views

Convexity of 0-1 loss

Problem The 0-1 loss is defined as $\ell(h(\mathbf{x}), y) = 1(h(\mathbf{x} \neq y)$. How could I show the convexity of this loss function? What I Have Done I tried to verify the Jensen's ...
0
votes
1answer
18 views

Determining Bayesian Classifier

Consider a 2-class Pattern Recognition problem with feature vectors in $R^2$. The class conditional density for class-I is uniform over $[1, 3]×[1, 3]$ and that for class-II is uniform over $[2, 4] × [...
0
votes
0answers
31 views

Why people like Hoeffding's inequality more than CLT?

I am reading some paper in information theory and machine learning, and I found that many people like to use the Hoeffding's inequality rather than the CLT. I know the Lindeberg Feller CLT can also be ...
0
votes
0answers
18 views

Average squared dot product between 2 distinct unit vectors.

Imagine you have to choose $N$ unit vectors in $D$ dimensional space, what is the lower bound of average squared dot product between all 2 different vectors? [Optional] My Solutions for Special Cases ...
1
vote
1answer
31 views

Derivative of autoencoder

Problem $$\nabla_{\mathbf{W}} \mathcal{L}(\mathbf{W})=\frac{1}{2}\Vert \mathbf{W}^T\mathbf{Wx} - \mathbf{x}\Vert_2 ^2$$ where $\mathbf{W} \in \mathbb{R}^{m\times n}\ (m < n)$. What I Have Done ...
0
votes
2answers
26 views

Computing the probabilty in a binary classification problem [closed]

I'm not sure how I should go about this question. I've tried looking through my lecture notes but can't seem to find any way of figuring out this question question link
0
votes
1answer
24 views

derivative for finite positive integral K

I can't figure out what this question is asking for, Compute the derivative with respect to x of the function $$ f(x) = \log\left(\sum_{k = 1}^K \exp(k x^k)\right)\:\text{ for finite, ...