Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [machine-learning]

How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

0
votes
0answers
5 views

Why can the best Fisher's linear discriminant vector be solved by $w_{lda}=S_W^{-1}(m_2-m_1)$?

Why can the best Fisher's linear discriminant vector be solved by $w_{lda}=S_W^{-1}(m_2-m_1)$? Background: https://www.cs.ccu.edu.tw/~wylin/publications/ieee_smc.pdf pages 11-12.
0
votes
0answers
14 views

Variance of the gradient function

I was reading the Xavier initialization technique used in deep learning. But I'm struct in the basic math used in the paper. For example for a dense artificial neural network using symmetric ...
0
votes
0answers
10 views

VC-dimension of parity functions

Consider the boolean hypercube $\{0,1\}^N$. For a set I $\subseteq$ {1,2,...N}, we define the parity function $h_I$ as follows. For a binary vector x = $(x_1, x_2, ...,x_N) \in \{0,1\}^N$, $h_I(x) = \...
0
votes
0answers
28 views

How to determine the convexity of multiple matrix variables function?

This formula is : $$f(W,V,B) =\|XW-V\|^2_F +\|Y-VB\|^2_F +\operatorname{tr}(V'LV) +2\operatorname{tr}(W'DW),$$ where $X$, $Y$ are constant matrices and $L$ is constant laplace matrix. Suppose $D$ is a ...
0
votes
0answers
18 views

time series data : Predict $Y$ with $X$, where $X$ is dependent on $Y$

Let $Y$ be a target variable which you want to predict on using $X$ (e.g with a statistical model), where $X,Y\in \mathbb{R}$. You are given data which looks like this : $$ data_t = (X_t, Y_t), \ t\...
0
votes
0answers
18 views

How can we get the equations for calculating mean, convariace, mixing probability value for Gaussian Mixture Model with EM algorithm?

I am currently learning EM algorithm and get some trouble with understanding EM algorithm for Gaussian Mixture Model. In the text book, it is written as the screenshot: (the link below) https://i....
1
vote
0answers
21 views

Mirror descent on a 1-ball

I have been recently reading about mirror descent, which essentially generalizes gradient descent to non-Euclidean spaces. Nearly every reference I find on this subject gives the same example for ...
1
vote
0answers
16 views

Convert Expected values of low frequency data to high frequency.

I have a frequency data (observations separated by 1hours) for daily sales of Product ‘A’ and Product ‘B’. 30,000 days of observations are recorded. In the data, the expected sales(Exp_A, Exp_B) ...
0
votes
0answers
16 views

How does the mean absolute error related to the relative error from log-transformed labels?

In machine learning we sometimes log-transform labels (or features). My understanding is that optimizing for the mean absolute (or squared) error using log-transformed labels optimizes for the ...
3
votes
1answer
21 views

Finding a smaller dataset with a similar covariance matrix

I'm interested in solving the following problem: Given a dataset $\mathcal{X} = \{x_i\}_{i=1}^n\subset\mathbb{R}^p$, find a smaller set $\mathcal{Y} = \{y_i\}_{i=1}^k\subset{\mathbb{R}^p}$ whose ...
0
votes
1answer
15 views

Linearly independence after feature embedding

Problem If feature vectors $\mathbf{x}_1, \mathbf{x}_2,\cdots,\mathbf{x}_m$ are linearly independent, argue whether or not their embedding $\psi(\mathbf{x}_1), \psi(\mathbf{x}_2),\cdots,\psi(\mathbf{...
0
votes
0answers
17 views

Proof of the hinge loss being 1-Lipschitz

Page 7 of https://web.stanford.edu/class/cs229t/scribe_notes/10_10_final.pdf I've tried finding a proof online, but haven't been able to find it. In the notes above which are provided as part of ...
0
votes
0answers
18 views

A research paper presents an equation for 'correlation distance between between two vectors'. But I cannot find information its derivation.

Hallo Mathematics StackExchange, I currently trying to pick apart a research paper titled 'Accounting for Label Unscertainty in Machine Learning for Detection of Acute Respiratory Distress Syndrome' (...
0
votes
0answers
6 views

Confusion about the notation of Rademacher Complexity

The online book: https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf In "Understanding Machine Learning:From Theory to Algorithms" by ...
0
votes
0answers
17 views

Help with Forward Stepwise Selection Algorithm

Below is a simplified forward stepwise selection algorithm and I am trying to understand the logic in step 2. Algorithm: Let $M_0$ denote the null model, which contains no predictors. For For $k=0,.....
1
vote
1answer
42 views

Piece-wise quadratic function [closed]

How can I find a minimum of a piece-wise quadratic function? (minorant of a set of quadratic functions) An example of this will be appreciated.
0
votes
1answer
6 views

Why having the loss matrix to be 1 - identity function yields a loss of $1-p(C_l|x)$ ? (Bishop exercise 1.24)

In the solution for exercise 1.24 in Bishop's PRML solutions book I see the following statement: For a loss matrix $L_{kj}=1-I_{kj}$ we have $\sum_kL_{kl}P(C_k|x)=1-p(C_l|x)$ Now I understand ...
1
vote
1answer
56 views

How to calculate $\frac{\partial\Theta}{\partial L}$ if I know $\frac{\partial L}{\partial\Theta}$?

How to calculate $\frac{\partial\Theta}{\partial L}$ if I know $\frac{\partial L}{\partial\Theta}$? Suppose I have a halved sum of squared errors loss: $$L(\Theta)=\frac{1}{2}\sum^{M}(y-h(X\circ\...
1
vote
0answers
19 views

A slight confusion about a particular step in Euclidean Transformations (Computer Vision)

On page 392 of Computer vision by Simon Prince (http://web4.cs.ucl.ac.uk/staff/s.prince/book/book.pdf), equation 15.2 has the following expression for the Euclidean transformation in homogeneous ...
0
votes
1answer
23 views

Convergence of Linear Neural Networks in the Easiest Framework

EDIT: Thanks to the first answer I'm lighting some assumptions: I'm trying to understand the basics of machine learning, and I have this theoretical question: I have a one layer linear neural ...
1
vote
0answers
27 views

Why is the Frank-Wolfe algorithm projection-free while gradient descent isn't?

While reading this article about the Frank-Wolfe algorithm, I did not understand why the Frank-Wolfe algorithm is projection-free, while the gradient descent is not. I think the problem is, that I do ...
0
votes
1answer
129 views

How to prove a matrix function is convex or nonconvex?

I have a function of three matrix variables. But now, the authors fix two of them, then update one, and I cannot understand how this function is convex in each iteration in the paper. This formula is ...
0
votes
1answer
33 views

Hessian-vector products

Can someone explain why this is true? $$g(x + \Delta x) = g(x) + H(x) \Delta x$$ where g is the gradient of function f(x) with respect to x, and H is the hessian of f(x) with respect to x. I would ...
1
vote
1answer
31 views

Confused by Kullback-Leibler on conditional probability distributions

I understand the Kullback-Leibler divergence well enough when it comes to a probability distribution over a single variable. However, I'm currently trying to teach myself variational methods and the ...
0
votes
1answer
32 views

What is the use of coefficient in in Regression

What is the meaning of coefficient values in Machine Learning. After I print model.print_summary() It shows, coefficient values of for each column. But I really ...
0
votes
0answers
11 views

Linear Discriminant Analysis project data on sum of eigenvectors

I know in Linear Discriminant Analysis (LDA) we project the data on a specific eigenvector or a matrix of eigenvectors to reduce the dimensionality of the data and separate classes. But what happens ...
2
votes
0answers
34 views

How to choose an overcomplete dictionary for dictionary learning in sparse coding?

Problem: I'm trying to understand how to choose the dictionary matrix in this paper http://yann.lecun.com/exdb/publis/pdf/gregor-icml-10.pdf. The paper is about Sparse Coding and trained Algorithms ...
0
votes
1answer
44 views

Struggling with Bayes network

Im in a machine learning course and bayes networks was presented in such an abstract way I find it really difficult to understand how to use it. And all examples I can find, the final numbers seem ...
1
vote
0answers
32 views

What does it essentially mean if the neural network has convex error surface?

Suppose if I am building a Linear Regression model with one fully connected layer and a sigmoid with minimizing mean squared error as objective. I understand that this network has a convex error ...
1
vote
2answers
45 views

What does this expected value notation mean?

From: Learning from Data, section 2.3.1 - Bias and Variance: Let $f : X \rightarrow Y$, and let $D=\{(x,y=f(x)) : x \in A \subseteq X\}$ where each $x \in A$ is chosen independently with distribution ...
-1
votes
0answers
15 views

Is class of multivariate sign functions VC-class?

For $\theta \in\mathbb{R}^n$ define $f_{\theta}:\mathbb{R}^n\rightarrow \mathbb{R}$ as $f_{\theta}(x)=\frac{\theta'x}{\|x\|_2}$ if $x\neq 0$ and $f_{\theta}(0)=0$. Is the collection of functions $\...
0
votes
0answers
13 views

Is it possible to output vectors/scalars from a neural network that are of a different type than the input

I have just begun researching convolutional neural networks on images and I see that they are useful for processes such as feature recognition and de noising etc. as they apply transformations to the ...
0
votes
0answers
42 views

uniform Effect of K-means Clustering

In the following link is discussed the uniform Effect of K-means Clustering: https://www.springer.com/cda/content/document/cda_downloaddocument/9783642298066-c2.pdf?SGWID=0-0-45-1338325-...
1
vote
0answers
19 views

Good reference for manifold learning

There are some references about manifold learning but I couldn't understand this theory from them properly. Is there a good reference about this topic? Thanks in advance.
1
vote
0answers
49 views

Is Banach fixed point theorem a necessary and sufficient condition for the existence of a fixed point

Banach fixed point theorem requires a contraction mapping from a metric space into itself, but when I was learning some machine learning algorithms, some questions rise above: k-means is an algorithm ...
0
votes
0answers
13 views

VC dimension bound of a union of mapping hypothesis sets

Consider the following feature transform, which maps a d-dimensional $x$ to a one-dimension $z$, keeping only the $k$th coordinate of $x$ $\phi_{(k)}(x)=(1,x_k)$ Let $H_k$ be the set of perceptrons ...
0
votes
0answers
16 views

Applying the strong law of large numbers to Random Forests

I'm trying to understand the proof in Appendix I from Random Forests (Breiman 2001) for convergence [https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf] . I am having difficulty adapting the ...
0
votes
1answer
16 views

Equivalent forms for a product notation

Context: See "2 Hoeffding’s Inequality" in : http://www.stat.cmu.edu/~larry/=stat705/Lecture2.pdf My particular question arises within 'section 2 Hoeffding's Inequality' is: $$ e^{-tn\varepsilon }\...
0
votes
0answers
9 views

Why is the matrix $L_{1}$ norm of this inverse matrix bounded by $\rho^{-1}p$?

Let $\Sigma_{n}$ be the sample covariance of size $p\times p$, and define $\Sigma_{n, \rho}=\Sigma_{n}+\rho I$ with $\rho>0$, then the matrix $L_{1}$ norm $$\| \Sigma_{n,p}^{-1}\|_{1}\le \rho^{-1}p$...
2
votes
0answers
10 views

What are the connections (if any) between “kernel” in “kernel density estimation,” “kernel of a matrix,” and “kernel method”?

I'm getting my understanding of kernel density estimation from pages 6-7 of this PDF. If there are conceptual relationships between the "kernels" in each of these topics, I'd like to understand them.
2
votes
0answers
12 views

SVM: Are all the support vectors necessarily used to construct the weights

I know what exactly the SVM is and very clear about the principles and algorithms. But I'm curious about are all support vectors necessarily used in constructing weights $w$. Let $w^Tx_i + b$ ...
0
votes
0answers
9 views

Why is the first derivative of a scalar-valued function with respect to a vector its gradient?

To train a word2vec model (vector representations of word meanings) the following expression needs to be optimized: $$p(\text{context words}| \text{current word}) = \frac{\exp(u_o^\top v_c)}{\sum_{...
0
votes
0answers
17 views

Designing a PDA to accept $wa^{n_a(w)}$ where $w$ is over $\{a,b\}$ and does not contain the string $bbb$.

I'm very new to this topic! I see that the PDA must push $a$ on the stack until after the last $b$ when it must pop $a$'s after that. I know that I'm looking to make sure the $a$ stack is empty at the ...
0
votes
0answers
11 views

Lipschitzness on a space of Turing Machines

I am trying to prove that $l$ is Lipschitz bounded and convex on the set $Z$ of all turing machines For some $h \in (0,1)$ and $T \in Z$ $\ell(h,T) = h \ell(0, T) + (1-h)\ell(1,T)$ where $\ell(0,T)$ ...
0
votes
0answers
10 views

Why transpose the weight vector in Perceptron Convergence Theorem

We have a training set D with a bunch of input vectors x and desired output of y. We are aiming to make a perceptron that can separate the input vectors x in accordance with their desired output y. ...
1
vote
0answers
16 views

Encouraging sparse output in optimization problem

I am trying to define an optimization problem: $$ \min_\theta \sum_{x\in X} L(x, \theta, f_\theta(x),\delta_\theta(x)) + \lambda_s S(\delta_\theta(x)) $$ where $X$ is the dataset, $\theta$ are the ...
1
vote
0answers
25 views

Elastic Net as LASSO

Good evening everybody, I need help with an excercise on Regularised Regression. What I need to do is turn an Elastic-net problem: \begin{equation} argmin_\omega \Vert y-\Phi(x)^T \omega \Vert_2^2 + ...
0
votes
1answer
20 views

Finding lipschitz bound for $F(x_1,\ldots,x_n) = (\tanh(x_1),\dots,\tanh(x_n))$

Given a vector-valued function $F(x) = (f_1(x),\ldots,f_m(x))$, where $x = (x_1,\ldots,x_n)$. Taking for example uniformly $f_j(x)=\tanh(x)$, how can I prove that $\lVert F(x) - F(y)\rVert_2 \leq\...
0
votes
0answers
7 views

Loss function : finding the criterion for which a given solution is the optimal classifier

For a binary classification problem, let $\eta(x) = P[Y=1 \mid x],$ and, for a given classifier $g$, we define the asymmetric cost : $$ L(g) = P[g(X)=0, Y=1] +\lambda P[g(X)=1,Y=0]$$ For this cost, ...
0
votes
0answers
19 views

Bayesian Interpretation for Ridge Regression and the Lasso

I'm learning the book "Introduction to Statistical Learning" and in the Chapter 6 about "Linear Model Selection and Regularization", there is a small part about "Bayesian Interpretation for Ridge ...