Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [machine-learning]

How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

1
vote
0answers
8 views

The loss function of stacked autoencoder

For a stacked autoencoder as when we train this autoencoder with loses function assume the vector at layer $i$ is $x_i$. Our loss function should be only $\|x_5 - x_1\|^2$ or all of loss in ...
1
vote
0answers
8 views

How to form a weighted index using observations from each data columns

I have a data set, which I have created by aggregating at top hierarchy level. My data is based on user's statistics in using a website for about an year. Some of the dimension which I have captured ...
0
votes
0answers
13 views

connection between RKHS norm and fourier transform

I've seen it stated that norms of some reproducing kernel hilbert spaces can be written in terms of fourier transforms, and this is often used to argue that a higher RKHS norm implies a less smooth ...
0
votes
0answers
27 views

How to calculate the partial derivative of first hidden layer in a neural network with 2 hidden layers

First off, this video explains how to retrieve the partial derivative of the neurons in the second hidden layer, but I am a little unsure if my calculations on how to retrieve the partial derivative ...
0
votes
0answers
22 views

hinge loss vs. square of hinge loss components

We can define the hinge loss to be $$ L(y,t) = \max\{0,1-yt\} $$ We can also have a variation such that the loss now becomes: $$ L(y,t) = \max\{0,(1-yt)^2 \} $$ When would you want to use one over ...
1
vote
0answers
25 views

Notation of expectation $\mathbb{E}_{x\,\sim\,p(x)}\left[f(x)\right]$

I want to study some topics in machine learning and the author often uses this notation $\mathbb{E}_{x\,\sim\,p(x)}\left[f(x)\right]$. I just wanted to check if this is the same as (for continuous ...
0
votes
0answers
5 views

High-dimensional classification of isotropic Gaussian categories

I'm trying to figure out if data from two isotropic Gaussian toy classes are harder or easier to classify in higher dimensions. If the inter-class-center distance is held constant, does the likelihood ...
2
votes
1answer
84 views

Second-Order Taylor Series Terms In Gradient Descent

My machine learning textbook states the following when discussing second-order Taylor series approximations in the context of Gradient descent: The (directional) second derivative tells us how well ...
0
votes
0answers
28 views

First-Visit vs Every-Visit Monte Carlo

I have recently been looking into reinforcement learning. For this, I have been reading the famous book by Sutton, but there is something I do not fully understand yet. For Monte-Carlo learning, we ...
0
votes
0answers
20 views

Does the theory of Gittins Indices solve the Multi-armed Bandit problem?

For example, both Wikipedia and Reinforcement Learning: An Introduction (page 33) seem to claim as much, which would suggest that the problem has been solved for over 40 years. However, doing as ...
3
votes
0answers
19 views

Recommended book on “Dynamic Model Learning” related to “System Identification”?

I need to take a course on so-called "Dynamic Model Learning" related to "System Identification" + "Machine Learning stuff". But my backrgound is more on Computer Science and Electrical Engineering ...
0
votes
1answer
36 views

Proving proposition regarding Rayleigh quotients?

I am having some trouble understanding what happens towards the bottom of this image. It is my understanding that $\mathbf{y}$ is a vector of length $1$; I can't quite see how this can be converted to ...
2
votes
1answer
38 views

regression optimisation problem with $l_1$ and $l_2$ norms for $x \in \Bbb R^n$

I'm trying to solve an optimization problem: $$\text{argmin}_{x \in \Bbb R^n}~ f(x),~ f(x) = ||x - a||_2 ^2 + \lambda ||x||_1,~ \lambda>0.$$ Any thoughts on how to solve it? Thanks in advance!
1
vote
0answers
21 views

On Applying the Strong Law of Large Numbers in a prediction model accuracy metric

During the last few weeks I've been having a light discussion with some peers at work about the applicability of the Strong Law of Large Numbers (SLLN) to a certain batch of data. Everybody mantains ...
0
votes
0answers
8 views

SVM Lagrangian Duality confusion

The following proof exists for the SVM Duality Primal Solution (http://cs229.stanford.edu/notes/cs229-notes3.pdf) Given: And I am unable to understand how we get suddenly, these i and j terms. ...
0
votes
0answers
7 views

SVM Kernels - $O(n)$ vs $(On^2)$ runtime

Referenced from Page 14: http://cs229.stanford.edu/notes/cs229-notes3.pdf I have the following function which I want to use in my SVM Kernel: We can define our kernel to be, where The article then ...
1
vote
0answers
28 views

Prove that two statements are equivalent for two nodes X and Y in an acyclic directed graph G

I'm taking a Probabilistic Graphical Model course and I got the following question: Prove that the following statements are equivalent for two nodes X and Y in an acyclic directed graph G: X and Y ...
1
vote
0answers
39 views

Binary coding of Numbers with Minimum Hamming Distance

I am looking for binary encoding of a set of integers that satisfy the following two properties: The number of 1s in the larger numbers is larger. The hamming distance between the encoding of two ...
0
votes
0answers
29 views

Does the Gaussian RKHS contain a nonzero constant function?

I am working with the reproducing kernel Hilbert space $\mathcal{H}$ of functions generated by the Gaussian kernel $k(x,y)=\exp{\left(-\frac{1}{2\sigma^2}|x-y|^2\right)}$ over an interval $[-1,1]$. Is ...
0
votes
1answer
52 views

Proving convexity of the negative log complementary probability: $-\log\left(1 - \frac{\exp(x_i)}{ \sum_j \exp(x_j)}\right)$

I am familiar with the convexity proof for \begin{align} f_i(x) &= -\log\left(p_i(x)\right) = -\log\left(\frac{\exp(x_i)}{ \sum_j \exp(x_j)}\right) = \log\left(\sum_j \exp(x_j)\right) -x_i. \end{...
2
votes
0answers
37 views

Absolute sum of partitioned Rademacher variables

$ \newcommand{\E}{\mathop{\mathbb{E}}} $ Hi, this is the first time I post a question here, so I'd be glad to have comments to make it better. So here it goes. The problem I am looking to ...
2
votes
0answers
18 views

Convert a vector of distances to a normalized vector of similarities

I'm struggling to find a way to solve this problem. I have derived a $m \times n$ matrix containing in each row the Mahalanobis distance from a certain centroid. So at the end I have $m$ rows each ...
0
votes
0answers
19 views

Training set using a cost function

The cost function is: $$ J(\Theta_1) = \frac{1}{2m} \sum_{i=1}^m \left( h\Theta(x^i)-y^i \right)^2 $$ We have a data set with $m=3$ examples. What is J(0)? Now I tried the following but it didn't ...
0
votes
1answer
26 views

Can SVM hyperplane always separate data?

According to this post SVM problem is a convex problem with convex constraints. Now what I am really struggling to understand if SVM is a convex problem, can we draw separating hyperplane for every ...
0
votes
0answers
61 views

Correct interpretation of the Bayesian rule when having a mix of (i) a joint and (ii) a conditional probability

In the Machine Learning Book by Tom Mitchell https://www.cs.ubbcluj.ro/~gabis/ml/ml-books/McGrawHill%20-%20Machine%20Learning%20-Tom%20Mitchell.pdf, Equation (6.8) is given as (pages 168-169) $$ P(D \...
0
votes
0answers
12 views

Relation between v(s) and q(s,a) in a Markov Decision Process?

I was solving questions related to backup diagrams from Reinforcement Learning: An Introduction by Barto and Sutton. Are these 4 equations mathematically correct ? Are there any shortcomings in terms ...
0
votes
0answers
13 views

Designing perceptron for given problem

I encountered following problem in one of the paper: Consider 2-class PR problems with n Boolean features. Consider two specific classification tasks specified by the following: (i) a feature ...
0
votes
0answers
19 views

Substitute a Fully-Connected layer with 1x1 Conv Layer

I am following Andrew Ng's class on ConvNets and I don't get the part where we try to replace a FC layer with a Conv layer and say that both are mathematically equivalent. Here is a picture from the ...
0
votes
0answers
33 views

Bayesian curve fitting, Biship Macchine learning

I have started to read Bishop's book but I can't understand a formula. I am at the introduction. Formula is the 1.68. I would like to understand its demostration. The author says he uses sum and ...
0
votes
0answers
32 views

Bishop - Pattern Recognition & Machine Learning, Exercise 1.4

I'm working on exercise 1.4 in Bishops Pattern Recognition & Machine Learning book. This exercise is about probability densities. I've two questions about this exercise. At first I don't ...
0
votes
1answer
27 views

Unsure of how to proceed through Trace Property

I'm having difficulty figuring out how to derive the following from Andrew Ng's CS229 lecture notes. $$\nabla_A \textrm{Tr } ABA^{T}C = CAB + C^TAB^T $$ where $\textrm{Tr }$ is the trace operator ...
0
votes
2answers
31 views

Clarification of Textbook Explanation of Hessian Matrix, Directional Second Derivative, and Eigenvalues/Eigenvectors

My machine learning textbook has the following section on the Hessian matrix: When our function has multiple input dimensions, there are many second derivatives. These derivatives can be collected ...
0
votes
1answer
43 views

Bishop - Pattern Recognition & Machine Learning, Exercise 1.2

I'm working on exercise 1.2 (Curve Fitting Problem) of Bishops Pattern Recognition and Machine Learning Book. You should write the linear equations, satisfied by the coefficients, that minimize the ...
0
votes
0answers
37 views

Difficulty Understanding the Last Step of Principal Component Analysis (PCA)

I'm having a tough time understanding the last "step" of Principal Component Analysis. Specifically, I'm trying to understand the following. $$Z^* = ZP^*$$ In the step above, you take the ...
2
votes
2answers
82 views

How to feed data into a polynomial basis function regression (unregularized) for degree n?

We know that polynomial base function Models is: $$t = \sum_{i=0}^n w^T\phi_j(x) = w_0*\phi_0(x) + w_1*\phi_1(x) + w_2*\phi_2(x)+ ....$$ $$\phi_j(x) = x^j$$ Problem: I am not sure how to pass the ...
1
vote
0answers
17 views

Minkowski distance to separate two classes

What Minkowski distance: $d_p(x,y) = (\sum^N_{n=1}|x_n-y_n|^p)^{1/p}$ is appropriate for separating two classes where class 1 is defined as a square and the second class is defined as all the points ...
1
vote
1answer
62 views

Why is there so little data-driven fluid dynamics research?

With the insane growth of data science, I notice that there's hardly any data-driven fluid dynamics research out there. What could explain this phenomenon? I have heard that fluid dynamicists ...
0
votes
0answers
18 views

Description of Convolution Commutative Property in Deep Learning Book

I am reading the Deep Learning Book chapter on convolutional neural networks (https://www.deeplearningbook.org/contents/convnets.html). On page 328 the authors outline the commutative property of ...
0
votes
0answers
43 views

No-free-lunch theorem and finite hypothesis class

I know that a finite hypothesis class $\mathcal{H}$ is PAC-learnable. Let's say I take a binary classification with a finite set of input : $\mathcal{X}$ finite and $\mathcal{Y}=\{0;1\}$. Then the ...
0
votes
0answers
12 views

Considering heteroskedasticity in Cp approach to adjusting training error rate in regression

I have been introduced to $C_p$ as a way to adjust the training error rate to account for bias due to overfitting regression models. $C_p$ is defined as such: $C_p = \frac{1}{n} (RSS + 2d\hat{\sigma}...
0
votes
1answer
54 views

What does it mean to integrate a parameter vector?

A naive question: I know a bout integrating a scalar function over values of x $$\int f(x)dx $$ and I'm trying to learn Machine learning now, however I face integrals integrating parameter vector w ...
0
votes
0answers
38 views

An Event with Unit Density Still Has Zero Information, Despite Not Being an Event That Is Guaranteed to Occur.

My textbook says the following in a section on information theory: The basic intuition behind information theory is that learning that an unlikely event has occurred is more informative than ...
0
votes
1answer
34 views

How to map quadratic programming formulation to dual soft margin SVM

I am trying to use quadratic programming for SVM and I am confused about how to map SVM formulation to quadratic programming formulation given in CVXOPT (Python package). This is what CVXOPT gives us ...
0
votes
0answers
18 views

How are the values found for the bias and variance of kNN from this website example?

At the website http://scott.fortmann-roe.com/docs/BiasVariance.html, in section 3.3, an explicit analytical expression is given for the bias variance trade off equation. Can someone please explain how ...
0
votes
0answers
32 views

The Bayes optimal predictor is optimal

I'm attending a lecture in Machine Learning Theory, and since I have almost no background in Probability Theory, I'm having problems with the following exercise: Let $X$ be a set and $Y = \{0, 1\}$ ...
5
votes
2answers
60 views

Linear Function: f(ax + by) = af(x) + bf(y) vs y = ax+b. [duplicate]

I studied a linear function definition in our Machine Learning course: f(ax + by) = af(x) + bf(y) Using this definition, we can prove that y = ax + b is NOT a ...
0
votes
0answers
11 views

Sample complexity in learning interval problem

Let the target concept class be $C=\{[a,b]:a<b,a,b\in\mathbb{R}\}$ and the hypotheses class $H=C$ and the version space be $VS_{H,D}$. $c\in C$ lables the points inside the interval positive and ...
1
vote
1answer
41 views

deriving the formula for multinomial linear regression

I am trying to understand why $\theta_{MLE} = (X^TX)^{-1}X^Ty$ for multinomial linear regression in which we have the Frobenius norm for $min||y-X\theta||^2$ Looking at this tutorial, I have hard time ...
1
vote
1answer
24 views

Intuitive difference between cosine similarity and bilinear similarity

Given a pair of strings in vector form $(s_i,s_j)$, I can find cosine similarity of pairs as follows: $cosine(s_i,s_j)=s_i.*s_j / (\|s_i\|\|s_j\|)$ Similarly, bilinear similarity is defined as: ...
1
vote
1answer
25 views

Question on derivation of probability proportion in Deep Learning Book

I am working through the Deep Learning Book, I am currently on the regularization chapter (https://www.deeplearningbook.org/contents/regularization.html). My question concerns the third step in the ...