The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [machine-learning]

How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

Filter by
Sorted by
Tagged with
-1
votes
0answers
28 views

What is the $75$th percentile of the data distribution? [on hold]

What would be the value at $75$th percentile for a data distribution series with mean $= 5$, variance $= 11.5$, skewness $= 13.5$, kurtosis = $224.5$, and $n = 4$?
1
vote
0answers
22 views

Lipschitz and $\mu$ constants of squared loss

Given $f(x)$ as follows and $x_i \sim U[-1,1]$ and i.i.d $$ f(x) = \frac{1}{2m}\sum_{i=1}^m (x-x_i)^2 $$ I'm trying to find the best constants $\mu$-convex and $L$-smooth. I think they're the ...
1
vote
0answers
10 views

How to count the maxwidth and layers' number of a feedforward neural network?

Definition $\phi$ is a feedforward neural network (FNN)$$\phi = W_L(\sigma(W_{L-1}(\sigma(\cdots > \sigma(W_1(x))\cdots))))$$ with affine linear maps $W_l:\mathbb{R}^{N_l-1}\to \mathbb{R}^{N_l},...
1
vote
0answers
14 views

Expected value of indicator variable PRML

I've been struggling trying to see the logic on how the author of a book arrives at an expected value. For reference, the book in question is Machine Learning by Bishop, Page 443, equation (9.38) and (...
0
votes
0answers
20 views

Logistic regression as a linear system [on hold]

The question Hi everyone, I have been thinking about the question b a while, but still cannot figure out how to solve the parameters with first order condition. Could someone help me with it? Thanks ...
0
votes
0answers
24 views

Probability and Optimization

I am reading a paper on optimization and came across the following formula: Now my questions are: Question 1: Does the integral of the expression inside the redbox evaluate to $p(\mathcal I_{t,0})$?...
0
votes
0answers
18 views

Optimization of a staircase surface

I am trying to find the minimum of some unexplored "staircase like" surface - image many Rastrigin functions side by side (some with a higher center and some with a lower center) but with flat parts ...
1
vote
1answer
22 views

How to plot value of loss function in time

Am working with linear regression $$\begin{bmatrix} y_1 \\ y_2 \\ ... \\ y_n \end{bmatrix} \approx \begin{bmatrix} x_{11} & x_{12} & ... & x_{1n} \\ x_{21} & x_{22} & ... & x_{...
0
votes
0answers
31 views

Is there any good transformation for piecewise constant functions?

I'm doing a machine learning task and my data looks like piecewise constant functions like this. I was wondering if I could transform the original data to make the learning faster or more efficient, ...
0
votes
1answer
20 views

how do we know these hyperplanes really seperate the data??

In the math behind Support Vector Machine : "Given a hyperplane H0 separating the dataset and satisfying:" w⋅x + b= 0 "We can select two others hyperplanes H1 and H2 which also separate the data ...
-3
votes
1answer
48 views

Are $A|B$ and $B$ independent events? [closed]

Suppose $A$ and $B$ are two dependent events, that is $P(A\cap B)>0$. We know that $P(A\cap B)=P(A|B)P(B)$. Is it true that $A|B$ and $B$ are independent? From my understanding, two events $X$ ...
0
votes
0answers
10 views

General method of finding the minimum of a function of multiple vectors?

For example, I want to find the max or min of $f(\vec{x},\vec{y})$. Do I just take gradient of both x and y separately? Do I concatenate x and y into z and then take the gradient by z?
1
vote
0answers
31 views

Matrix calculus for RNN equations

In the deep learning book we have the standard RNN with these equations. It calculates various derivatives, including one for W. I understand that: $1 - {(h^{(t)})}^2 $ is coming from the ...
3
votes
0answers
60 views
+50

What is the motivation for using cross-entropy to compare two probability vectors?

Define a "probability vector" to be a vector $p = (p_1,\ldots, p_K) \in \mathbb R^K$ whose components are nonnegative and which satisfies $\sum_{k=1}^K p_k = 1$. We can think of a probability vector ...
1
vote
1answer
102 views

How to Show that AdaBoost Weighted Error is Exactly 1/2

I am trying to prove that in an AdaBoost model $Y \rightarrow [-1,1]$ $err_t'= \frac{\sum_{i=1}^{N}w'_i1\{h_t(x^{(i)})\neq t^{(i)}\}}{\sum_{i=1}^{N}w'_i} = \frac{1}{2}$ here, $w_i' = w_i exp(-\alpha ...
1
vote
0answers
23 views

Derivative of Cross Entropy of Finite Discrete Random Variables

Consider the following definition of cross entropy on 2 independent finite discrete random variables $X$ and $Y$ with respective probability mass functions $p$ and $q$ defined from set $\{a_i|i=1,...,...
1
vote
2answers
38 views

proof: maximum likelihood and least square is pseudo-inverse

Why does does the following gradient equal transpose of phi? $\nabla_w \bigg[w^T\phi\bigg] = \phi^T$ instead of just phi? $\nabla_w \bigg[w^T\phi\bigg] = \phi$ as in minimizing sum-of-squares ...
2
votes
2answers
89 views

Solving constrained minimisation problem using unconstrained optimization of the generalized Lagrangian

My textbook, Deep Learning by Goodfellow, Bengio, and Courville, says the following in a section on constrained optimization: The Karush-Kuhn-Tucker (KKT) approach provides a very general solution ...
0
votes
0answers
34 views

Show Unbiased Learner

I want to show that $g_{\tau}(\mathbf{x}) = \mathbf{x}^T\hat{\beta}$ where $\hat{\beta} = \mathbf{X}^+\mathbf{y}$ and $\tau$ denotes a fixed training set is an unbiased learner, in the sense that: $$\...
0
votes
0answers
14 views

Choosing best expressions from all possible combinations on variables, unary operators and binary operators along with hyper parameters

I have a few financial variables of a stock universe like OHLC prices, volume, and other fundamentals with varying time-frequency. Using this set I'm creating an expression that gives the weights to ...
0
votes
0answers
13 views

Custom cross entropy loss function

I want to define custom cross entropy loss penalizing different class errors. Categorical cross entropy loss = $\sum_{i=1}^K y_i log(p_i)$ I want to give different weights to different prediction ...
0
votes
0answers
13 views

Choosing parallel hyperplanes in SVM

In formulation for cost function for SVM , we take the equation of parallel planes as W'x+b =1 and W'x+b =-1 but two parallel ...
-1
votes
0answers
23 views

Perturbation of a 1-rank matrix to give a full-rank matrix

For any matrix of the form $A=(x_1e,...,x_ne)^T$ where $x_1<...<x_n$ and $e^T=(1,...,1)$, we can find a matrix of the form $B=(b_1e,...,b_ne)$ such that Relu(A-B) is full rank. Indeed, if we ...
0
votes
0answers
8 views

How does Locally Weighted Regression work for test sets far outside the train set bound

I was following CS229 machine learning course where I came across the Locally Weighted Regression algorithm. We have to minimize $\sum\limits_{i}w^i(y^i - \theta^T x^i)$ and output $\theta^Tx$ ...
0
votes
1answer
29 views

Proof that gradient descent algorithm escapes saddle points exponentially

https://arxiv.org/pdf/1705.10412.pdf I was going through this paper, and understood the crux of it. But in appendix, the complete proof of it is given which was a bit tough for me mathematically. So ...
0
votes
0answers
11 views

What is the difference between a confounding variable and a latent variable in stats?

In statistics, a confounder (also confounding variable) is a variable that influences both the dependent variable and independent variable, causing a spurious association. Does it mean a confounder ...
1
vote
0answers
13 views

Proof to find C in isomap

So I am trying to study Isomaps for non-linear dimensionality reduction, and I cannot understand how we reach the formula for C as : $$C = − \frac{1}{2m}HD^{2}H$$ So we have the centering matrix, $$H ...
0
votes
0answers
16 views

probability of misclassifying using knn classifier

Given that we choose $n$ random labeled point $X_i$ in $R^m$ with labels in $\{1,-1\}$ as follow: so given a random label for $X_i$ with probability of $0.5$ of being either $1$ or $-1$. If the label ...
0
votes
0answers
28 views

Partial derivative of operator

Let $\epsilon \in \mathbb{R}$ and $f \in H$, with $H$ being a reproducing kernel Hilbert space. Define $$G(\epsilon, f) := 2\lambda f + \operatorname{E}_{P\epsilon}\left[D_3L(x,y,f(x))\Phi(x)\right]$$ ...
1
vote
0answers
14 views

Conditional Probability and KL Divergence

Let $\mathbb{P}_A$ denote the distribution conditioned on $A$ such that for any measurable set $C$, we have $\mathbb{P}_A(C)=\mathbb{P}(A\cap C)/\mathbb{P}(A)$. On page 80 of the citation, author ...
2
votes
1answer
100 views

Gradient-based optimization: A small change in the input obtains a corresponding change in the output

My textbook, Deep Learning by Goodfellow, Bengio, and Courville, says the following in a section on gradient-based optimization: The derivative of this function is denoted as $f'(x)$ or as $\dfrac{...
1
vote
1answer
16 views

Constrained optimization: Imposing norm constrain on input to find small solutions

My textbook, Deep Learning by Goodfellow, Bengio, and Courville, says the following in a section on constrained optimization: Sometimes we wish not only to maximize or minimize a function $f(\...
2
votes
1answer
25 views

Clarifying with notation

I try to implement this paper. It's about an instrument that provides explanations on how the Graph Neural Network makes its prediction. Reading this, I got the clear understanding of author idea ...
1
vote
1answer
23 views

For an estimate $\hat{f}$ of a regressor $f$, showing $\mathbb{E}[(f(x)-\hat{f}(x))^2]$ is equivalent to another expression.

In the context of regression for machine learning, suppose I have a function from an instance space $I$ to $\mathbb{R}$, say $f:I \rightarrow R$, and that I have an estimator $\hat{f}:I \rightarrow R$....
0
votes
0answers
13 views

Why is calculating the denominator of likelihood even a problem?

I'm reading up on Conditional Random Fields here (https://towardsdatascience.com/conditional-random-field-tutorial-in-pytorch-ca0d04499463) and the author basically says that CRF is basically a "trick"...
2
votes
1answer
102 views

Derivative of matrices product AXA^T with respect to A. (Plus result when A is a vector.)

I want to know how to find an expression for $$\frac{\partial (AXA^T)}{\partial A}$$ where no information is given a priori on the dimensions of $A$ and $X$. The question is related to machine ...
1
vote
1answer
50 views

Newton's method and gradient descent in deep learning

My textbook, Deep Learning by Goodfellow, Bengio, and Courville, says the following in a section on numerical computation: Newton's method is based on using a second-order Taylor series expansion ...
1
vote
1answer
28 views

Simplifying risk equation for 0/1 loss in machine learning

I'm currently studying machine learning using the textbook Introduction to Machine Learning 3e (Ethem Alpaydin, 2015) and had a question regarding the derivation of a particular equation. For those ...
1
vote
1answer
27 views

Step Activation function…why does it work?!

I enjoy watching a channel called "Coding Train" because he does his work very informative and with an energy I can only envy. He did a video on neural networks and I'm kinda stumped on why it works. ...
0
votes
0answers
8 views

Jacobian factor: linear vs. non-linear transform of probabilty density and position of mean

Consider a probability density $p_x(x)$ defined over a continuous variable x, and suppose that we make a non-linear change of variable using: $$x=g(y)$$ so that the density transforms according to ...
0
votes
0answers
34 views

Overdetermined system of nonlinear equations

The following system follows from a maximum likelihood estimation: $\alpha_1$ = $18c(\beta_2+\beta_3)^{-1}$ $\alpha_2$ = $5c(\beta_1+\beta_3)^{-1}$ $\alpha_3$ = $8c(\beta_1+\beta_2)^{-1}$ $\beta_1$...
0
votes
0answers
33 views

Prove the posterior of linear discriminant analysis admit Sigmoid form

As we all know, the Sigmoid Function is: 1/[1 + exp(-(wTx + b))] Is it possible that the constant "b" can be ignored? Here is the original question: Prove that in binary classification, the ...
1
vote
0answers
44 views

ML vs Math terminology

I read often in machine learning the termonilogy "smooth" and "differentiable" referring to a property to commonly used activation functions. I really struggle sometimes, because for example the ReLU ...
0
votes
0answers
43 views

Interview question: What are the chances of the user being Genuine?

This is an actual interview question.: A website has a system called The "findoo" that blocks brokers from using the website. It is empirically observed that the chances of a user of website is a ...
1
vote
2answers
30 views

Variance and E[X^2] of Dirichlet distribution

Definition of Dirichlet distribution: $$Dir(\vec{x}|\vec{a})=\frac{\Gamma(a_0)} {\Gamma(a_1)\Gamma(a_2)...\Gamma(a_M)}\prod_{k=1}^{M}x^{a_k - 1}$$ Where: $$\vec{x}=(x_1, x_2, ... x_M)^T$$ $$\vec{a}=(...
1
vote
1answer
19 views

Please explain what this notation means in K-means clustering.

I am reading up on K-means clustering. I understand the procedure to an extent (I think!), but I am looking to understand the mathematical formulation better. Given $m $ data points {$ x^1, x^2, ........
0
votes
0answers
26 views

Prove that expected value of the training error is equal to the true error.

Can't solve this exercise from Understanding of Machine Learning book . Can somebody help me with this proof please. Let $H$ be a class of binary classifiers over a domain $X$ . Let $D$ be an ...
0
votes
0answers
18 views

Orthogonality in Orthogonal Matching Pursuit

Why is this Algorithm called ORTHOGONAL Matching Pursuit. I know that the residue is changed in each iteration so that it is orthogonal to the features already found. Is there any way to prove this ...
1
vote
1answer
34 views

Measure of convexity for point clouds

I want to measure "how convex" a set of points $x_n\in\mathbb R^K, n=1\ldots N$ is. Both $K$ and $N$ are potentially large $N\gtrsim 10^6$, $K\gtrsim 10^3$. One way would be to compute the volume of ...
1
vote
0answers
19 views

Parzen density estimation

Given a collection of data points $(x_1, ..., x_n)$, we assume they are drawn from some distribution with known parameters (say normal). Parzen density estimate is defined as $p(x) = (1/(nh))*\sum_{i=...