Questions tagged [machine-learning]
How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?
3,316
questions
2
votes
0
answers
17
views
Expected value and variance of Sigmoid and SiLU on a normally distributed random variable for variational approximation
I am trying to apply Assumed Density Filtering (ADF) according to the paper Lightweight Probabilistic Deep Networks to my own model, and I need to implement the variational approximation layer of ...
0
votes
0
answers
14
views
Bernstein type PAC bounds
I am reading Understanding Machine Learning: From Theory to Algorithms and I don't know how to prove the second part of their Lemma B.10 (page 427).
Specifically, let $S=\{(x_i,y_i)\}_{i=1}^m$ be a ...
0
votes
0
answers
18
views
Gradient Steepest Descent
In the book I am currently reading, the steepest descent is described as follows:
$$\min_{\mathbf{x}} \frac{1}{2}x'Qx - x'b$$
Let this quadratic problem be the initial position and Q must be positive ...
0
votes
0
answers
25
views
How to show that the Gaussian Process parameters decreases with more training data
The posterior mean prediction of a Gaussian Process is given by
$$\mu(x_*) = \sum_{i=1}^n\alpha_i k(\mathbf{x}_i,\mathbf{x}_*) $$ where
$$\alpha = (K + \sigma_n^2I)^{-1} \mathbf{y}$$
Can we show that $...
0
votes
1
answer
36
views
Comparing $R^2$ in multivariate regression
Given that $R^2$ from a linear regression of $Y$ on $X_1$ and $X_2$ is denoted as $R_ 1^2$ , and $R^2$ from first regressing Y on X_1 and then using the residuals to regress on $X_2$ is denoted as $...
2
votes
0
answers
51
views
Proving MGF Bound for squared weighted sum of Uniform(-1,1) random variables
In proving the Jonhson-Lindenstrauss Lemma with $\mathrm{Uniform[-1,1]}$ random projections I used the following fact which I have been trying to prove:
$$ \mathbb{E}[e^{\lambda(3Z(w)^2-1)}] \leq e^{2\...
0
votes
2
answers
57
views
How does hyperplane work?
The definition of a plane in $R^3$ is intuitive : $Ax + By + Cz = D$ , where $(A, B, C)$ is a normal vector to the plane and $D$ is some bias.
I can visualize it dividing the space into two halves (...
0
votes
1
answer
23
views
Logistic map type function with controlled steepness on either side
I am looking for a function that must have the following requirements:
$f(1) = f(-1) = 0$
$f(x) > 0, \forall x \in (-1,1)$
$f$ is differentiable.
Additionally, I would like it to be ...
0
votes
0
answers
23
views
Linear algebra/multivariable calculus books dedicated for ML/DL purposes [closed]
I am looking for a book that have good exercises with solutions which have direct implementation in ML/DL algorithms (I hope I am clear, like for example differentiating the mean square function etc......
0
votes
1
answer
40
views
Large Quadratic Program with Approximately Low Rank Structure [closed]
I want to solve a general, bound-constrained quadratic program with linear equality constraints:
$$
\min_x \frac{1}{2} x'Qx + c'x \quad \text{subject to} \quad \begin{cases}
Ax=b,\\
l \leq x \leq u
\...
1
vote
0
answers
30
views
Utility of the definition of non-uniform learnability
The question concerns the book Understanding Machine Learning by Shalev-Schwartz & Ben-David and the provided definition of non-uniformly learnability. Suppose $\mathcal{H}$ is non-uniformly ...
0
votes
0
answers
20
views
generating the graph of allowed moves in two-player games
I recently came up with a method of constructing the graph of allowed moves in two-player boardgames like tic-tac-toe or chess, and I would like to turn this into a research project with a goal of ...
0
votes
1
answer
22
views
What assumptions are needed to "un-marginalize" this conditional distribution?
I want to prove the identity:
$$p(y|D,x)=\int p(y|W,x)p(W|D)dW$$
using the conditional independences given by one of the following two graphical models:
I believe I have a proof that assumes model (A)...
0
votes
1
answer
31
views
Maximum Likelihood - Information Matrix Identity Derivation
I try to derive the information matrix equality for the Poisson distribution with the log-Likelihood:
$$\mathcal{L}(\lambda; x_1, x_2, \ldots, x_n) = \sum_{i=1}^{n} \left[-\lambda + x_i \log(\lambda) -...
0
votes
0
answers
15
views
Missing step in computing the derivative of a logged pdf
I'm having trouble following how the authors got from equation 1 to equation 2.
We have a score $\nabla \log p(y)$, measurement density, $p(y|x)$ (which is gaussian), and a posterior density $p(x|y)$. ...
0
votes
0
answers
18
views
Check My Solution -> Problem: Deriving Bias term in the context of Support Vector Machines given the Weight Vector.
Please check my solution/understanding of deriving the bias term in the context of Support Vector Machines.
Information given,
(1.) $t_i(\mathbf{w}^T\mathbf{x}_i + b)=1$
(2.) $\mathbf{w} = \sum_{i=1}...
1
vote
0
answers
40
views
Bold 1 without subscript - indicator function?
I'm trying to understand the notation I've found in a paper which doesn't seem to be defined within the text. I can't quite recreate it in MathJax but it looks something like:
$$\propto \Bbb{1}(\...
0
votes
1
answer
55
views
How to prove or disprove a function is Lipschitz continuity?
Because I am not majoring in math, I wonder if there is a standard approach to prove or disprove Lipschitz continuity.
In my case, I want to prove that the Mean Squared Error (MSE) loss function for ...
1
vote
1
answer
126
views
No free lunch theorem and understanding of distributions
I am currently studying "Understanding Machine Learning from Theory to Practice" written by Shai Shalev-Shwartz and Shai Ben-David. And i have trouble in understanding the presented proof of ...
1
vote
1
answer
53
views
Understanding Sutton's definition of the Projected Bellman Error
I am reading Richard Sutton's textbook Reinforcement Learning, chapter $11.4$, and I am confused by his definition of the Projected Bellman Error.
He defines a norm on value functions $v: S \mapsto \...
0
votes
2
answers
58
views
Logit Gradient/Hessian derivations
I'm trying to follow the algebra leading from the gradient function to the Hessian in Logistic Regression, but I can't quite understand where I have gone wrong.
I have the gradient function as:
$$
\...
1
vote
0
answers
61
views
Is every convex cone a subset of a half space?
I have come across a proof for this statement(link to paper at end), which however I do not understand. It makes use of Lemma 5.5, which I've also included.
Lemma 1. The interior of the complement of ...
1
vote
0
answers
32
views
Expression for derivative of neural net output w.r.t edge weight
This is a question on the mechanics of backpropogation derivatives in NNs. I've seen plenty of analysis on the derivative of a NN output with respect to a given input, but am unsure on how to compute ...
1
vote
0
answers
296
views
Calculating $\min(x_1,x_2)$ and $\max(x_1,x_2)$ using a two-layer neural network
Suppose that $\vec{x}\in\mathbb{R}^2$ is a vector and we want to find the minimum and the maximum of its components, using a two-layer neural network: $$\vec{y} = f_2(W_2f_1(W_1\vec{x}+b_1)+b_2), \ \ ...
3
votes
0
answers
64
views
A self-proof of Vapnik - Chervonenkis theorem
Theorem: For every $\varepsilon >0$, with the probability greater than $1-\varepsilon$
\begin{align*}
R_p(\hat{g}_{n,\mathcal{G}}) - R_{p}(g^*_{p,\mathcal{G}}) \le 2 \sqrt{\dfrac{2V_{\mathcal{G}...
-1
votes
0
answers
24
views
Infinity reward for unreachable state in reinforcement learning
I'm new here. So pardon me if my question is irrelevant. In Chapter 3 (page 49) of RL book by Sutton and Barto, the expected rewards for state-action-next-state of an MDP is defined as a three-...
0
votes
0
answers
6
views
Methods for Efficient Feature Aggregation Maintaining Prediction Accuracy
Given a high-dimensional dataset X containing potentially redundant features, how can we efficiently aggregate and/or select features to achieve accurate prediction of target variable Y while reducing ...
0
votes
1
answer
31
views
Why does a shift in the probability distribution over a label space naturally trigger a shift in the distribution over input space?
Can anyone explain this statement?
"Firstly, let’s define the input space as
X (sensory observations) and the label space as Y (semantic categories). The data distribution is represented
by the ...
0
votes
0
answers
26
views
When to use chi square law for confidence intervals with mahalanobis distance?
So right now i'm reading this paper: Distance-based detection of out-of-distribution silent failures for Covid-19 lung lesion segmentation, available here: https://arxiv.org/abs/2208.03217
In brief, ...
0
votes
0
answers
48
views
modified random walk problem
so there is a problem of a machine which has a screen that can show integers. starting at $0$ at each time step the machine either increases the number by $1$ or decrease it by $1$ (the probability of ...
1
vote
1
answer
29
views
Why do you take (1-the area under the ROC curve) when the area is less than .5?
I'm taking a course in which the ROC curve is specified by plotting points on an XY plane such that x is the false positive rate and y is the true positive rate at a certain threshold in binary ...
0
votes
0
answers
52
views
Find a solution of a classification problem
Find a solution $\hat{\varepsilon}$ of the following minimization problem
\begin{align*}
&\min_{\varepsilon \in \mathbb{R}^M} \sum_{h=1}^M \varepsilon^h \hat{R}^h+\beta \sum_{h=1}^M \...
0
votes
0
answers
43
views
How to minimise a function containing double integrals
I'm trying to minimise this function. $$R(h)=\int_{(x,y)\in(\chi,Y)}{L(h(x),y)dP_0(x,y)}$$
where L(z,y) is just $(z-y)^2$
I tried to differentiate wrt h and set the derivative to zero, but I'm getting ...
0
votes
0
answers
28
views
Question Intuition behind mathematics of activation function in a neural network. [migrated]
Does this intuition behind why an activation function is used in a neural network make sense mathematically :
For this example lets consider a fully connected (NOT CONVOLUTIONAL) network that ...
1
vote
2
answers
124
views
Relationship between $L^2$-distance and cosine similarity
Given a vector space $X$, the cosine-similarity can be defined as:
$$c(x,y)= \frac{\langle x, y\rangle}{\| x \| \| y \|} $$
and distance is:
$$d(x,y) = \| x - y\| $$
First, I expect to estimate some &...
0
votes
2
answers
134
views
Using Matrix Calculus in Backpropagation derivation. Rules of order of matmul and transposition when taking derivatives in different layouts.
I define a neural network with $L$ layers ($L-1$ hidden layers). The forward pass is as follows:
$$
\mathbf{a}^{(l)} = f(\mathbf{W}^{(l)}\mathbf{a}^{(l-1)}+\mathbf{b}^{(l)})
$$
Where $l \in [0,L]$ and ...
0
votes
1
answer
58
views
Understanding Friedman’s H-statistic
In "Interpretable Machine Learning: A Guide For Making Black Box Models Explainable", I found the following for Friedman's H-statistic:
$$PD_{jk}(x_j, x_k) = PD_j (x_j) + PD_k (x_k),$$
where ...
0
votes
0
answers
17
views
Gaussian Kernel outputs a 2*m feature map?
I am currently writing my masters thesis on the Double Descent Curve in Neural Networks and as I was doing some research, I came across the paper "On the Double Descent of Random Features Models ...
1
vote
0
answers
28
views
Is there a Closed-Form Solution for L2 Regularization Raised to a Power?
Recently, I came across a modified L2 regularization term as stated in the equation below, where $\gamma$ is a positive number.
$$
\lambda'(w^Tw)^\gamma
$$
I'm curious if a closed-form solution ...
0
votes
1
answer
122
views
Statistical framework for using MSE for linear classification
We learned in class to use a linear model to to predict a real target value y. We made the assumption that $$ y = w^Tx + w_0 + \epsilon $$ where $x$ is the input vector,$w$ is the vector of the ...
0
votes
0
answers
13
views
Metrics for document clustering with measure of synonyms
I asked this question on Data Science stack exchange, but didn't get any responses there.
I have a (finite) vocabulary which is a metric space, where the metric measures how antonymous the words are. ...
0
votes
1
answer
68
views
What are a set of example of tasks or problems where the Kolmogorov Complexity is Known -- ideally numerical values can be obtained?
Is there a machine learning task (or any task/problem) that one can by construction know the Kolmogorov Complexity (or minimum description length)? I know the Kolmogorov Complexity is uncomputable but ...
0
votes
0
answers
50
views
Posterior probabilities in a GMM
This is a statistics/probability question formulated in the context of machine learning (problem 6.17 in Bishop's 'Deep Learning' book). We are modelling the conditional distribution $p(\mathbf{t}|\...
0
votes
1
answer
65
views
Application of KKT-Theorem Does Not Give Expected Result
I've got this exercise for uni that I've been absolutely racking my brain over for the past couple hours, the problem goes as follows:
Let $\theta_1, \cdots, \theta_d \in (0, \infty)$. Consider the ...
0
votes
1
answer
52
views
Multiclass classification by hand - how to use gradient descent?
I am learning logistic regression. Having learnt something about binary classification, I came across this article on multiclass classification:
Given a set of $9$ training data with $2$-dimensional ...
3
votes
0
answers
40
views
Minimizaiton of $f\mapsto\int\frac12\|f\|^2+\nabla\cdot f\:{\rm d}\mu$, when $\mu$ is only given by i.i.d. samples
I know this question is quite vague, but I need some indication. I have a problem where I have a probability distribution $\mu$ on $\mathbb R^d$ and I want to find a differentiable function $f:\mathbb ...
3
votes
0
answers
100
views
How do I understand this aspect of Shapley Values?
I have read a few of the posts on here regarding Shapley values and have started to form an intuition surrounding it, especially in connection with explainability of ML models. However, I am still ...
1
vote
0
answers
35
views
When does the optimal model exist in learning theory?
In the context of learning theory, we usually have: data $(x,y)\sim P(x,y)$, with $x\in\mathcal{X}\subseteq\mathbb{R}^d$ and $y\in\mathcal{X}\subseteq\mathbb{R}^k$, a hypothesis class $\mathcal{F}\...
0
votes
1
answer
44
views
Why to take "uniformly distributed" random numbers in $[0,1]$ to get desired random variable in inverse transform method of synthetic data generation?
I am learning probability for ML and came across a concept called inverse transform method to generate artificial data that transforms the random numbers (probabilities) in range $[0,1]$ into actual ...
0
votes
0
answers
38
views
How to find the Lipschitz constant of the function below?
The function is $f(w)= \displaystyle\frac{1}{n} \displaystyle\sum_{i=1}^n\left ( N \left ( \displaystyle\sum_{j=1}^{7} x _{ij}w_j\right) - G_i\right )^2$ where N is the sigmoid function. w is a vector ...