Questions tagged [machine-learning]
How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?
3,215
questions
0
votes
1
answer
29
views
What is the growth function of the hypothesis set of linear classifiers on $\mathbb{R}^2$?
Recently I've been studying machine learning. I want to find out the growth function of the following hypothesis set:
$$
\mathcal{H}=\left\{h_\mathbf{w}\equiv\mbox{sign}\big(\langle\mathbf{w}, \cdot\...
- 2,196
0
votes
0
answers
27
views
differential-privacy: show $\epsilon$ -differentially privacy
In this problem we consider a sensitive dataset $x \in \{−1, 1\}^n$. We consider the bounded setting where neighboring n-dimensional datasets differ in one coordinate. $A$ mechanism is available that ...
- 656
-2
votes
1
answer
24
views
When is my lightgbm going to find cut points in random variables that reduce entropy more than a naturally correlated variable with the target? [closed]
In machine learning sometimes we build models using hundreds of variables/features that we don't know (at least at first) if they might have a relation with the target. Usually we find that some of ...
- 2,628
0
votes
1
answer
140
views
Lagrange Multipliers and Hard Margin SVMs
With hard margin support vector machines (SVMs), it suffices to find the critical points of the Lagrangian $L = \frac{1}{2}||\theta||^2 - \sum_{n=1}^{N} \alpha_n (y^{(n)} (\vec{\theta}^T\vec{x}^{(n)} +...
- 629
0
votes
0
answers
10
views
What's the probability of one binary classifier better than another knowing the results over a sample of size $N$?
Assuming that classifier_1, classifier_2 have an unknown hit ratio $α, β$, what is the probability that $α>β$ if after doing an experiment on a sample of size 20 classifier_1 gets a hit ratio of 80%...
- 2,463
0
votes
1
answer
34
views
Show that the equivalence of MAP in Bayesian estimation to Structural Risk Minimization (SRM)
I'm sorry that I didn't explain my problem clearly😭I would like to add something.
I saw this problem in Machine Learning Method written by Li Hang(p16).
The book states the problem as below: "...
- 3,802
1
vote
0
answers
57
views
Optimization of the sum of a quadratic form and the L1-norm of logarithm
Given the symmetric positive definite matrix $W\in\mathbb{R}^{n,n}$ and the positive scalar $\lambda $, the objective reads
$$\min_{x} x^{T}Wx + \lambda \|\log(x)\|_1, $$
where $\|\cdot\|_1$ denotes ...
- 21
0
votes
1
answer
101
views
Proving a function satisfying Bellman's equation is optimal
The Question
I'd like to prove that a function $V$ (like in reinforcement learning) is optimal iff it satisfies the Bellman equation. A lot of places online reference this fact, but none prove it.
...
- 249
-2
votes
0
answers
12
views
Can Midas regression involve independent variables of different frequnecies?
Can the dependent variable (GDP) be quarterly and it dependent on mix of monthly and weekly independent variables?
- 1
-1
votes
0
answers
40
views
How to find an "distance maintaining" mapping $f:X \to Y$, where $X \subset \mathbb{R}^m$, $Y \subset \{0,1\}^n$, and $m<n$ in a relatively easy way?
As stated in title, the function can map data points from a lower dimensional Euclidean space into a higher dimensional space, and by "isometric" (may not be strictly correct in this context ...
- 93
0
votes
0
answers
7
views
Log probability of following a trajectory under an optimal policy
In reinforcement learning, is the log probability of following a trajectory under an optimal policy equal to the sum of rewards for that trajectory? i.e.
$\log(p(\tau)) = \sum^T_{t=1}r(s_t,a_t)$
I've ...
- 2,463
2
votes
0
answers
29
views
Empirical distribution learns w.r.t total variation distance
I am trying to prove or disprove that the empirical distribution can learn any continuous distribution w.r.t the total variation distance. The context is the one of statistical learning.
I am quite ...
- 21
1
vote
0
answers
25
views
Are we finding the density of $x$ or evaluating the density of $\theta$ at $x$? | Alpyadin Machine Learning
In section $4.4$ The Bayes Estimator of Alpaydin he discusses the use of the prior density of $p(\theta)$ to construct a posterior density for $\theta$. This is standard Bayesian estimation to get a ...
- 1,163
0
votes
0
answers
14
views
Modeling Regional Quality Based on SUM of GCL / SUM of TAP - Comparing Two Approaches
I'm learning Data Science and I'm currently working on analyzing loan data across different regions. My goal is to build a model that can assess the quality of a given region based on the ratio $$\...
- 38.2k
0
votes
0
answers
19
views
Optimizing function defined by integral
Let the two functions $q: \mathbb{R}^d \rightarrow\mathbb{R}^{+}$ and $s: \mathbb{R^d} \times \mathbb{R^d} \rightarrow \mathbb{R}^{+}$, $d \in \mathbb{N,}$ where both are assumed to be continuous and ...
4
votes
4
answers
525
views
Large Deviation, Optimal Transport and Machine Learning Reference
I am looking for references (books/sites/articles) on the following three subjects: Large Deviation, Optimal Transport and Machine Learning References. I would like works which involve any of them ...
- 653
1
vote
1
answer
69
views
$\min$-entropy for the uniform distribution on $[𝑛]$
The min-entropy of a distribution $\nu$ on $[n]$ is given as:
$$H_{\infty}(\nu)=\min_{i} \log(\frac{1}{\nu(i)})$$
Now we will prove that that for every distribution $\nu$ on $[n]$ and for $U$ being ...
- 656
0
votes
0
answers
17
views
variance maximization in PCA: doubt for the correctness of an Algerian passage.
I know trying to perform the steps omitted in Bishop's machine learning book (from point 12.4 to 12.5):
there is only one step that leaves me with a doubt of correctness namely whether I can, taken ...
- 235
5
votes
1
answer
162
views
ODE's: Continuity Equation
The context of this question is Machine Learning (more specifically, my question results from this paper, yet I have a math question, so I'm posting it here). First of all, some definitions (Sec. 2 of ...
- 1,820
1
vote
1
answer
108
views
Shapley Kernel Proof
I read the paper about SHAP.
I think this paper is very interesting ! I would like to understand the algorithm, but I cannot follow the below fact.
\begin{equation}
X^T WX=\dfrac{1}{M-1}I+cJ
\end{...
- 51
1
vote
0
answers
12
views
Unclear maximization step in the PCA for machine learning [closed]
In Bishop's book (Pattern Recognition and Machine Learning, chapter PCA) there is this passage for calculating the gradient with respect to $\vec{u}_{1}$. In the passage, $\vec{u}_{1}^{T}$ are ...
- 235
2
votes
0
answers
42
views
How to make the income distribution of a country follows the 80:20 rule?
My main question:
Let's imagine a country with a population of $n$ people. Each person has a certain amount of income in a certain year. When we calculate the income distribution of this country, we ...
0
votes
1
answer
19
views
How is this reinforcement learning value formula read / understood?
$$V_\pi(s) = E[R_t|s_t=s,\pi]$$
This is a value function for state s under policy $\pi$ where $R_t$ is the return value, all of which occurs at time t. I was wondering how I should read/ understand ...
- 1,403
0
votes
0
answers
62
views
VC dimension of indicator functions is equal to pseudo dimension
I am reading the "Foundation of machine learning" by Mehryar Mohri (https://cs.nyu.edu/~mohri/mlbook/). In the proof of Theorem 11.8, it said the following statement, which I can not ...
- 659
0
votes
0
answers
27
views
Limiting probability of classifying correctly with the k-NN algorithm as the number of data points increases.
Let $x_1, \ldots, x_n$ be random variables that are uniformly distributed on [0,2]. If $x_i \leq 1$, we'll classify it as green ($y_i=0$), and if $x_i > 1$, we'll classify it as red ($y_i=1$). We ...
- 413
0
votes
0
answers
9
views
KKT Conditions for SVM Problem
I am reading about SVMs and want to confirm that I understand the optimality conditions. Details below:
Consider the $n$ points $x_1, x_2, \dots, x_n$, each with $ d$ dimensions, and consider $ n$ ...
- 71
0
votes
0
answers
16
views
Finding an algorithm EF[1,1] and PO division for more than two agents
From this research paper I want to write an algorithm for finding envy-freeness(EF) and Pareto optimality(PO) division for more than two agents.
We consider the problem of fairly and efficiently ...
- 111
0
votes
0
answers
23
views
Maximizing the summation of reward of some users (waiting in different positions and lines) using reinforcement learning or other learning methods
There is a mathematical problem that I think can be solved using reinforcement learning and it would be great if you could help me with it.
Some users are standing in some lines. There are N lines. In ...
- 1
2
votes
0
answers
37
views
Recommendations for Information Geometry in Machine Learning
I am fairly new to machine learning, but I have a 22-dimensional dataset, which I would like to increase the interpretability of by dimension reduction. I am relatively familiar with principal ...
- 3,083
0
votes
0
answers
12
views
Kernel density estimators in Bishop book unclear formulas
In Bishop's book (Pattern recognition and machine learning, pag 122) there is an unclear passage for me in deriving certain formulas:
$E[K/N] = P$ and $var[K/N] = P(1-P)$
Considering binomial ...
- 235
1
vote
0
answers
36
views
Approximating an isometry with a neural network
I'm trying to link a machine learning framework to more theoretical considerations about linear isometries.
Let's say we have an input dataset $\mathcal{D}=\{(\textbf{x}_1,y_1), (\textbf{x}_2,y_2), \...
- 1,884
0
votes
0
answers
46
views
About the inputs of the Wasserstein Distance $W_1$
Introduction (this is just supporting my questions, but you can skip it and go directly to the questions).
Let's consider the following Proposition from "Ramdas & Trillos(2015) On ...
- 255
-1
votes
0
answers
9
views
Why does gradient vector update parameters based on how the loss changes due to the parameters? [closed]
Essentially, I want to know why the gradient vector points in the direction of steepest increase. Like why does a vector that moves in the x direction the same amount as the loss changes due to x ...
1
vote
0
answers
35
views
Is this smooth linear ramp function already a thing?
In machine learning (particularly with regards to Neural Nets), there's a bunch of "ramp" functions that're used. For example, the ReLU is $0$ for $x\leq 0$ and $x$ for $x>0$.
I was ...
- 2,060
2
votes
2
answers
1k
views
how to show the K-nearest-neighbor density model is an improper distribution: Bishops 2.61
In Bishop's pattern recognition and machine learning, KNN is defined from a starting distribution:
$$p(x) = \frac{K}{NV}$$
where K is the number of observed points in a region of measure V, out of a ...
- 1
0
votes
0
answers
11
views
Expectation of linear form multiplied by quadratic form for MVN distribution
Assume that $\bf{x}$ is a random vector that is distributed multivariate normal with mean $\boldsymbol{\mu}$ and covariance matrix $\boldsymbol{\Sigma}$. Let $\bf{A}$ be a matrix of constants.
I'm ...
- 194
0
votes
0
answers
23
views
How do I obtain the primal and dual for the regression estimator $\min _\beta[\|\beta\|^2+\sum_{i=1}^n \xi_i^2]$ s.t. $\xi_i=y_i-h(x_i)^\top \beta$?
I am working on a statistical learning exercise that requires some knowledge of convex optimization which I am unfortunately lacking.
Consider the linear regression model
$$y_i=h(x_i)^\top\beta+\...
- 61
9
votes
1
answer
3k
views
Reconciling Donsker-Varadhan definition of KL divergence with the "usual" definition
Let $\mu$ and $\lambda$ be probability measures on a measurable space $(X, \Sigma)$.
In my experience, the usual definition of the Kullback-Liebler divergence of $\mu$ with respect to $\lambda$ is
$$
\...
- 4,703
0
votes
1
answer
26
views
Define 'accuracy' for numerical data?
Normally, people use 'accuracy' to describe the output quality (from a model or methodology https://en.wikipedia.org/wiki/Precision_and_recall) for categorical data.
However, I am wondering could the ...
- 3,213
1
vote
1
answer
89
views
Understanding a statement in Sutton's Reinforcement Learning Section 5.5 on Importance Sampling
I am trying to understand chapter $5.5$ of Sutton's Book on Reinforcement leaning, in a particular a statement on page $104$ related to off policy prediction via importance sampling.
Supposing $b$ is ...
- 16.3k
0
votes
0
answers
45
views
Partial derivate of scalar-valued function of a vector variable
Working through Mathematics for Machine Learning and got stuck on the end-of-chapter exercises 5.8a):
Calculating $\frac{\partial g}{\partial y}$ for
$$g(y) = y^T S^{-1} y$$
with $y \in \mathbb R^{D}$...
0
votes
0
answers
38
views
Derivative Least Square Regression (Numerator vs denominator layout)
Assume I have the following expression:
$$\frac{\partial}{\partial w} \lVert X^Tw - y \rVert^2 = 0$$
which is trying to find the solution for the Least Squares Approach in Regression.
Let's assume $X \...
- 11
1
vote
0
answers
37
views
Coefficient for the gradient term in stochastic gradient descent (SGD) with momentum
I'm studying SGD with momentum and have come across two versions of the update formula.
The first is from a wiki same as from the original paper:
$$
\Delta w^t = \alpha * \Delta w^{t-1} - lr * \nabla ...
- 23
0
votes
0
answers
12
views
Does skipgram model uses backpropagation? [migrated]
I just started to get interested in natural language processing and I was trying to understand the skipgram model from word2vec. I was reading this interesting website. However, in the mentioned ...
- 1,208
0
votes
1
answer
47
views
Is this a closed-form analytical solution for the hard-margin SVM dual problem?
I have been searching, without much success, for some dicussion on the possibilities (or impossibilities) of a general closed-form analytical solution for the hard-margin (only) support vector ...
- 498
0
votes
2
answers
1k
views
Similarity metric between two sets of points with varying densities
How can I create a similarity metric that describes the top left set of points as more similar to the bottom left set of points than the top right set of points? Clearly least-squares distance doesn't ...
CommunityBot
- 1
0
votes
0
answers
50
views
How can I understand the (no) independence property in this very simple setting of first order logic?
My understanding of logic is limited to first order logic without functions with finite set of domain constants, and with herbrand semantics. Now in this setting, I would like to understand the ...
- 1,777
2
votes
3
answers
262
views
Implementing multiclass logistic regression from scratch
This is a sequel to a previous question about implementing binary logistic regression from scratch.
Background knowledge:
To train a logistic regression model for a classification problem with $K$ ...
- 3,315
0
votes
1
answer
1k
views
Scaling Cumulative Probability Distribution function values
We have a cumulative probability distribution function (cdf), we want to scale it down for using it in anomaly detection. The mapping should look like this.
CDF value: 0.1 ... 0.5 ... 0.9 ... ...
CommunityBot
- 1
4
votes
0
answers
76
views
Book suggestion on "Banach space geometry for machine learning"
Is there any book for a Mathematics student who can learn Machine learning in the aspect of Banach space geometry? Or, one can understand the connection between Geometry of Banach spaces and Machine ...
- 129