Questions tagged [machine-learning]
How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?
3,211
questions
1
vote
0
answers
41
views
Optimization of the sum of a quadratic form and the L1-norm of logarithm
Given the symmetric positive definite matrix $W\in\mathbb{R}^{n,n}$ and the positive scalar $\lambda $, the objective reads
$$\min_{x} x^{T}Wx + \lambda \|\log(x)\|_1, $$
where $\|\cdot\|_1$ denotes ...
- 21
-2
votes
0
answers
10
views
Can Midas regression involve independent variables of different frequnecies?
Can the dependent variable (GDP) be quarterly and it dependent on mix of monthly and weekly independent variables?
- 1
-1
votes
0
answers
40
views
How to find an "distance maintaining" mapping $f:X \to Y$, where $X \subset \mathbb{R}^m$, $Y \subset \{0,1\}^n$, and $m<n$ in a relatively easy way?
As stated in title, the function can map data points from a lower dimensional Euclidean space into a higher dimensional space, and by "isometric" (may not be strictly correct in this context ...
- 93
0
votes
1
answer
23
views
Show that the equivalence of MAP in Bayesian estimation to Structural Risk Minimization (SRM)
I'm sorry that I didn't explain my problem clearly😭I would like to add something.
I saw this problem in Machine Learning Method written by Li Hang(p16).
The book states the problem as below: "...
- 9
0
votes
0
answers
7
views
Log probability of following a trajectory under an optimal policy
In reinforcement learning, is the log probability of following a trajectory under an optimal policy equal to the sum of rewards for that trajectory? i.e.
$\log(p(\tau)) = \sum^T_{t=1}r(s_t,a_t)$
I've ...
2
votes
0
answers
28
views
Empirical distribution learns w.r.t total variation distance
I am trying to prove or disprove that the empirical distribution can learn any continuous distribution w.r.t the total variation distance. The context is the one of statistical learning.
I am quite ...
- 21
0
votes
0
answers
19
views
Optimizing function defined by integral
Let the two functions $q: \mathbb{R}^d \rightarrow\mathbb{R}^{+}$ and $s: \mathbb{R^d} \times \mathbb{R^d} \rightarrow \mathbb{R}^{+}$, $d \in \mathbb{N,}$ where both are assumed to be continuous and ...
0
votes
0
answers
17
views
variance maximization in PCA: doubt for the correctness of an Algerian passage.
I know trying to perform the steps omitted in Bishop's machine learning book (from point 12.4 to 12.5):
there is only one step that leaves me with a doubt of correctness namely whether I can, taken ...
- 227
1
vote
0
answers
12
views
Unclear maximization step in the PCA for machine learning
In Bishop's book (Pattern Recognition and Machine Learning, chapter PCA) there is this passage for calculating the gradient with respect to $\vec{u}_{1}$. In the passage, $\vec{u}_{1}^{T}$ are ...
- 227
1
vote
1
answer
102
views
Shapley Kernel Proof
I read the paper about SHAP.
I think this paper is very interesting ! I would like to understand the algorithm, but I cannot follow the below fact.
\begin{equation}
X^T WX=\dfrac{1}{M-1}I+cJ
\end{...
- 51
2
votes
0
answers
41
views
How to make the income distribution of a country follows the 80:20 rule?
My main question:
Let's imagine a country with a population of $n$ people. Each person has a certain amount of income in a certain year. When we calculate the income distribution of this country, we ...
0
votes
1
answer
19
views
How is this reinforcement learning value formula read / understood?
$$V_\pi(s) = E[R_t|s_t=s,\pi]$$
This is a value function for state s under policy $\pi$ where $R_t$ is the return value, all of which occurs at time t. I was wondering how I should read/ understand ...
0
votes
0
answers
27
views
Limiting probability of classifying correctly with the k-NN algorithm as the number of data points increases.
Let $x_1, \ldots, x_n$ be random variables that are uniformly distributed on [0,2]. If $x_i \leq 1$, we'll classify it as green ($y_i=0$), and if $x_i > 1$, we'll classify it as red ($y_i=1$). We ...
- 413
0
votes
0
answers
9
views
KKT Conditions for SVM Problem
I am reading about SVMs and want to confirm that I understand the optimality conditions. Details below:
Consider the $n$ points $x_1, x_2, \dots, x_n$, each with $ d$ dimensions, and consider $ n$ ...
- 71
0
votes
0
answers
61
views
VC dimension of indicator functions is equal to pseudo dimension
I am reading the "Foundation of machine learning" by Mehryar Mohri (https://cs.nyu.edu/~mohri/mlbook/). In the proof of Theorem 11.8, it said the following statement, which I can not ...
- 659
1
vote
0
answers
25
views
Are we finding the density of $x$ or evaluating the density of $\theta$ at $x$? | Alpyadin Machine Learning
In section $4.4$ The Bayes Estimator of Alpaydin he discusses the use of the prior density of $p(\theta)$ to construct a posterior density for $\theta$. This is standard Bayesian estimation to get a ...
- 11
0
votes
0
answers
23
views
Maximizing the summation of reward of some users (waiting in different positions and lines) using reinforcement learning or other learning methods
There is a mathematical problem that I think can be solved using reinforcement learning and it would be great if you could help me with it.
Some users are standing in some lines. There are N lines. In ...
- 1
2
votes
0
answers
37
views
Recommendations for Information Geometry in Machine Learning
I am fairly new to machine learning, but I have a 22-dimensional dataset, which I would like to increase the interpretability of by dimension reduction. I am relatively familiar with principal ...
- 3,083
0
votes
0
answers
12
views
Kernel density estimators in Bishop book unclear formulas
In Bishop's book (Pattern recognition and machine learning, pag 122) there is an unclear passage for me in deriving certain formulas:
$E[K/N] = P$ and $var[K/N] = P(1-P)$
Considering binomial ...
- 227
1
vote
0
answers
36
views
Approximating an isometry with a neural network
I'm trying to link a machine learning framework to more theoretical considerations about linear isometries.
Let's say we have an input dataset $\mathcal{D}=\{(\textbf{x}_1,y_1), (\textbf{x}_2,y_2), \...
- 1,884
0
votes
0
answers
46
views
About the inputs of the Wasserstein Distance $W_1$
Introduction (this is just supporting my questions, but you can skip it and go directly to the questions).
Let's consider the following Proposition from "Ramdas & Trillos(2015) On ...
- 255
-1
votes
0
answers
9
views
Why does gradient vector update parameters based on how the loss changes due to the parameters? [closed]
Essentially, I want to know why the gradient vector points in the direction of steepest increase. Like why does a vector that moves in the x direction the same amount as the loss changes due to x ...
1
vote
0
answers
34
views
Is this smooth linear ramp function already a thing?
In machine learning (particularly with regards to Neural Nets), there's a bunch of "ramp" functions that're used. For example, the ReLU is $0$ for $x\leq 0$ and $x$ for $x>0$.
I was ...
- 2,060
0
votes
0
answers
16
views
Finding an algorithm EF[1,1] and PO division for more than two agents
From this research paper I want to write an algorithm for finding envy-freeness(EF) and Pareto optimality(PO) division for more than two agents.
We consider the problem of fairly and efficiently ...
- 111
5
votes
1
answer
161
views
ODE's: Continuity Equation
The context of this question is Machine Learning (more specifically, my question results from this paper, yet I have a math question, so I'm posting it here). First of all, some definitions (Sec. 2 of ...
- 642
0
votes
0
answers
10
views
Expectation of linear form multiplied by quadratic form for MVN distribution
Assume that $\bf{x}$ is a random vector that is distributed multivariate normal with mean $\boldsymbol{\mu}$ and covariance matrix $\boldsymbol{\Sigma}$. Let $\bf{A}$ be a matrix of constants.
I'm ...
- 194
0
votes
0
answers
23
views
How do I obtain the primal and dual for the regression estimator $\min _\beta[\|\beta\|^2+\sum_{i=1}^n \xi_i^2]$ s.t. $\xi_i=y_i-h(x_i)^\top \beta$?
I am working on a statistical learning exercise that requires some knowledge of convex optimization which I am unfortunately lacking.
Consider the linear regression model
$$y_i=h(x_i)^\top\beta+\...
- 61
1
vote
1
answer
65
views
$\min$-entropy for the uniform distribution on $[𝑛]$
The min-entropy of a distribution $\nu$ on $[n]$ is given as:
$$H_{\infty}(\nu)=\min_{i} \log(\frac{1}{\nu(i)})$$
Now we will prove that that for every distribution $\nu$ on $[n]$ and for $U$ being ...
- 648
0
votes
0
answers
44
views
Partial derivate of scalar-valued function of a vector variable
Working through Mathematics for Machine Learning and got stuck on the end-of-chapter exercises 5.8a):
Calculating $\frac{\partial g}{\partial y}$ for
$$g(y) = y^T S^{-1} y$$
with $y \in \mathbb R^{D}$...
0
votes
0
answers
38
views
Derivative Least Square Regression (Numerator vs denominator layout)
Assume I have the following expression:
$$\frac{\partial}{\partial w} \lVert X^Tw - y \rVert^2 = 0$$
which is trying to find the solution for the Least Squares Approach in Regression.
Let's assume $X \...
- 11
1
vote
0
answers
35
views
Coefficient for the gradient term in stochastic gradient descent (SGD) with momentum
I'm studying SGD with momentum and have come across two versions of the update formula.
The first is from a wiki same as from the original paper:
$$
\Delta w^t = \alpha * \Delta w^{t-1} - lr * \nabla ...
- 23
0
votes
0
answers
12
views
Does skipgram model uses backpropagation? [migrated]
I just started to get interested in natural language processing and I was trying to understand the skipgram model from word2vec. I was reading this interesting website. However, in the mentioned ...
- 1,208
0
votes
1
answer
41
views
Is this a closed-form analytical solution for the hard-margin SVM dual problem?
I have been searching, without much success, for some dicussion on the possibilities (or impossibilities) of a general closed-form analytical solution for the hard-margin (only) support vector ...
- 1
1
vote
1
answer
89
views
Understanding a statement in Sutton's Reinforcement Learning Section 5.5 on Importance Sampling
I am trying to understand chapter $5.5$ of Sutton's Book on Reinforcement leaning, in a particular a statement on page $104$ related to off policy prediction via importance sampling.
Supposing $b$ is ...
- 3,738
0
votes
0
answers
49
views
How can I understand the (no) independence property in this very simple setting of first order logic?
My understanding of logic is limited to first order logic without functions with finite set of domain constants, and with herbrand semantics. Now in this setting, I would like to understand the ...
- 1,777
0
votes
0
answers
14
views
Modeling Regional Quality Based on SUM of GCL / SUM of TAP - Comparing Two Approaches
I'm learning Data Science and I'm currently working on analyzing loan data across different regions. My goal is to build a model that can assess the quality of a given region based on the ratio $$\...
0
votes
0
answers
17
views
convert a 1-dimensional set of points to a 2-dimensional parabola with explicit embedding
I am trying to rephrase to better understand concepts regarding discriminant functions for classification using explicit embedding. I report a very easy diagram found online that from 1-dimension ...
- 227
4
votes
0
answers
76
views
Book suggestion on "Banach space geometry for machine learning"
Is there any book for a Mathematics student who can learn Machine learning in the aspect of Banach space geometry? Or, one can understand the connection between Geometry of Banach spaces and Machine ...
- 129
3
votes
1
answer
174
views
Rearrangement property of diagonal matrices [closed]
Is it true that for a diagonal matrix $B\in\mathbb{R}^{n\times n}$, a matrix $A\in\mathbb{R}^{n\times p}$, the following property holds:
$$
A^\top B A = A^\top A B,
$$
where in essence I am asking ...
- 664
0
votes
0
answers
16
views
Reason behind objective function in Linear Discriminant Analysis
I don't really understand the objective function to be optimized in Linear Discriminant Analysis (LDA). My question is centered around the same concepts mentioned in this this other one.
The analysis ...
- 453
0
votes
1
answer
41
views
Unclear passages in Bishop's machine learning book for Probabilistic Generative Models
I am trying to prove the relationship (4.65) expressed in Bishop's book (Pattern Recognition and Machine Learning) in chapter (4.2.1) Continuous inputs:
I found the following passages but there is ...
- 227
0
votes
0
answers
17
views
Margin width and normalization in SVM
Let $x_1$ and $x_2$ be two support vectors and $w$ an orthogonal vector to the decision hyperplane.
To find the width of the margin, I don't understand why we have to calculate the dot product ...
- 15
0
votes
1
answer
26
views
Define 'accuracy' for numerical data?
Normally, people use 'accuracy' to describe the output quality (from a model or methodology https://en.wikipedia.org/wiki/Precision_and_recall) for categorical data.
However, I am wondering could the ...
- 113
0
votes
0
answers
30
views
Perceptron Convergence: Monotonically Approach Solution?
I'm new to learning about perceptrons, but I saw a proof (for perceptron for binary classification with the caveat of forcing the separator through the origin) that, assuming the data is linearly ...
- 101
0
votes
1
answer
26
views
Unclear formula in the book of linear model for classification in the Bishop book (machine learning)
From Bishop's book pag 183 - (4.11) (Pattern Recognition and Machine Learning, 2009), a mathematical relationship is immediately shown without an explanation of where it is derived from:
I cannot ...
- 227
1
vote
1
answer
34
views
Understanding an inequality related to Diffusion Processes in Machine Learning
I'm trying to understand an equality from a blog post about diffusion models here in the Reverse diffusion process section.
Here $\beta_t \in (0, 1)$, $\alpha_t = 1-\beta_t$, and $\bar{\alpha_t} = \...
- 3,738
0
votes
0
answers
48
views
Converting Cosine Similarity Range from $[-1, 1]$ to $[0, 1]$
I'm working on computing the cosine similarity between two vectors in my project. While I understand that the standard cosine similarity ranges from $-1$ to $1$, I'm interested in transforming the ...
1
vote
1
answer
42
views
PAC Learning Definition with 'if and only if' statement
In this question, the top answer references a definition of PAC learning from the book
Understanding Machine Learning: From Theory to Algorithms by Shai
Shalev-Shwartz, Shai Ben-David"
The ...
- 13
1
vote
0
answers
36
views
Error propagation and Gradient Descent
I was looking at error propagation (or propagation of uncertainty in wikipedia: https://en.wikipedia.org/wiki/Propagation_of_uncertainty)
My primary concern is getting an estimate of error of ...
- 194
0
votes
0
answers
39
views
Using Q-Learning to find best mixture of algorithms.
I'm working on an optimization procedure where one out of N strategies (actions) is run at each time step, leading to a decrease (reward) in a global optimization function. Exactly which strategy ...
- 129