Skip to main content

Questions tagged [machine-learning]

How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

Filter by
Sorted by
Tagged with
0 votes
0 answers
41 views

How to estimate the inverse of a non-invertible matrix?

So I'm working on a machine learning problem where my solution requires taking the inverse of a matrix at some point. The problem is that this matrix is sometimes non-invertible. In theory the the ...
Dr.'s user avatar
  • 11
-1 votes
0 answers
30 views

Distribution of two combined ML models

Due to the complexity of the problem, the problem was divided into two models: a stationary model and a model that corrects the stationary model for temporal effects, i.e. $X = X_{stat} + X_{time}$ ...
xbc68's user avatar
  • 1
-2 votes
1 answer
96 views

EM algorithm for estimating worker ability [closed]

I cannot understand how to get results in M-step. This formulation is from https://papers.nips.cc/paper_files/paper/2012/file/cd00692c3bfe59267d5ecfac5310286c-Paper.pdf
Hwang Jeong Yeon's user avatar
0 votes
0 answers
29 views

Formulating a solution ansatz for the 1D heat equation in polar coordinates to learn the PDE in a PINN setting

Hello Math Stack Exchange Community, I am working on solving a partial differential equation (PDE) with a neural network in a PINN-like fashion, and I am seeking advice on identifying an appropriate ...
alighato's user avatar
6 votes
1 answer
68 views

What is the collection of functions that a given finite neural network can approximate with ease?

To my understanding, one version of the universal approximation theorem runs as follows: Let $\Phi$ be the family of (trained) feedforward neural networks of bounded width, arbitrary depth, and mild ...
SapereAude's user avatar
0 votes
1 answer
23 views

Question about likelhood function of discriminative models

Im a little confused with the likelihood function. For discriminative models, we have a hypothesis function $h_{\theta}(x) = p(y \mid x ; \theta)$. Using the principles of maximim likelihood we want ...
Joe Jameson's user avatar
0 votes
0 answers
26 views

Calculating functional derivative for a Physics-Informed Neural Network (PINN) using Automatic Differentiation

I'm working with a Physics-Informed Neural Network (PINN) to approximate the solution of a 1D Poisson equation: $\frac{d^2u}{dx^2} = f$ Here, I have an MLP with weight parameters $\theta$ that takes a ...
Yanyan Wang's user avatar
1 vote
0 answers
19 views

How many vectors can be placed in $n$ dimensions given max cosine similarity? [duplicate]

In machine learning we usually use the concept of cosine similarity to compare things. Similar things should have embeddings with high cosine similarity and different things should have embeddings ...
F. Bruno Dias's user avatar
0 votes
0 answers
48 views

Self-Organizing maps: why input vectors (x) are dependent on steps (t)?

Based on the paper Essentials of the Self-Organizing maps, I rephrase paragraph 4.1. ->The original, stepwise recursive SOM algorithm: In the mathematical framework $\{\mathbf{x}(t)\}$ represents ...
Nauel's user avatar
  • 51
1 vote
0 answers
42 views

FGSM for logistic regression

In arXiv:1412.6572 (https://arxiv.org/pdf/1412.6572, a seminal article), it is stated that $$\mathrm{sgn}(\nabla_{\mathbf{x}} L(\mathbf{x},y,\mathbf{w})) = -\mathrm{sgn}(\mathbf{w})$$ for the softplus ...
SEJ's user avatar
  • 47
0 votes
0 answers
22 views

Sample complexity bounds of $L_S(h)$

Fix $\mathscr{H} \subset \mathscr{Y}^\mathscr{X}$ and a loss $\ell : \hat{Y} \times Y \to [0,1]$. Fix $S \in (\mathscr{X} \times \mathscr{Y})^{2m}$. Assume for now that $S$ is not random. Suppose we ...
isaac's user avatar
  • 41
1 vote
1 answer
48 views

sorting functions by amount of conditions for a random dataset to be described using it?

Given a finite dataset like 1, 2, 3, 4 You could find infinite functions, for simplicity I found 2: Add 1 for the next data point, so the sequence continues as 5, 6, etc. 2.Cycle through 1, 2, 3, 4, ...
Anonymous's user avatar
0 votes
0 answers
31 views

Strong convexity and Lipschitz-continuous gradients, how restrictive are these assumptions in practice?

I am reading a paper on stochastic gradient descent and different varieties of it. For all the convergence proofs the author assumes strong convexity and Lipschitz-continuous gradients for the ...
Sen90's user avatar
  • 453
0 votes
0 answers
14 views

Information coefficient as loss function of XGBoost

$$ IC = \frac{\frac{1}{n}\hat{y}^Ty-\mathrm{E}\left[ \hat{y} \right] \mathrm{E}\left[ y \right]}{\sigma \left[ \hat{y} \right] \sigma \left[ y \right]} $$ XGBoost requires a gradient and a Hessian of ...
atlantic0cean's user avatar
1 vote
0 answers
71 views

Relation between values of $ξ_i$ and $\alpha_i$ in SVM?

I have a question in about a property of support vectors of SVM which is stated in subsection "12.2.1 Computing the Support Vector Classifier" of "The Elements of Statistical Learning&...
hasanghaforian's user avatar
0 votes
0 answers
11 views

Paired bootstrap test p-value formula in binary classification

Background For a binary classification task, let $M(A, Z)$ denote an evaluation metric, such as accuracy, for classifier $A$ and test examples $Z.$ Then, let $$ \delta(Z) = M(A, Z) - M(B, Z) $$ denote ...
sunspots's user avatar
  • 802
0 votes
0 answers
39 views

least squares minimum test error solution

assume we want to learn a model $y=x^T \beta + \varepsilon $ where $\beta \in \mathbb{R}^d$ is constant $ x \in \mathbb{R}^d$ is the input vector with Gaussian distribution $\mathcal{N}(0,\Sigma_x)$ ...
Elad Elmakias's user avatar
0 votes
0 answers
24 views

Does clustering actually reduce the number of rows in a dataset? [migrated]

I am reading the book "grokking Machine Learning" by Luis G. Serrano and came across the following sentence: "It seems that clustering and dimensionality reduction are nothing like each ...
Leox's user avatar
  • 8,204
0 votes
0 answers
13 views

Defining Unsupervised Learning Problem

I have read a paper that says Unsupervised learning concerns modeling and understanding the structure of complex data ... Of course, the distribution over such complex data as images and sounds ...
Pond-nj's user avatar
  • 31
0 votes
0 answers
23 views

Analyzing a deep learning model by constructing a Matrix from input and output data

I am currently completing my bachelor’s thesis, and recently my supervisor suggested that I could strengthen my arguments by explaining why deep learning models perform so well in my case. To give you ...
glaand's user avatar
  • 109
1 vote
0 answers
98 views

Why does learning theory study generalization bounds?

Disclaimer: I know that mathematics needs no external motivation to be developed, and that such view is (in the long term) helpful even for applications. Nonetheless, I believe it is crucial for ...
Alek Fröhlich's user avatar
0 votes
0 answers
18 views

minimization involved $l_2$ norm

I am trying to find the minimum of the following problem $$\frac{\theta}{2}\lVert\beta-x\rVert_2^2+\lambda\lVert x\rVert_2-\frac{1}{2\tau}\lVert x\rVert_2^2+\alpha^Tx$$ by taking the derivative with ...
Simple's user avatar
  • 3,633
1 vote
0 answers
29 views

Understanding of KL divergence

I am learning machine learning and encountered KL divergence: $$ \int p(x) \log\left(\frac{p(x)}{q(x)}\right) \, \text{d}x $$ I understand that this measure calculates the difference between two ...
Dmitry_IT_03's user avatar
1 vote
1 answer
42 views

Solve the Soft SVM Dual Problem with L1 Regularization

I'm considering a support vector regression model with a prediction $$ \hat{y}(\mathbf{x}_\star)=\boldsymbol{\theta}^{\top} \boldsymbol{\phi}(\mathbf{x}_\star)$$ where $\boldsymbol{\theta}$ are the ...
oweydd's user avatar
  • 239
2 votes
0 answers
21 views

Would like to validate whether the AUC equation is correct or not

I found a paper "Chapi, Kamran, et al. "A novel hybrid artificial intelligence approach for flood susceptibility assessment." Environmental modelling & software 95 (2017): 229-245&...
Simon's user avatar
  • 95
1 vote
0 answers
24 views

Minimizing a function with absolute L^q penalty in one variable

I would like to solve the following minimization problem to analyze LASSO-type estimators: $$\arg \min_{x} \quad f(x) = (x-a)^2 + b|x|^q $$ $$\text{where} \quad a \in (-\infty, \infty), b \in (0,\...
ludwig's user avatar
  • 11
0 votes
0 answers
33 views

Showing axis-aligned rectangles with noise are PAC-learnable

In what follows, an axis-aligned rectangle is an element of the set $\mathcal{C}:=\{[l,r]\times [b,t]\in\mathscr{P}(\mathbb{R}^2)\mid l,r,b,t\in \mathbb{R}\}$. This will be our concept class and our ...
Choripán Con Pebre's user avatar
2 votes
1 answer
53 views

What is so interesting about the Armijo-Rule or the Wolfe-Conditions for choosing the right step size?

Right now I am taking a course on nonlinear optimization where we currently talk about step size rules(Armijo-Rule and Wolfe-Conditions). I also had a course 1 year ago about statistical machine ...
Sen90's user avatar
  • 453
1 vote
1 answer
28 views

Min-max optimization and prediction of a parameter in a mathematical model

Context Hello, everyone; let me preface this by saying that my background is in CS and not mathematics, but I do have a background in calculus, statistics, and discrete mathematics. The issue at hand ...
A.Sal's user avatar
  • 13
1 vote
0 answers
30 views

Clarification on the Concept of i.i.d Datasets in Machine Learning

I am currently delving into the field of machine learning, and I’ve encountered a point of confusion regarding the concept of i.i.d. datasets. I have only studied rigorous measure-theoretic ...
Bill's user avatar
  • 4,503
1 vote
0 answers
15 views

Corollary to Dvoretzky's stochastic approximation theorem extension

I am looking into proofs of Q learning convergence. Specifically, I am looking at Jaakkola, Jordan and Singh's proof of Q learning convergence from their paper On the convergence of stochastic ...
user11728899's user avatar
-4 votes
1 answer
55 views

Neural network that learns ODE's 'refuses to learn' initial conditions [closed]

I have implemented a simple network that for now i'm just trying to teach the ODE: $$\frac{\text{d}x}{\text{d}t} = x$$ Using the simple code below: ...
somemathperson's user avatar
0 votes
1 answer
16 views

Understanding the Reasoning Behind the Growth Function $m_{\mathcal{H}}(N)=2^N$ for Convex Sets

I am currently reading Learning from Data by Abu-Mostafa et al. and I am struggling to understand the reasoning behind the growth function $m_{\mathcal{H}}(N)=2^N$ for convex sets. Here is the ...
bruno's user avatar
  • 425
2 votes
0 answers
99 views

Generalization error bound for Empirical Risk minimizer on Gaussian noisy data

I have datapoints that are sampled from a distribution $\mathbb{D}$. Each datapoint is a tuple $(t,y)$ of a time $t \in [0,T]$ that is sampled uniformly and a value $y(t) \sim u(t) + \mathcal{N}(0, \...
Paul Joh's user avatar
  • 559
1 vote
1 answer
86 views

Fitting a parabola through two points in which the derivatives are known with the aim of finding a minimum.

For a project I am designing an accelerated gradient descent algorithm that uses the first derivative of two points of the unknown function (as opposed to just using a single point with its derivative)...
Carlo Wood's user avatar
0 votes
0 answers
14 views

How to find a linear decision boundary of a linearly separable problem with unlimited class evaluations?

I have a binary classification problem, where my goal is to find a linear decision boundary (which I assume exists). The context of the problem is that I have an iterative optimization process, where ...
oskar0711's user avatar
1 vote
0 answers
46 views

How do generative models generate?

I am trying to understand the difference between discriminative models and generative models in machine learning. One of the helpful answers at Stack Overflow is here: https://stackoverflow.com/...
Oatmeals's user avatar
  • 111
1 vote
1 answer
35 views

Is there any algorithm to calculate this ranking method quickly?

I want to rank the 20 teams in the English Premier League. Say that each team are assigned to the number 1 through 20, defining their ranking, no ties. There would be $20!$ number of permutations for ...
Germaniac's user avatar
0 votes
0 answers
19 views

How to project an unseen vector to the lower rank SVD subspace?

I think I might be missing something fundamental when it comes to SVD, or perhaps the answer to my question actually isn't straightforward. Say we have a Matrix $A$ of rank n. SVD, as far as I ...
user3346532's user avatar
0 votes
1 answer
37 views

Estimating the conditional entropy of a discrete variable conditioning on continuous variable

I am doing a machine learning project and I am trying to select the best features by computing their mutual information and select the ones with the highest information gain. I was looking at this ...
Ishigami's user avatar
  • 1,727
1 vote
1 answer
65 views

Why is the numerator-layout Jacobian transposed in backpropagation calculation?

In the derivation of the backpropagation algorithm in Neural Network Design by Hagan et al., we consider the derivative of the scalar-valued sample loss function $\hat{F}$ with respect to a vector of &...
aas's user avatar
  • 11
3 votes
0 answers
118 views

Kernel Regression, Similarity-Based Modeling & Weights Normalization

Let be $D=[x_j(t_i)]_{i,j} \ M_{n,p}(\mathbb{R})$ a state matrix of $n$ $x(t_i) \in M_{1,p}(\mathbb{R})$ observations of $p$ sensors $X_j$ representing the normal conditions of a system for some ...
Mistapopo's user avatar
  • 101
0 votes
0 answers
19 views

Calculation of the projection w from Linear Discriminant Analysis

In an assigment focused on Linear Discriminant Analysis(LDA) there is this theoritical exercise: A dataset has been derived from two classes $ \omega_A$ and $\omega_B$, the distributions of which are ...
Constantinos Pisimisis's user avatar
0 votes
0 answers
11 views

F-statistic comparision.

My goal is to find a good combination of features for a machine learning algorithm. I have two multivariate normal distributions and I want to find out what combination of features (x or y or both) is ...
Rekmix's user avatar
  • 1
-1 votes
2 answers
60 views

Sigmoid vs heaviside step function

I am curious about a relationship between the elementwise sigmoid function $\sigma$ and unit heaviside step function $H$. I know that as the slope of $\sigma$ increases, it gets closer to $H$. However,...
Johny's user avatar
  • 27
0 votes
0 answers
13 views

Hypothesis testing of Precision-Recall curve AUCs

In recent times, I have been about learning classification models (e.g., logistic regression) and how to evaluate them. While learning about the Precision-Recall (PR) curves, it occurred to me that ...
Yat-Hon's user avatar
  • 21
0 votes
0 answers
32 views

How to Upper Bound the Spectral Norm of $\left(XX^T\right)^{-1}\left(XX^T\right)^{-1}X$?

I have an observation matrix $ X \in \mathbb{R}^{n \times n}$. Considering $XX^T$, this matrix can be seen as a correlation matrix between individuals, so it generally has elements close to the ...
Tool's user avatar
  • 1
1 vote
0 answers
20 views

Self-Attention: Rank of Matrixes

I was reading this paper and could need some help understanding an inequality in Sec. 3.2. The starting point is the following matrix $A_{i}$ from Eq. (13): $$A_{i} = QW_{i}^Q{W_{i}^{K}}^{T}K^{T},$$ ...
Hermi's user avatar
  • 702
1 vote
0 answers
173 views

Explain the proof of Kolmogorov Arnold representation theorem

Can someone explain the outline of proof strategy of Kolmogorov Arnold representation theorem? Any proof of any variant (eg. George Lorentz's variant) would suffice. I would be grateful if you could ...
HIH's user avatar
  • 451
0 votes
0 answers
39 views

Definition of tensors used in deep learning/machine learning

“Tensors” used in deep learning or machine learning are defined as N-dimensional arrays of data that can take any value and be operated on by in any way. This differs from the tensor definition used ...
joshlk's user avatar
  • 286

1
2 3 4 5
68