Questions tagged [machine-learning]

How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes?

Filter by
Sorted by
Tagged with
1
vote
2answers
22 views

Equality Constraint Optimizations

I have an optimization problem of this sort $min_{(x_1, x_2) } 4x^2_1 + x^2_2$ subject to $x_1 -2x_2 + 5 = 0$ I am trying to solve for the optimal points using the first and second derivative tests ...
-1
votes
0answers
13 views

What is the difference between MSE and LSE when it comes to linear regression?

Specifically I want to know the difference between these two equations. Is one them used for a specific scenario while the other is used for different conditions? Least Squares Error Mean Squared ...
0
votes
0answers
8 views

Questions on Logistic Regression.

I have a few questions about Logistic Regressions. we can get a probability that certain 'y' happen in the situation of certain 'X'. And, Is there any error range of that probability? And Is there ...
2
votes
0answers
9 views

Finding the conditional probabilities of a latent dirichlet allocation model

Let's say I'm defining a LDA as the following: For each doc $m$: Sample topic probabilities $\theta_m \sim Dirichlet(\alpha)$ For each word $n$: Sample a topic $z_{mn} \sim Multinomial(\theta_m)$ ...
0
votes
0answers
52 views

I'm asked to differentiate this $\dfrac{C}{2} \sum^{m}_{j' = 1} \| W^{j'} \|^{2}_{2}$ but I barely understand the notation.

I'm asked to differentiate this : $$\dfrac{C}{2} \sum^{m}_{j' = 1} \| W^{j'} \|^{2}_{2}$$ according to $w^{j'}_{k}$ which is the $k$th weight of the vector of weight of $j'$. It seems that $\| W^{j'} ...
1
vote
0answers
20 views

Questions about polynomial algbras

If $x$ is vector of dimension 3, say $(x_1,x_2,x_3)^T$, then $S^d$ is an operator such that $$ S^d(x)=\left( \begin{matrix} x_2 & x_1 & 0 &0 & 0& ...& 0 &0&0\\ ...
0
votes
1answer
26 views

Vector Calculus questions from the Mathematics for Machine Learning Book

I have been working through the exercises for the vector calculus section to gain some practice and have got various solutions I would be grateful if the community could check (There is no solution ...
0
votes
0answers
26 views

Finding gradient descent with inner product of matrices [closed]

In this problem, we consider the matrix sensing problem that was mentioned briefly during the lecture. Suppose that we want to learn an unknown positive semidefinite matrix $X^* = U^*U^{*T}$ > , ...
0
votes
1answer
61 views

Prove that the k-fold cross validation error is an almost unbiased estimate of the risk

In the setting of linear regression: The prediction error using k-fold cross-validation is given by : \begin{equation}\label{eq1} \begin{split} err_{k-fold} = \frac{1}{k} \sum_{i=1}^{k} \frac{1}{n/k}\...
0
votes
1answer
54 views

Maximum Entropy Continuous Distribution

In Pattern Recognition and Machine Learning Ch 1.6, the author derives the distribution which maximises the differential entropy; $$H(\textbf{x})-\int p(\textbf{x}) \ln (p(\textbf{x})) d\textbf{x}$$ ...
0
votes
0answers
8 views

How should I proceed and treat the Sigmoid in proving the Lipschitz condition for regularized logistic regression?

I have come across this question to prove the gradient of a regularized logistic regression is lipschitz continuous. I have derive the gradient, I can easily see the proof for linear regression ...
0
votes
1answer
20 views

How to minimize the KL divergence with respect to fixed parameters?

I read the LDA paper multiple times but I'm having trouble with the following. Let's say I define a LDA model as: For each doc $m$: Sample topic probabilities $\theta_m \sim Dirichlet(\alpha)$ For ...
1
vote
0answers
19 views

Enforcing local invariance in a covariance (kernel) function

I am doing some Bayesian inference over a particular domain and the functions to be inferred are known to posses certain invariances (symmetries). In particular there are 2 types of invariance that I ...
1
vote
0answers
15 views

Hierarchical Clustering with Ward Distance

I know how hierarchical clustering (with a certain definition of inter-cluster distance) works. And I know that Ward's procedure is based on the goal of minimizing the sum of the squared errors ...
0
votes
1answer
25 views

Performance of linear regression models

Linear regression models can have accuracy vs performance issues. If we have a linear regression model implemented in each of the following ways: 1)By solving Normal equations 2)Stochastic Gradient ...
0
votes
1answer
42 views

Training error in linear regression can always be made zero, given certain condition?

I was going through the derivation of linear regression and wanted to verify that what I understood is indeed correct? We minimize this $$ {\lVert y -Xw \rVert}^2 $$ where y is a $n*1$ dimensional ...
0
votes
1answer
35 views

How to check if the inverse exist by analyzing the matrix?

I was going through the solution for least square linear regression that comes out to be $$ w_{ls} = {(X^TX)}^{-1} X^Ty $$ Now, this exist when ${(X^TX)}^{-1}$ exist, which means ${(X^TX)}$ is ...
1
vote
1answer
45 views

why we don't calculate global minimum or maximum with looking the plot [closed]

Sorry I'm kind of newbie on optimization. As you know we can plot the multivariate functions. example plot So we can see the global minumum and maximum. so why we run gradient descent to calculate it ?...
0
votes
0answers
11 views

Minimizing The Expected Loss Of a Classification Problem

I found the following squared-loss function for the regression of classification problem: I learned that the expected loss of any classification problem is given by the following: $\begin{gather*} ...
0
votes
1answer
13 views

How is iteratively reweighted least squares used for $L^p$ norm linear regression?

The iterative scheme that I see everywhere in this context is $$\theta _{k+1}=\left(X^{\:t}W_k\:X\right)^{-1}\left(X^{\:t}W_k\:Y\right)$$ With the weight $W_k$ being a diagonal matrix of $$w_i=\left(...
0
votes
0answers
35 views

Prove Convexity Multinomial Logistic Loss

I have been asked to prove the multinomial logistic loss is convex with respect to the model parameters. I have managed to compute the first gradient: $$\nabla_W-\log[softmax(z)]_j = -(\phi(x_j) - \...
0
votes
0answers
9 views

k-NN complexity

Let the sequence $\mathcal{X} = \{ \mathbf{x}^t\}_{t=1}^N$ and the estimation of a density function ($k$-NN) $$\hat{p}(x) = \frac{k}{2N d_k(x, \mathcal{X})}$$ where $d_k()$ is just a measure of ...
1
vote
0answers
21 views

Applications of Gradient Descent Methods

I'm trying to find applications of the batch gradient, stochastic gradient, and mini-batch gradient descent method. Most applications (on Google Scholar) seem to be too theoretical or, in the area of ...
1
vote
0answers
23 views

Is the construction of gradient for this infinite dimensional Hilbert space well-defined?

Motivated by functional gradient in the paper Functional Gradient Techniques for Combining Hypotheses about Machine Learning. I tried to make it rigorously. Could you please verify if my understanding ...
1
vote
1answer
20 views

Query about the the $ℓ_k$ norm, where $k$ = $0$.

I am reading through Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, by Aurélien Géron. Chapter 2 he introduces the idea of a general vector norm as follows: "More generally, ...
0
votes
0answers
35 views

Is this prove of the basic simple hypothesis by induction correct or not?

I have a well-known hypothesis in Machine Learning, which I have to prove it. I tried to prove it by induction, but not sure if it's correct like this: The Hypothesis: Suppose that all the ...
0
votes
1answer
36 views

How should I interpret constraint $x^T u_1=0$ in a quadratic function optimization?

In a linear algebra book, the following is stated with proof shown, so I get it. Let A be a symmetric matrix and so $x^TAx$ the quadratic form of a quadratic function. Arrange the eigenvalues such ...
0
votes
2answers
48 views

What is special about $(1-\alpha )\cdot f(x) + \alpha \cdot f(y)$?

I see the expression $(1-\alpha )\cdot f(x) + \alpha \cdot f(y)$ in many places: In the definition of a concave function: $$ {\displaystyle f((1-\alpha )x+\alpha y)\geq (1-\alpha )f(x)+\alpha f(y)}$$ ...
1
vote
0answers
36 views

Visualizing data using vectors

Say there are 10 houses and we have three pieces of information for each of them, area, nbedrooms, price I can view this as 10 different vectors in space where there are 3 axes. Basically 10 arrows ...
2
votes
1answer
77 views

Understanding an Inequality from the AdaGrad paper involving expectation

Reading the paper on AdaGrad, an optimization method for machine learning, I am coming across an inequality I do not understand on page 5, available here Denote $g_{1:T}$ as a matrix $G_T=[g_1, \ldots ...
3
votes
1answer
68 views

Maximum number of weights in a neural network

Suppose we have a neural network with an input layer, $n$ hidden layers, and an output layer. The input layer has $d_o$ units ($d_o-1$ inputs and a bias). Each hidden layer has $d_i$ units (bias is ...
0
votes
0answers
20 views

calculating distance between two distributions from samples of different size

I'm looking for some general guidance on where to look if I want to solve this problem I encountered: I have two unknown distributions- p and q, from the same family of distributions. Just for the ...
0
votes
1answer
15 views

Convergence analysis of Optimisation algorithms using Regret Bound.

I was reading a research paper https://arxiv.org/pdf/1707.01647.pdf which talked about the convergence analysis of different Machine learning optimisation algorithms (in convex domain). i am not able ...
0
votes
0answers
18 views

How to calculate $v_{k}$ in reinforcement learning?

Given the above grid world reinforcement learning problem and solution, I wonder how to calculate the $v_{k}$ for it? (Solutions are attached above but I don't know how the numbers are calculated)
6
votes
3answers
158 views

Application of the chain rule to $3$-layers neural network

Consider the differentiable functions $L^1(x,\theta^1),L^2(x^2,\theta^2),L^3(x^3,\theta^3)$, where every $x_k,\theta^k$ are real vectors, for $k=1,2,3$. Also define $\theta=(\theta^1,\theta^2,\theta^3)...
2
votes
3answers
219 views

Application of chain rule, and some recursion

Consider the differentiable functions $L^1(x,\theta^1),L^2(x^2,\theta^2),...,L^l(x^l,\theta^l)$, where every $x_k,\theta^k$ are real vectors, for $k=1,...,l$. Also define $\theta=(\theta^1,...,\theta^...
1
vote
0answers
19 views

Using a poset or directed graph as input for a neural network.

As the title states, I'm trying to train a neural network using some unconventional input. I'm wondering if anyone has any experience or has read any papers that involve using a partially ordered set ...
2
votes
1answer
55 views

$\lim\limits_{m, l\to\infty} \frac{m+a}{m+a+l+b}$ where $m + l = N$

I have been working on Bishop's book, Machine Learning and Pattern Recognition. The page 73 says, the limit of an infinitely large data set $m, l \rightarrow \infty$, the result $\frac{m+a}{m+a+l+b}$ ...
0
votes
1answer
32 views

How can one show that $argmin_{a,b}\:\left\{\:\sum \:\left|y-\left(a+bx\right)\right|\right\}$ can lack a unique solution

I'm trying to wrap my head around minimizing the L1 norm, it often possesses infinite solutions despite being a convex function(multiple global optima.) Does that property have a name? and how can one ...
0
votes
1answer
17 views

Loss Function for $l_0$ norm

Suppose we have $n$ samples of data with feature $Y$. Derive solution for the optimal constant $a^\star_0$ \begin{align*} a^* \in argmin_{a \in \mathcal{R}} \frac{1}{n}\sum_{i=1}^n |Y(i)-a|^0 \end{...
0
votes
0answers
13 views

Finding threshold to separate two distributions

I am following this paper, that aims to find anomalous images. For that they came up with an anomaly score for each image. This metric is not normalized. I do not fully understand how they specify an ...
0
votes
0answers
41 views

Optimal Set Partitioning

Consider a set of integers $1$ to $n$, $$ \mathcal{S} = \{1,2,3,\ldots,n\} $$ initialised with no-partitions. Now let $L[f(\mathcal{S})]$ be a cost function we wish to minimise that depends on some ...
4
votes
0answers
34 views

sigmoid function prove

in machine learning, sigmoid function is used to maximize the likelihood, right ? $a(x) = \frac{1}{1+e^{-x}}$ {sigmoid function} which will give me the probability of success, now it's used when you ...
3
votes
2answers
63 views

Reference Request: Machine Learning from Scratch

I have a background in Software Engineering and the question is a bit related to CS theory, but I find it more appropriate to post my question here. I want to get started on Machine Learning but haven'...
2
votes
0answers
16 views

VC-Dimension Combination of Restructions

Let $F\subseteq C([0,1]^n,\mathbb{R})$ be a finite family of functions, which is non-empty. Let $A,B$ be subseteq of $[0,1]^n$, again non-empty, and let $vc-dim(C)$ denote the Vapnik–Chervonenkis ...
2
votes
0answers
40 views

Why scaling down the parameter many times during training will help the learning speed be the same for all weights in Progressive GAN?

The title is one of the special things in Progressive Gan, a paper of the NVIDIA team. By using this method, they introduced that Our approach ensures that the dynamic range, and thus the learning ...
0
votes
0answers
16 views

Calculate Gini Index Using a data Table

I have a question about calculating Gini index from a given table. Given Problem I dont need the answer, but I do need to see if the work I have done so far is the step in right direction. I dived the ...
0
votes
0answers
15 views

Analytic solution for $x = \frac{n}{1 + e^{-ax + t}}$ i.e. When is the output of a parametrised logistic function equal to the input?

I would like to know when is the input to a parametrised logistic function (the right hand side) equal to its output. I've been trying to solve the following equation: $$x = \frac{n}{1 + e^{-ax + t}}$$...
0
votes
2answers
22 views

Plot the decision boundary for Bayes

Let $X=(X_1,X_2) \in [0,1]×[0,1]$ and $Y \sim Bernoulli(p=X_1⋅X_2)$. The Bayes decision boundary $\{(x_1,x_2):P(Y=1|X=(x_1,x_2))=0.5\}$ in the regions $[0,1]×[0,1]$ whose points would be classified as ...
1
vote
0answers
15 views

Linear regression feature is combination of other features, but has OPPOSITE correlation?

We're examining "salesperson cost per sale," where cost is just what the agent is paid in compensation. In other words, our target variable is the ratio of (salesperson pay)/(count of sales)....

1
2 3 4 5
48