0
votes
2answers
25 views

Exposition of solving the quadratic programming problem for SVMs

I'm looking to find a mathematically rigorous exposition on how to solve the quadratic programming problem $$\min ||x||^2 \textrm{ subject to } Ax\leq b$$ where $x\in\mathbb{R}^n$, ...
1
vote
0answers
16 views

Baseline predictors model

I implemented baseline predictors model (like it is told in Recommender systems handbook pp 148-149). b_ui = mu + b_i + b_u where mu is overall average rating ...
0
votes
2answers
37 views

optimization problem: finding an hyperplane separating one point from a set of pointy maximizing the distance

I have this problem: I have a set of n-dimensional points $P$. I have one more n-dimensional point $q$. The points in $P$ are linearly separable from $q$ (i.e. it always exists an hyperplane $n^t x ...
0
votes
0answers
28 views

Proof that feature normalization cause faster convergence of gradient descent

How to prove that if I do feature normalization (scaling of the $x_1,\ldots,x_n$ to be all in range $[0,1]$) to a convex function $f(x_1,\ldots,x_n)$ that returns real scalar, then gradient descent ...
0
votes
1answer
22 views

Gradient descent with adaptive learning ratio.

I have a neural network, trained with SGD (stochastic gradient descent) with learning ratio $\alpha$. Each iteration I try to recalculate the weights with a rule: $$\Delta \vec{w} = -\alpha ...
0
votes
0answers
11 views

Learning a multivariate polynomial with dependent coefficients

I have a polynomial of the form of $ K^2((a-i)^2 + (b-j)^2 + c^2) = (ct)^2$ where $a,b,c,t$ are unknowns. I have multiple observation points for the values of $i,j,K$. Can I use some technique to ...
1
vote
1answer
33 views

Locally minimizing a concave function

What will happen if we minimize a concave function via gradient descent? Where does it get stuck? Intuitively a concave function has more structure than an arbitrary function, and seem to be easier ...
0
votes
1answer
99 views

How do you minimize “hinge-loss”?

A lot of material on the web regarding Loss functions talk about "minimizing the Hinge Loss". However, nobody actually explains it, or at least gives some example. The best material I found is here ...
0
votes
1answer
15 views

Maximize marginal log likelihood

I was reading this tutorial on expectation maximization and in section 3 the author mentions that a marginalization inside a log function is difficult (impossible?) to take the derivative of. I am ...
0
votes
0answers
23 views

Deriving the optimal value for the intercept term in SVM

I was reading andrew ng's machine learning lecture notes on SVM. I came across the following equation (finding the optimal value for the intercept term $b$ in the SVM problem): However, I have no ...
2
votes
0answers
41 views

Simplify huge optimization task (updated)

Shorten and simplified explanation of a problem: We have a sequence of variables from R of length S : $SEQ = [x_1^{(1)} \ x_2^{(2)} \ x_4^{(3)} \ x_2^{(4)} \ x_{12}^{(5)} \ x_6^{(6)} \ x_{15}^{(7)} ...
3
votes
0answers
39 views

Mathematical analysis of e-shop

I'm ukrainian student, studying applied mathematics in Kiev. I have an online store and some statistics data on it's work. Also I've learned a bit about optimization problems and operation reasearch. ...
0
votes
0answers
10 views

Loss specific inference in graphical models

As far as I have seen, in graphical models, the inference (for training or parameter estimation) is done via maximizing likelihood. While in many applications people need loss specific optimization of ...
7
votes
1answer
397 views

Gradient Descent on Non-Convex Function Works But How?

For Netflix Prize competition on recommendations one method used a stochastic gradient descent, popularized by Simon Funk who used it to solve an SVD approximately. The math is better explained here ...
1
vote
1answer
73 views

Analytical Method to Minimise a Cost Function

I've been researching optimisation methods used to minimise the cost function of a neural network, such as: Back propagation, which calculates the error at each node in order to calculate the partial ...
6
votes
0answers
542 views

Optimization / personalization within clusters

I have the following optimization problem: I have a (random and very noisy) objective function f(A, P), where A is a vector of "observable" parameters of the input and P is the parameters that I can ...
0
votes
2answers
80 views

machine learning optimization

I was studying SVM and I am having problems in the conversion of this optimization problem into another : and gamma_hat is defined by I had to paste the images because I was having troubles with ...
0
votes
0answers
38 views

Online machine learning algorithm for dynamical system

I have a complex dynamical system which takes input as x1, x2, x3 and gives output as y1, y2, y3. I don't have any mathematical model of the system. x(k) is the present input to the system and y(k) is ...
1
vote
0answers
30 views

Is the Support Vector Classifier in some sense optimal?

My question is, is the original hard-margin support vector classifier optimal in some sense? If you have an answer that refers to the soft-margin SVC instead, I'd also be interested. I know that the ...
0
votes
0answers
39 views

Finding an optimal set of weights for combining correlated classifiers

In order to combine classifiers that are correlated with one another, I would need to solve the following optimization problem: Find a vector $\mathbf{w}$ that minimizes $\mathbf{w}^T M \mathbf{w}$ ...
0
votes
0answers
17 views

Confusion related to least squares

I was reading this paper where they have modeled the ys given some samples xs,ys as The paper states that the above optimization problem is equivalent to a least squares problem. I didn't get how ...
0
votes
1answer
69 views

online learning to maximize profit

I have a software which takes input as investment and gives the output as return on a particular stock. Now profit metric $x_i$ is defined as the ratio of return $g_i$ to maximum possible return ...
0
votes
1answer
55 views

SVM - Min square norm

All Support Vector Machine litterature mentions that optimal hyperplane is found as: max 1/∥x∥ (st. constraints) which translates directly to: min ∥x∥ or equivalently min $ ∥x∥^2 $. Here ...
1
vote
1answer
119 views

Gradient descent/ nonlinear optimization intuition needed

all. I'm taking an introductory AI class, and we're using the gradient descent algorithm to find the optimized/ lowest cost of a set of thetas (variable coefficients) to best fit a regression line. In ...
5
votes
3answers
701 views

Lagrange multipliers and KKT conditions - what do we gain?

I'm working through an optimization problem that reformulates the problem in terms of KKT conditions. Can someone please have a go at explaining the following in simple terms? What do we gain by ...
0
votes
0answers
66 views

Lagrangian Duality Complementary Slackness solution

If $$\alpha^*,\mu^* $$ is the solution of optimalization $$\max_{\alpha, \mu}\mathcal{L}(\omega, \xi, b, \alpha, \mu)$$ How I can show that "complementary slackness condition" $$\alpha_i^* = 0$$ ...
1
vote
0answers
200 views

Gradient Descent for Primal Kernel SVM with Soft-Margin(Hinge) Loss

Given the primal objective $$F({\bf a})=L\sum_{i,j}a_{i}a_{j}k(x_i,x_j) + \sum_{i}max(0, 1-y_i \sum_{j}a_jk(x_i,x_j)$$ for the soft margin SVM, where ${\bf a}=(a_1,...,a_N)$, N being the number of ...
0
votes
1answer
196 views

Subtraction of slope in gradient descent

In the gradient descent algorithm say $f(x)$ (quadratic function) is the objective function. SO the algorithm is defined as $$x_i = x_i - a\frac{\partial f(x)}{\partial x_i}$$ I Just dont quite ...
2
votes
2answers
220 views

Gradient descent vs ternary search

Consider a strictly convex function $f: [0; 1]^n \rightarrow \mathbb{R}$. The question is why people (especially experts in machine learning) use gradient descent in order to find a global minimum of ...
4
votes
3answers
733 views

Batch vs incremental gradient descent

I am studying Machine Learning, but I believe you guys should be able to help me with this! Basically, we have given a set of training data $\{(x_1,y_1), x(x_2,y_2), ..., (x_n, y_n)\}$, and we need ...
1
vote
1answer
275 views

how to find the input for this optimization problem?

Suppose I have a neural network, with input variables $a,b,c,d,f,g$ and output variables $m,n,o,p,q$. Given different input values, the neural network will output corresponding $m,n,o,p,q$. Now I ...