0
votes
1answer
18 views

Expanding variance

Could someone please expand on line 2 and 3 of: Thank you.
0
votes
0answers
17 views

Deriving the optimal value for the intercept term in SVM

I was reading andrew ng's machine learning lecture notes on SVM. I came across the following equation (finding the optimal value for the intercept term $b$ in the SVM problem): However, I have no ...
0
votes
1answer
37 views

Maximum of two skewed normal distributions

Does there exist a means to approximate the maximum of two skewed normal distributions in terms of another skewed normal distribution? To make it clearer, given 2 skewed normal distributions ...
0
votes
0answers
20 views

Randomized optimization: confidence bounds

I am not sure whether the question is appropriate for MSE, so please feel free advise another website in the network. Suppose I have a Lipschitz continuous function $f:X\to \Bbb R$ where $X$ is a ...
0
votes
0answers
30 views

Why is Expectation Maximization algorithm guaranteed to converge to minimum, even local?

I have read a couple of explanations of EM algorithm (e.g. from Bishop's Pattern Recognition and Machine Learning and from Roger and Gerolami First Course on Machine Learning). The derivation of EM is ...
0
votes
0answers
64 views

Minimizing the Kullber-Leibler divergence between two multivariate normal distributions

Take two zero-mean multivariate normal distributions: $p=\mathcal{N}(\mathbf{0},\boldsymbol\Sigma)$ and $q=\mathcal{N}\left(\mathbf{0},\left(\mathbf{A}^{T} \boldsymbol\Omega ...
2
votes
0answers
185 views

Mana Maximization (Hearthstone)

I recently started playing Hearthstone and a statistic / probability question came up my mind. Here's a quick breakdown: The game is a turn-based card game which involves "points" that you can used ...
0
votes
1answer
23 views

Zisserman Lecture and $x_{MLE}$

In the Zisserman Lecture below http://www.robots.ox.ac.uk/~az/lectures/est/lect34.pdf page 36, he derives $x_{MLE}$ for Gaussian sensor fusion. There are two noisy measurements $z_1$ and $z_2$ ...
1
vote
1answer
51 views

Example calculation of estimating GMM parameters using EM

I'm trying to study expectation maximization and I've almost got the idea. What I'm missing is a concrete example. Could someone familiar with the subject give me a concrete example how one would ...
0
votes
5answers
123 views

How to find the minimum of the function?

How to find the minimum of the following function $$ {\rm f}\left(w\right) = {1 \over 2}\sum_{i = 1}^{n}\left({1 \over 1 + {\rm e}^{-x_{i}\,w}} -y_{i}\right)^{2} $$ where $x_{i}, y_{i} \in \left(0, ...
1
vote
0answers
24 views

Is the Support Vector Classifier in some sense optimal?

My question is, is the original hard-margin support vector classifier optimal in some sense? If you have an answer that refers to the soft-margin SVC instead, I'd also be interested. I know that the ...
0
votes
0answers
28 views

Probability of Sampling Matching of Two Independent Schemes in Time

I have encountered a very specific problem, that I don't even know the keywords to search for it. Here is the explanation: There are two independent sampling schemes like in the image below: I want ...
0
votes
0answers
35 views

Using Newton's method to find an optimized matrix

I'm trying to apply Newton's method to find a local optimum of the matrix $\Sigma$ to minimize the objective function: $f(\Sigma) = -\sum_{n=1}^{N}\left(-\ln{2\pi} - ...
0
votes
1answer
77 views

Minimization of Sum of Squares Error Function

Given that $y(x,{\bf w}) = w_0 + w_1x + w_2x^2 + \ldots + w_mx^m = \sum_{j=0}^{m} w_jx^j$ and there exists an error function defined as $E({\bf w})=\frac{1}{2} \sum_{n=1}^{N} \{y(x_n, w)-t_n\}^2$ ...
1
vote
0answers
48 views

Reformulating objective function of canonical correlation analysis

Given two column vectors $X = (x_1, \dots, x_n)'$ and $Y = (y_1, \dots, y_m)'$ of random variables with finite second moments, canonical-correlation analysis seeks vectors $a$ and $b$ such that the ...
1
vote
0answers
33 views

Simplifying a difficult expression to input it in R

The function is $\Large f(X,Y|\mu_1,\mu_1,\theta)=\frac{\phi (X-\mu_1)\phi (Y-\mu_2)\theta(1-e^{-\theta})e^{-\theta (\Phi(X-\mu_1)+\Phi(Y-\mu_2))}}{[1-e^{-\theta}-(1-e^{-\theta \Phi ...
0
votes
1answer
64 views

online learning to maximize profit

I have a software which takes input as investment and gives the output as return on a particular stock. Now profit metric $x_i$ is defined as the ratio of return $g_i$ to maximum possible return ...
1
vote
0answers
26 views

Minimization values for a function [duplicate]

I have got function - non linear(I thk), and a set of variable S=[(x1,y1),(x2,y2)...]. The objective is to find the value for ...
-3
votes
2answers
93 views

Guess the functional form of a graph

Can you guess the functional form of the following curve y is 0 at x= Infinite ; y is very small ( +ve near to zero) at x=0 Thanks and regards
1
vote
0answers
84 views

When would Least Square Estimate equals Maximum Likelihood Estimation?

Under what situations, MLE(Maximum Likelihood Estimation) would equal to LSE(Least Square Estimate)? I got an impression that under norm 2, MLE and LSE is equal. For example the question $min || ...
4
votes
1answer
242 views

Maximum of absolute value of linear combinations with i.i.d random variables

Suppose $x_{1},\dots,x_{n}$ are i.i.d random variables with density $p(x_{i})=exp(-|x_{i}|)/2$. Denote column vector $x=(x_{1},\dots,x_{n})^{T}$ Let $C\in\mathbb{R}^{n\times n}$ be a matrix with unit ...
0
votes
0answers
34 views

Confidence Interval for finding a minimum value

I have a machine (neural network) that is tuned by a parameter p, and outputs an error e depending on the parameter choice. The error curve is non-linear. I iterate through the valid p parameters ...
0
votes
0answers
23 views

Equivalent form of Dantzig selector / optimization problem

The Dantzig-selector is the solution to $\min_\beta \|\beta\|_1 \;\;\; s.t. \;\;\; \|X^T(y - X\beta)\|_\infty \leq s$ where $X \in \mathbb{R}^{n\times p}$, $y \in \mathbb{R}^n$, and $\beta \in ...
3
votes
1answer
54 views

Rewrite constrained optimization objective

I wanted to ask, under which conditions can one rewrite the optimization objective $\min_x f(x)\;\;\;s.t.\;\;\;g(x) \leq s$ as $\min_x g(x)\;\;\;s.t.\;\;\;f(x) \leq t$ I have particular interest ...
2
votes
1answer
89 views

The relationship between fisher information and EM algorithm?

I wonder what is the relationship between fisher information and EM algorithm? When I read papers about EM algorithm, people sometimes discussed about fisher information, and there are algorithms ...
0
votes
0answers
73 views

L1 penalty can serve as a convex surrogate for an L0 penalty. Why?

I have heard machine learning practitioners say that the $L_1$ penalty is a (or can serve as) convex surrogate for an $L_0$ penalty (in the context of optimization and statistical fitting). What do ...
1
vote
1answer
263 views

Maximum likelihood estimators, hypergeometric and binomial

I'm trying to solve a two part problem. The set up is as follows: consider a bag with $\theta$ red marbles and $7-\theta$ blue marbles, with $\theta$ being unknown. Let $x$ denote the number of red ...
2
votes
3answers
118 views

Packing radios into cartons - why is my solution wrong?

A manufacturer of car radios ships them to retailers in cartons of $n$ radios. The profit per radio is $\$59.50$, minus shipping cost of $\$25$ per carton, so the profit is $59.5n-25$ dollars per ...
1
vote
1answer
61 views

Non-trivial solution for $2*a^k = b^k + c^k$

I have a data set where I want its median to be the arithmetic average of maximum and minimum by multiplying every value with a factor $k$ and then applying the exponential function. This leads to the ...
6
votes
1answer
555 views

Why are additional constraint and penalty term equivalent in ridge regression?

Tikhonov regularization (or ridge regression) adds a constraint that $\|\beta\|^2$, the $L^2$-norm of the parameter vector, is not greater than a given value (say $c$). Equivalently, it may solve ...
-1
votes
1answer
117 views

Inverse transform sampling

I know the basic idea is to generate a random number from $U(0,1)$, find the inverse cumulative distribution function $F^{-1}$ and then take $x = F^{-1}(U)$. If you were plot a histogram of say 1000 ...
1
vote
1answer
73 views

How to sample point from triangle where vertex is not in origin

This link http://mathworld.wolfram.com/TrianglePointPicking.html gives an overview of how to sample points from either a quadrilateral or triangle given one vertex is at the origin. The standard ...
1
vote
0answers
55 views

Statistical significance test in polygaussian fitting, using Levenberg-Marquardt

I have a set of dihedral angle values that I have fitted using a polygaussian function via the Levenberg-Marquardt algorithm http://en.wikipedia.org/wiki/Levenberg-Marquardt. Specifically, the ...
1
vote
1answer
44 views

minimum of absolute value

If we consider the following problem $$ \mathbb{E}[(Y-y)^2 | X=x] $$ I can easily show that the minimum with respect to $y$ occurs at $$ y=\mathbb{E}[Y |X=x] $$ How can I find the minimum of $$ ...
0
votes
1answer
66 views

Linear optimization of line under points.

I have a set of points in two-dimensional space. I'd like to find the line of best fit that minimizes the distance between the line and the points such that the line is below all points. How can I ...
2
votes
1answer
60 views

Regression with arbitrary norm

I have a function $f(X,Y,Z)$, which I know is polynomial of degree $3$. I have a set of samples $(X_i,Y_i,Z_i)$ and corresponding values of $f$. My task is to find (the best approximation of) the ...
2
votes
1answer
155 views

Optimization, Gradients, and Multivariate Data

I would like to learn gradient based optimization for multivariate data. For example, assume the data I have is $X = (x_0, ..., x_n)$ where $x_i$ are some random variables and $f$ a function ...
1
vote
0answers
178 views

Gradient Descent for Primal Kernel SVM with Soft-Margin(Hinge) Loss

Given the primal objective $$F({\bf a})=L\sum_{i,j}a_{i}a_{j}k(x_i,x_j) + \sum_{i}max(0, 1-y_i \sum_{j}a_jk(x_i,x_j)$$ for the soft margin SVM, where ${\bf a}=(a_1,...,a_N)$, N being the number of ...
0
votes
1answer
22 views

online estimation of autoregressive process

I am interested about online estimation of autoregressive models. Is there anything I can find in the literature about this topic?
0
votes
1answer
52 views

How to find optimal border that defines, who is “friend”

I have the data about usage of several services in the population and the data about interactions among users. The idea is to determine, who is user's friend and who has interacted just ...
3
votes
2answers
69 views

Closing 3 numbers

I have 3 numbers that physically must add up to zero. Unfortunately, each is obtained from a noisy measurement and they don't add up exactly. Assuming the noise is Gaussian and given 3 corresponding ...
0
votes
1answer
106 views

Maximizing the expectation of a function with a constraint

I am interested in computing the maximum of the following function $E[ ((XB) - E(XB))^TR ] $ subject to the constraint $E(XB)=1$ where $X$ is a $n\times m$ random matrix, $B$ is a $m \times 1$ ...
2
votes
0answers
63 views

Root Convergence rate of Iterative Scheme [closed]

I have an iterative sequence for optimizing an EM algorithm based loss function $L(X)$ with $t$ being the iteration number as: $X_t=ABX_{t-1}+CX_{t-1}+X_{t-1}$ where $A$ is a diagonal matrix, $B$ and ...
0
votes
1answer
36 views

Confusion regarding ML estimate

I was going through this article and they have this log likelihood given by $$ LL = \sum_{i=1}^n A_i\log p_i + \sum_{i=1}^n A'_i\log(1-p_i).$$ Basically this is the loglikelihood of a logistic ...
1
vote
0answers
219 views

A convex programming problem involving sum of logarithms of linear functions

Here is a convex programming problem I encountered while working on an estimation problem for a mixture of multinomial distributions. We have a matrix $A_{m \times n}$ containing non-negative real ...
1
vote
2answers
38 views

Maximizing the time we reach to a threshold in a series of numbers

I have a problem and I really don't know what kind of mathematical method should I apply to solve or model my problem. I would be thankful If anyone can give me some answer or help. Suppose we have ...
0
votes
0answers
158 views

BUE (Best Unbiased Estimator)

Suppose we are given a matrix $V$ and our goal is to find non-negative matrices $W$ and $H$ such that $V \approx WH$. So we want to minimize $K(V || WH)$ (Kullback-Leibler Divergence) where $$K(V||WH) ...
2
votes
1answer
74 views

Drawing samples from an LP program

Say I have an LP program in standard form: \begin{equation*} \begin{array}{rl} \mathbf{x}^* = \underset{\mathbf{x}}{\text{arg}\;\text{min}} & \mathbf{c}^T\mathbf{x} \\ \mbox{s.t.} ...
1
vote
1answer
92 views

Derivatives with respect to a symmetric matrix, with an application to maximum likelihood

I am quite unsure about this whole matter of differentiation with respect to a matrix. First, I'd like a good (online hopefully) reference for getting up to speed on the theory - as opposed to a bunch ...
1
vote
0answers
217 views

Point-wise error estimate in polynomial regression

In our application we wish to estimate the actual path of objects. We have a set of samples of object locations $(t_i, x_i, y_i, P_i)$ where $t_i$ is the sample time, $(x_i, y_i)$ is the 2D location, ...