5
$\begingroup$

Cross-posted on Operations Research SE

I'm reading a paper where the goal is to determine the weights of a weighted arithmetic mean to estimate a new sample from a random variable. These weights weigh a vector of samples of a known portion of the samples' population to estimate it:

$$\hat{y}_k = \frac{w^Ty}{w^T1}\tag{1}$$, where w, y and 1 are all vectors of size $N$.

Now every sample of the vector y is described by some numerical features of size $m$. Let's call this collection matrix X. And also the sample we are trying to estimate has this vector of features.

What they are trying to do is to estimate the weights by looking how similar each sample of the samples in vector y is compared to the new sample we are trying to estimate. Then a more similar sample will have more weight and will thus have more impact on the value of our estimate of the new sample.

To measure the similarity, they use the Euclidean distance on the vector of features that describes each sample. In the paper, they normalize each feature!
This will give us an Euclidean distance matrix A of size $N \times m$.

$$A^{(n)} = (X - 1x^n)\space\odot\space(X-1x^n) $$ With the superscript (n) to indicate that the result has been calculated wrt the new sample (the one we are trying to estimate) and $x^n$ the vector of features that describes the new sample.

But every numerical feature has its relative importance, so we also want to find a vector v that scales the matrix A so that you eventually get a vector of size $N$.

So the more similar, i.e. the lower the scaled Euclidean distance is for a sample, the more weight we want to give it. So the weights are inversely proportional to the scaled Euclidean distance which is obvious.

And here comes the question:

In the paper they take the inverse square root of the Euclidean distance matrix to define the relationship between the weights and the similarity:

$$\text{weight}_j = \frac{1}{\sqrt{a^iv}}\tag{2}$$, where $a^i$ is the $i$th row of the matrix A and $j$ represents each of the known samples.

But why do they use the square root? Why not just:

$$\mathrm{weight}_j= \frac{1}{a^iv}$$ or $$\mathrm{weight}_j= \frac{1}{(a^iv)^2}$$

Why do we use the inversed square root and not just the inverse or the inversed square to define the proportionality ?

I can think of some reasons but it's not clear if these are the right ones:

  • A is a matrix of squared distances and thus maybe they use the square root to convert back to unit distance. But in optimization theory, often squared distances are preferred, so that's kinda weird.
  • They want to use the squared distance but the relationship between the weights and the squared distance is assumed to be non-linear (else it would just be the inverse) and twice the squared distance between two events means the impact on the forecast will be lower than two (else it would be the inverse square root).
  • Numerical stability (because of the normalized features and the squared distance matrix that's being built).
  • The square root has some mathematical properties in optimization that these other two don't have (feel free to elaborate on this).

So the answer I'm looking for depends on the reason behind using the square root,

or it necessary to be mathematically consistent; then I want to know: how so?

or it is just a model assumption and best practice: then I want to know: why is it best practice to use the square root?

Important:

To be complete: the paper continues with using Eq. (2) and substituting it to the Eq. (1) to get an optimization problem which is used to find the vector v: $$\hat{v} =\operatorname{argmin}_{v^*}\left[ \left\lVert y_k - \frac{\sum_{i=1}^n \frac{1}{\sqrt{a^iv}}y_i}{\sum_{i=1}^n \frac{1}{\sqrt{a^iv}}}\right\rVert^2 \right]$$ Once v is found, we calculate the weights with Eq. (2):

$$\text{weight}_j = \frac{1}{\sqrt{a^iv}}$$ and using this in the Eq. (1):

$$\hat{y}_k = \frac{w^Ty}{w^T1}$$ to eventually get our estimate.

Reference of the paper: https://oa.upm.es/67130/1/INVE_MEM_2019_334744.pdf

$\endgroup$
6
  • $\begingroup$ It would help your question if you explicitly showed the equations in which these weights are used. It would help your formatting if you used \text or \mathrm macros to render weight! $\endgroup$
    – FShrike
    Dec 18, 2021 at 20:11
  • $\begingroup$ See you edited the question, thanks! I edited the question's content as well. $\endgroup$ Dec 18, 2021 at 20:38
  • 1
    $\begingroup$ You can also use the macro \operatorname{x} to render nonstandard functions and operators properly, e.g. \operatorname{argmin}_{v^*}, and someone more MathJax savvy than I probably knows how to get the $v^*$ fully underneath in this natural way (without using \underset) $\endgroup$
    – FShrike
    Dec 18, 2021 at 21:10
  • $\begingroup$ More specific titles are good too. I read and upvoted your post but many people would glance at the title “why do we use the square root for a problem” and not click on it for more details! I did my best based on my limited knowledge of the topic, but if the optimisation problem is non-linear or if you can be more specific please edit the title further accordingly. $\endgroup$
    – FShrike
    Dec 18, 2021 at 21:13
  • $\begingroup$ If the paper has been published, it would be nice to have a reference, as it might shed some light on the reasons of this choice, or at least give some context. $\endgroup$ Dec 20, 2021 at 10:11

0

You must log in to answer this question.

Browse other questions tagged .