Cross-posted on Operations Research SE
I'm reading a paper where the goal is to determine the weights of a weighted arithmetic mean to estimate a new sample from a random variable. These weights weigh a vector of samples of a known portion of the samples' population to estimate it:
$$\hat{y}_k = \frac{w^Ty}{w^T1}\tag{1}$$, where w, y and 1 are all vectors of size $N$.
Now every sample of the vector y is described by some numerical features of size $m$. Let's call this collection matrix X. And also the sample we are trying to estimate has this vector of features.
What they are trying to do is to estimate the weights by looking how similar each sample of the samples in vector y is compared to the new sample we are trying to estimate. Then a more similar sample will have more weight and will thus have more impact on the value of our estimate of the new sample.
To measure the similarity, they use the Euclidean distance on the vector of features that describes each sample. In the paper, they normalize each feature!
This will give us an Euclidean distance matrix A of size $N \times m$.
$$A^{(n)} = (X - 1x^n)\space\odot\space(X-1x^n) $$ With the superscript (n) to indicate that the result has been calculated wrt the new sample (the one we are trying to estimate) and $x^n$ the vector of features that describes the new sample.
But every numerical feature has its relative importance, so we also want to find a vector v that scales the matrix A so that you eventually get a vector of size $N$.
So the more similar, i.e. the lower the scaled Euclidean distance is for a sample, the more weight we want to give it. So the weights are inversely proportional to the scaled Euclidean distance which is obvious.
And here comes the question:
In the paper they take the inverse square root of the Euclidean distance matrix to define the relationship between the weights and the similarity:
$$\text{weight}_j = \frac{1}{\sqrt{a^iv}}\tag{2}$$, where $a^i$ is the $i$th row of the matrix A and $j$ represents each of the known samples.
But why do they use the square root? Why not just:
$$\mathrm{weight}_j= \frac{1}{a^iv}$$ or $$\mathrm{weight}_j= \frac{1}{(a^iv)^2}$$
Why do we use the inversed square root and not just the inverse or the inversed square to define the proportionality ?
I can think of some reasons but it's not clear if these are the right ones:
- A is a matrix of squared distances and thus maybe they use the square root to convert back to unit distance. But in optimization theory, often squared distances are preferred, so that's kinda weird.
- They want to use the squared distance but the relationship between the weights and the squared distance is assumed to be non-linear (else it would just be the inverse) and twice the squared distance between two events means the impact on the forecast will be lower than two (else it would be the inverse square root).
- Numerical stability (because of the normalized features and the squared distance matrix that's being built).
- The square root has some mathematical properties in optimization that these other two don't have (feel free to elaborate on this).
So the answer I'm looking for depends on the reason behind using the square root,
or it necessary to be mathematically consistent; then I want to know: how so?
or it is just a model assumption and best practice: then I want to know: why is it best practice to use the square root?
Important:
To be complete: the paper continues with using Eq. (2) and substituting it to the Eq. (1) to get an optimization problem which is used to find the vector v: $$\hat{v} =\operatorname{argmin}_{v^*}\left[ \left\lVert y_k - \frac{\sum_{i=1}^n \frac{1}{\sqrt{a^iv}}y_i}{\sum_{i=1}^n \frac{1}{\sqrt{a^iv}}}\right\rVert^2 \right]$$ Once v is found, we calculate the weights with Eq. (2):
$$\text{weight}_j = \frac{1}{\sqrt{a^iv}}$$ and using this in the Eq. (1):
$$\hat{y}_k = \frac{w^Ty}{w^T1}$$ to eventually get our estimate.
Reference of the paper: https://oa.upm.es/67130/1/INVE_MEM_2019_334744.pdf
\text
or\mathrm
macros to render weight! $\endgroup$\operatorname{x}
to render nonstandard functions and operators properly, e.g.\operatorname{argmin}_{v^*}
, and someone more MathJax savvy than I probably knows how to get the $v^*$ fully underneath in this natural way (without using\underset
) $\endgroup$