2
$\begingroup$

Consider the softmax function $f \colon \mathbb{R}^k \rightarrow \mathbb{R}^k $ defined by $f((x_1, \ldots, x_n)) = (y_1, \ldots, y_n)$ where $y_i = \frac{e^{x_i}}{\sum_j e^{x_j}}$.

Can you show that for every $x, \Delta y,\delta$, there exists $\epsilon$ and $\Delta x < \delta$ such that $f(x + \Delta x) = f(x) + \epsilon \cdot \Delta y$? Also, is there a name for this property for general function? Assume there exists $x'$ such that $y + \Delta y = f(x')$.

$\endgroup$
0
$\begingroup$

I'm not so sure that you can make that statement. By the construction of the softmax, the sum of its entries is equal to $1$, in other words:

$$\sum_{i=1}^k [f(x + \Delta x)]_i = 1$$

and

$$\sum_{i=1}^k [f(x)]_i = 1$$

However, from what you've written,

$$ f(x+\Delta x) = f(x) + \epsilon \Delta y $$

It would imply that

$$\sum_{i=1}^k [f(x) + \epsilon \Delta y]_i > 1$$

which leads us to a contradiction if we sum over the entries of $f(x + \Delta x)$:

$$ \sum_{i=1}^k [f(x+\Delta x)]_i = \sum_{i=1}^k [f(x) + \epsilon \Delta y]_i > 1 $$

$\endgroup$
  • $\begingroup$ updated the question. I meant for any valid vector $\Delta y$. $\endgroup$ – listener Feb 1 at 5:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.