1
$\begingroup$

I have been reading a paper about dilated convolutions in neural networks and there is an equation which I don't understand:

However, we found that random initialization schemes were not effective for the context module. We found an alternative initialization with clear semantics to be much more effective: $$ k^b(\mathbf{t}, a) = 1_{[\mathbf{t}=0]}1_{[a = b]} \tag{4}\label{eq4}$$ where $a$ is the index of the input feature map and $b$ is the index of the output map. This is a form of identity initialization, which has recently been advocated for recurrent networks (Le et al., 2015). This initialization sets all filters such that each layer simply passes the input directly to the next.

I do not understand the notation the authors use. In particular:

  1. Superscript of the function $k$

I am only familiar with superscripts used for derivatives of functions

  1. Subscript of a number with square brackets and some condition

I find this part odd as if $\mathbf{t} = 0$ and $ a = b$, then the result is still one (I assume multiplication) but if neither of them are true, no value is specified.

  1. Bolded text for (I believe) non-vectors

I assume $\mathbf{t}$ is not a vector since the author had written $\mathbf{t} = 0$, and $0$ is not a vector

I can only guess that the kernel could be one that always returns 1, but that could have been written much more simply (like $k(x) = 1$ or something like that). I would like to know what the notations represent and understand what the author is trying to say.

Searching for answers is difficult, as "superscript of function" usually returns results about derivatives or exponents which would not make sense for a function that returns a kernel. "Subscript of number" or anything similar also usually explains that subscripts are used to denote different but related values, which is not what is used here.

$\endgroup$
1
$\begingroup$
  1. Superscript of the function $k$

The superscript is used to index the sets of kernels. In this particular context, it plays the same role as a subscript, that is, you have a collection of kernel functions and you indicate to the $b$-th one with the notation $k^b$.

Note that the authors also use subscript notation, but it is used to indicate the layer number (eq. 3). So, $k_i^b$ means the $b$-th kernel at layer $i$.

In the machine learning community using the superscript notation for indexing is somewhat common; see here a discussion.

  1. Subscript of a number with square brackets and some condition

This notation is known as Iverson bracket; it "casts" a boolean expression to an integer (either $0$ or $1$):

$$ \mathbf{1}_{[a = b]} = \begin{cases}1 &\text{ if } a = b, \\ 0 &\text{ otherwise.} \end{cases} $$

  1. Bolded text for (I believe) non-vectors

I think $\mathbf{t}$ denotes a two-dimensional vector (as explained at the beginning of §2) and the expression $\mathbf{t} = 0$ should be read as $\mathbf{t} = [0, 0]$.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.