In a recent article at Distill (link) about visualizing internal representation of convolutional neural networks, there is the following passage (bold is mine):

If neurons are not the right way to understand neural nets, what is? In real life, combinations of neurons work together to represent images in neural networks. Individual neurons are the basis directions of activation space, and it is not clear that these should be any more special than any other direction.

Szegedy et al.[11] found that random directions seem just as meaningful as the basis directions. More recently Bau, Zhou et al.[12] found basis directions to be interpretable more often than random directions. Our experience is broadly consistent with both results; we find that random directions often seem interpretable, but at a lower rate than basis directions.

I feel like they are talking about linear algebra representations, but struggle to understand how one neuron can represent a basis vector.

So at this point I have 2 main questions:

  1. A neuron has only a scalar output, so how that can be a basic direction?
  2. What is an activation space and how to intuitively think about it?

I feel like understanding these can really broaden my intuition about internal geometry of neural nets. Can someone please help by explaining or point me in the direction of understanding internal processes of neural nets from the linear algebra point of view?

  • 1
    $\begingroup$ I'm not sure of what it meant. A single neuron with sigmoid activation is a smooth approximation of $1_{w^\top x +b > 0}$ with $x$ the input, $w$ the weights, $b$ the bias. $\endgroup$ – reuns Nov 13 '17 at 16:39
  • $\begingroup$ The sentences seem to mix up real life neurons and our limited attempts to model them. (Unfortunately scientists are always verbally mixing up their models of reality for reality itself. Engineers tend to be more modest in this regard). Ordered plane layers of model neurons create physical activation surfaces. What happens at these surfaces is either synchronous, asynchronous or a mixture of the two depending on how the model is built up. There can be multiple layers of model neurons and thus multiple physical activation surfaces. After that no idea. $\endgroup$ – James Arathoon Nov 13 '17 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.