0
$\begingroup$

I was reading a book on deep learning [1] and came across an equation like this:

$$ C_{i,j} = A_{i,j} + b_{j} $$

This is an equation for showing how an addition of a matrix and a vector is done. But I think instead of $b_j$ it should be $b_i$ since vectors doesn't have a $j$ index. Please tell me if I am wrong and why? Thanks.

Quote from the book:

In the context of deep learning, we also use some less conventional notation.

We allow the addition of matrix and a vector, yielding another matrix $$C = A + b$$ where $ C_{i,j} = A_{i,j} + b_{j} $. In other words, the vector $b$ is added to each row of the matrix. This shorthand eliminates the need to define a matrix with $b$ copied into each row before doing the addition. This implicit copying of $b$ to many locations is called broadcasting.

[1] Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron, Deep learning, Adaptive Computation and Machine Learning. Cambridge, MA: MIT Press (ISBN 978-0-262-03561-3/hbk; 978-0-262-33743-4/ebook). xxii, 775 p. (2016). ZBL1373.68009. Free Online Version and Errata on https://www.deeplearningbook.org

$\endgroup$
8
  • $\begingroup$ You can't add a matrix and a vector. Either you are misunderstanding what the author is saying, or else the author was drunk. $\endgroup$ Jan 28, 2022 at 11:38
  • 1
    $\begingroup$ Can you provide more context for the statement in the book so we can try and understand the intent? $\endgroup$ Jan 28, 2022 at 11:53
  • $\begingroup$ As I mentioned in the question this is a deep learning book. It says "We allow the addition of matrix and a vector" so this addition is under scope of the deep learning field. If you curious about how addition of matrix and a vector is possible, you can check the "MIT Deep Learning Book" by Ian Goodfellow, Yoshua Bengio and Aaron Courville. @GerryMyerson $\endgroup$ Jan 28, 2022 at 11:59
  • $\begingroup$ I edited it @GrapefruitIsAwesome $\endgroup$ Jan 28, 2022 at 12:03
  • 2
    $\begingroup$ The index $j$ in $b_j$ just refers to the fact that you are adding the $j$-th entry of $b$ to the $j$-th column of $A$. Therefore the number of columns in $A$ should be the same as the number of entries in $b$. $\endgroup$ Jan 28, 2022 at 12:18

1 Answer 1

1
$\begingroup$

I suspect that the author is thinking in Python. The proper matrix notation would be $$ {\bf C} = {\bf A} + {\bf 1} {\bf b}^\top $$

$\endgroup$
3
  • $\begingroup$ Pardon my confusion, but is this what the author meant? Maybe I misunderstand but the identity multiplied by the vector is either just the diagonal (which does not add all of b to each row) or you mean some other product which I'm unfamiliar with - an outer product here may result in a matrix which is too large for component-wise addition. Help? $\endgroup$
    – SubSevn
    Mar 8, 2023 at 4:29
  • $\begingroup$ Ahhh is this an outer product between a column of 1's and $\textbf{b}^T$ then? $\endgroup$
    – SubSevn
    Mar 8, 2023 at 4:49
  • 1
    $\begingroup$ @SubSevn Replying to the 2nd comment, yes, that is correct. $\endgroup$ Mar 8, 2023 at 6:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .