How would I compute this matrix in Matlab? I'm trying to compute the H$_\infty$ norm of a matrix using the paper:
Bruinsma, N. A.; Steinbuch, M., A fast algorithm to compute the $H{\infty}$-norm of a transfer function matrix, Syst. Control Lett. 14, No. 4, 287-293 (1990). ZBL0699.93021. 
It requires calculation of a matrix derived from the original system matrices A,B,C,D:
the required matrix
and I just can't seem to figure out the sequence of operations this would require in matlab, after multiple tries, the closest I've gotten to something being even close to be workable is this:
R = D'*D - Ylb^2*eye(2);

S = D*D' - Ylb^2*eye(2);

H(y)=[A-C*D'*(R^-1)*B -Ylb*B'*(R^-1)*B; Ylb*C*S^-1*C' -A'+B'*(R^-1)*D*C']

but this throws out an horizontal concatenation error, as the dimensions don't match.
If it helps the system matrices I'm using are: 
A = [0 1; -10 -1];
B = [0; 1];
C = [1 0];
D = 0;

Any help here would be nice.
 A: Well...it might help to compute the four blocks one at a time, and look at them as you do so, comparing to the values you compute by hand. Something like
NW = A - B * inv(R) * D' * C
NE = -ga * B * inv(R) * B'
...
H = [NW, NE ; SW, SE]

And then when you notice that the inverse of $R$ keeps appearing, you can clean it up with 
Ri = inv(R);
NW = A - B * Ri * D' * C
NE = -ga * B * Ri * B'
...
H = [NW, NE ; SW, SE]

As it happens, what you've written is this:
H(y)=[A-C*D'*(R^-1)*B -Ylb*B'*(R^-1)*B; 
      Ylb*C*S^-1*C' - A'+B'*(R^-1)*D*C']

where I've inserted a newline to make it fit better. The "northwest" entry in your matrix is $A - CD^t R^{-1}B$, where it should be $A - BR^{-1}D^TC$, so you've got the order of matrices wrong in that entry at least. You should fix that and check the other three. 
Your next problem is that you're assigning this to H(y), which is not a variable name. It could be interpreted as a subscripted matrix, but my best guess is that $\gamma$ (that's a "gamma", not a "y") is a real number, and real numbers aren't allowed as subscripts in Matlab. 
Gratuitous advice about things like this in general:
I suspect that you need to 


*

*Get a better grasp on the order of operations in matrix multiplication, both in mathematics and in Matlab

*Learn Matlab's basic syntax better before you try to write comparatively complex mathematics in it

*Make certain you actually understand the paper whose results you're hoping to implement in software. In particular, you should be able to carry out at least one example by hand. 

