Frequency Response of a Linear Constant-Coefficient Difference Equation I've just started self-studying Oppenheim and Schafer's book, Digital Signal Processing and I'm having trouble solving problem 20 from Chapter 1. The problem is stated as follows:
A causal linear shift-invariant system is described by the difference equation
$$ y(n) - ay(n-1) = x(n) - bx(n-1) $$
Determine the value of $b$ ($b \neq a$) such that this system is an allpass system; i.e., the magnitude of its frequency response is constant, independent of frequency.
I started trying to solve the problem by substituting the Dirac-delta function $\delta(x)$ for $x(n)$ in order to find the unit-sample response $h(n)$ of the system. I found
$$ h(n) = (a-b)a^{n-1} u(n), $$
where $u(n)$ is the unit-step function. Plugging in the unit-sample response I found the frequency response to be
$$ H(e^{j\omega}) = (a-b) \sum_{n=0}^{\infty} a^{n-1} e^{-j \omega n} $$
I cannot figure out where to go from here or where I made a mistake earlier. Any help would be greatly appreciated. Thanks and have a great day.
 A: From the discrete-time Fourier transform (DTFT) relationship
$$x[n-k]\Longleftrightarrow X(e^{j\omega})e^{-jk\omega}\tag{1}$$
the DTFT of the given difference equation is
$$Y(e^{j\omega})(1-ae^{-j\omega})=X(e^{j\omega})(1-be^{-j\omega})\tag{2}$$
from which the frequency response is obtained as
$$H(e^{j\omega})=\frac{Y(e^{j\omega})}{X(e^{j\omega})}=\frac{1-be^{-j\omega}}{1-ae^{-j\omega}}\tag{3}$$
Its squared magnitude is given by
$$|H(e^{j\omega})|^2=\frac{1-2b\cos(\omega)+b^2}{1-2a\cos(\omega)+a^2}\stackrel{!}{=}c^2\tag{4}$$
where $c$ is some constant. From $(4)$ we get
$$1-2b\cos(\omega)+b^2=c^2-2ac^2\cos(\omega)+a^2c^2\tag{5}$$
Excluding the trivial solution $a=b$, equating both sides of $(5)$ gives
$$a^2c^2=1\\ac^2=b\\c^2=b^2$$
from which
$$a=\frac{1}{b}\tag{6}$$
follows.

Note that the impulse response and frequency response that you found is not entirely correct. However, it is of course possible to obtain them the way you tried, i.e. by setting $x[n]=\delta[n]$:

$$\begin{align}n&&y[n]\\{}&&{}\\0&&1\\1&&a-b\\2&&a(a-b)\\\vdots&&\vdots\\k&&a^{k-1}(a-b)\end{align}$$
So we obtain
$$h[n]=a^{n-1}(a-b),\qquad n\ge 1$$
This is basically what you got, but your solution does not include the value for $n=0$. The correct expression taking the value at $n=0$ into account is
$$h[n]=\delta[n]+a^{n-1}(a-b)u[n-1]$$
which  corresponds to the frequency response
$$H(e^{j\omega})=1+(a-b)\frac{e^{-j\omega}}{1-ae^{-j\omega}}$$
It can be easily shown that this expression equals the one given in Eq. ($3$) above.
