Derivative of Convolution with Respect to One of the Arguments of the Convolution Continuous Case
Let $ z \left( t \right) = \left( h \ast x \right) \left( t \right) $. What is the derivative of $ z \left( t \right) $ with respect to $ x \left( t \right) $?

Discrete Case
Given $2$ vectors $ x \in \mathbb{R}^{n} $ and $ h \in \mathbb{R}^{m} $, their convolution given by
$$ z = h \ast x $$
What would be the gradient of $ z $ with respect to $ x $? And what would be the gradient w ith respect to $ x $ of the following quadratic cost function?
$$ \frac{1}{2} {\left\| h \ast x - y \right\|}_{2}^{2} $$ 
 A: Discrete Case
For $ x \in \mathbb{R}^{n} $ and $ h \in \mathbb{R}^{m} $ (Assuming $ n \geq m $ The convolution is defined as:
$$ {y}_{n} = \sum_{i = 1}^{m} {x}_{n - i + 1} {h}_{i} $$
Hence
$$ \frac{ \mathrm{d} {y}_{n} }{ \mathrm{d} {x}_{j} } = \begin{cases}
h \left[ m \right] & \text{ if } n - m + 1 = j \\ 
h \left[ m - 1 \right] & \text{ if } n - m + 2 = j \\ 
\vdots & \text{ if } \vdots \\ 
h \left[ 1 \right] & \text{ if } n - m + m = j \\ 
0 & \text{ else } 
\end{cases}  $$
As one could see, this flips the Convolution Kernel.
So the derivative is a matrix which in each row has a shifted version of the flipped kernel.
This matches the the Matrix Form of convolution:
$$ y = H x $$
Where $ H \in \mathbb{R}^{\left( n + m - 1 \right) \times n} $ is the convolution matrix with Toeplitz Form which suggests the gradient is given by:
$$ \frac{ \mathrm{d} {y}_{n} }{ \mathrm{d} {x}_{j} } = { \left( {H}^{T} \right) }_{j} \Rightarrow \frac{ \mathrm{d} y }{ \mathrm{d} x } = {H}^{T} $$
Where $ {H}_{j} $ is the $ j $ -th column of $ H $ (Hence $ { \left( {H}^{T} \right) }_{j} $ is the $ j $ row of $ H $).
The above is using the Denominator Layout for Matrix Calculus.
This also easily solve the Least Squares equation:
$$ f \left( x \right) = \frac{1}{2} {\left\| h \ast x - y \right\|}_{2}^{2} = \frac{1}{2} {\left\| H x - y \right\|}_{2}^{2} \Rightarrow \frac{ \mathrm{d} f \left( x \right) }{ \mathrm{d} x } = {H}^{T} \left( H x - y \right) = h \star \left( h \ast x - y \right) $$
Where $ \star $ is the discrete correlation operator (Convolution with the flipped kernel).
Remark
As Rodrigo de Azevedo noted, for $ y = h \ast x $ the function is Vector Function. Hence $ H $ is its Jacobian.
A: Given $h : \mathbb R \to \mathbb R$, let
$$z (t) := (h \ast x) (t) = \int_{\mathbb R} h (t - \tau) \, x (\tau) \,\mathrm d \tau$$
Fixing $t$, say, $t = t_0$, we obtain a functional that takes $x : \mathbb R \to \mathbb R$ and returns $z (t_0) \in \mathbb R$
$$z (t_0) = \int_{\mathbb R} h (t_0 - \tau) \, x (\tau) \,\mathrm d \tau = \langle (\mathcal S_{t_0} \circ \mathcal R)(h), x \rangle$$
where $\mathcal R$ is the reversal operator and $\mathcal S_{t_0}$ is the shift-by-$t_0$ operator. Therefore, the functional derivative of functional $z (t_0)$ is $(\mathcal S_{t_0} \circ \mathcal R)(h)$, i.e., the reversed-then-shifted version of $h$.
A: Note that the operation 
$$f: x\mapsto (t\mapsto z(t):=(h*x)(t))$$
is linear in $x$, so the derivative of $f$ in $x$ is just itself. (For analogy think about finite dimensional spaces and matrices.)
For the discrete case, note that you can easily view the vectors $z,h,x$ just as functions on the domain $\Bbb R$: let's say originally the vector $h=(h_1,\cdots,h_m)^T$, then you can identify $h$ with a function 
$$\hat h:\Bbb R\to\Bbb R,\quad t\mapsto \sum_{i=1}^m I(t=i)h_i$$
where $I$ is the indicator function. Likewise you can identify $\hat x, \hat z$. You'll then find that, under such identifications, the so-called "discrete case" is trivially compatible with the "continuous" case. So the derivative is the map $x\mapsto h*x$ itself, or just $H$ if you use Toeplitz matrix $H$ to represent the convolution.
