Number of positive integer solutions of a linear equation with maximal value of two summands being restricted. My question is how many possible solutions there are for the equation subject to the condition that $x_1 \leq k_1$ and $x_r \leq k_r$, where $x_i$ are positive integers.
The equation is
$$
x_1 + x_2 + \ldots + x_r = n.
$$
I have tried to find the number of solutions so that if L = number of solutions we can obtain the combinatorial number but i haven't reached any conclusion.
Any tip would be much appreciated, thanks.
Edit:
I solved it using the inclusion-exclusion principle.
 A: Let us consider the general case where all $x_i$ are restricted: $0<x_i\le k_i$.
In terms of generating functions we are looking for the coefficient at $x^n$ in the product:
$$\begin{align}
x^r\prod_{i=1}^r\sum_{j=0}^{k_i-1}x^j
&=x^r\prod_{i=1}^r\frac{1-x^{k_i}}{1-x}\\
&=\frac{x^r\prod_{i=1}^r(1-x^{k_i})}{(1-x)^{r}}\\
&=x^r\prod_{i=1}^r(1-x^{k_i})\sum_{j=0}^\infty\binom{-r}{j}(-x)^j\\
&=x^r\prod_{i=1}^r(1-x^{k_i})\sum_{j=0}^\infty\binom{r+j-1}{j}x^{j}\\
&=\sum_{\boldsymbol{\mu}}(-1)^{|\boldsymbol{\mu}|}x^{\boldsymbol{\mu}\cdot\mathbf{k}}
\sum_{j=0}^\infty\binom{r+j-1}{r-1}x^{r+j},
\end{align}$$
which is
$$
\sum_{\boldsymbol{\mu}}(-1)^{|\boldsymbol{\mu}|}\binom{n-1-\boldsymbol{\mu}\cdot\mathbf{k}}{r-1},\tag1
$$
where $\boldsymbol{\mu}$ are all $2^r$ binary vectors of the length $r$, $|\boldsymbol{\mu}|=\sum_{i=1}^r\mu_i$ and $\mathbf{k}=(k_1,k_2,\dots,k_r)$.
In (1) the equality $\binom pq=0$ is assumed for $p<q$.
If some $x_i$ are unrestricted one can simply ignore them while using (1). For example if (as in your question) only two values are restricted the answer is:
$$
\binom{n-1}{r-1}-\binom{n-1-k_1}{r-1}-\binom{n-1-k_r}{r-1}+\binom{n-1-k_1-k_r}{r-1}.
$$
This can be obtained also using inclusion-exclusion principle.
