Closed form for $\sum_{k=1}^\infty(\zeta(4k+1)-1)$ Wikipedia gives
$$\sum_{k=2}^\infty(\zeta(k)-1)=1,\quad\sum_{k=1}^\infty(\zeta(2k)-1)=\frac34,\quad\sum_{k=1}^\infty(\zeta(4k)-1)=\frac78-\frac\pi4\left(\frac{e^{2\pi}+1}{e^{2\pi}-1}\right)$$
from which we can easily find $\sum_{k=1}^\infty(\zeta(2k+1)-1)$ and $\sum_{k=1}^\infty(\zeta(4k+2)-1)$. From here it's natural to ask the following

Question: Is there a known closed form for $\sum_{k=1}^\infty(\zeta(4k+1)-1)$?

Related: Closed form for $\sum\limits_{k=1}^{\infty}\zeta(4k-2)-\zeta(4k)$ and Closed form for $\sum_{k=1}^{\infty} \zeta(2k)-\zeta(2k+1)$.

Progress
Note that we are done once we have a closed form for $\sum_{k=1}^\infty(\zeta(4k+1)-\zeta(4k+3))$. So I tried the same approach as in one of the questions above and this is what I got:
We have $$\zeta(4k+1)-\zeta(4k+3)=\sum_{n\geq2}^\infty\left(1-\frac1{n^2}\right)\frac1{n^{4k+1}}$$
Hence 
$$\sum_{k\geq1}\zeta(4k+1)-\zeta(4k+3)=\sum_{n\geq2}\left(1-\frac1{n^2}\right)\sum_{k\geq1}\frac1{n^{4k+1}}=\sum_{n\geq2}\left(1-\frac1{n^2}\right)\frac1{n^5}\frac1{1-\frac1{n^4}}\\=\sum_{n\geq2}\frac1{n^3+n^5}.$$
In the same way I get $$\sum_{k\geq1}\zeta(4k+1)-\zeta(4k)=-\sum_{n\geq2}\frac1{n+n^2+n^3+n^4}$$ and $$\sum_{k\geq1}\zeta(4k+1)-\zeta(4k+2)=\sum_{n\geq2}\frac1{n^2+n^3+n^4+n^5}.$$
It suffices (in fact it is equally hard) to evaluate any of these series.
 A: Just expanding nospoon's comment, it is well-known that
$$ g(x)=\sum_{n\geq 2}\left(\zeta(n)-1\right) x^{n-1} = \frac{x}{x-1}-\psi(1-x)-\gamma\tag{1}$$
so, in order to compute series like $\sum_{n\geq 2}\left(\zeta(an+b)-1\right)$, it is enough to apply a discrete Fourier transform to the middle series and to the RHS of $(1)$. So $\sum_{n\geq 2}\left(\zeta(an+b)-1\right)$ just depends on the RHS of $(1)$ evaluated over the $a$-th roots of unity. The value of $b$ chooses if there is some simplification in the sum of digamma values or not.
A: By expressing the infinite sum as a double sum and then switching the order of summation, we get
$$\begin{align} \sum_{n=1}^{\infty} \left[\zeta(4n+1) - 1 \right] &= \sum_{n=1}^{\infty} \sum_{m=2}^{\infty} \frac{1}{m^{4n+1}} = \sum_{m=2}^{\infty} \frac{1}{m}\sum_{n=1}^{\infty} \left( \frac{1}{m^{4}} \right)^{n} \\ &= \sum_{m=2}^{\infty} \frac{1}{m} \frac{\frac{1}{m^{4}}}{1-\frac{1}{m^{4}}} = \sum_{m=2}^{\infty} \frac{1}{m} \frac{1}{m^{4}-1}. \end{align}$$
One can use the series representation $$\psi(2+z) = - \gamma +1 + \sum_{n=2}^{\infty} \frac{z}{n(n+z)}, \tag{1}$$ where $\psi(x)$ is the digamma function, to show
$$\psi(2+z) + \psi(2-z) + \psi(2+iz) + \psi(2-iz) = - 4 \gamma +4 -4z^{4} \sum_{n=2}^{\infty} \frac{1}{n(n^{4}-z^{4})}.$$
Letting $z=1$, we find
$$ \begin{align} \sum_{m=2}^{\infty} \frac{1}{m(m^{4}-1)} &= 1-\gamma - \frac{1}{4} \Big(\psi(3) + \psi(1) + \psi(2+i) + \psi(2-i) \Big) \\ &= 1-\gamma  - \frac{1}{4} \left(\frac{3}{2} - \gamma - \gamma + \psi(2+i)+\psi(2-i) \right) \\ &= \frac{5}{8}-\frac{\gamma}{2}  - \frac{1}{4} \Big(\psi(2-i)+\psi(2+i) \Big) \\ &\approx 0.0390670072. \end{align}$$
EDIT:
$(1)$ I find it easier to prove that series representation for $\psi(2+z)$ using the difference equation $$\psi(N+1+z) - \psi(z) = \sum_{n=0}^{N} \frac{1}{n+z} $$ as opposed to manipulating the series representation for $\psi(1+z)$.
$$ \begin{align} \sum_{n=2}^{N} \frac{z}{n(n+z)} &= \sum_{n=2}^{N} \frac{1}{n} - \sum_{n=2}^{N} \frac{1}{n+z} \\ &= \Big( H_{N}-1 \Big) - \Big(\psi(N+1+z) - \psi(z) - \frac{1}{z} - \frac{1}{1+z} \Big)\\ &= \Big( \psi(N+1) + \gamma -1 \Big) - \Big(\psi(N+1+z) - \psi(2+z) \Big) \end{align}$$
Letting $N \to \infty$ leads to the result since $\psi(x+a) = \ln(x) + \mathcal{O} \left(\frac{1}{x} \right)$ as $x \to \infty$.
A: I don't know if this will be useful, but thought that is might help.  So, here we go ...
We can write the sum of interest as 
$$\begin{align}
\sum_{k=1}^{\infty}\left(\zeta(4k+1)-1\right)&=\sum_{k=1}^{\infty}\sum_{n=2}^{\infty}\frac{1}{n^{4k+1}} \tag 1\\\\
&=\sum_{n=2}^{\infty}\left(\frac{n}{2(n^2+1)}+\frac{1}{4(n-1)}+\frac{1}{4(n+1)}-\frac{1}{n}\right) \tag 2\\\\
&=\sum_{n=2}^{\infty}\left(\frac{n}{2(n^2+1)}-\frac{1}{2n}\right) \tag 3\\\\
&+\sum_{n=2}^{\infty}\left(\frac{1}{4(n-1)}-\frac{1}{4n}\right) \\\\
&+\sum_{n=2}^{\infty}\left(\frac{1}{4(n+1)}-\frac{1}{4n}\right) \\\\
&=\frac18-\frac12\sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}\tag 4
\end{align}$$
In arriving at $(1)$, we used the series definition of the Riemann-Zeta function.
In going from $(1)$ to $(2)$, we changed the order of the series, summed the interior geometric series, and expanded in partial fractions.
In going from $(2)$ to $(3)$, we merely split the sum of three convergent series.  
In arriving at $(4)$, we noted that the latter two series in $(3)$ are telescoping series whose sum is $\frac18$.
Now, this last series can be written in terms of the Digamma Function as 
$$\sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}=-1+\gamma+\frac12\psi^0(2-i)+\frac12\psi^0(2+i)\approx 0.17186598552401 \tag 5$$
Therefore, we have
$$\sum_{k=1}^{\infty}\left(\zeta(4k+1)-1\right)=\frac58 -\frac12 \gamma-\frac14\psi^0(2-i)-\frac14\psi^0(2+i)\approx 0.039067007237995$$

NOTE:
Here we show that the series in $(5)$ is indeed given by
$$\sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}=-1+\gamma+\frac12\psi^0(2-i)+\frac12\psi^0(2+i)$$
Expanding in partial fractions yields
$$\sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}=\frac12\sum_{n=2}^{\infty}\left(\frac1n-\frac{1}{n+i}\right)+\frac12\sum_{n=2}^{\infty}\left(\frac1n-\frac{1}{n-i}\right) \tag 6$$
Next, using the series definition of the digamma function, we find 
$$\begin{align}
\psi^0(z+1)&=-\gamma+\sum_{n=1}^{\infty}\frac{z}{n(n+z)}\\\\
&=-\gamma+\sum_{n=1}^{\infty}\left(\frac1n-\frac{1}{n+z}\right)\\\\
&-\gamma+\sum_{n=1}^{\infty}\left(\frac1n-\frac{1}{n+1}+\frac{1}{n+1}-\frac{1}{n+1+(z-1)}\right)\\\\
&=-\gamma+1+\sum_{n=2}^{\infty}\left(\frac{1}{n}-\frac{1}{n+(z-1)}\right) \tag 7\\\\
\end{align}$$
Using $(7)$ in $(6)$, we can write
$$\begin{align}
\sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}&=\frac12 \left(-2\gamma+2+\psi^0(2+i)+\psi^0(2-i)\right)\\\\
&=-1+\gamma +\frac12\psi^0(2+i)+\frac12\psi^0(2-i)
\end{align}$$
as was to be shown!
