I've never seen anyone vocalize any expectation for any of the nontrivial zeros to have an explicit symbolic expression, presumably from which one could in principle then mathematically prove said expression is a zero of $\zeta(s)$ ("plug in and see"). That sounds too good to be true. Perhaps there are cheap ways to express them analytically in terms of $\zeta(s)$ that basically amount to defining them to be zeros of the function in the first place in some roundabout fashion.
The trivial zeros though one can show are in fact zeros, using the functional equation and the fact that the gamma function has no zeros in the complex plane but it has poles at negative integers.
Beyond this, and the values of $\zeta$ at positive even integers (which derive from harmonic analysis essentially), you'll have to settle for numerical approximation instead of symbolic computation.
Chris notes in the comments that the series $\sum n^{-s}$ only defines $\zeta(s)$ for ${\rm Re}(s)>1$; obviously it doesn't converge to the left of this by comparison with the harmonic series. The Riemann zeta function is defined as the analytic continuation of the function defined by this series. It is the magical functional equation that relates $\zeta$'s values on the left half-plane to its values on the right.
For reference, the functional equation is given by $\xi(s)=\xi(1-s)$, where $\xi$ is the "completed" zeta function defined by $\xi(s)=\pi^{-s/2}\Gamma(s/2)\zeta(s)$ (sometimes a factor like $s(1-s)$ is introduced to make the proofwork easier). The explanation for the mysterious "gamma factor" lies in heavy algebraic number theory machinery, which is the content of Tate's thesis.
The theory of zeta functions definitely has connections with Fourier theory. Coefficients can be twisted by characters, zeta functions can be attached to modular forms which themselves can be defined by $q$-series, the explicit formula for the growth of primes in terms of the nontrivial zeros can be interpreted as "the music of the primes," and so on. I am not sure how high the observation that the $n^{-s}$-series looks like a Fourier series in $\tau={\rm Im}(s)$ ranks on the ladder of significance.
As a physicist, you may also be interested to know that the zeros have been empirically observed to obey certain statistical spectral laws, keywords: "GUE ensemble," "random matrix theory." There's a nice collection of information related to these ideas here. Another direction of interest is that on the Riemann hypothesis the nontrivial zeros form a one-dimensional quasi-crystal, see here.
Hagen notes in the comments that even though we are forced (in the critical strip in particular) to settle with numerical approximation, we can in fact test whether discovered nontrivial zeros have real part exactly half. See this MSE question for the relevant math behind this.
(Sorry I can't be of much help.)