There is a connection.
An algebra over a set is actually a boolean ring: if we interpret $+$ as symmetric difference $\triangle$ and $\cdot$ as intersection $\cap$, they satisfy all (unital) ring axioms (with $0=\emptyset$ and $1$ equal to the universe), and moreover $x+x=x\triangle x=\emptyset=0$. (However, it is never a field, unless it has two elements, despite being called a field of sets: no element except $1$ itself has a $\cap$-inverse.)
Every boolean ring is trivially an algebra over the two-element field, so there (moreover, every ring is an algebra over the ring of integers, but that is not a field).
But this is, I think, mostly incidental. As far as I can tell, historically, the term algebra was used in a sense closer to the one used in universal algebra now, which is rather more abstract than modern "algebra over a field".