I believe this can be done with $16$ players over $4$ rounds. Start by labeling the players according to the first table at which they play, so at table $0$, the first round will see players $A_0,B_0,C_0,D_0$, and the other tables $1,2,$ and $3$ have similar players $A,B,C,D$.
For the next three rounds, keep the $A_j$ players at table $j$. At the tables of even index, bring in a $B$, a $C$, and a $D$ player whose indices are all distinct and start with the next available number counting up, with the restriction that you start over at $0$ after reaching $3$, and skip the label of the table (the players with the same index have already seen each other). At the tables of odd index, start with the next available number counting down.
For example, at table $2$, the second round will see $A_2,B_3,C_0$, and $D_1$. The next round will see $A_2,B_0,C_1$, and $D_3$.
In other words, at table even $j$, the indices of the $B$, $C$, and $D$ players will cyclically progress through rounds $2$ through $4$ respectively (where everything is reduced modulo $4$) as
At table odd $k$, the indices of the $B$, $C$, and $D$ players will cyclically progress through rounds $2$ through $4$ respectively (where everything is reduced modulo $4$) as
For the final round, put all the $A$ players at a table, $B$ players at another, $C$ at another, and $D$ at the final table.
I imagine there's some nice way to extend this to other specific cases of the general problem, but it sounds like you are trying to organize an actual tournament of 16 players rather than solve a maths problem in the abstract.