Split the $80$ people into $5$ clusters of $16$.
Then, split each cluster of $16$ into $4$ groups of $4$, and have each person within any group of $4$ listen to all the other persons in all the other $3$ groups of $4$ within that cluster of $16$.
If you don't want to allow two people listening to each other, consider this alternative: have the $80$ people sit in a circle and have everyone listen to the $12$ people on their right.
Notice how this alternative will work for any number of people and any number of 'listening to and being listened to'. For example, with $37$ people in a circle, and everyone lostening to the $11$ people to their right, you are by symmetry guaranteed that everyone will be listened to by $11$ people as well.
In fact, you can even do weird things like 'have every listen to the 6th, 13th, 21st, 22nd, 34th, 41st, 43rd, 47th, 49th, 56th, 63rd, and 72nd person to their right': by symmetry, it will still be true that everyone will be listening to and be listened to the same number of people each.