Darwin would say that behaviour that reproduces in greater numbers will reproduce the same traits in the next generation.
Women who have more than one male partner avoid the chance of mating only with a sterile partner. Men who have more than one partner spread their genes far and wide, and avoid the chance of mating only with a sterile partner.
Therefore, from a Darwinian point of view Swinging is an activity that is a more successful means of an individual reproducing. Does that make it normal?
I know swingers do not fuck to reproduce, but who does know, we can avoid reproducing so easily. Does that stop the swinging being normal?