What if he said that privately and you somehow found out. Would it be OK? Because a huge amount of people say similar things privately among their friends and the world keeps turning.

What you describe sounds more like being the victim of gossip, which is probably a far more common and serious problem in workplaces. Should gossip be banned?