I see that too, but in that case the social data is served from Facebook and doesn't go through their server unless you authorize it. Unless there are exceptions I'm not aware of.
So when I see a Like button on a site I've never visited before, that displays my name, FB avatar + social data - you're saying that the site I'm on has no way to know that I've visited it unless I click the Like button? Only Facebook knows that and is displaying it in a way that is undetectable to the owner of the site?
This is a bit over my head programmatically but that doesn't seem possible. If Facebook is serving something to visitors on my site, surely there must be a way for me to capture that data?