You guys aren't alone. I also have way more friends on XBL than PSN. (200+ vs. 8. Yes, eight). It could be we're statistical outliers. But when it comes to market trends, previous purchase habits and consumer mindshare can trump pure statistics. Even with MS screwing up the Xbone launch badly, their online momentum, better online infrastructure, arguably more naturally social ergonomics, etc...is important.
So I don't think there's as much of a mystery here as we might think at first glance. At least in the United States, XBL was far more successful than PSN at building an online social gaming experience during the PS360 Gen. That's a lot of momentum from millions of gamers who became attached to how XBL works and "feels". After Microsoft did their best launch effort to piss off the entire gaming universe, millions of those XBL gamers "switched" to PS4 early in this gen. Most probably expected PSN to be similar to XBL but found out that for better or worse it doesn't have a similar layout. My personal online experience with PS4 has been similar to PS3 in that the way it works just isn't as conducive to adding/making new friends on the fly when gaming. Some of this is subjective preference, but I think a bit of it is objective in that the PS4's UI just isn't very good (yet) when it comes to certain functions.
The end result for me: I own both new Gen systems. Due primarily to the PS4's significant ~30%-40% horsepower advantage + my disgust for MS's anti-consumer bullshit I initially intended to buy all multiplats on PS4. But over the past year PSN has not done enough to make itself a better, more social experience. So, provided there aren't huge visual/performance differences I'm thinking of going Xbone for MP multiplats and PS4 for SP multiplats. A lot of that is due to all the XBL friends I've carried over from 360, but a lot of it is also that the PS4's online experience just isn't quite as easily social for my tastes.
Edit: I'm sure also has a lot to do with preferences for certain titles and exclusives. Titanfall may be a lot less than many of us hoped for, but so far its been the best online experience I've had by far. By comparison, Destiny came off as a repetitive grind and not to my tastes. So I added a bunch of new XBL friends when playing Titanfall but ended up returning Destiny for a refund after adding just a handful of friends.