Are you switching games while still connected to the party? Because that's where the problem is. You can connect, disconnect, and reestablish from game to game, but it doesn't hold the session if you, for example, move from killer instinct to dead rising, which usually causes the xbox to freak out and attempt to create two separate sessions at the same time which causes all sorts of craziness to occur. Literally every live stream I watched had this problem at one time or another, most prominently with Ryse and Dead Rising.
And not to be glib, but your anecdotal experience really doesn't hold as much weight as evidence that's been documented by an assortment of journalistic institutions (see how that can be flipped on you?).
Just because it's not happening to you specifically doesn't mean it's not a problem. That's a very head-in-the-sand attitude to take regarding your 500 dollar machine.