People don't care about specs, but they do care about noticeable visual differences. If this was a PS3 to 360 situation I would agree with you. It isn't though, and reminds me more of the PS1 and Saturn. I remember wanting a Saturn to play all the arcade game conversions, so I took my mother along to the game store. We specifically ask about the Saturn, and immediately the assistant there begins talking us out of it, because of how much the graphics suck comparatively, pointing us to both copies of Tomb Raider running on each machine. It took all the 'individual preference" reasoning in the world to convince my mother to get me the damn Saturn after that lol. If the libraries between the two consoles had been as similar as they are between X1 and PS4, I probably wouldn't have even bothered to persuade her. Just because casual buyers don't have all the information about the two machines, isn't going to stop those facts from influencing their decisions later on. It definitely doesn't help that the PS4 gets a sizeable headstart due to the current price.
Right now I can't think of many ways MS could try to convince people to pick up a $400 Xbox One without the Kinect over an equivalent $400 PS4. Maybe later games like Halo, a new Gears, Quantum Break etc are enough to get people on board, but I feel like they are equally countered by Sony's offerings, putting us back at square one. I honestly think they would have a better chance at selling the device based on the other functionalities it can offer at the moment, but are maybe not enough to overcome the price difference. Take the Kinect out and Snap, Skype, Kinect Fitness, One Guide and whatever they're planning to do with NFL all become pointless, whilst even things like Twitch and Upload Studio lose some of the advantages they have against the PS4 equivalents. It just doesn't make sense to me.