their $39.99 games are keeping tons of people away. even if they do buy the system, they're not buying much software for it. there's about 4-5 nintendo games above 500k in the united states, and then the next closest thing are a handful of games selling 250k-300k. that is terrible considering these games are traditionally strong on nintendo brands (sonic, lego, kingdom hearts, etc).
the $169.99 price tag is a huge barrier too. at the very least, it needs to be $149.99 next year, especially since there's no signs of a savior in the near-future, and i doubt animal crossing will do the trick.
and the problem is less about how they're fucking up the 3ds and more about how the dedicated handheld market is being fucked overall. if nintendo had gotten a grip and stuck with its formula in the handheld world, they'd probably be making some good money off a portable dreamcast with an emphasis on 2d games and some gimmick. and even if it meant the vita was where it was now, there'd be an argument that people want dedicated handhelds to stick around. right now it looks like consumers are pretty lukewarm to the idea in the united states and europe.
next gen, nintendo may give dedicated handheld gaming another go, but it'll be from a weakened position.