1. The average consumer just doesn't want to constantly fiddle with bullshit like docks or HDMI outs. You can see this based on how quickly sales of iPod/iPhone media docks have dropped in favor of wireless solutions. Heat, battery and latency issues also make streaming less practical for a game console.
2. When it comes to dedicated gaming devices, people generally don't like paying for features they don't see themselves using. The Xbox One's sales improved noticeably after Microsoft started offering a bundle without Kinect. There are posters on this very board who refuse to buy a Wii U unless Nintendo offers a Gamepad-less SKU.
The same could happen with a handheld-console hybrid; handheld-centric consumers will see the console features as a waste of money and vice-versa. There's a very real risk of alienating a sizable chunk of both markets, especially since...
3. A hybrid device will always end up more expensive than either form factor, because it will need to house specific components to function well as both a handheld and home console. A home console that can double as a handheld will need a decent display, a battery pack and low-power mobile hardware. A handheld that doubles a home console will need to be bundled with at least one traditional controller, hardware that supports 4+ player MP at home and most importantly, a processor that can be overclocked to handle the higher quality graphics that are usually expected from a home console, or some kind of dock that boosts the handheld's capabilities.
In both cases, the cost of those extra components could instead be put towards making two (or more) separate devices that are more attractive for their respective markets. Remove the home console parts from the hybrid and you can sell a much more affordable handheld - low prices were always key to Nintendo's past successes in the handheld market. Remove the handheld parts from the hybrid and you can either put their cost towards building a much more powerful home console or, again, sell it cheaper.
Even if you're interested in both the portable and home experience, why would you want to drop $250-300 on a handheld that doubles as a sub-par console instead of spending slightly more ($300-400 tops) on a handheld and a console that's more powerful than what a hybrid would have been? Even the most powerful hybrid they could put together within the next three years would be well behind a new Nintendo console that's nearly a generation behind the PS5/XB2 itself.
4. Even if they diversify their lineup and improve their marketing next time around, Nintendo will still want a piece of the family market. Why does this matter? A handheld is a personal device while a home console usually ends up being shared until the kids get to a certain age.
For any family with two or more children, you'll likely end up with constant bickering over who gets to take the hybrid with them, who gets to play on the TV and a bunch of other crap. Which means that you'll probably need to buy a $250 gaming device for everyone. At that point, many parents would already opt for a hand-me-down or even brand new iPod, smartphone or tablet instead - not only do they offer more functionality than a dedicated gaming device, but two of the three are also cheaper than $250.
In the separate handheld ($150) and home console ($200-250) scenario, buying a handheld for both of your kids and a Nintendo box for the living room would cost you the same, or just slightly more than the price of two hybrids ($500).
5. And finally, if your hybrid is rejected by the market (which is quite likely to happen taking all of the above drawbacks into account), you're pretty much fucked. Having a separate handheld and home console gives you a plan B in case one of the two underperforms and keeps your foot in the door of both markets. A failed hybrid console would turn Nintendo into a non-factor as a handheld and console manufacturer for several years, and make it that much harder to pull off a comeback.
Everything a hybrid console could do, a lineup of Nintendo devices with a shared architecture and software library could do better with less risk of putting their hardware business in jeopardy.