What screwed the Vita over the most was actually Sony themsleves.
They made so many stupid decisions with the system from the start they positioned it as both a competitor to mobile and the 3DS and it ended up being spread to thinly to really compete with either.
Then you have the needless and outright useless 3G support (which was not cheap and only through AT&T, they literally got booed when they announced this lol)
Then you have the starting price tag of 250$ which sounded great at the time (the 3DS was still 250$ at this point) but the 3DSes lackluster sales pretty much already proved that price point was simply too high for a handheld.
Then you have the memory cards, which were proprietary (unlike the 3DS) and expensive as all hell (seriously at launch the prices of these cards were insane 20$ for the smallest one and these were REQUIRED to do anything with the system and non-bundles did not come with one)
Then you had the games, launch had a few decent games but third party support was (like the Wii U) pretty much lackluster from the beginning, Sony even had a bundle for both Call of Duty and Assassin's Creed liberation, Call of Duty is still one of the worst games of the system (if not THE worst) and Liberation was... decent at least? Instead of highlighting some of the better titles the system would receive with bundles (like Persona 4 Golden or Teraway or Killzone) they went for mass market appeal which ended up dropping flat shortly after those bundles were released, and despite the rather good sales of both of those bundles neither game ever got a sequel on the Vita so any of the people who Sony tried to bring in have been left in the dark without games to play.
The Vita itself is an amazing piece of tech but the way it was ultimately designed and the decisions they made at the top were what ultimately killed it.
If Sony had been as smart with the Vita as they have been so far with the PS4 it might be a very different story, but the way Sony approached the Vita was very similar how the first approached the PS3, and that's what largely screwed them over.