2 was just such a huge jump from 1, and had some of the best plot/cast, as far as villains/etc. 4 introduced new characters & fighting styles and really helped develop the series, i just adore it.
i don't know why 5 gets knocked down by some, though. i loved the balls off of that one. 3 just gets hurt because some folks didn't like the orphanage stuff as much (related: ya'll should watch Sonatine), and Mine/CIA was a bad villain thing to go with.
I definitely like 5 more than 3, and any negative opinion I give about any Yakuza game should be prefaced with an acknowledgement that I like every Yakuza game I've played. I even think Dead Souls is kinda neat.