Faggot doesn't even need to apply to sexual orientation. A person who has their ice cream taken by their older sibling could be called a faggot, and it would have nothing at all to do with sexism or homophobia. If you're going by just the porn/4chan version, than maybe it has to do with those. But most people aren't using faggot as a way to get their outdated, sexist, homophobic view points out without saying crazy shit.
Fixed that for you.
I hope my point is made clear.
In case it wasn't: the same justification is always used to justify slur use, yet it falls through when it is noticed that the reason the insult version of the slur is even considered an insult is because the slur refers to a group maligned by bigots