Dude, if you're sitting in front of my house at 1:30 am in the morning, I'm taking down your license plate and asking you to leave with a bat in my hand. For all I know you are just another group of heroin addicts that have been breaking into cars and houses in the neighborhood.
You have to understand how bad that looks.
You have zero right to ask them to leave. The public street isn't your property, no matter how much you might think otherwise.