The filter turns people into babies because in aggregate women retain more youthful features when they go through puberty (voice never drops, no facial hair, skin doesnt thicken, jawline doesnt change etc), whereas men tend to look haggard and older by comparison. It makes sense that male features deviate more from youth as female is the default human body plan. If you have a baby face (round or heart shaped face, big eyes, few wrinkles) already, it will group you in with images of children and the result is a baby.
No need to feel bad though as it just means youll age better.