You're asking me to map out something that developed in my mind over the better part of 23 years from the time I was 6 and first played Link's Awakening. This wasn't something I deliberately decided upon but simply formed in my mind and grew from there, so sorry if I'm not specific enough for you. And again people place different levels of importance on aspects of characters. Things I find unimportant may be critical to your perspective of what makes a character, while things I find important you see as irrelevant. I won't feign that there is a clear logic to this. Especially in a case like this where there is often a lack of traditional character development due to the minimal narrative structure of the series.
For me the way Link reacts to the world around him and how different encounters and scenes in each game play out always felt like the same character to me. Just different circumstances. His background almost never plays a role in his character development. It's a minor detail in the story and his characterization, at best, as the result is always the same. And I'll admit he doesn't have a lot of traditional characterization, but it's always been enough for me to see him as independent of me and consistent across titles.
The way he emotes since OoT has really solidified him for me. Constantly reacting to whats going on around him and often in ways that I would not, showing affection and other emotions towards characters that I don't, or at least not in the same way as me. Across each game every Link feels very consistent with his greater character. Brave, naive, curious and transparent to name a few. Swap out OoT Link with TP Link and they would react the same way to the situations, characters and events they are presented with. Swap out MM Link and WW Link and the same thing would occur. The specific origins and motivations that propel them on their journey are different, but everything else about them remains the same. His character and personality, what little there may be, is not dependent on any specific game, background or familial situation. His gender and general appearance also play into this image of him, the latter less so for me, even if they don't have any significant bearing on his character. It's simply come be a part of who he is to me, no different than any other character.
All that and likely a lot more I'm not thinking of factored into Link becoming a defined and established character for me. Sorry if that's not good enough for you or doesn't mesh with your sensibilities. It's also only a part, but a major one, as to why I personally don't care to see a Female Link. My opinion on the series as a whole and where I would like it to go in the future, my view of Zelda as a character and other things also play a big part into that as well.
.