That is essentially what i'm doing, yeah. We must accept that to some ends the premise of including all races may be such an impediment that no progress is made. Whether to that end we may rethink what "inclusive" means in any given context, that is likely true as well.
Your point of historical discrimination is not quite as specific as what i was describing, i was more pointing to the effects of historical discrimination, namely the dynamics of hierarchies in groups, specifically the perception that the "white people" group have accrued through historical discrimination a privilege that portrays them as hierarchically higher than other racial groups (i.e. black people).
I'm trying to point this out in a way that makes it understandable and applicable in other contexts of privilege (and some which we already take for granted). For example, i think the same is applicable for labor rights. Though i think bosses should be part of the conversation in some higher capacity, i also think laborers should be able to organize and create places of conversation where bosses are not present, because bosses have obviously different interests in the discussion that run against what laborers might want (and e.g. it's perfectly normal for higher ups in companies to exclude their employees from discussions because they are keenly aware that what they want may go against the interests of their employees).
That is in essence what i mean when i say that you can't pretend those hierarchies aren't practiced by racial groups, i don't think it's particularly wrong to say that white people can be expected to have a kind of social behavior that displays their accrued privilege (intentionally or not, racism is also propagated by systemic discrimination which may go unaware to those who benefit from it (this too is something that helps propagate systemic racism because it's inherent immorality is not immediately evident to those it puts in power)), and to say that you simply cannot make spaces where people try to protect themselves from that friction between interests coming from different perceived hierarchies in essence denies that they exist, when in fact they do.