I certainly understand (and, in fact, agree with) YOUR definition/intent when referring to "colorblindness" (even though I personally choose not to use the term).
The problem is that there are people who will corrupt the term/idea as a way to make the case that systemic racism doesn't exist...and/or to brush off a majority of examples of race-based adversity.
That bastardization is equally as bad as the #Woke folks who gleefully trumpet the mantra of "You are White [and/or male], so, therefore, you need to shut up because your life has been so good [compared to other groups]!"
Both are cases where language and thought is corrupted to serve individuals' (or groups') nefarious agendas.