I agree with you 100% that "I don't see you as a color" is the preferred premise over "I don't see your color." The problem is that the latter is implied (even if not intended) by the term "colorblind."

Looking at it another way: the term "prejudice" has become mangled because lazy "intellectuals" on the Left want to simplify the term "systemic racism" to be just "racism" (I guess three extra syllables is too burdensome for them to utter). Since systemic forms of racism aren't experienced by White people (as opposed to social or cultural forms of racism, which are experienced by everyone), they want to reserve the term "prejudice" for any racially-motivated violence or gaslighting directed at paler-skinned folks (or, in some cases, that people of color inflict upon one another). That's also problematic, because the word "prejudice" implies passive, internalized beliefs...rather than behavioral actions.

So I use "color-savvy" or "color-conscious" (or "color-aware") to convey the message of "I see your race, acknowledge any struggles you've experienced based on your race, but will also strive to get to know you as an individual" -- rather than "colorblind," because "colorblind" could be construed as meaning, "Racism is no longer much of a problem, so your skin color shouldn't even be a part of the discussion."

The same way I say "systemic racism" when referring to racially-charged institutional discrimination faced by people of color, but I use "social racism" (individually-inflicted) or "cultural racism" (group-inflicted) when referring to racist acts committed against anyone regardless of their skin color. While any of these systems or interpersonal acts do indeed stem from prejudice, the term "prejudice" itself is a reference to the mindset rather than the behavior.