They are the racist ones for ASSUMING that you were White, in the first place.
They are trying to use the Association Fallacy, by saying, for example, only Black people are allowed to comment on issues predominantly affecting the Black community (unless, of course, you're "strange fruit" who isn't "Black enough," in their subjective opinion).
If you haven't already, you'll probably get accused of "tone-policing" by many of these hyperwoke archetypes. The very simple remedy, here, is to turn it back around on them and point out how they are "thought-policing" you.