The problem is that a lot of people who promote the concept of "antiracism" simply want to control the entire narrative from the opposite end (as opposed to white supremacy) of the ideological spectrum. If you disagree with any tenets of their academically-elitist dogma, even in a respectful or thoughtful manner, you get automatically branded as "racist."
This has led to a toxic climate where moderate and centrist voices are demonized by factions on the Left as "racist" (while, ironically, being simultaneously demonized by factions on the Right as "too politically-correct").