You can't say that unless you're the one against whom it's being aimed.

I also wonder if we're talking about two different dynamics.

For Dave Chappelle to make a joke about the White people who cross the street out of paranoia/bigotry when they encounter a Black person on the sidewalk...*THAT'S* funny because he is making fun of the ridiculousness of their behavior.

If he, by contrast, remarks generically about a White person simply walking into a restaurant chain that's traditionally popular amongst Black people...it's racist for him to "joke" a one-liner such as "What are YOU doing here?" when the underlying subtext is that they had no right to enter the restaurant in the first place.

There's a clear difference between disparaging someone's ignorance versus disparaging a person's identity IN AND OF ITSELF. Regardless of the purported direction ("up" or "down").

I'm a working-class person...so if I make a joke mocking a wealthy person who is complaining obnoxiously about the fact that their server forgot to put an umbrella in their drink...THAT is using humor to illustrate how ridiculous they are being.

It's neither funny nor fair for me to "joke" about how a wealthy person complains about malpractice from botched appendectomy their child endured. In fact, I'd argue that would be classist, on my part.