Well, how would you respond to people of African descent who embrace the label of "Black"?
When I use "Brown" while talking about race relations, I'm broadly referring to people of Latino/a, Asian, Middle Eastern, Oceanic, and Indigenous descent. Their cultures aren't uniformly "brown" in hue, but rather, featuring skin colors along a spectrum of lighter brown shades.
There's also the matter of asymmetry/indignity when it comes to selectively capitalizing certain racial/ethnic groups while selectively leaving others in all lowercase. That's a byproduct of political correctness that warrants its own entirely separate article.