In general, I think the major grievance tends to be with people who take credit for cultural elements outside of their identities and/or people who disregard the sanctity that a given culture associates with specific symbols or attached meanings.
Which begs the question...
Who gets to be the ultimate arbiter over whether something is "cultural appropriation" or not?
If I grow out my hair and then style it with braided plates specific to a culture that's part of my own genetic lineage and ancestry...do Black people have legitimate grounds for criticizing my new hairstyle because I, as a White man, have supposedly tried to emulate a hairstyle that "sort of" resembles variations of Black dreadlocks? Even if I'm taking the time to learn about the history behind the element that I'm adopting?
At what point would the slippery slope end?