If you're saying the problem is the subjectivity inherent when trying to identify "hate speech"...then, yes, I agree with you.

I think intent and impact both need to be taken into consideration. But we shouldn't go around accusing people of "hate speech" when we don't know much about their track record, let alone when we're unwilling to attempt to have an actual conversation with them about what they said and what they meant.