Well, a lot of people share your perspective...that one shouldn't care what someone's sexual orientation is, unless it's harming others in some specific way.

Unfortunately, not everyone possesses that level of maturity. Many people are more than willing to weaponize a person's sexual orientation (even if it's mere SUSPICION) against that person, for their own ego or self-gratification or cerebral personal gain.

Same concept applies to race, class, sex, gender, religion, etc.

Which is why, when someone berates me or another person due to an attribute of their identity per se, my first question to them is always some variation of...

"What are you trying to accomplish by throwing this person's identity back in their face?"