The reason it bothers me is because it reduces us to that attribute of ours, as a label. I feel the same way about a lot of adjectives-repurposed-as-nouns, based on racial, sexual, and generational descriptors.

It's sort of similar to how most transgender people dislike being referred to as "transgenders" (as a plural noun).