I'd argue that, in many cases, they are being microaggressive toward YOU in terms of their unconscious ableism (assuming they are neurotypical).