Well, in this hypothetical case, it sounds as though the middle-class Person of Color is telling the working-class White person that they are "privileged," universally speaking. They are centering their own skin color in the context of "life in general" -- and that's dangerous.

What they need to do is distinguish racial privilege from class/income privilege.