I'd also like to add something else. Whether or not we can "control" having a certain type of privilege is beside the point. This see-saw of value-judgements that some people try to make when it comes to "mutable vs. immutable attributes" is really irrelevant.
If we recognize a privilege that we have, and we actively use that privilege to lift others up, then IT DOESN'T MATTER whether the privilege was nature, nurture, inborn, behavioral, psychological, physiological, LMNOP -- it's there, and the focus must be on using it for good rather than evil.
Because if you "give up privilege" just to make some sort of performative gesture...how long before a nefarious soul steps into the void that you left behind?