This isn't always true. Many men will go out of their way to give deference to women, by default, intentionally ("and they say 'chivalry' is dead"...). Many men will close themselves off to listening to OTHER MEN regardless of what we say (or try to communicate). I should know...I'm surrounded by them!
Also, *I* didn't build any system. Ever. This is the language and verbosity that loses all of the hypothetical "antisexism allies" that you purportedly want to attract. Now, are there ways in which I can unknowingly/unconsciously add to the perpetuation of patriarchy, whereupon I can learn to correct my behavior? Of course. But that's the case for everyone, regardless of what type of attribute we're examining.
And, by that same token, can I use my privilege to stand up for women (and nonbinary people) who are the targets of misogyny? -- or to stand up for other men who are trying to help eradicate misogynistic policies? Yes, of course. But I have to be realistic about how far that privilege can extend, as well as balancing my own safety (as it relates to other intersectional identities that I possess).