"Do you not think its noble to make life better for 0.1% of people?"

It's not within reason or pragmatism to change how everyone on the planet refers to gender - and our entire 'bathroom social norms' to accommodate a tiny minority.

Here is a practical, non-ideological solution:

It seems as though every edifice has to provide special facilities for handicapped people.

Change the sign on the handicapped door, to something considerably more generic, like 'a bathroom for whoever' - and then those who are not comfortable with male/female - or for any other reason - can use it.

Now the 99.9% have their bathrooms - and 'everyone else' who doesn't fit perfectly into that paradigm, can use the 'other' facility.