You'r not failing at life. You need to understand that people generally don't care what you do. If you put yourself in a colleagues situation, would you really be that bothered if they had 3-weeks...

I wouldn't assume that they are going to be sexist or that your worse case scenario will come to light....If it does then there are channels you can go down, they are in the wrong, not you.