I'm happy to stand corrected, although it does appear that it's at least advisable to carry ID at all times. Anyway, back to the OP's original question -- your Swiss permit is certainly sufficient ID and does not state your gender.