Initially this idiom referred to only women at work wearing their hair in buns, who would let their hair down arriving home. But now all people, even bald-headed guys might "let his hair down" in order to relax.

+++