Men have more rights to female-only spaces than women do. When women complain that a man is using female-only spaces it is the women who are thrown out or banned and the men who are allowed to remain. When a space goes from being female-only to mixed sex it is the women who lose access to those spaces.