Do you honestly think that if James Dale had been living with a girl that he would have been drummed out of the Boy Scouts? All too often the definition of 'morality' depends on what suits the agenda of the person passing judgement.
If he were an avowed fornicator, you bet! See, heterosexuals have the good taste to understand that immorality is not something to flaunt in front of children. Those who don't have no business being leaders to children anywhere.