There's also the matter of what kind of person is able to know the Word and vigorously ignore it.
That's a good observation. In fact, I think if Douglass had assessed the situation a bit more, he would have found that the "irreligious" were actually religious--they just didn't make quite as big a brag of it.