They’re saying that they knew Mary was pregnant before Joseph married her. The time of His birth didn’t synch with the wedding.
Therefore, they accused His parents (or so they think) of being fornicators before they were married. They simply did not comprehend the Holy Ghost overshadowing Mary.
And of course, later we see they, or many of them, knew Who He was and were simply trying to fight Him for some odd reason (hence His accusation that they were of the Devil).
An exception being Nicodemus who turned out to be sincere.
To be fair neither did Mary or Joseph until an Angel explained it to them personally.