“The secret to random sample surveys is to keep them random. Once you get into weighting, or attempting to fit results to a turnout model, they are no longer random.”
Absolutely true, but random samples will not work in this type of statistical study. A random sample from a population of humans will have a significant, and often hard to understand or measure, sample selection bias. Sample selection bias is the biggest factor in gutting an otherwise properly constructed statistical study. I remember at school many years ago we had a presentation from a visiting economist about a study she did about the benefits of a government study program providing free pre-natal vitamins. Early on in the presentation I raised my hand and asked whether she had accounted for a sample selection bias (meaning that those who already would have cared about the health of their babies are the ones who would take the time to show up for the free vitamins). I was not trying to be rude, the question just sort of came out. Well, one could tell from the tone in the room for the rest of her presentation that we thought her study was essentially worthless.
A random sample would work if, for example, we want a sample from a series of products being produced at a manufacturing plant. In this case, either a random sample or a periodic sample would be fine.
However, a random sample of voters is not so easy. If we use phones, we will have a bias. If we find voters on the street, we will have a bias, etc. Therefore, the method is to divide the voting population into many sub-categories, get a random sample from each category, and then add the categories together with a weighted average. And then we have the rub: how does one allocate the weights to the various samples? This is where the years of experience and prior data is vital to the poll. But still, weighting accurately something that will happen in the future is really not so easy to do.
The most accurate method is to get a random sample of voters as they are leaving a voting booth. Of course, even in this case we may have sample selection bias (a Republican voter at a Democratic box may, for example, not want to participate in the survey; or the voting patter of early or absentee voters may be very different than the pattern on election day; etc.).
By the way, in our Texas county a young election worker for a local candidate worked up our early voting turnout numbers. By looking at the turnout at our local voting precinct level (voting box level), and comparing to prior turnout, we can tell that the precincts with a higher percentage of republicans are turning out higher than the precincts with a lower percentage of republicans.
Now, an older method with quite a bit of success was called Augury.
Supposedly based mostly on the flight of birds, the augur, or priest, would turn loose a bunch of birds from a box and watch which way they flew ~ although many have proposed it was more sophisticated than that based on the observation that if you keep a box of birds overnight they will defecate upon release from the box ~ as birds usually do when they take off to fly somewhere.
By doing this in a plaza filled with regular stones the Augur could simply walk around and note how many white spots were in which stones ~ and how they clustered, or didn't cluster. Direction of flight could readily be inferred this way as well ~ if questioned by higher authority.
The Auguries could be used to displace laws considered INAUSPICIOUS!
I believe my observation on the fall of the effluvia has probably not been associated with political prognostication before, but that could been a bit of secret knowledge among the Augurs.
It's certainly not secret among pollsters ~ that the flow and accommodations of the BS may well be more important than the head counting.
But whatever, as soon as you stray from the question of randomness and introduce the value of eliminating bias from a random poll, you begin degrading the quality of the statistical validity of your poll ~ and become, to a degree, not different from the Roman Augurs!.
The Roman Augurs over a period of time ~ centuries actually ~ sought to avoid being unduly influenced by NEGATIVE OUTCOMES which might show up from time to time as they applied their art to determine that which was Auspicious and that which was not.
There seem to have been some rules or observations built up that guided the Augur in dealing with the negative outcomes. To wit;
Against the negative auspicia oblativa the admitted procedures included:
1. actively avoiding to see them.
2. repudiare refuse them through an interpretative sleight of hands.
3. non observare by assuming one had not paid attention to them.
4. naming something that in fact had not appeared.
5. choosing the time of the observation (tempestas) at one's will.
6. making a distinction between observation and formulation (renunciatiatio).
7. resorting to acknowledging the presence of mistakes (vitia).
8. repeating the whole procedure.
I do believe you would recognize these principles at work in modern use of what had been random sample surveys of opinion ~ love that 'interpretive sleight of hands' ~ that sucker is still a major way of dealing with bad news ~ before the election anyway.
Very nice explanation of sample selection bias.
I would like to hear you discuss a little more how to balance the various samples in order to get an accurate reflection of reality.