Just let statistics do the work. When a sample is big enough, it's makeup is the same or very close to the same as the total population. No need to know any particular individual.
Assume the margin of victory is 2,000, and the race is really close, 50.01% to 49.99%. That means a little more than half of the total ballots cast went to the winner. If you grab 4,000 ballots, without looking at ANY of them, you can figure a majority went to the winner. If you grab 5,000, the range of votes to the winner is probably in the range of 2,400 to 2,600, with 99.9999999% confidence. More than enough to overcome the margin of victory.
If the winner won by a fat margin, then more of any given sample is winner's votes, and it takes a smaller sample (maybe 110% of the margin of victory) to conclude, with high confidence, the contest was decided wrongly.
I follow your logic - to a certain extent - but I don’t see how one uses statistics to conduct a recount. What you say may convince a judge that a recount is needed but how does one actually compute the new results? It would seem that would have to be done one vote at a time.