How the Probability of a False Positive Affects the Value of DNA Evidence

Guy sat in jail 4 years before he could get a second test to prove him innocent. Original sample had DNA from two people and test resulted in a "match" with an entirely different person!

Excerpt is from a journal paper and a bit dense, but you can skip the math to get an idea of what they're talking about.

Don't have blind faith in DNA testing to ID suspects through database searches. It's one thing if you ID a suspect and THEN do a DNA test, quite another for a DNA "dragnet" because in that case, circumstantial evidence is believed to be more credible.

But, but, but what did the woman say?? That is the most important thing /s

Exactly. This proves that DNA can lie; but we know of course that women never lie. Except when they’re accusing a liberal.

Especially a Lib named BillyJeff!

Interesting.

As a former leader of a hazmat team I do not have experience with DNA testing, but I do have experience using sophisticated equipment to identify unknown substances.

This may not be completely relevant to the specifics being discussed. But to me it feels similar. Using various sophisticated and incredibly expensive equipment and techniques we would sometimes find evidence of a very small amount of a very harmful substance and my crew would fixate on that and lose sight of the larger amount of a much less hazardous substance that actually made up the bulk of the threat to the public.

It reminds me of drowning in polluted water. Who cares about the pollution in the water when the real problem is that you can’t breath?

It reminds me of drowning in polluted water. Who cares about the pollution in the water when the real problem is that you can’t breath?.

But it feels so icky! Gasp! Gasp!

And I just KNOW that I will get cancer when I’m 65! Gasp!

It is just like all of the talk about glyphosates and cancer lately.

We are so wealthy and well fed these days that people have become completely oblivious to what the problems of human existence really are.

Hunger and starvation once were universal problems. Just a century ago living to the age of 70 was a rare thing for anyone but the very wealthy.

Wondrous chemicals and modern farming methods have solved the problem of having enough to eat so well that obesity has become a problem of the poor.

And now we have lawyers out there try to kill the goose that laid the golden egg. People have really lost the sight of the big picture and a focusing on the gnat on the wine glass.

I’ve got a long response for you. Hope you don’t mind.

It depends on the nature of the test itself. Is the test truly deterministic, or does it depend on probability?

In your case, if you use something like a mass spectrometer and you test for, let’s say, selenium, and if its spectrum is unique, meaning no other substance will produce that spectrum, if the spectrum for selenium appears, then you have selenium. The only error would arise from miscalibration.

On the other hand, if your tests are field tests, typically designed to be fast and affordable, and depend on more generic parameters shared by classes of substances, you might get results such as “90% chance of X, .01% chance of Y, etc..

In that case, the chemicals with low probability have a better chance of being a false positive, meaning, the substance was reported to be present, but actually is not.

You also have to know what the result actually means, as in, when it says 90% X and .01% Y, does that mean what you tested has 90% X in it and .01% Y, or is expecting a single substance and only reporting the odds that the entire substance is X or Y?

In the former case (and if it’s deterministic), you’d want to estimate how much X and Y is present on site and take measures based on what is considered a dangerous level of exposure (drowning in dirty water). In the latter (probabilistic), you’d want further testing or different evidence as to whether or not Y is actually present.

The problem with DNA testing (forget everything you’ve seen on TV), is that it’s probabilistic from the get go, so to say.

It’s easier to start with fingerprints. You can use a computer to get a “point” match on the imagery of a crime sample and the source of a booked suspect. You can then overlay images to get a better feel of how close the two prints are. Probability only comes into play when you have partial prints or just plain ol’ crappy ones. Even so, this can be ameliorated with image overlays.

In a loose analogy, DNA works off the point match, but there is no overlay step. DNA has millions upon millions of genes, and they only look at some. From the get go, based on the nature that test, the crime scene sample would “match” tens of thousands of people and that’d be only if the testing process and interpretation were “perfect”.

So the point of the article is, in a sense, that if your test is so dependent on probability, you MUST continue to factor the impact of that early decision throughout the exercise. Put another way, you have to finish the “math problem” you’ve created for yourself, not just stop half way and think you have the full answer.

What arises out of probability is that “false positive”, and cannot be ignored, but usually is, especially in the latter stages of criminal cases (jury explanations). E.g., a prosecutor will invariably point out that “the odds of matching a random sample are a billion to one”. That statement alone is enough to convince juries of the “perfection” of DNA evidence. However, if you account for false positives, there are situations where those same odds can be brought down to 1 in ten (see article).

False positives can be ameliorated by subsequent retesting, but that depends on the quality and quantity of the crime scene sample. I.e., you have plenty of victim sample, but usually very small perp samples.

Forgive my graphic language, but even if you have a “big load” in a rape case, problems arise due to the fact that you have to attempt to separate the victim’s DNA and possibly, the DNA of other “partners”. This is the case of my excerpt where someone sat in jail 4 years before exoneration (and he was never even a “partner”).

I post this article because I keep seeing other articles posted about people being “caught” by DNA database searches with data from ancestry.com, etc.. Many hail this as a great tool but if people understood the nature of DNA testing (and the math - which is the biggest obstacle) far fewer would.

You not only have to account for the problems with testing, but also the realities of how criminal cases are processed. It’s one thing to use DNA as “the straw on the camel’s back” to lock down a case, it’s quite another to get your first suspect with it through a database search.

Many cases do not have good leads. If you have a dead woman in an alley and suspect the husband (who claims he wasn’t with her at the time), and find a blood drop nearby that matches him, you’ve got a pretty sensible case.

However if you start out with a database search, then narrow it down to proximity (are they in the same town), you’re more likely to pick up innocent people.

Think of it in terms of mugshots. As is often the case, victims are shown mugshots to help ID a perp. Since many people do actually have similarities, it is well known that victims have identified the wrong person in this fashion, and are often very convincing in their testimony that “that’s him”. Part of the problem with the mugshot approach is that you’re not looking at everybody, just people with prior convictions. Simply by having your mugshot in the “book”, you are more likely to be fingered for a crime you did not commit.

So having DNA in a database is like having your mugshot in the book. You get a “hit” on your sample, not necessarily reliable, and the next thing you know you are in cuffs.

It’s possible you’d get exonerated later, but that takes a finite amount of time, much longer than the duration of a television episode. At a minimum you have the time to wait to appear before a judge for bail. If the DA asks the judge to hold you at least while they’re waiting for the sample taken at booking to be processed, you might be waiting in a cell a bit beyond “monday morning”.

And before you think, “hey wait, my retest will exonerate me”, again remember that this is not a TV show. It’d be up to you and your lawyer to push for it and pay for it. The DA may fight back (because they want a closed case), the judge may not allow it, and you may not have the money to do it. Again the excerpt - guy waited 4 years post-conviction for retest.

Even though the mugshot scenario is well known, many don’t seem to mind because they’re closing on a case with an already known bad guy so who cares? This guy that probably should’ve been in prison longer anyway, is thrown back in the slammer.

But DNA databases are loaded with innocents who are now leaving their freedom to chance. Not a bet I want to make.

Thanks for the long and well reasoned post!

Our mass spectrometer was one of ou most useful tools, and we had a large database of common compounds that we were constantly updating. If we had a good match from a “clean sample” our computer program gave us statistical data as to the likely hood of our mystery substance being common talcum powder, powdered sugar, or something more nefarious. Of course the problem is that many times our sample could be a mixture of talcum powder, powdered sugar, and possibly substances which were more nefarious. If you are dealing with a perpetrator who is knowledgeable they may understand how our process works and mix in a substance that will mask the real threat.

As you stated calibration was a constant concern, especially considering that our machine although incredibly expensive was still setup in the back of motorhome like lab. We also had access to more sophisticated and even more expensive stationary equipment and personnel that we shared with other agencies. It was interesting work and some people have much better aptitude than others at it. Of course the majority of our experience in analysis was from our practice drills set up by our peers.

The issue that sometimes would arise is that my crew would sometimes interpret readings that combined with their initial assumptions would cause the direction of their results to be misleading. In other words we could waste a lot of time following a theory that was skewed by “conformational bias”.

This is obviously a problem that commonly plagues any type of police work when the real goal of the lab work typically is to confirm a theory on who the guilty party is. In the case of my crews and I, our goal was to evaluate hazards to the public. The exact identity of our mystery compound was typically not as important as to whether its properties suggested a serious threat and then to find the most efficient means to mitigate possible threats.

When establishing guilt or innocence, it is important that the lab technicians not have a slanted interest in the results.