Monthly Archives: May 2010

DNA versus Fingerprints

A recent edition of The Economist reported on the perceived “CSI Effect.” [1] The following paragraph raised some eyebrows:

In reality, scientists do not deal in certainty but in probabilities, and the way they calculate these probabilities is complex. For example, when testifying in court, a fingerprint expert may say that there is a 90% chance of obtaining a match if the defendant left the mark, and a one in several billion chance of a match if someone else left it. In general DNA provides information of a higher quality or “individualising potential” than other kinds of evidence, so that experts may be more confident of linking it to a specific individual. But DNA experts still deal in probabilities and not certainties. As a result of all this reality checking, trials are getting longer and more cases that might previously have resulted in quick convictions are now ending in acquittals.

I am skeptical of the claim that television shows have converted quick convictions into acquittals, but even if that is true, how is DNA is more “individualising” than fingerprints? Fingerprints of identical twins are distinguishable. DNA profiles, in their current form, are not. The latter fact has been a problem in some investigations and prosecutions. Police investigating a notorious jewel heist in Berlin found DNA “in a drop of sweat on a latex glove discarded next to a rope ladder.” [2] When “they ran the material through the German [DNA] database, . . . [t]he computer identified two 27-year-old identical twins . . . .” Id. A German court released them, explaining that “[f]rom the evidence we have, we can deduce that at least one of the brothers took part in the crime, but it has not been possible to determine which one.” Id. According to a Time Magazine story:

Identical twins share 99.99% of their genetic information, and the tiny differences are impossible to isolate because of their nature; they tend to be spontaneous mutations limited to certain organs or tissues. “Identifying those [differences] would amount to dissecting the suspects,” says Peter M. Schneider, a University of Cologne forensic expert. “Our hands are tied in a case like this,” says criminal-law expert Hans-Ullrich Paeffgen of Bonn University. “The law doesn’t allow us to detain someone indefinitely just because he is suspected of a crime. This may be different elsewhere. But I’d rather live in a country where someone guilty is not convicted for lack of conclusive evidence than in a place where innocent people are locked up.”

This isn’t the first time an identical twin has proved impossible to pin down. The genetic material can thwart paternity tests if both twins claim — or deny — fathering a child. In the U.S., a jury in a rape trial in Houston deadlocked in 2005 when the DNA recovered at the crime scene matched identical twins who had kidnapped their victim together. A year earlier in Boston, a suspected rapist blamed his identical twin when confronted with the matching DNA. Although he was already serving a sentence for a rape conviction, the jury could not agree on a verdict, and the judge declared a mistrial. Earlier this year, an identical twin suspected of drug-smuggling and sentenced to death in Malaysia was set free when the court could not prove beyond doubt whether he or his brother had committed the crime.

Thus, fingerprint examiners like to claim that their information is more individualizing. But this overlooks the issue of whether today’s subjective comparisons of latent prints to clear, complete ones taken under controlled conditions are any more precise than STR profiling. Latent print examiners declare a match when they have a large subjective probability that the latent print and the exemplar originated from the same finger. In the United States, at least, the usual testimony for a subjective match is that the items came from the same finger. Do examiners ever testify to the likelihoods of competing hypotheses, as illustrated by The Economist‘s statement “that there is a 90% chance of obtaining a match if the defendant left the mark, and a one in several billion chance of a match if someone else left it”? Lacking the data to support statements like these and being trained to give categorical opinions about the source of a latent print, fingeprrint examiners do not seem to testify in terms of likelihoods. However, research supporting likelihood-based testimony exists, and the time is ripe for a change.

References

1. The “CSI Effect”, Economist, Apr. 22, 2010, http://www.economist.com/science-technology/displaystory.cfm?story_id=15949089

2. Claudia Himmelreich, Despite DNA Evidence, Twins Charged in Heist Go Free, Time, Mar. 23, 2009, http://www.time.com/time/world/article/0,8599,1887111,00.html

An Open Letter to the Arizona Forensic Services Advisory Committee

Over a year ago, the National Research Council — the operating arm of America’s preeminent scientific, medical, and engineering societies — issued an overdue report on Strengthening Forensic Science in the United States. An article in the Arizona Republic promptly declared that

America’s forensic-science operations are so flawed that a federal watchdog agency should be created to regulate crime labs and certify expert witnesses, according to a report issued Wednesday by the National Academies of Science.

The report, requested by Congress, says there is little research to verify the integrity of scientific protocols used in criminal investigations. Because of that, it warns, prosecutions are vulnerable to shoddy lab work, unfounded testimony and false convictions.

The 255-page report may stun those whose notions about criminology are derived from television shows portraying forensic science as infallible. . . .

Dennis Wagner, National Agency Sought for Forensic Sciences, Ariz. Republic, Feb. 19, 2009.

Certainly, the pivotal recommendation of the NRC was to “[e]stablish a National Institute of Forensic Science (NIFS) to develop Best Practices and Standards [with] a culture that is strongly rooted in science . . . .” But the report did not stop there. A second recommendation for institutional reform was to “[i]mprove the scientific bases of forensic science examinations to maximize independence from or autonomy within the law enforcement community [by removing] all public forensic laboratories and facilities from the administrative control of law enforcement agencies or prosecutors offices.” Other recommendations included certification of all forensic examiners, accreditation of all laboratories, and standardized reporting and testimony consistent with scientific knowledge. The first recommendation is directed specifically to the federal government. The others are more broadly applicable.

On April 29, 2010, the supervisor of the Department of Public Safety’s laboratories, Todd Griffith, summarized the NRC report for the Arizona Forensic Services Advisory Committee. He assured the committee that the DPS labs perform every test in every area of forensic science with “controls, blanks, and standards.” No erroneous results can occur, he suggested, because a second examiner “verifies” every conclusion. Other than an acute shortage of funding, there was, he seemed to say, nothing for the state to worry about. Likewise, in a newspaper interview a year ago, he assured the public that “the study recommendations . . . are already met or exceeded in the DPS crime lab.” Id.

These statements warrant scrutiny. This letter identifies some differences between (a) the ideal that appealed to a large, national committee composed of forensic scientists, medical examiners, research scientists, lawyers, and other individuals and (b) the state of affairs as sketched by DPS staff.

I. Lack of Mandatory Certification

The NRC Report insists that “[n]o person (public or private) should be allowed to practice in a forensic science discipline or testify as a forensic science professional without certification. Certification requirements should include, at a minimum, written examinations, supervised practice, proficiency testing, continuing education, recertification procedures, adherence to a code of ethics, and effective disciplinary procedures.” See also Frank Fitzpatrick & Kenneth Martin, The Need for Mandatory Accreditation and Certification, The Police Chief, Sept. 2009. But see Joseph Polski, Forensic Science: A Critical Concern for Police Chiefs, The Police Chief, Sept. 2009 (“Because of the costs associated with accreditation and certification efforts, the IACP is strongly opposed to proposals that would institute ‘mandatory’ accreditation/certification requirements in the absence of secure, sustainable, and stable federal assistance funding.”).

As I understood the DPS presentation, its laboratories–to their credit–are ISO-accredited, but individual certification is not mandatory. Instead, laboratory administrators encourage (in unspecified ways) personnel to become certified. Whether these certifications meet the criteria listed above and the percentage of Arizona practitioners who are certified is not clear.

II. Lack of Supporting Research for Categorical Conclusions

The NRC Committee found that forensic science laboratories use techniques of analytical chemistry and biochemistry (for toxicology, drug analysis, and DNA identification) and these techniques rest on well researched and well understood technologies. It found an inadequate research base to support examiners’ reports of individualization in the analysis of bitemarks, latent fingerprints, and toolmarks (the pattern-matching disciplines) as well as the working theories of arson investigators. The committee reasoned that valid individualizations would require (1) systematic studies of the variability in marks from the same source, (2) extensive studies of variability across different sources, and (3) the success of analysts in determining whether the differences between questioned and known samples fell within the measured range of intrasource variability and outside the measured range of intersource variability. Only then would it be possible to state with confidence that two marks originated from the same source.

The Advisory Committee was informed that the forensic sciences have their own journals with many different studies that demonstrate the validity of all current practices. According to DPS staff, despite the presentations and submissions from many groups and individuals within the forensic science community who presented their views and described the research literature to the NRC committee, the forensic scientists and other members of that committee somehow overlooked much of the relevant research literature.

Of course, the dearth of extensive, published studies of variability of the kind sought by the NRC committee does not mean that analysts frequently err in their individualizations. There is research showing that, under controlled conditions, handwriting analysts can match a set of handwriting exemplars to the authors of the samples, that firearms examiners can match bullets fired from a small set of guns to the guns that fired them, and so on. Such studies demonstrate the potential value of forensic science evidence, but they do not constitute the kind of research the NRC committee deemed necessary to support existing testimony of individualization. This testimony fails to meet the standards of validation used in some other scientific fields, such as evidence-based medicine.

III. Lack of Separation Between Laboratories and Police

The NRC recommendation to alter the administrative structure of most crime laboratories so that they would be more like the independent medical examiners’ offices in various jurisdictions has not been popular with many police agencies and administrators of laboratories working under the existing regime. E.g., Joseph Polski, Forensic Science: A Critical Concern for Police Chiefs, The Police Chief, Sept. 2009. This reaction is not surprising. Whether this opposition reflects a careful analysis of the costs and benefits of the independent examiner model is beyond the scope of this letter.

IV. Lack of Controls and Blinding in Laboratory Work

If DPS laboratories use “controls, blanks, and standards” in every test they perform, then they depart from conventional practice in, say, latent friction ridge analysis. When a questioned latent print is compared to a ten-print card (using the ubiquitous ACE-V method), there are no experimental controls and no well-defined standard for declaring when the number and quality of similarities (and the extent of the differences that might arise from distortion or other causes) meet the threshold for an individualization. It is a question of expert judgment.

Expert judgment is a good thing. It is part of the process of classifying objects in many sciences (such as botany). But Arizona laboratories could alter their procedures (if they have not done so already) to follow the recommendations of forensic scientists and psychologists to ensure that these judgments are as free as feasible from potential biases. For example, validation by a second examiner could be done blindly, that is, without knowledge that a colleague has made an identification or an exclusion.

Experimental controls could be part of the protocol for some comparisons. Just as a well-conducted lineup to secure an eyewitness identification is superior to a single showup, the first analyst could be provided print cards from other individuals with roughly similar prints to serve as controls. I mention this modification to indicate how true controls might be incorporated into current practice and not to argue that any particular procedure would be cost-effective.

The broad point is that instituting procedures to avoid bias and introducing meaningful controls in some situations would bring classifications of various marks more in line with standard practices in other areas of applied science such as clinical pathology and drug safety and efficacy trials. Seriously considering such steps would permit Arizona’s forensic service providers  to assure the criminal justice community that they are responding to the NRC committee’s concerns by strengthening as well as defending their current practices.

Voltaire wrote that Le mieux est l’ennemi du bien — the best is the enemy of the good. But the good should not be the enemy of the better. It would be unfortunate if the good work of the DPS and other forensic services providers in Arizona were to stand in the way of the even better work that they could do.

The Spin Doctors

The National Research Council February 2009 report on forensic science was widely perceived as an indictment of laboratories and expert witnesses across the United States. Newspaper and magazine articles, op-ed headlines, television news, and radio talk shows referred to the field as “seriously deficient,” “dismal science,” “in disarray,” even “clueless.” [1, p. 8]. The NRC report concludes that many fields of forensic science are under-researched, under-regulated, and oversold.

Parts of the forensic science establishment have responded with various forms of denial. One ploy is to challenge the thoroughness of the report. For example, on April 29, 2010, I listened to the supervisor of the Arizona Department of Public Safety’s laboratories, Todd Griffith, spin an interesting story about the report. According to Mr. Griffith, the National Academy of Science (NAS) committee correctly endorsed the well validated methods of forensic toxicology and DNA profiling but incorrectly rejected the equally well validated procedures for other forensic disciplines. To prove that the committee did not adequately examine the immense body of research supporting current testimony, Mr. Griffith provided the names of the many journals with articles on forensic science.

In a similar vein, the Scientific Working Group on Friction Ridge Analysis, Study, and Technology (SWGFAST) issued a position paper “respectfully disagee[ing]” with the NRC’s observation that “[w]ith the exception of nuclear DNA analysis, . . . no forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source.” According to this group of fingerprint analysts, “[h]istory, practice, and research have shown that fingerprints can, with a very high degree of certainty, exclude incorrect sources and associate the correct individual to an unknown impression” [2].

To a degree, the NRC committee set itself up for this criticism by the lack of full documentation of many of its conclusions. The report could have contained an analysis (at least in an appendix) of the relevant studies that have been published in quality journals. Other NRC committees (on polygraphs in lie detection, on compositional analysis of bullet lead, and on DNA evidence, to name three) were less conclusory — and hence more convincing — in their assessments of the state of the science.

Still, if one must choose between the NRC’s evaluation and the denials of practicing forensic scientists, there is no contest. Where is the published, rigorous scientific research that shows that latent print examiners can tell that only one individual in the world could be the source of a latent print (as these examiners assert day in and day out in the courtroom)? It does not appear among the seven references cited by SWGFAST. Where is the proof that marks on bullets permit them to be matched to the barrel of a single gun? Mr. Griffith’s response was a study that showed that examiners could correctly match ten bullets to ten barrels. Such research is encouraging. It demonstrates that firearms examiners are not just guessing (although no one ever said they were). But it does not begin to address firearm examiners’ claims of discernible uniqueness [3, 4].

Responses like those of the Arizona DPS and SWGFAST prompted NAS Committee co-chairman and United States Court of Appeals Judge Harry T. Edwards to remark at a conference last week that “no one has meaningfully refuted the Committee’s finding” about individualization [5, p. 3]. Moreover, Judge Edwards replied to the “they did not read our research” complaint by explaining that

The Committee spent an enormous amount of time listening to testimony from and reviewing materials published by numerous experts, including forensic practitioners, heads of public and private laboratories, directors of medical examiner and coroner offices, scientists, scholars, educators, government officials, members of the legal profession, and law enforcement officials. Not only did we examine how the forensic disciplines operate, we also carefully considered any peer-reviewed, scientific research purporting to support the validity and reliability of existing forensic disciplines. Additionally, we invited experts in each discipline to refer us to any pertinent research. Committee members and staff spent countless hours reviewing these materials. And before the Report was released, it was peer-reviewed by outside experts in the fields of science, law, and forensic practice.

[5, p. 2].

To avoid confrontations it cannot win, the forensic science community needs to consider alternative ways to present the findings of fingerprint analysts, toolmark examiners, handwriting analysts, and, yes, even DNA profilers. (Claims of probabilities in the quadrillions and septillions have their own problems.). There is considerable information in the forensic evidence. It is time to present it fairly and with a frank recognition of its possible limitations.

References

1. David H. Kaye, The Good, the Bad, and the Ugly: The NRC Report on Strengthening Forensic Science in America, 50 Sci. & Justice 8 (2010).

2. SWGFAST, Strengthening Forensic Science in the United States: A Path Forward, 2009, Aug. 3, 2009 (www.swgfast.org/Comments-Positions/SWGFAST_NAS_Position.pdf).

3. Michael J. Saks & Jonathan J. Koehler, The Individualization Fallacy in Forensic Science Evidence, 61 Vand. L. Rev. 199 (2008).

4. David H. Kaye, Probability, Individualization, and Uniqueness in Forensic Science Evidence: Listening to the Academies, 75 Brook. L. Rev __ (2010, in press).

5. Harry T. Edwards, The National Academy of Sciences Report on Forensic Sciences: What it Means for the Bench and Bar, Address at the Superior Court of the District of Columbia Conference on the Role of the Court in a Developing Age of Science and Technology, Wash. D.C., May 6, 2010 (http://www.cadc.uscourts.gov/internet/home.nsf/AttachmentsByTitle/NAS+Report+on+Forensic+Science/$FILE/Edwards%2C+The+NAS+Report+on+Forensic+Science.pdf).