Over a year ago, the National Research Council — the operating arm of America’s preeminent scientific, medical, and engineering societies — issued an overdue report on Strengthening Forensic Science in the United States. An article in the Arizona Republic promptly declared that
America’s forensic-science operations are so flawed that a federal watchdog agency should be created to regulate crime labs and certify expert witnesses, according to a report issued Wednesday by the National Academies of Science.
The report, requested by Congress, says there is little research to verify the integrity of scientific protocols used in criminal investigations. Because of that, it warns, prosecutions are vulnerable to shoddy lab work, unfounded testimony and false convictions.
The 255-page report may stun those whose notions about criminology are derived from television shows portraying forensic science as infallible. . . .
Dennis Wagner, National Agency Sought for Forensic Sciences, Ariz. Republic, Feb. 19, 2009.
Certainly, the pivotal recommendation of the NRC was to “[e]stablish a National Institute of Forensic Science (NIFS) to develop Best Practices and Standards [with] a culture that is strongly rooted in science . . . .” But the report did not stop there. A second recommendation for institutional reform was to “[i]mprove the scientific bases of forensic science examinations to maximize independence from or autonomy within the law enforcement community [by removing] all public forensic laboratories and facilities from the administrative control of law enforcement agencies or prosecutors offices.” Other recommendations included certification of all forensic examiners, accreditation of all laboratories, and standardized reporting and testimony consistent with scientific knowledge. The first recommendation is directed specifically to the federal government. The others are more broadly applicable.
On April 29, 2010, the supervisor of the Department of Public Safety’s laboratories, Todd Griffith, summarized the NRC report for the Arizona Forensic Services Advisory Committee. He assured the committee that the DPS labs perform every test in every area of forensic science with “controls, blanks, and standards.” No erroneous results can occur, he suggested, because a second examiner “verifies” every conclusion. Other than an acute shortage of funding, there was, he seemed to say, nothing for the state to worry about. Likewise, in a newspaper interview a year ago, he assured the public that “the study recommendations . . . are already met or exceeded in the DPS crime lab.” Id.
These statements warrant scrutiny. This letter identifies some differences between (a) the ideal that appealed to a large, national committee composed of forensic scientists, medical examiners, research scientists, lawyers, and other individuals and (b) the state of affairs as sketched by DPS staff.
I. Lack of Mandatory Certification
The NRC Report insists that “[n]o person (public or private) should be allowed to practice in a forensic science discipline or testify as a forensic science professional without certification. Certification requirements should include, at a minimum, written examinations, supervised practice, proficiency testing, continuing education, recertification procedures, adherence to a code of ethics, and effective disciplinary procedures.” See also Frank Fitzpatrick & Kenneth Martin, The Need for Mandatory Accreditation and Certification, The Police Chief, Sept. 2009. But see Joseph Polski, Forensic Science: A Critical Concern for Police Chiefs, The Police Chief, Sept. 2009 (“Because of the costs associated with accreditation and certification efforts, the IACP is strongly opposed to proposals that would institute ‘mandatory’ accreditation/certification requirements in the absence of secure, sustainable, and stable federal assistance funding.”).
As I understood the DPS presentation, its laboratories–to their credit–are ISO-accredited, but individual certification is not mandatory. Instead, laboratory administrators encourage (in unspecified ways) personnel to become certified. Whether these certifications meet the criteria listed above and the percentage of Arizona practitioners who are certified is not clear.
II. Lack of Supporting Research for Categorical Conclusions
The NRC Committee found that forensic science laboratories use techniques of analytical chemistry and biochemistry (for toxicology, drug analysis, and DNA identification) and these techniques rest on well researched and well understood technologies. It found an inadequate research base to support examiners’ reports of individualization in the analysis of bitemarks, latent fingerprints, and toolmarks (the pattern-matching disciplines) as well as the working theories of arson investigators. The committee reasoned that valid individualizations would require (1) systematic studies of the variability in marks from the same source, (2) extensive studies of variability across different sources, and (3) the success of analysts in determining whether the differences between questioned and known samples fell within the measured range of intrasource variability and outside the measured range of intersource variability. Only then would it be possible to state with confidence that two marks originated from the same source.
The Advisory Committee was informed that the forensic sciences have their own journals with many different studies that demonstrate the validity of all current practices. According to DPS staff, despite the presentations and submissions from many groups and individuals within the forensic science community who presented their views and described the research literature to the NRC committee, the forensic scientists and other members of that committee somehow overlooked much of the relevant research literature.
Of course, the dearth of extensive, published studies of variability of the kind sought by the NRC committee does not mean that analysts frequently err in their individualizations. There is research showing that, under controlled conditions, handwriting analysts can match a set of handwriting exemplars to the authors of the samples, that firearms examiners can match bullets fired from a small set of guns to the guns that fired them, and so on. Such studies demonstrate the potential value of forensic science evidence, but they do not constitute the kind of research the NRC committee deemed necessary to support existing testimony of individualization. This testimony fails to meet the standards of validation used in some other scientific fields, such as evidence-based medicine.
III. Lack of Separation Between Laboratories and Police
The NRC recommendation to alter the administrative structure of most crime laboratories so that they would be more like the independent medical examiners’ offices in various jurisdictions has not been popular with many police agencies and administrators of laboratories working under the existing regime. E.g., Joseph Polski, Forensic Science: A Critical Concern for Police Chiefs, The Police Chief, Sept. 2009. This reaction is not surprising. Whether this opposition reflects a careful analysis of the costs and benefits of the independent examiner model is beyond the scope of this letter.
IV. Lack of Controls and Blinding in Laboratory Work
If DPS laboratories use “controls, blanks, and standards” in every test they perform, then they depart from conventional practice in, say, latent friction ridge analysis. When a questioned latent print is compared to a ten-print card (using the ubiquitous ACE-V method), there are no experimental controls and no well-defined standard for declaring when the number and quality of similarities (and the extent of the differences that might arise from distortion or other causes) meet the threshold for an individualization. It is a question of expert judgment.
Expert judgment is a good thing. It is part of the process of classifying objects in many sciences (such as botany). But Arizona laboratories could alter their procedures (if they have not done so already) to follow the recommendations of forensic scientists and psychologists to ensure that these judgments are as free as feasible from potential biases. For example, validation by a second examiner could be done blindly, that is, without knowledge that a colleague has made an identification or an exclusion.
Experimental controls could be part of the protocol for some comparisons. Just as a well-conducted lineup to secure an eyewitness identification is superior to a single showup, the first analyst could be provided print cards from other individuals with roughly similar prints to serve as controls. I mention this modification to indicate how true controls might be incorporated into current practice and not to argue that any particular procedure would be cost-effective.
The broad point is that instituting procedures to avoid bias and introducing meaningful controls in some situations would bring classifications of various marks more in line with standard practices in other areas of applied science such as clinical pathology and drug safety and efficacy trials. Seriously considering such steps would permit Arizona’s forensic service providers to assure the criminal justice community that they are responding to the NRC committee’s concerns by strengthening as well as defending their current practices.
Voltaire wrote that Le mieux est l’ennemi du bien — the best is the enemy of the good. But the good should not be the enemy of the better. It would be unfortunate if the good work of the DPS and other forensic services providers in Arizona were to stand in the way of the even better work that they could do.