By Matthew B. Devaney
In recent years, we in the law enforcement canine community have been faced with court challenges by the defense bar attacking our training and certification processes, within many detection related disciplines, using purported canine experts. These experts have varying backgrounds. Some are, or were, from the law enforcement canine community while others hail from the world of academia. If not effectively challenged, these experts are frequently successful in raising an issue as to the validity of a canine team’s training. Many of these experts assert that a training and/or certification process is not valid unless a double-blind methodology is followed in administering the exercises. This article will provide you with the issues associated with the use of double-blind testing of detection canine teams and counter measures and arguments that can be implemented to refute this defense challenge.
What is double-blind? – Double-blind testing is designed to eliminate the possibility that an evaluator can cue the canine team as to the existence and location of a training aid. In a doubleblind test, a person sets up the training exercise and places training aids, if any. That person then leaves the exercise location and a separate evaluator, who has no knowledge of the exercise design, administers the testing to the canine team. This evaluator documents the results of the exercise and then confers with the person who set up the testing at a later time to determine the success or failure of the team. As the evaluator has no knowledge as to the existence and location of training aids while observing the team, he or she cannot provide any conscious or subconscious visual cues to the handler or canine regarding this information.
Double-blind testing is used extensively in scientific research to guard against both experimenter bias and placebo effects. The often-cited justification for double-blind testing is to prevent what is referred to as “The Clever Hans Effect.” This reference dates back to 1891 when William von Osten, a German, started displaying his horse “Clever Hans” before the public. His claim was that Hans could answer questions by tapping his hoof. The horse’s purported intellectual abilities included performing mathematical calculations, telling the time, identifying musical intervals, and naming people. Hans’ abilities expanded over time to include complex calculations, such as square roots. Many people tested Hans, one even claiming that the horse had the intellectual ability of a fourteen-year-old boy. In 1907, a group of thirteen scientists known as the “Hans Commission,” tested the horse. The first thing that became apparent was that Hans needed visual contact with the questioner in order to answer correctly. It was also found that, when the questioner was visible, Hans’ correct answers diminished in direct correlation to the distance the person was standing from the horse. The other major finding was that Hans could only answer correctly if the questioner also knew the answer to the question. What the psychologists realized was that a person’s or an animal’s behavior can be influenced by subtle and unintentional cueing on the part of a questioner. This was the first scientific identification of the subconscious cueing that every handler needs to be mindful of while working a canine. As a result of this research, the double-blind method of testing was developed.
William von Osten and Clever Hans
While the scientific community highly endorses double-blind testing, it can have disastrous effects on canine teams. As there is no way to determine whether the canine responded correctly at the time of the alert, the question is, should the handler reward the canine? This is a vital question due to the fact that detection canines are largely trained on a fixed ratio reward system where the canine is rewarded immediately for almost every correct response. Therefore, if the canine is correct and the reward is withheld, the canine will become confused and frustrated. Conversely, if the canine team is wrong and the canine is rewarded, the canine will again become confused and, as incorrect behavior has now been reinforced, false indications will increase. This factor was dramatically demonstrated during a double-blind study conducted by Dr. Lisa Lit at the University of California at Davis. In this study, entitled Handler Beliefs Affect Scent Detection Dog Outcomes, eighteen drug and/or explosive detection canine teams each completed two searches of the same four rooms. Handlers were falsely told that two conditions contained a paper marking scent location. In reality, no conditions contained drug or explosive scent and any alerting reported by the handlers would be incorrect. In addition, food samples and reward objects were also concealed in the search areas, with some of them marked with the paper marker and others not. There were 225 incorrect responses. Defense experts frequently cite this study when asserting the possibility of handler cueing in criminal proceedings involving a canine alert. It should be noted that The Scientific Working Group on Dog and Orthogonal Detector Guidelines (SWGDOG), evaluated this study. This group was a federally funded work group tasked with developing consensus-based best practices to enhance the performance of detector dog teams, and was comprised of noted experts from the law enforcement canine and scientific communities. The group found many shortcomings in the study and posted a commentary that can be found within the SWGDOG website at https://swgdog.fiu.edu/. The study does not indicate whether the handlers rewarded the canines but it is immaterial as, based on the outcome, the canines clearly became confused and frustrated, and the repetitive double-blind testing was a probable cause.
Some defense canine experts have asserted that the SWGDOG guidelines mandate double-blind testing in the certification of detection canine teams. This is a patently erroneous declaration. Within the SWGDOG guidelines, which are also available on the aforementioned website, under the canine team assessments section, three methodologies are identified to be used in evaluating a detection canine team: (1) odor recognition assessment, (2) comprehensive assessment, and (3) double-blind assessment. With the odor recognition and comprehensive assessments, “The evaluating official shall know the desired outcome of the search.” Referring specifically to the certification of detection canine teams, the SWGDOG guidelines state, “The certification shall be comprised of a comprehensive assessment together with either an odor recognition assessment or a double-blind assessment, or both.” For a defense expert to assert that doubleblind testing is mandated by SWGDOG demonstrates either the expert’s lack of knowledge and understanding of the SWGDOG guidelines or a deliberate attempt to mislead a trier of fact.
When a defense canine expert asserts that a canine team’s training and/or certification is not reliable due to the lack of double-blind testing, an effective rebuttal argument can be offered using the below points against employing double-blind testing procedures in the training and certification of detection canine teams:
Results in confusion and frustration in the canine – As stated previously, under doubleblind testing, the determination of the team’s accuracy cannot be made at the time of the alert. Therefore, to provide or withhold a reward from the canine under such a condition can result in confusing and frustrating the canine, which will lead to an increase chance of incorrect responses. This is especially true in the case of repetitive double-blind testing, as was done in the Lisa Lit study.
Does not effectively evaluate the canine team – In spite of defense expert assertions to the contrary, a canine training or certification exercise is not a scientific experiment where the data collected is confined to the answer of a simple yes/no question. The fact that a canine team locates, or fails to locate, a training aid is only part of an effective evaluation. The more important factor is WHY a given team failed to perform correctly. If a team fails to locate a training aid, was it because the canine did not respond to trained odor or was it a result of the handler never working the canine in that part of the search area? If a handler calls an alert incorrectly, was it because the canine incorrectly responded or was it because the handler misread the canine’s behavior? Lastly, if a canine team calls an alert to a location, is the team at source or did the canine fail to trace the odor to true source? If the latter is the case, does the handler have the ability to analyze the wind currents and conditions and effectively work the canine to source? Evaluator knowledge as to the existence and location of training aids is required to make these type of determinations and prescribe corrective actions to address any noted deficiencies. As the evaluator in a doubleblind testing format does not have this information when the team is actively working the problem, these critical determinations cannot be made.
Is logistically impossible when testing multiple teams – While the double-blind method appears attractive at face value, its tenets are unobtainable in the “real world” of canine team training and testing. A cornerstone to the validity of the double-blind method is that the evaluator have no idea as to the existence and location of any training aids. So, let us say we are going to certify eight canine teams using a double-blind format. A test exercise is set up as described earlier with another person placing a training aid in a trashcan and leaving. The evaluator enters with the first canine team who searches the area and correctly calls an alert on the trashcan. The next two teams conduct the exercise with the same result of identifying the trashcan. Is this a true double-blind test for the remaining five teams? Does not the evaluator have a preconceived idea that there is a training aid in the trashcan and, based on this belief, could he or she not subconsciously provide cues to the remaining teams being tested? The answer to the first question is obviously “no” as the answer to the second question is obviously “yes.” Therefore, to fully comply with the philosophy and
intent of double-blind testing would require a separate evaluator for every one or two canine teams, which would be logistically impossible in practical application. The possibility of an evaluator’s actions influencing a canine team’s responses is a factor that needs to be addressed and certain practices should be included in training and certification exercises, in lieu of double-blind testing, that will serve to mitigate this possibility. These practices appear below. The first two should be practiced at all times while the last two can be
employed whenever logistically possible:
Monitor evaluator behavior – Evaluators of canine teams should be cognizant that their behavior and actions can provide cues to the team. Evaluators should make every effort to remain neutral in their actions while observing canine teams working and display the same behavior in an exercise that has a training aid to be located as in one that does not.
Maintain distance from the team and training aid – As the researchers on the “Hans Commission” noted, distance serves to mitigate the possibility that researcher actions can influence a subject’s responses. Maintaining an effective distance from the canine team will serve to diminish the possibility that the handler or canine will identify subtle subconscious visual cues that an evaluator may demonstrate. In addition, evaluators should ensure that they are not divulging the location of a training aid by their positioning and actions within the search area.
Vary evaluators – As repetition is required for the handler or canine to associate any subconscious cues given by an evaluator with a corresponding test result, and different individuals will display different subconscious cues as a result of their own emotions, varying evaluators will serve to mitigate the possibility of cueing the canine teams.
Position evaluators behind barriers – As the scientists on the “Hans Commission” discovered, for the subject of an experiment to be influenced by cues provided by a researcher, the subject must be able to see the researcher. By simply positioning evaluators at locations where they can observe the team, but neither the handler nor canine can observe them, the possibility of evaluator cueing is eliminated. Some examples include; observing an exterior exercise from within a vehicle, observing a team through a louvered window blind or one-way mirror, observing the team from behind a portable cloth blind with observation slits in the fabric, or employing closed circuit video monitoring. By using barriers, the intent of double-blind testing is fulfilled while the evaluator can continue to provide guidance to the handler with regard to canine accuracy and associated rewarding practices. In reality, the possibility of evaluator actions influencing a canine team’s responses, especially while the team is actively conducting a search problem, is extremely remote. It should be remembered that the double-blind method was originally developed to prevent subconscious evaluator cueing under testing conditions where a researcher is asking questions of a subject, whose only responsibility at the time is to answer the questions posed. Under such conditions, the subject is completely focused on the researcher; hence, the possibility of the researcher influencing subject responses by subconscious body actions is extremely high. This is a distinctly distinguishable testing condition from that found in the evaluation of detection canine teams. As anyone that has actually handled a detection canine during a search can attest, when a team is in the process of searching, the handler is completely focused on directing the canine’s search path while simultaneously observing the canine for any alert behavior, and the canine is completely focused on following handler direction while actively sniffing. This leaves little time for either the handler or canine to observe and note the raised eyebrow or puckered lip of a nearby evaluator and discern from these actions whether a training aid is present. When the above listed negative aspects of double-blind testing are weighed against the extremely remote possibility of evaluator influence the method is designed to prevent, the reasoning for rejecting this testing methodology in the training and certification of detection canine teams becomes clear. In addition, the employment of any or all of the above listed methods designed to mitigate the possibility of evaluator influence will serve to rebut defense challenges regarding the lack of double-blind testing, buttress the assertion that the training and certification program is valid, and serve to establish that the canine teams produced are reliable.
In a recent evidentiary hearing in U.S. District Court, a federal agency’s canine training practices were questioned by a noted defense expert, who opined that the canine team in question was not reliable, as double-blind testing was not used in the team’s training and certification. A government canine expert provided rebuttal testimony using the above information. In a lengthy written decision, the court ruled: “The Court finds [the government witness’s] opinions as to the practicalities and the goal of evaluating canine teams being a basis for not utilizing a double blind method as frequently as recommended by [the defense witness] to be well-taken. Moreover, this must be considered in conjunction with the fact that, in practice, during the initial training and certification process, a modified double blind method is used – the observer is behind a barrier which mitigates cuing to a canine. While this procedure contrasts with the use of observers being close to a canine team during maintenance training, the alternate procedures allow observations of different scenarios and allow the canine teams to be evaluated in different ways.” United States v. Ruiz, No. CR 16-121-TUC-CKJ, 2017 WL 1031137, at *17 (D. Ariz. Mar. 16, 2017).
The Court went on to rule the canine team reliable and denied the motion to suppress. Defense canine experts will continue to be utilized as part of our adversarial court process. It is only through continued education and communication between agencies facing these defense challenges that we can all improve and thwart such assaults on the credibility of law enforcement canine training.
Disclaimer: The views expressed in this article do not necessarily represent the views of Customs and Border Protection or the United States.