Fuel-cell breathalyser use for field research on alcohol intoxication: an independent psychometric evaluation

Participants

Two hundred and fifty individuals in total participated across four studies. Of the 42 participants used in study 4, 31 of them had also been used in study 2. This leads to 229 unique participants in this investigation of FCBs. All data were collected in the Australian state of Queensland from the night-time entertainment district (NED) of Brisbane Central Business District (CBD) and Fortitude Valley on Thursday, Friday, and Saturday nights between the hours of 9 p.m. and 4 a.m.

Apparatus and materials

Diagnostics and selected analyses

SPSS v.24 (IBM, St. Leonards, NSW, Australia) and Statistica v13 (TIBCO Software, Palo Alto, CA, USA), were used for data screening and analyses. All analyses were tested at the standard α = .05 criterion for significance. Before the main analyses, variables were screened for clerical errors, missing values, and assumption violations.

In order to see whether two fuel-cell technologies are giving similar results, or whether a FCB is giving similar results to an evidentiary system, we need to calculate the instrument measurement error. This is approximated using the standard error of measurement (SEM). Using the current case, the SEM of a breathalyser is obtained by first computing the standard SEM between two administrations of the test. The actual formula (see Eq. (1a)) accounts for the test-retest reliability of the breathalyser (r_xx): if the test-retest correlation was 1, then there would be no error in the instrument; if r = 0.9 there would be .1 missing from being a perfect measure (assuming the time to take the measurements was not a factor). This error estimate is then standardised using the standard deviation (usually from time 1; SD) of the test-retest data. This SEM then needs to be computed for two administrations, as demonstrated in Eq. (1b). This creates a standard error of the difference (SED) figure. This “SED describes the spread of the distribution of change scores that would be expected if no actual change had occurred. (A change) larger than 1 .96 would be unlikely to occur (p < .05) without actual change.” (p. 14; Jacobson & Truax, 1991). This leads us to Eq. (1c), where we can say that the difference between two tests is reliable (i.e., not due to measurement error; Score_t2–Score_t1) if it is greater than 1.96 times the SED (for 95% confidence). For 99% confidence we can say that difference between two tests is greater than measurement error if it is greater than 2.58 times the SED. This produces a reliable change index (RCI) whereby a participant’s score could fluctuate if there was measurement error between the instruments. (1a)${\displaystyle SEM=SD\sqrt{1-r_{xx}}}$ (1b)${\displaystyle SED=\sqrt{2{\left(SEM\right)}^2}}$ (1c)${\displaystyle Scor{e}_{t2}-Scor{e}_{t1}>1.96SED\left(95\text{\%}\mathrm{CI}\right).}$

We used ClinTools (Devilly, 2007) for the calculation of the breathalysers’ RCI). If the difference between aBAC samples was within a 95% RCI then we assumed the difference between aBAC readings was not due to measurement errors, because there was no intervening effect that would warrant the difference being outside this interval. When the opposite was observed and the difference was greater than the 95% RCI—we assumed this difference between aBAC readings was affected by measurement error from the breathalysers.

Study 1 method

Apparatus and materials

Study 1 results

Table 1 presents the descriptive statistics and correlations between samples with and without .000 readings. We analysed with and without .000 readings to reduce systematic bias in our results, because a zero rating would likely equal a 100% hit rate.

The first and second readings, on average, differed by .001 aBAC and had an excellent test-retest correlation. The time between samples was associated with a larger aBAC, which meant participants with a higher aBAC took longer to provide a second sample—likely due to difficulty concentrating on the task. The time since their last drink was not significantly associated with aBAC. aBAC was analysed further between each breathalyser and sample (see Table 2).

There was little difference between each first and second reading, and the test-retest correlation was excellent across all breathalysers. The differences between each first and second reading were calculated and transformed into an absolute value—because aBAC change could not be interpreted as a negative value in later analyses. Participant characteristics had limited influence on aBAC readings: the absolute difference between samples was not influenced by age (r =  − .07, p = .47), time since last drink (r =  − .10, p = .28), and there was a non-significant small effect between genders (t(117) = 1.14, p = .26, g = .21). To check if the breathalysers were reliable on average across test-retest we ran a One-Way Analysis of Variance (ANOVA) with the absolute difference between aBAC samples as the outcome variable. The absolute mean difference between test-retest was consistent across the breathalysers, F(3, 115) = .35, p = .79.

Next, we computed a 95% RCI for each breathalyser’s test-retest reliability. To calculate the number of aBAC samples that were impacted by measurement error we used the RCI of each breathalyser and the absolute aBAC difference between samples. If the absolute aBAC difference was less than the 95% RCI, then any difference between aBAC readings was judged as not impacted by measurement error and this was coded 1 = No Change. If the absolute aBAC difference was greater than the 95% CI then measurement error impacted the difference between aBAC readings and this was coded 2 = Change. Table 3 presents the RCI of each breathalyser, the absolute mean aBAC difference, frequency, and percentage of samples that were impacted/unaffected by measurement error.

The RCI of each breathalyser found a high percentage of readings were consistent across the test-retest period. The readings that were impacted by measurement error had a higher mean aBAC than the no-change group. As a final analysis, we conducted a chi-square test to check if the test-retest change was different between breathalysers. There was no difference between the change/no-change groups of each breathalyser, X² (3, n = 119) = .83, p = .84. This suggested that FCBs such as the LE5 produce similar aBAC readings across time. Indeed, the agreement rate of test-retests showed an overall 93.28% of samples were unaffected by measurement error.

Study 2 method

Apparatus

Study 2 results

Table 4 presents the descriptive statistics and correlations of aBAC readings between LE5s’. Three participants were excluded because they provided .000 aBAC readings. Each breathalyser found similar aBAC readings and excellent correlation.

We ran a Repeated Measures ANOVA to check the differences within the subjects’ aBAC measurements. Mauchely’s test of sphericity was not violated and there was a significant difference with a small effect size in the ‘within subjects’ factor of aBAC readings (F(2, 54) = 6.84, p < .01 ${\acute{\eta }}_p=.20$). This suggested that 20% of the variance was attributed to the difference between the breathalyser aBAC readings.

Despite this small variance between the breathalysers’ readings, we placed more importance on the 95% RCI established in study 1 (i.e., RCI = .0117) to account for the measurement error between breathalyzers. We used the same coding specification from study 1 to recode the absolute difference between aBAC readings and calculate the number of samples that were impacted by measurement error. Table 5 displays the mean and standard deviation of the absolute aBAC difference between breathalyzer readings, as well as the frequency and percentage of samples impacted/unaffected by measurement error.

Five samples were impacted by measurement error. These readings had a higher overall mean than the unaffected readings. Ninety-four percent of the readings were reliable between replica FCBs.

Study 3 method

Apparatus and materials

Study 3 results

Eight participants were excluded because they provided .000 aBAC readings. The FC20 recorded similar aBAC readings ($\overline{x}=.074$, σ = .042, n = 34) to the LE5 ($\overline{x}=.073$, σ = .041, n = 34). There was minimal difference between the different FCB readings (${\overline{x}}_{\mathrm{difference}}=.002$, σ_difference = .006) and the relationship between aBAC readings was excellent (r = .991, p < .001). To check for order and sampling effects we ran a Mixed 2 (aBAC: FC20, LE5) × 2 (Order of testing: FC20 used first or second) ANOVA. There was no significant between-subjects main effect for order of testing F(1, 32) = .39p = .54, and no interaction with aBAC readings F(1, 32) = 1.75, p = .20. The within subjects effect of aBAC was approaching significance with a small effect size, F(1, 32) = 3.57, p = .07, ${\acute{\eta }}_p=.10$. Ten percent of the variance between different fuel-cell aBAC readings was possibly due to instrumental differences.

Consistent with the two previous studies we used the 95% RCI (i.e., .0117) established from the test-retest of the LE5. We used the same coding specification from the previous studies to recode the absolute difference between the FC20s’ and LE5s’ aBAC readings, and calculate the number of samples that were impacted by measurement error. Thirty-two samples displayed no difference between different FCB readings¹ (94.12%; $|\overline{x}{|}_{\mathrm{difference}}=.005$, |σ|_difference = .003), while the difference between 2 participants’ readings were impacted by measurement error ($|\overline{x}{|}_{\mathrm{difference}}=.012$, |σ|_difference = .000; 5.88%). The two samples impacted by measurement error recorded a higher aBAC with both breathalysers.

Study 4 method

Apparatus and materials

Study 4 results and discussion

Six participants were excluded because they provided .000 aBAC readings. The Intoxilyzer recorded similar aBAC readings ( $\overline{x}=.080$, σ = .044, n = 36) to the mean BAC of the LE5s’ ($\overline{x}=.078$, σ = .044, n = 36). There was little difference between the FCB readings and the EB readings (${\overline{x}}_{\mathrm{difference}}=-.001$, σ_difference = .005) and the relationship between aBAC measurements was excellent (r = .995, p < .001). To check for order and sampling effects we ran a Mixed 2 (aBAC: Intoxilyzer, LE5) × 2 (Order of testing: Intoxilyzer used first or second) ANOVA. There was no significant between-subjects main effect for order of testing F(1, 34) = .53p = .47 and no interaction with aBAC readings F(1, 34) = .05, p = .94. The within subjects effect of aBAC was non-significant, F(1, 34) = 2.16, p = .15, suggesting little difference between the two technologies.

To conclude our validity analysis, we calculated the absolute difference between all Intoxilyzer readings and LE5 readings. The 95% RCI (i.e., .0117) based on the test-retest of the LE5 in study 1 was used to conservatively estimate the measurement error for both types of breathalysers. We used the same coding specification from the previous studies to recode the absolute difference between the Intoxilyzer and LE5s’ aBAC readings, and calculate the number of readings that were impacted by measurement error. One reading between the EB and FCB was impacted by measurement error (2.86%; Intoxilyzer aBAC = .199; LE5 aBAC = .184), while 35 samples were unaffected by measurement error between the different measurement methods (97.14%; ${\overline{x}}_{\mathrm{Intox}}=.076$, σ_Intox = .039; ${\overline{x}}_{\mathrm{LE5}}=.075$, σ_LE5 = .039; $|\overline{x}{|}_{\mathrm{difference}}=.003$, |σ|_difference = .003).

Implications for field research

To our knowledge this research constitutes the first non-forensic and independent investigation on the psychometric properties of portable breathalysers for field research. Previous investigations (e.g., Gullberg, 2003; Zuba, 2008; Schechtman & Shinar, 2011; Leonard, 2012) analysed the precision and reliability of various breathalysers in forensic and laboratory studies. We have built upon this research by conducting studies which are generalisable to the population field researchers intend to sample. Our findings imply that fuel-cell technology is reliable and valid for researchers who plan to use portable breathalysers for future investigations of alcohol intoxication in large representative populations.

With that said, future field researchers should consider which breathalyser technology to employ in their research designs. We found the FCBs much quicker, easy to administer and cheaper than the evidential system. FCBs would be useful for large-scale alcohol research that require quick aBAC assessments—e.g., research establishing baseline intoxication trends in NEDs. However, it has been argued that FCBs will have greater variation from environmental conditions than infrared systems (Zuba, 2008). We recommend using infrared systems with smaller sample sizes and where researchers are seeking to test a new FCB, while using a different technology as a comparison. It is important to note, however, that an EB is more expensive, requires training to operate and takes longer to administer and test aBAC. In contrast, the EB is generally less susceptible to mouth alcohol (Leonard, 2012).

Mouth alcohol is a large threat to aBAC validity. This is the presence of alcohol that remains in the lining of the mouth, which exaggerates the aBAC (Black, 2017). Fuel-cell detectors are confounded by mouth alcohol, because the detector cannot differentiate between mouth or breath alcohol (Leonard, 2012). In general, mouth alcohol has a greater influence on individuals with low BAC, because it has an inversely proportional dissipation rate with actual BAC (Gullberg, 1992). Individuals who also rinse alcohol around their mouth (e.g., wine drinkers) can influence higher mouth alcohol aBAC (Wigmore & Leslie, 2001). If future researchers suspect mouth alcohol is inflating aBAC they should corroborate their own observations of the participant and the time since their last drink, before re-testing the participant.

Mouth alcohol can produce a 20% decrease between first and second aBAC readings (Sterling, 2012). Our readings of error across the four studies demonstrated² a small average decrease between the first and second aBAC readings, but were much lower than Sterling’s (2012) 20% inference. While we are not certain mouth alcohol entirely explains this decrease, the second aBAC readings often decreased—as shown by Sterling (2012). However, this decrease could be attributed to irregular breath patterns, error at high aBAC readings and other external influences (Gullberg, 2003). Researchers could reduce the impact of mouth alcohol by setting a minimum 10-minute exclusion period since their last drink before participants are ready for breath sampling (e.g., Leonard, 2012) or by having water available for mouth rinsing.

Future researchers should also consider their exclusion criteria when sampling participants, being respectful of the participant’s informed consent (Aldridge & Charles, 2008). A representative sample of any NED will inevitably include heavily intoxicated individuals and those who are uncomfortable providing aBAC samples. Considering we encountered greater measurement error at higher aBAC ratings, one could argue that omitting heavily intoxicated individuals might be an option. However, not all high aBAC readings were impacted by measurement error, which suggests external influences fluctuated aBAC readings (Gullberg, 2006). We argue that participants should not be excluded at the first-point of contact based on experimenter observation because intoxication impacts all individuals differently and this approach will bias the sample (see Devilly, 2018). Understandably, if the participant is not able to proceed with the research—because of extreme intoxication and cognitive impairment—then it would be reasonable for the researcher to terminate the testing and count this as a ‘refusal’ or ‘omission due to impairment’ in the study attrition data. In our study we did not have anyone so intoxicated that they could not complete a breath test, although we had people who walked past us when offered the test.

Limitations

Our four-part study was not without its limitations. We used a RCI derived from the test-retest reliability of the LE5 and extended this index to estimate the measurement error between different breathalysers. The range of measurement error (i.e., RCI = .0117 aBAC) was better than ±.5 of a standard drink—if we assume that 1 standard drink roughly equals .025 BAC g/dL. Still, we concede the evidence for the FCBs convergent validity must be viewed as the ‘best estimate’, because the test-retest reliability of the FC20 and the Intoxilyzer were unknown at the time of analysis. Efforts to contact the manufacturers of the FC-20 were unsuccessful. Future researchers could establish the test-retest for all comparison breathalysers. A second limitation concerns the inference of aBAC readings in place of blood-sampled BAC. This inference is problematic because aBAC readings were argued to underestimate true BAC by 15% on average (Coyle, Field & Starmer, 2010; Kriikku et al., 2014). We did not compare aBAC to blood analysis taken from blood samples, because we conducted field testing of intoxicated patrons in the night-time entertainment districts—an unsuitable environment to collect blood samples from inebriated participants.

Conclusion

Overall, our study featured a strong design and clear analyses to assess the psychometric properties of fuel-cell breathalyser technology. We first established the test-retest reliability of the instrument (i.e., the LE5) to check reliability across time. To evaluate the psychometric properties of these breathalysers we accounted for measurement error between aBAC readings and validated the breathalyser against itself, a similar portable FCB (i.e., FC20), and an EB (i.e., Intoxilyzer 8000)—three breathalysers endorsed by rigorous domestic and international measurement standards. Each breathalyser was regularly calibrated by the distributors to ensure accurate readings were obtained. Our samples were collected from patrons that frequented the NED, which established external validity of these instruments for field research designs. In sum, our results provided excellent reliability and validity for fuel-cell technology and we recommend the use of these breathalysers for field studies of alcohol intoxication.