Is "the Sober Truth" about AA sober and true? Considering a recent challenge to AA. (Part 1)

Is Alcoholics Anonymous being subjected to unfair criticism--perhaps partly rooted in unconscious anti-religious bias against AA's Christian roots?

This was the question I found myself asking after reading a recent Salon piece by Lance Dodes and Zachary Dodes, authors of the recent book, The Sober Truth. The primary author, Lance Dodes, is a clinical psychiatrist who's been treating addiction for decades--and he seems to be on something of a crusade. In a recent NPR interview, he asserted that "AA has probably the worst success rate in all of medicine" and is "harmful to the 90 percent who don't do well."

This is quite a scathing indictment of one of the most respected and venerable organizations for fighting alcoholism and addiction. And as I read the Salon piece (an excerpt from the recent book), I started out skeptical, became irritated, and ended up a bit angry. Dodes does make a number of good points about the way our culture has appropriated AA's 12-step model--making it the heart of for-profit rehab centers, and endorsing it to the exclusion of other approaches to promoting sobriety. But when it comes to Dodes' direct challenge to the value of AA, the argument in the Salon piece amounts to little but sophistry.

In fact, it reminded me a bit of Richard Dawkins' rhetorically excessive and intellectually sloppy "take-down"of religion in The God Delusion. Which might, I suppose, mean that Dodes and Dodes have a bestseller on their hands. But it doesn't mean that the millions who benefit from AA should pay much attention to them.

In the next couple of posts on this blog, I want to consider some of the problems I found with the Dodes' supposed take-down of AA, at least as laid out in the Salon piece. While it's possible that some of these problems are resolved in their book, my decisions about what books to spend time on need to made in the light of the information available prior to investing those resources. And the Salon piece leads me to be very skeptical that the investment would be worth it.

Here's why.

1. Inconsistent Approach to Anecdotes

Part of my initial skepticism when I started reading the piece was based on my experience: the people I know who've made a sincere effort to work the program have had great success with it, which is not what I'd expect if Dodes' 5-10% success rate were accurate. This is "anecdotal" evidence, of course--specific personal experiences used to assess a general claim.

But anecdotal evidence is not always bad. If a general claim is at odds with one's personal experience, this doesn't mean the general claim is false. But it does give one reason to be skeptical--especially if one's body of personal experiences would be very surprising or unlikely were the general claim true.

But while Dodes is happy to sweepingly dismiss the anecdotal evidence in support of AA with a few gestures (pointing out that the general public only hears from the AA success stories), when it comes to his own astonishing claim that AA is "harmful to the 90% who don't do well," Dodes resorts to...anecdotal evidence.

He tells the story of "Dominic," a patient of his who got out of AA the same mangled understanding that Dodes puts forward as if it were the only one (more on this in the next post). Dominic wasn't able to achieve sobriety when he tried to follow his mangled understanding of AA's principles, but he did manage to uncover what was really triggering his drinking after eight months of (potentially expensive) therapy. Apparently, since pursuing the AA path kept Dominic from getting the therapy he needed (as if pursuing therapy and AA were mutually exclusive, as if there aren't tons of people who do both), AA delayed his enlightenment--and may even have triggered him (when, for example, he misinterpeted AA's tradition of giving out tokens to celebrate milestones in sobriety as a tally system that played into his fears of failure).

I'm sorry. When you make a claim to the effect that AA is bad for 90% of people who pursue it, you need more than a dubious anecdote or two. An anecdotal experience might lead you to be suspicious of a program--but to decide whether that suspicion is justified, you need more than anecdotes. In the Salon piece, that's all we get.

2. Dubious Statistics

But what about Dodes' assertion that AA has a low success rate, of only 5-10%? In the Salon piece, Dodes assures us that this figure comes from peer-reviewed studies. I'm sure there are such studies. But what exactly are these studies measuring? What is their definition of "success"? According to a Scientific American article, a 16-year study of problem drinkers published in 2006 found the following:

Of those who attended at least 27 weeks of AA meetings during the first year, 67 percent were abstinent at the 16-year follow-up, compared with 34 percent of those who did not participate in AA. Of the subjects who got therapy for the same time period, 56 percent were abstinent versus 39 percent of those who did not see a therapist—an indication that seeing a professional is also beneficial.

At least on one reading of "success rates," this could be taken to mean that AA has a 67% success rate. But, of course, to simply assert that AA has been shown to have an almost 70% success rate based on this study would be seriously misleading--especially since not participating in AA has, by this study, a 34% success rate.

Dodes' figure, without the proper context and qualifications, is likely to be similarly misleading. (From what I gather here, one common source of the 5% success figure is that it's a measure of retention in the program after a year, and has its roots in a misconstrual of an internal AA study--with at least two studies indicating a substantially higher retention rate. But are retention rates really the best way to measure success? How many show up at a meeting or two well before they're ready to commit to the challenge of getting sober?)

Even if we can settle on a useful definition of success, another problem is this: In measuring the success of a twelve-step program, you need to distinguish between the success of those who actually work the steps from the success of those who just show up. It's like the difference between measuring how many of those using an antibiotic properly are cured of their infection, and measuring how many of those sent home with a prescription for the antibiotic are cured. If many of those prescribed the antibiotic don't actually use it or use it properly, it's hard to assess the antibiotic's efficacy under proper use--unless we have some way to determine who is using it properly and only count their cure-rate.

Dodes actually invokes the antibiotic analogy at one point in the Salon piece, in response to AA's claims that those who really work the steps are very successful at remaining sober. Dodes takes this to imply that if AA doesn't work for you, the fault is yours. He finds this an irresponsible assertion that would be dismissed out of hand were it made with respect to an antibiotic or new chemotherapy treatment. "In professional medicine, if a treatment doesn’t work, it’s the treatment that must be scrutinized, not the patient."

Here, Dodes has failed to pay any attention to the crucial distinction made above. In professional medicine, if prescribing a course of treatment isn't working, good doctors would first ask why. Are the patients actually following through on the treatment plan as prescribed? If not, then that may be the problem. In other words, in professional medicine you do need to "scrutinize" the patient: Is the patient skipping doses or stopping treatment before the course of antibiotics has been completed? If you don't, you are in danger of dismissing a valuable treatment based on patient error.

Likewise, when it comes to AA, we need to focus on the success-rate of those who actually follow the 12 steps properly if we want to know how successful they are in helping alcoholics stay sober.

But doing that is going to be difficult in the case of AA for a couple of reasons. And this leads to my third concern about Dodes' Salon piece.

3. Treating Absence of Evidence as Evidence of Absence.

In the case of AA, the nature of the beast makes it very difficult, if not impossible, to come up with definitive conclusions concerning the success rate of those who actually work the program. One reason for this is that AA's "treatment plan" is very different from a course of antibiotics. When it comes to antibiotics, all you need to do is swallow the pills on schedule until they're gone. While your attitude might have some placebo effect, it's going to be relatively minor.

But working the steps in AA is more like following the practice guidelines of your music teacher: commitment and focus matter at least as much as plodding through all the assigned etudes and scales and practice pieces. Put simply, difficult-to-measure subjective phenomena play a much bigger role when it comes to making proper use of what AA offers than they do when it comes to making proper use of antibiotics. In AA, there's a difference between working the program and going through the motions, and it's a difference that matters a lot for one's chances of success. It's also a difference that's difficult to take into account in a study.

That difficulty is compounded by the "anonymous" feature of AA. Here's how one overview of studies into AA puts the problem:

Accurate reports about the success rates of 12-step programs like AA and NA are notoriously difficult to obtain. The few studies that have attempted to measure the effectiveness of the program have often been contradictory. Fiercely protective of their anonymity, AA forbids researchers from conducting clinical studies of its millions of members.

In the face of that commitment to anonymity, it is hard to do any kind of study that takes into account the distinction between the committed twelve-stepper and the guy who's going through the motions. Because if you want to take that distinction into account you're going to need inside information about each person in your study--the kind of qualitative information that humans use to determine who is really committed and who is just floating along. And I can't see how you could get that kind of information without violating the confidentiality of the AA model.

Finally, there's the problem of accounting for variation in understanding and approach among AA groups. Just as with religious communities, there are different ways of understanding the 12 steps, different attitudes towards the Big Book, and different attitudes towards alternative, non-AA approaches to sobriety. Some AA groups are similar to fundamentalist exclusivists, and some are more like religious progressives. With that kind of variability, does it even make sense to ask about the success of AA as a single, monolithic entity?

What does all of this mean? It means that it would be very difficult, given the limitations of the real world, to come up with a reliable scientific way to measure the success rate of committed participation in AA. That doesn't mean no measures can be taken. But it does mean that they will typically fail to account for all the important variables, may not measure the right things all the time, etc.

But Dodes seems to treat this limitation of science as if it were the failure of AA. In this respect, he reminds me of those among the New Atheists who, under the influence of logical positivism, dismiss as unbelievable any assertion that resists scientific testing.

In the Salon piece, Dodes notes the following:

In 2006, one of the most prestigious scientific research organizations in the world, the Cochrane Collaboration, conducted a review of the many studies conducted between 1966 and 2005 and reached a stunning conclusion: “No experimental studies unequivocally demonstrated the effectiveness of AA” in treating alcoholism.

This, of course, is the very conclusion one would expect given what I've said above--but Dodes presents the conclusion of the Cochrane Collaboration as if the failure to unequivocally demonstrate AA's effectiveness were reason to suppose that it's ineffective.

This is logical positivism on steroids: if it resists unequivocal proof in an experimental study, don't believe it--even if eighty years of human experience give reason to think it has considerable merit.

Let's be clear: there are lots of truths that we have yet to demonstrate are true. There are lots of truths that we will never be able to demonstrate are true. And there are some things that we have reason to think are true based on collective wisdom and a rich body of anecdotal evidence, but which we haven't been able to unequivocally demonstrate as true using experimental studies.

The rigor of science is wonderful when it can be applied--but there are some questions that fall in principle outside the scope of scientific inquiry and others where a scientific approach faces formidable practical hurdles. The question of AA's success rate falls into the latter category. And so, even if AA is the most successful addiction-recovery program around, we shouldn't be surprised if, to this point, "no experimental studies unequivocally demonstrated the effectiveness of AA."

We should, however, be very surprised when someone who quotes this conclusion then asserts--in the same paragraph--that AA has a 5-10% success rate, as if some experimental studies have been able to unequivocally demonstrate the ineffectiveness of AA.

Rather than quoting dubious numbers out of context, what we should do is wrestle with what to do when we have to make a decision but don't have precise "scientific" answers. Looking into better ways to find those answers makes sense--but in the meantime, should you check out AA if you or a loved one is struggling with addiction?

The Dodes, at least in their Salon piece, don't offer much reason not to. The best they offer are some anecdotes, dubious statistics out of context, and the fallacious insinuation that absence of clear scientific evidence of success is evidence of failure. While they are surely right that therapy can often provide useful insights in the pursuit of sobriety, there is nothing preventing those who attend AA from pursuing therapy as well--unless they are prevented by financial constraints, in which case it's a good thing that they at least have AA, which is free.

In the next post in this series (which I may not get to for a few days), I will look at how the Dodes walk through the 12-steps, focusing on how their understanding of those steps seems to be distorted by (perhaps unconscious) anti-religious bias and a tendency to read the steps through the lens of fundamentalist theology.