Supporters of the Montana Meth Project have loudly proclaimed the success of the multimillion-dollar advertising campaign.
Tom Siebel, the billionaire software mogul who conceived of the project and put up millions of dollars to launch it, told a U.S. Senate committee in 2007 that results of the Meth Project in Montana "have been more significant than any drug prevention program in history."
Former Attorney General Mike McGrath, now chief justice of the Montana Supreme Court and an original member of the Meth Project board, said the ad campaign "is very simply changing the nature of crime control in Montana."
Created in 2005 to reduce the use of the devastating drug methamphetamine, the Montana Meth Project has won multiple advertising awards, has been lauded by the White House Office of National Drug Control Policy and has expanded from Montana into six other states.
In the midst of what appears to be universal praise, though, some people have questioned how successful the Meth Project really has been and whether it deserves the millions of dollars in taxpayer support it has been receiving in Montana and elsewhere.
Gov. Brian Schweitzer, who used his line-item veto in May to whittle away half of a $1 million legislative appropriation for the Montana Meth Project, said he has heard from many people involved in addiction treatment who are "a little dubious" about the project's claims of success. And though he calls himself a supporter, Schweitzer said he has heard from experts in the field that meth use was declining in Montana even before the project began, making the impact hard to gauge.
Mona Sumner, chief operating officer for the Rimrock Foundation, a nationally known addiction treatment business based in Billings, said she has never trusted figures cited by the Montana Meth Project.
"I think this playing games with the numbers - I'm fed up with it. They've been doing it since the outset," she said.
The project's use of statistics was also questioned by Russ Lord, who was the researcher for the Billings component of a multiple-site clinical trial of meth treatment models, which was sponsored by the federal government between 1999 and 2004.
Lord, a professor of health and human performance at Montana State University Billings, said the Montana Meth Project has made big claims based on scraps of data and has not bothered to do the kind of analysis that would give its findings validity.
"If we're going to be putting health care dollars and those kinds of resources to something, it really needs to have the solid, empirical research," he said.
Joan Cassidy, chief of the Chemical Dependency Bureau for the state health department, said the severe effects of meth use shown in some of the project's ads "did pose some concerns."
In some cases, she said, "the severity of the media didn't match what the adolescents perceived." Cassidy noted that there has been a decrease in the self-reported use of meth by Montana teens, but she asked: "Was it connected to the Meth Project? I don't say yes it was or no it wasn't."
The most detailed criticism of the Montana Meth Project came from halfway around the world.
David Erceg-Hurn, a doctoral student in clinical psychology at the University of Western Australia, reviewed the effectiveness of the Meth Project's advertising campaign in the December issue of Prevention Science. He dissected the project's claims, accusing the organization of suppressing negative findings, cherry-picking data and taking unwarranted credit for favorable results.
He flatly concluded that "there is no compelling evidence that teenage meth use has declined as a result of the ads."
A blizzard of ads
Siebel, a part-time resident of Montana, founded the Montana Meth Project in 2005, using his own millions to launch a blizzard of in-your-face TV, radio, print and billboard ads based on the project's slogan, "Not even once."
The ads pulled no punches, starkly portraying rape, prostitution, robbery and beatings, all as a result of getting hooked on meth. Other ads showed meth users with rotting teeth, gaunt, scarred bodies and hollow eyes.
Then came the surveys gauging the impact of the ad blitz. Six months into the project, in March 2006, the Meth Project released a survey showing that there was significantly more awareness of the dangers of meth, more discussions about the drug between parents and children and stronger disapproval of meth use.
A year and a half into the project, in January 2007, McGrath, then the attorney general, issued his own report on "Methamphetamine in Montana." It concluded that the Montana Meth Project's educational campaign, "combined with other enforcement and prevention efforts," was having "dramatically positive results."
The study pointed to changing attitudes, apparent major declines in meth use and substantial decreases in meth-related crime.
"With the continued educational and prevention efforts of the Montana Meth Project," the report said, "it is logical to conclude that meth use will continue to decline."
Perhaps the most sweeping claims of the project's success were made by Seibel himself when he appeared before the Senate Finance Committee, chaired by Sen. Max Baucus, D-Mont., on Sept. 18, 2007. Siebel told the panel that in two years, the Meth Project blanketed Montana with 45,000 television ads, 35,000 radio ads, 10,000 print ads and 1,000 billboards.
He then spoke of the project's "market results." Among them, he said, was that Montana dropped in one year from fifth in the nation to 39th for levels of meth abuse, according to the Office of National Drug Control Policy, and that adult meth use declined by 70 percent, meth-related crimes were down 53 percent and teen meth use "has declined very dramatically."
Siebel asked Congress to consider funding an expansion of the project into other states. He suggested that $40 million a year spent in the 10 states represented by senators on the committee "could achieve dramatic reductions in teen methamphetamine use."
Critics of the Montana Meth Project generally concede that the organization has produced powerful advertising and that its message has reached virtually everyone in Montana. They also concede that attitudes have been changed, and that awareness of the dangers of meth has increased.
What they question is whether the project is meeting its fundamental goal of significantly reducing meth use.
The Montana Meth Project, in numerous press releases and promotional materials, has said that teen meth use in Montana went down 45 percent between 2005 and 2007. That figure comes from the Youth Risk Behavior Survey, given every two years by the Centers for Disease Control and Prevention to high school students across the United States, including Montana.
The YRBS surveys showed a decline, in the percentage of Montana teens who reported ever having used meth, from 8.3 percent in 2005 to 4.5 in 2007, a drop of 45 percent. Just last week, the 2009 YRBS results were released, showing a further decline, to 3.1 percent.
Erceg-Hurn, however, said the Meth Project consistently failed to mention that the YRBS also showed a 38 percent decrease in meth use among Montana teens between 1999 and 2005 - which might suggest that factors other than the Meth Project were driving a steady decrease in the drug's popularity.
Also, national YRBS results show that meth use among high school students across the country dropped 55 percent between 2001 and 2007, from 9.8 percent to 4.4 percent.
To make a connection between the survey numbers and the effectiveness of the Meth Project, Erceg-Hurn said, the project should have used standard experimental testing to gauge the impact of the ads, rather than relying on the use of focus groups.
Sumner said many people in the treatment community were predicting a decline in meth use in Montana even before the Montana Meth Project was launched. Drug abuse traditionally spreads across the country in epidemics or waves, she said, and there were indications that meth use was peaking many years ago, replaced by steep increases in the use of opiates.
Erceg-Hurn said the YRBS data, as presented by the Montana Meth Project, is possibly misleading in another way. Expressing changes in terms of relative percentages rather than absolute percentages "can make small changes in meth use sound like massive changes," he said. In absolute percentages, the YRBS survey showed a drop in teen meth use of 3.3 percent between 2001 and 2003, two years before the launch of the Montana Meth Project. The decline was 3.7 percent between 2005 and 2007, only slightly larger than the earlier decline, and yet that is the number the project chooses to highlight as a noteworthy achievement, he said.
Another of Erceg-Hurn's key points is that the project has ignored a surprising finding in its own surveys - that 98 percent of Montana teens strongly disapproved of meth use in 2005, before the ads began running, and that the number dropped to 91 percent by 2008.
"This change is in the opposite direction to what the Meth Project has been trying to produce," Erceg-Hurn said.
Geoff Feinberg, vice president of GfK Roper, the New York polling and research firm that conducted Meth Project surveys in 2007 and 2008, argued that the small drop shown on that survey question is far outweighed by other survey responses showing steep increases in the perceived risk of the specific dangers of meth use.
He and Peg Shea, the first director of the Montana Meth Project and now a volunteer consultant to the organization, said that in response to numerous statements - meth makes you more intelligent, helps you study, helps you lose weight, etc. - teens clearly showed they were becoming aware of the dangers of the drug.
"I don't believe a few small findings undercut the much larger body of data demonstrating positive trends," Feinberg said.
A 'flawed' premise
Erceg-Hurn said the fact remains that 98 percent of Montana teens disapproved of meth use in 2005, suggesting that the premise of the Montana Meth Project - that teens needed to learn of the dangers of the drug - was "seriously flawed." Also, he said, the teen respondents' "global, overall perceptions of risk" are more likely to drive behavior than beliefs about specific risks.
A person might know that drinking a case of beer a day is bad for your health, Erceg-Hurn said, "but this doesn't mean that you think drinking beer, per se, is bad for you, and that you will abstain from it."
Lord, the MSU Billings professor, made a similar point. He said scare campaigns aimed at smoking, drunken driving, meth use and other problems are based on the assumption "that people engage in these behaviors as perfectly rational decision-makers."
"But humans aren't perfectly rational," he said.
In general terms, Lord said the Montana Meth Project should heed Erceg-Hurn's study because it was published in a peer-reviewed journal published by the Society for Prevention Research. It may well have more validity than the Meth Project's own commissioned findings, he said.
"I think his study is very instructive, because when someone gets something into a magazine like Prevention Science, it's been vetted in a way that in-house data hasn't been vetted," Lord said.
Erceg-Hurn said the Montana Meth Project doesn't want to hear about flaws in its methods.
"They've conducted poor-quality research, and used this to portray the ads as an unqualified success," he said in an e-mail to The Gazette. "When they've been criticized or encouraged to alter their approach by people like me, they've reacted angrily and aggressively rather than taking a hard look at what they've been doing and trying to improve it."
That was a conclusion drawn as well by an Arizona state senator, John Huppenthal, who sponsored a bill to appropriate $4 million to bring the project to Arizona in 2006. He later withdrew his support for his own bill, though the state did subsequently establish the Arizona Meth Project.
Huppenthal said he changed his mind after looking into Montana Meth Project advertising and concluding that the "appearance of exaggeration" was too high for many of the ads.
"You can be dramatic, but you have to be spot-on as far as correct description goes," he said. "Many of the ads appeared to have problems." He also noticed some of the same things as Erceg-Hurn, including the Meth Project's own findings that the number of teens who recognized the risks of meth use had actually dropped after the campaign started.
Huppenthal called Siebel to talk about his concerns.
"It wasn't a really pleasant conversation," he said. "He dismissed the survey stuff. He wasn't a guy who wanted conversation. … He was a guy who was certain of where he was going." Siebel did not respond to interview requests from The Gazette.
Schweitzer was unhappy with the Meth Project's reaction to Erceg-Hurn's criticism.
"I'm a scientist, and I understand statistics and I understand studies," he said. Erceg-Hurn examined the Meth Project's own reports and saw that "they weren't laying their own data out," Schweitzer said, and it should have responded to his findings.
"I saw some quotes from Montana Meth Project people saying, 'he doesn't know what he's talking about because he's not from Montana.' That was disappointing," Schweitzer added.
'Dramatic' results seen
McGrath, who is still on the Montana Meth Project board, said he didn't read Erceg-Hurn's study. "I didn't really feel I needed to," he said, calling Erceg-Hurn "just a guy from Australia."
McGrath said the YRBS figures on teen meth use "were incredibly dramatic" and backed up the findings of his office when he was attorney general. The 2008 edition of the AG's meth report documented steep declines in the number of meth addicts admitted to state treatment programs - a 32 percent drop between 2005 and 2007 - and even sharper drops in the number of crimes in which meth was involved - a 62 percent drop in the same period.
In the area of human services, that report showed, meth was involved in 53 percent of out-of-home placement of children who were abused or neglected in 2005. By 2007, that number had fallen to just 26.4 percent.
McGrath also cited a 72 percent drop in the number of workers who tested positive for meth in workplace drug testing. That was the figure seized on by Siebel when he credited the Montana Meth Project for a 70 percent drop in the number of adult meth users.
Even McGrath's own study, however, included this caution: "It should be noted that the number of workers tested is relatively small, so it's possible that the trends may be less dramatic than they appear."
Beyond the statistics, McGrath said, there was the evidence that people in law enforcement and the judiciary were seeing every day.
Methamphetamine became the focus of a multimillion-dollar prevention campaign because it was a uniquely dangerous drug, he said. During his 30-year career in law enforcement, "we hadn't seen the dramatic effects caused by meth that we saw with other drugs."
There were more crimes associated with meth than there had been with other drugs, as well as more violence - and violence of a particularly gruesome kind that was new to Montana. McGrath said he has always considered the Montana Meth Project just one aspect of the war on meth, which includes treatment programs and law enforcement.
The 2005 Legislature made big strides by banning over-the-counter sales of a key ingredient in meth, which drastically reduced the problem of homegrown meth labs, and it also established treatment programs, drug courts and other responses to the meth epidemic, McGrath said. Those steps were important, but they needed the addition of a prevention program, he said, and the Montana Meth Project more than filled the void.
As a board member, McGrath said, he is not concerned by charges that Siebel is overselling the success of the project.
"Tom's proud of the program," he said, and for good reason: "It's dramatic how much has been done and how things have changed. … This thing is unparalleled."
A 'powerful combination'
Billings Gazette Publisher Mike Gulledge, chairman of the Montana Meth Project board, said he hadn't read Erceg-Hurn's report, either, although he said he was open to having a board discussion of it.
Like many other supporters of the project, Gulledge said one of its key accomplishments has been a documented increase in the number of parents who now talk to their children about meth. One thing that has made the project so successful has been the "powerful combination" of Siebel's vision and knowledge of business with Shea's expertise in prevention, he said.
Before becoming director of the Montana Meth Project, Shea was executive director of Western Montana Addiction Services, with nearly 30 years' experience in the field.
Asked whether there was any conflict of interest in his serving on the board of a campaign that spends millions on advertising, including newspaper ads, Gulledge said, "I am not involved in any decisions on where to place advertising." Also, he said, the Montana Meth Project has always been primarily a broadcasting campaign.
Cascade County Attorney John Parker, a three-term member of the Montana House of Representatives, was only recently named to the board, but he has been involved in the fight against meth as a deputy county attorney and legislator. And though he hadn't read all of Erceg-Hurn's report, Parker said, "the things I've read seem to ignore the nature of our problem locally in Montana, as well as the need to deter people from meth use."
There is enough evidence that the project is having an impact to warrant its continuation, he said, and he knew from his own experience that the problem was too big to postpone the search for a solution.
"The risk to people's lives isn't going to wait for the perfect data," he said.
He said the 45 percent reduction tracked by the YRBS study is important, as are the multiple findings that kids are learning the dangers of meth use.
"I don't know how you can quantify the far-flung benefits that are going to result from keeping kids off meth. I think it's more important that we try and demonstrate what success we can, rather than do nothing," he said.
A skeptic no more
Shea criticized Erceg-Hurn for suggesting that the Meth Project should have done studies involving control groups. That would have meant intentionally failing to expose some Montana children to the advertising, she said, and Siebel and other members of the project wouldn't have been willing to do that. Their goal was to reach as many kids as possible.
Shea said she was skeptical at first, having come from the treatment industry, but she has long since become a believer in the project's methods.
"What I was doing wasn't working. It wasn't changing behavior. We weren't cutting it with kids," she said. The Montana Meth Project listened to teens and parents and public relations and treatment experts and changed its ads accordingly.
"I truly believe that the campaign raised awareness," she said.
Sumner, of the Rimrock Foundation, said her mistrust of Meth Project figures stems partly from what she saw in an HBO documentary on methamphetamine in Montana, which received $500,000 in funding from the Montana Meth Project. Sumner's recollection was that the documentary said 98 percent of meth addicts admitted to Montana treatment centers had relapsed.
Shea said the language that "flashed across the screen" merely said 93 percent of addicts, not 98 percent, "in traditional treatment relapse into meth abuse."
Sumner said that in any case, she and other members of a panel who were present when the documentary was screened in Billings came away with the idea the idea that the figure applied to Montana. And she knew, she said, "that data is not collected in Montana in a way that would permit a statement like that to be made."
"It's just an example of the kind of over-generalization and, I think, attention-seeking, that has characterized this campaign from its beginnings," Sumner said.
Shea said the figure was based on the clinical trials sponsored by the federal government and was considered legitimate because it was the only statistic on relapse rates available at the time.
Bill Slaughter, appointed director of the Montana Meth Project in May, asked Kathy Sabol, a preventive health specialist, to comment on the 93 percent figure because she coordinated the Billings component of the federal study.
Sabol said in an e-mail that she considered the figure meaningless "because there is no definition of terms." She also said it was irresponsible to cite any relapse or recovery rates because "there is no way to responsibly link treatment/prevention/law enforcement efforts (or anything else) to relapse/recovery rates."
Beyond the debate over specific accomplishments, there is some dispute about the premise of the anti-meth campaign, the idea that kids can be persuaded to avoid meth by graphic advertising.
Even before he read Erceg-Hurn's study, Schweitzer said, there were things that "made my antennas go up." One was that teenagers, including his own children, were telling him that that claims made in the Meth Project ads "were probably not completely true."
The mantra was "not even once," but teens knew of peers who had experimented with meth without losing their lives or their teeth, and they quit using on their own.
In Schweitzer's view, the advertising campaign worked, "but not necessarily for the target audience. … It worked for their parents. It scared the heck out of us and we sat down and started talking to them about meth." That's one reason, despite his qualms, that he thinks the project was worthy of continued state funding, Schweitzer said.
Lord echoed that point, saying the "scare-tactic strategy" has mainly had an effect on parents.
"We're vulnerable to things that play on our fears," he said. "We don't know if it works, but we're so scared we'll spend money on anything."