SD2's Davis Demographic study was broken from the start

2012-03-02T00:15:00Z 2014-08-25T16:16:20Z SD2's Davis Demographic study was broken from the startBy ROB ROGERS rrogers@billingsgazette.com The Billings Gazette

The problem was shaped like a hockey stick.

Enrollment forecasts calculated by Davis Demographics & Planning — included in a study the company presented to Billings School District 2 last summer — showed years of flat student growth followed by a dramatic upward spike in 2010 that continued to rise into perpetuity.

Laid out as a bar graph, the projection looks like a hockey stick.

For analysts, that shape is a big red flag that shows the data may be flawed, said Andy Wildenberg, an associate professor of computer science at Rocky Mountain College.

Wildenberg, a parent in the district, became involved after he learned of some of the issues with the study and attended a board meeting two weeks ago.

"When Davis first came up with their projections, Richard saw there was something wrong," Wildenberg said.

He was talking about Richard Dews, a business teacher in SD2's adult education department and a computer systems expert.

The shortcomings in the study that Dews discovered and the subsequent efforts by SD2 staff members, board trustees and Davis Demographics to address those problems exposed troubling fault lines all through the district.

As Dews tells it, the troubles began two years ago in Thomas Harper's office. At the time, Harper was SD2's chief financial officer and was building the district's budget forecast and needed enrollment projections. 

The state allocates money according to how many students a district has enrolled. The better a district can plan for enrollment, the more accurate its budget forecasts will be.

Harper worked off a spreadsheet with enrollment projections supplied by the district. According to Dews, one afternoon the spreadsheet program that Harper was using failed.

Harper called Dews' boss, Woody Jensen, who is director of adult education, and asked Jensen if he knew someone who could help fix the spreadsheet.

Jensen recommended Dews, whom Jensen describes as brilliant.

Dews said that as he worked on the spreadsheet, he immediately saw that what Harper wanted to do couldn't be done with the software he was using.

He discovered also that the reason Harper's spreadsheet wasn't working was because it had been corrupted with enrollment projection data pasted onto the document.

The data had come from SD2's executive directors office, and Dews believes it's the same data the district gave Davis Demographics, and that that's what ultimately led to the hockey stick graph.

The district's data showed extensive growth on the city's West End.

The problem, Dews said, is that when Davis compiled its study, it used those numbers and created an average on which it then based its findings rather than doing its own research. 

That specifically is what causes a hockey stick-shaped graph, Dews said.

"To me, it looked like it was an end run," Dews said, a way for some district officials to justify building a school.

Reached last week by telephone, Greg Davis, president of Davis Demographics & Planning, declined to comment.

Dews wanted to take raw numbers and create data sets without averages that could then be used to test the predictions Davis had come up with.  

Wildenberg said that was the right move. The ability to test findings and cross-verify conclusions is what makes a study solid and trustworthy, he said.

"The core numbers are numbers that they just eyeballed," Wildenberg said of Davis. "They didn't check their answers."

Kathy Olson, who was director of elementary education at the time, said the district had no sophisticated way to create enrollment projections.

She concedes there were mistakes in the data given to Davis but said that it wasn't intentional, nor was it an effort to justify building a new school.

The intent was to be able to answer enrollment questions, she said.

A year or two before the study was commissioned in 2010, SD2 had purchased software from Davis in an effort to better map where students in the district lived and how best to redraw school boundaries with the information.

"It went far beyond that after I left," said Jack Copps, interim superintendent.

Copps retired in June 2010 and was replaced by Keith Beeman. When the board commissioned Davis for the study in June 2010, the scope expanded from redrawing boundaries to including enrollment projections.

Dews didn't know that the district had software from Davis when he came in to help Harper. So he created his own program using years' worth of birth rates and mobility data to create his enrollment projections.

When Dews finally learned that the district had the Davis software, he couldn't get permission to use it, so he downloaded his own copy as a 30-day free trial and, working with SD2 staffer Daniela Walsh, crafted his own enrollment forecasts.

Dews' forecast showed some growth, but it was far less than what the Davis study had shown in the hockey stick graph.

At that point, district leaders met to decide how best to move forward. When they pointed out the errors to Davis that had been discovered by Dews, the Davis demographers volunteered to just cut their growth projections in half.

Dews said he was taken aback by the response. He believed that Davis should have created a new set of projections using the raw data, as he had done.

As a last-ditch effort to salvage the study, the district agreed to the change proposed by Davis, and Dews was cut out of the project. 

At the end of the school year, in June 2011, Beeman ordered that Dews' contract not be renewed by the district. The study he and Walsh had created was buried.

"The only message that I got was, 'You will not renew his contract,'" Jensen said.

It bothered Jensen. He really liked Dews.

"I had no bones with him, loved his work," Jensen said. "I thought he was highly productive."

In the meantime, Beeman asked Barbara Bryan, then the board chairwoman, to rewrite the study in an effort to salvage what was left.

Bryan is a technical writer by profession, and the belief was she could do the rewrite and save the district some money. By this point, the study's cost had ballooned from $40,000 to nearly $60,000. The cost of the software purchased a few years earlier brought the total the district had paid to Davis to about $100,000.

Beeman and Bryan never informed the rest of the board of the troubles with the study or that Bryan had significantly rewritten portions of the study. In her rewrite, she jettisoned recommendations related to the significant growth projected on the West End and added tables to illustrate the data.

"There were a lot of us who didn't know what the right hand or the left hand was doing," said board Chairwoman Teresa Stroebe.

That led to accusations earlier this month that Bryan had purposefully misled the board and manipulated the data. In response, Bryan, who resigned in October, released to the board all of her correspondence with Davis and Beeman regarding the study and their efforts to salvage it. 

The correspondence showed that she and Beeman had learned from Dews about the faults in the Davis study and that she had worked with Davis to find a way to correct it. In the end, the two decided she should tackle the rewriting.

For Dews, the handling of the study shows SD2's penchant for making decisions and then gathering data to justify those decisions. He said he went before the board last week to explain his role in the study with the hope of effecting some change in the district.

"If you don't get these things out, you're likely in peril to repeat" these mistakes, Dews said.

On Monday night, trustees directed Copps to work with Dews and Wildenberg to figure out what can be used from the study and report back to the board in March.

Trustees also decided to use a local firm from now on as the district continues to build enrollment forecasts. The district will continue to use the Davis software.

Still, the experience has left some trustees frustrated.

"To me, it's like two years of waste," said Trustee Lindy Graves.

Contact Rob Rogers at rrogers@billingsgazette.com or 657-1231.

Copyright 2014 The Billings Gazette. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.

Follow The Billings Gazette

Popular Stories

Get weekly ads via e-mail

Deals & Offers

Featured Businesses