A number of services have cropped up over the years to help schools answer a question that, in an age of information overload, scarce resources, and new technology, is becoming central to how they move forward: What works?
Whether it鈥檚 aggregating the most trustworthy studies on education or culling user reviews on products, as Yelp does with restaurants, none of these available services, both privately and publicly financed, has gained national scale. But a new proposal from two economists is perhaps the most ambitious鈥攐r, if you ask some of its potential users, flawed鈥攁ttempt at creating a Consumer Reports for education technology to address the lack of independent evidence available on such products鈥� efficacy.
The proposal is called Edu Star, a technology tool that would allow schools to conduct rapid, randomized evaluations of education products, collect and analyze the results, and publish the data to the public. By doing so, schools would make better-informed purchasing decisions, and entrepreneurs would have evidence that their products worked, the economists hope.
鈥淣ot surprisingly, when no one knows what works, schools are unlikely to buy, and innovators are unlikely to create,鈥� the economists, Aaron Chatterji, a professor at Duke University, and Benjamin Jones, a professor at Northwestern University, write in sponsored by the , an economic-policy initiative of the Brookings Institution. The paper at a Sept. 27 forum hosted by the Washington think tank.
A new proposal from two economists would create a Web-based system to rapidly conduct randomized trials among students on the efficacy of education technology products.
SOURCE: The Hamilton Project, 鈥淗arnessing Technology to Improve K-12 91制片厂视频鈥�
Innovate Faster?
Mr. Chatterji and Mr. Jones came up with the idea while both were on the White House Council of Economic Advisers two years ago. Edu Star is a convergence of three factors they believe drive much of the economy鈥檚 health: education, innovation, and 鈥渂uilding things,鈥� as Mr. Jones put it in an interview.
鈥淚f there is a weak link in this interconnected chain, it really seems to be K-12 education,鈥� he said.
K-12 education accounts for just 0.2 percent of the research and development expenditures in the United States, according to a 2010 report by the President鈥檚 Council of Advisers on Science and Technology, despite $600 billion in total annual K-12 expenditures. A low-cost way to conduct randomized trials could help education technology innovate at the same speed as other fields, where products are rolled out and tested at a much faster rate, Mr. Jones said.
To understand how Edu Star鈥檚 trials would work, Mr. Chatterji and Mr. Jones suggest thinking of Google鈥檚 rapid product development rather than, say, the U.S. Food and Drug Administration鈥檚 trial studies. 91制片厂视频 technology companies would offer their products to be tested, and schools would agree to test the products. After logging in to Edu Star, students would take a pre- and post-assessment on a skill tied to the Common Core State Standards.
In between, students would be assigned exercises in either one of two products built to teach that skill or a placebo product. All of the results would be recorded in Edu Star and reported on its website.
Schools would have an idea of whether a product improves academic outcomes, proponents of the plan say. Entrepreneurs without major sales forces could shift focus from wooing district leaders to sign contracts to proving a product鈥檚 value to its end user, the teacher, they say.
鈥淚t鈥檚 not an enterprise sale, it鈥檚 the bottom-up approach,鈥� Eric Westendorf, the chief executive officer of LearnZillion, a Washington-based video-lesson-sharing tool, said at the recent forum. 鈥淭hat, working in collaboration with an Edu Star-type platform, is really exciting.鈥�
But early responses to the proposal suggest Edu Star takes a limited view of what鈥檚 required to evaluate educational products. Factors such as professional development, ease of implementation, and long-term academic performance are also important to purchasing decisions, said Mark Edwards, the superintendent of the 5,500-student Mooresville, N.C., school district.
鈥淚n my experience, there鈥檚 a level of complexity around using digital resources that would not align with a quick and efficient means of evaluation,鈥� he said.
Other Services Emerge
A commonly referenced barrier for companies in the education market is the country鈥檚 approximately 14,000 school districts, which essentially make up 14,000 distinct buyers. But many of those barriers are reduced because technology companies can enter the market cheaply.
The common-core standards, adopted by all but four states, are likely to both simplify the market for vendors and create confusion for school-level customers.
鈥淎lmost everything you see now says, 鈥榓ligned with common core.鈥� I don鈥檛 believe that,鈥� Dennis Van Roekel, the president of the 3.2 million-member National 91制片厂视频 Association, said at the Washington forum. 鈥淏ut everyone says so because it鈥檚 a nice marketing line.鈥�
Some services have cropped up to help make sense of the market, and most are geared toward teachers, indicative of their increasing role in the marketplace.
, a Silicon Valley startup news service that covers the education technology industry, offers selective reviews of digital learning tools from educators who have used them. , another Silicon Valley-based company, offers a directory, reviews, and videos of digital learning tools.
, a K-12 learning platform from CFY, or Computers for Youth, a New York City nonprofit organization, rates free digital educational content using reviews from teachers, students, and parents. The organization is in talks with Edu Star about potential partnerships, said CFY co-founder Elisabeth Stock. Using both anecdotal user reviews and empirical randomized trials would help give a fuller picture of the education technology marketplace, she said.
鈥淵ou can have a teacher who uses a product and has great results, but that doesn鈥檛 mean that teacher has tried that product against three other products,鈥� Ms. Stock said. (The Bill & Melinda Gates, Eli and Edythe Broad, and W.K. Kellogg foundations contributed a total of $7 million to CFY. The Gates Foundation provides grant support for 91制片厂视频 Week鈥榮 coverage of K-12 business and innovation.)
Another effort, the , funded by publishing associations and foundations, including Gates, aims to categorize the open learning materials available on the Web for easier search.
The , a website funded by the U.S. Department of 91制片厂视频, evaluates the credibility of education research. But there are few up-to-date items on education technology on the site and, overall, it鈥檚 been criticized for a lack of user-friendliness.
鈥楾he Right Answers鈥�
But would Edu Star itself work?
Right now, it is only a proposal, though it鈥檚 one that Mr. Chatterji and Mr. Jones intend to pursue, by raising $5 million in foundation funding and hiring a small staff. (Neither plans to work on Edu Star full time.)
The idea has drawn early critics, meanwhile.
Betsy Corcoran, EdSurge鈥檚 chief executive officer, doesn鈥檛 believe the Edu Star ratings would work for modern technology products constantly in flux. 鈥淚t won鈥檛 accurately capture the usefulness of young and emerging tools because those tools will be in transition a lot,鈥� she said.
Mr. Westenberg of LearnZillion said it is 鈥渟cary鈥� that a rating of an early version of his product could be published.
There is also criticism that the rapid assessments the proposal envisions aren鈥檛 the best way to judge a product, let alone student learning in general.
鈥淎h, educational research. Ah, test scores. Ah, common core. Ah, what a very limited definition of 鈥榣earning,鈥� 鈥� education technology blogger Audrey Watters.
Mr. Jones, the Northwestern economist, argues that because Edu Star would test students on the common standards, what was being assessed would have legitimacy and would provide an apples-to-apples way of judging disparate groups of students.
But in general, he acknowledged, there is still a lot to figure out, including how to provide incentives for participation for entrepreneurs and schools, and how to navigate the , which protects the confidentiality of students鈥� education data.
Mr. Jones also stressed that the proposal doesn鈥檛 aim to reinvent learning or the education marketplace; it鈥檚 just another tool for helping produce better decisionmaking.
鈥淭he real innovation isn鈥檛 Edu Star,鈥� he said. 鈥淚t鈥檚 not about knowing what the right answer is, it鈥檚 about creating a system where the right answers can emerge.鈥�
