In the competition for $4 billion in grants, states have made their best pitches, a secret jury has debated and scored their applications鈥攁nd now U.S. Secretary of 91制片厂视频 Arne Duncan must decide who鈥檚 good enough to make it to the final round.
Mr. Duncan鈥檚 announcement of the Race to the Top Fund finalists, which is expected as early as this week, caps a dash by the Department of 91制片厂视频 to recruit, vet, and train peer reviewers who wield tremendous power in determining who will win this high-stakes education reform competition.
In this first round of competition, all but 10 states applied for awards financed by the economic-stimulus package that will range from an estimated $350 million to $700 million鈥攂adly needed money that would help cash-strapped states.
Once the first-round awards are made in April, the department will publicize the scores and comments for each applicant from the 49 peer reviewers who have been given responsibility for judging 30 criteria that measure a state鈥檚 commitment to improving teacher quality, low-performing schools, data systems, and academic standards.
Despite all that influence, the ultimate power rests with Mr. Duncan, who will have the final say in awarding those cash prizes. But if the secretary deviates from the highest scores in awarding the Race to the Top grants, department officials have said he pledges to justify his reasons.
Each of 49 peer reviewers evaluated four or five state applications, which were randomly assigned. Comments and scores were due February 8, 2010.
IMAGES: iStockphoto.com/auras
SOURCES: U.S. Department of 91制片厂视频; 91制片厂视频 Week
Mr. Duncan also has sole discretion in choosing how many finalists will come to Washington the week of March 15 to make presentations to the peer reviewers.
The purpose of the presentations, said Joanne Weiss, the department鈥檚 Race to the Top director, is not only to verify and clarify what鈥檚 in the applications, but also to ensure that those who must implement the plan know and understand what鈥檚 been promised.
鈥淭he real issue is, we鈥檙e giving away hundreds of millions of dollars, and we don鈥檛 want to just rely on what you wrote on a piece of paper,鈥� Ms. Weiss said. 鈥淲e want to look you in the eye.鈥�
Looking for Conflicts
In August, the 91制片厂视频 Department put out a call for peer reviewers to judge the applications and drew 1,500 applicants. They will pull from this pool for the second round of the competition, which starts in June when applications are due and ends in September, when the final dollar is awarded.
Vetting the list for potential conflicts of interest was no easy task. Any nominee who works for a state or district was automatically eliminated. So was anyone who helped out in any way on a state鈥檚 application, or who has a financial interest in the outcome of the competition.
Lawyers and ethics officers from the department did most of that work, although four career and four political staff members also reviewed the applicants to make sure they had expertise in the four school improvement areas emphasized in Race to the Top.
The department also hired a contractor to do a public-records search鈥攅ven a Google search鈥攊nto nominees to uncover information not previously disclosed. The department was also on the lookout for anyone who might have 鈥渟ignificant identification with a specific pedagogical or philosophical viewpoint that might give an unfair advantage or disadvantage to an applicant,鈥� according to department documents.
鈥淚t鈥檚 wasn鈥檛 like, 鈥楧o you have a belief?鈥� 鈥� Ms. Weiss said. 鈥淚t was more about, 鈥楧o you have an agenda?鈥� 鈥�
Of the 49 peer reviewers, the department granted 12 waivers for 鈥渋ndirect鈥� conflicts of interest, most commonly that a reviewer had worked for a state or school district within the past 12 months.
The reviewers receive $5,000 for what amounts to hundreds of hours of work that will span at least two months and involve tedious reading of applications that can exceed 1,000 pages each, counting appendices.
Though the department released basic demographic information on the reviewers鈥攎ost, for example, have doctoral degrees and at least a dozen served on a state board of education or as a local or state school superintendent鈥攆ederal officials have caught criticism for keeping their identities secret.
One reason the reviewers鈥� names should be made public is because of Mr. Duncan鈥檚 pledge to run a transparent competition, said Frederick M. Hess, the director of education policy studies for the American Enterprise Institute and hosted by 91制片厂视频 Week. 鈥淲e鈥檙e talking about an extraordinary sum of cash in play, and that鈥檚 why the identities of the judges are so important.鈥�
In fact, the peer reviewers had to sign confidentiality agreements declaring they won鈥檛 disclose their identities or talk to the press鈥攊n effect, a gag order that applies until the department lifts it. Department officials have said they will make the identities public once the winners are announced, although it does not intend to reveal which reviewers scored which application.
Process So Far
In the course of the judging process, each peer reviewer received four or five applications to score; they were not allowed to judge applications from the states where they live. Then, the reviewers came together in Washington to form five-member panels to discuss and debate each application.
Since the reviewers rotate among panels, forming a unique one for each state, the department hopes to smooth out any scoring variances that might occur with different groups reviewing different applications.
Peer reviewers could refine their scores and comments based on their panel discussions, and the five scores were averaged to arrive at a final score. Ms. Weiss said any outliers whose grades were particularly low or high compared with the other panelists were not pressured to come to consensus.
Throughout the process, peer reviewers were instructed to do no independent research and to rely entirely on what鈥檚 written in the application.
Checks and balances were built into the system to fact-check the states鈥� applications. Ms. Weiss said, for example, that making the applications public, including the scores, comments, and video from the finalists鈥� presentations in Washington, will help ensure accountability.
Although the department has established a detailed, 500-point scoring guide, peer reviewers鈥� judgment will still ultimately come into play. For example, a peer reviewer will have to decide how to weigh an overly ambitious application, with a price tag that far exceeds the department鈥檚 nonbinding award estimates, against a less ambitious plan that stays within those estimates. 鈥淚t鈥檚 entirely the peer reviewers鈥� judgment,鈥� Ms. Weiss said. 鈥淵ou will see the reviewers made comments such as, 鈥楾his is underfunded. Or this is overfunded.鈥� 鈥�
Wild Card
Such judgment is the wild card in the competition, said Michael J. Petrilli, a vice president of the Thomas B. Fordham Institute, a think tank in Washington.
鈥淲hat this comes down to is people. And invariably there are going to be judgment calls being made even as prescriptive as the criteria are,鈥� said Mr. Petrilli, who worked with some peer-reviewed grant programs while at the department under President George W. Bush. 鈥淭hat鈥檚 when policy leanings are going to come into play.鈥�
Still, the peer-review system was probably the best choice for the department in judging Race to the Top, said Mark Schneider, the vice president for education, human development, and the workforce program at the American Institutes for Research.
鈥淧eer review is probably the best system we鈥檝e come up with for judging products on merit,鈥� said Mr. Schneider, a former commissioner of the National Center for 91制片厂视频 Statistics. 鈥淚f it鈥檚 done right, many people, especially losers, are unhappy but it does have a justification and a long history of support.鈥�
There鈥檚 been less clarity to how Secretary Duncan and his staff will decide how many finalists will be selected鈥攁nd how many awards will be given out.
The department has said that the applications will be ranked in order by score, with the top scorers getting money, until the money runs out. And Mr. Duncan has promised to be tough in setting a high bar for states to win.
Ms. Weiss said she couldn鈥檛 talk specifically about the process the department would use in determining the number of finalists or awards. But factors will include whether there are 鈥渘atural breaks鈥� that separate the superior applications from the not-so-good鈥攁nd how much money is involved in awarding the grants.
One example: If a few big states with large grants are top scorers, there might be fewer finalists and fewer winners in the first round because the department has promised to leave a sizable pot of money for Phase 2.
And what will the losers get?
The same feedback as the winners, an average of seven to 10 pages of comments from each reviewer, which will serve as a roadmap for Phase 2.
