Corrected: This article presented data on student test scores incorrectly. The study found that 4th grade students in a nationally representative sample of charter schools scored 4.2 points behind students in regular public schools on the National Assessment of 91制片厂视频al Progress in reading, and 4.7 points behind regular public-school students in mathematics, when various student characteristics were considered. Those score differences were measured on the NAEP achievement scale of zero to 500.
In discussing a new federal study on charter schools last week, the commissioner of statistics for the U.S. Department of 91制片厂视频 reiterated that his office should not be initiating analyses such as that one and a recent comparison of public and private schools, both of which he believes rely too much on subjective judgments.
The two studies鈥攚hich used data from the National Assessment of 91制片厂视频al Progress that were adjusted for student and school differences鈥攔eported generally favorable academic results for regular public schools when compared with private schools and public charter schools. The studies, though, did not track scores over time, and could not take into account students鈥� prior achievement.
Mark S. Schneider, the commissioner of the National Center for 91制片厂视频 Statistics, said of the private school study earlier this month that it was not proper for the NCES to have directed a report that went so far in making judgments about how to interpret raw school data.
He repeated such concerns during a conference call last week on the charter study, saying that both analyses, which were initiated under his predecessor, could lead outsiders to question the statistics center鈥檚 impartiality.
鈥淲e do not have plans to replicate this analysis using the 2005 or eventually the 2007 data,鈥� Mr. Schneider said of the charter study during the Aug. 22 call. 鈥淭hat task is best left to individual researchers, who should be competing with each other in the marketplace of ideas.鈥�
Read the related story,
The charter study, after taking into account a range of student and school characteristics such as race, family income, and other socioeconomic variables, found that charter schools on average trailed regular public schools in 4th grade achievement in both reading and mathematics. The study found the average reading score for the 150 charters sampled was 4.2 percentage points behind that for a much larger sample of regular public schools. For mathematics, charter schools were 4.7 percentage points behind. (Reanalysis of NAEP Scores Finds 91制片厂视频 Schools Lagging, this issue.)
The commissioner initially had voiced reservations about the NCES鈥� involvement in the private school study at an Aug. 4 meeting of the National Assessment Governing Board in McLean, Va., and elaborated on those comments in an interviewafterward. The governing board sets policy for the National Assessment of 91制片厂视频al Progress, or NAEP, which the NCES administers.
The report on public and private schools, which received wide attention when it was released last month, showed public school students generally performing as well as or better than their private school peers in reading and math, when researchers adjusted for race, socioeconomic status, and other characteristics. One exception was 8th grade reading, in which private school children did better than their public school peers. (鈥淧ublic Schools Fare Well Against Private Schools in Study,鈥� July 26, 2006.)
鈥淭his is not what we should be doing,鈥� Mr. Schneider said of the private school study in an interview with 91制片厂视频 Week. 鈥淚t鈥檚 one thing for [an academic researcher] to put out a study. That鈥檚 a totally different story than if NCES, as a statistical agency, is weighing in on that front.鈥�
The NCES is one of three research centers housed within the Institute of 91制片厂视频 Sciences, the 91制片厂视频 Department鈥檚 research arm as reorganized by Congress in 2002. Although the institute is a part of the department, it is supposed to conduct its work free of political influence.
The IES director can prepare and publish any research without the approval of the education secretary. While the heads of two of the institute鈥檚 research centers are appointed by the IES director, the NCES commissioner is appointed by the president. Mr. Schneider had no say in the NCES鈥� decision to conduct the private school and charter school studies. Those projects were begun under Robert Lerner, Mr. Schneider鈥檚 predecessor. Mr. Lerner could not be reached for comment.
Like Mr. Lerner, Mr. Schneider was appointed by President Bush; his term expires in June 2009.
A Departure for NCES
Henry I. Braun, a senior educational researcher for the 91制片厂视频al Testing Service, a nonprofit research and testing organization in Princeton, N.J., led both studies for the federal statistical agency.
Each of them relied on a technique known as hierarchical linear modeling, or HLM, which allows scores to be compared while simultaneously taking into consideration multiple characteristics of students and schools. This method allowed for a more detailed analysis than was used during a 2004 NCES comparison of charter and regular public schools.
But using the HLM technique requires researchers to choose which variables to consider鈥攕uch as race, gender, and various measures of socioeconomic status鈥攁s the ETS did on the study of public and private schools and the charter study. Although it is common for outside researchers to crunch federal data using the HLM method, the NCES has typically not engaged in such work, Mr. Braun and others said.
鈥淚 think it was clear from the beginning that this was something of a departure for NCES,鈥� said Mr. Braun of his private school study.
Peggy Carr, the statistical agency鈥檚 associate commissioner, said the NCES had made only 鈥渧ery limited鈥� use of the HLM technique in previous studies.
The ETS conducted the private and charter school studies under an existing, five-year contract with the NCES, Ms. Carr said. Both studies underwent peer review by outside experts, she said. Each study cost about $130,000.
While the NCES publishes such reports and provides general guidance to their authors鈥攕uch as the decision to use the HLM method鈥攖he research itself was conducted entirely by the ETS, and decisions about what variables to consider and other procedures were left to the contractor, Ms. Carr said.
IES Chief Concurs
Mr. Schneider emphasized that he was not questioning the merits of the NCES reports, which he described as high-quality efforts. But the commissioner said that in the future he would not support having the NCES produce a report in which researchers made similarly subjective judgments about which student characteristics to consider.
鈥淥ur job is to collect the data and get it out the door,鈥� Mr. Schneider said in an interview after the NAEP governing board meeting earlier this month. The problem, he said, is that in studies like the private school report, 鈥測ou鈥檙e always having to make choices about variables.鈥�
After the publication of the study on public and private schools, Paul E. Peterson, a professor of government at Harvard University, analyzed the data using different variables and found that private school students鈥� performance topped that of public school students. (鈥淒ata Reanalysis Finds Test-Score Edge for Private Schools,鈥� Aug. 9, 2006.)
The head of the Institute of 91制片厂视频 Sciences echoed Mr. Schneider鈥檚 opinion. In an e-mail, IES Director Grover J. 鈥淩uss鈥� Whitehurst said studies using hierarchical linear modeling and similar methods of analysis are 鈥渘ot a priority鈥� for the NCES, because they take time away from other projects, and private-sector researchers have the ability to conduct the same work. Mr. Whitehurst also noted that such interpretive reports almost always produce debates about the methodology used.
鈥淭here are typically several legitimate ways to carry out complex data analyses,鈥� Mr. Whitehurst wrote in the e-mail. 鈥淓xperts will differ on which approach is best and how the results should be interpreted. 鈥� Advocacy groups that don鈥檛 like the results will frequently assert that the analytic approach was biased. It is best for NCES, as a federal statistical agency, to stay out of that fray.鈥�
Sensitivity Cited
Gerald E. Sroufe, the director of government relations for the American 91制片厂视频al Research Association, a 25,000-member professional association based in Washington, essentially agreed.
鈥淚t should provide the raw material,鈥� Mr. Sroufe said of the NCES. Having the statistics center produce studies using HLM and similar methods 鈥渋s probably not a good idea, particularly involving work that is so politically sensitive,鈥� he said.
The study on public and private schools came out the week before Secretary of 91制片厂视频 Margaret Spellings and Republican lawmakers unveiled a $100 million proposal to make vouchers for private school tuition available under the No Child Left Behind Act. In questioning the wisdom of that proposal, critics cited the NCES report showing public school students outperforming their private school peers.
Despite that debate, Mr. Schneider said that his concerns were his own, and that Bush administration officials did not try to influence his views on the subject. 鈥淭he NCES,鈥� he said, 鈥渋s quite insulated from that kind of political pressure.鈥�
