How can states tell if their tests adequately reflect their academic-content standards? What鈥檚 the best way to determine the value that schools add to student learning? How can researchers capture the quality of instruction going on in classrooms?
Those were just some of the questions posed at a national conference here Sept. 9-10 for which researchers are still seeking answers.
鈥淲e need a systematic research agenda addressing both assessment design and accountability design,鈥� said Daniel Koretz, a professor at Harvard University鈥檚 graduate school of education. Finding the answers to such questions is particularly pressing now, given the demands of the federal No Child Left Behind Act, researchers said during the annual conference of the National Center for Research on Evaluation, Standards, and Student Testing, or CRESST.
In particular, researchers stressed the need to focus more on classroom assessments that can actually promote student learning.
鈥淚 think we ought to attend to, when we build assessments, classroom performance more than we ever have,鈥� said Robert Glaser, a professor of education at the University of Pittsburgh.
Such assessments, he said, should not only provide teachers with information about what to do next, but also help students reflect on their own thinking.
Lorrie A. Shepard, a professor of education at the University of Colorado at Boulder, said that both large-scale and classroom assessments should embody the same curriculum standards and the same model of learning. But she argued that for assessments to serve as good instructional tools, they have to involve rich, complex tasks that large- scale, standardized testing cannot typically get at.
One of the central tenets of standards-based education is that states should align standards, tests, and instruction so that they all reflect the same learning goals. But pinning down what 鈥渁lignment鈥� means and how states know when they鈥檝e got it isn鈥檛 easy.
鈥淎lignment is a moving target,鈥� said Joan Herman, a co-director of CRESST, located at the University of California, Los Angeles.
Existing methods for studying the alignment between standards and assessments are often time-consuming and cumbersome, and they don鈥檛 always yield the same results.
One study presented at the conference, by Richard S. Brown, an assistant professor at the University of Southern California, examined whether it would be possible to streamline the process.
Mr. Brown and his team compared three states鈥� math exams with their content standards, using an alignment method devised by Norman L. Webb at the University of Wisconsin-Madison. They also compared the test objectives to a compendium, or index, of statements that had previously been aligned to the standards documents in multiple states.
While the latter approach produced more matches between the testing objectives and the state standards, Mr. Brown said, that was primarily because the index statements were so broadly written that a single statement matched multiple test items. The index statements also tended to be written at a much lower level of cognitive demand than either the state standards documents or the testing objectives.
He cautioned against the fidelity of alignment claims based on the use of such multistate compendiums.
During the conference, researchers also expressed cautious enthusiasm about 鈥渧alue added鈥� measures that try to determine how much schools contribute to gains in student learning.
鈥淲e would like to move in that direction eventually,鈥� said Edward Haertel, a professor of education at Stanford University. 鈥淭he technical problems are just enormous.鈥�
One problem, said Kilchan Choi, a senior researcher at CRESST, is that depending on which model you use, 鈥測our ranking is different.鈥� Some models, for example, compare each student鈥檚 initial test performance against his or her performance a year later without accounting for differences in school characteristics. But using data from 72 elementary schools in a Pacific Northwest district, Mr. Choi and his colleagues found that models that also considered the average socioeconomic status of a school鈥檚 students yielded a much different picture of how a school was doing.
Derek Briggs, an assistant professor of education at the University of Colorado warned against giving a causal interpretation to such value-added analyses, by simply identifying which schools or teachers are making larger-than-expected learning gains than a comparison group.
That 鈥渞esidual,鈥� as it is often called, 鈥渏ust basically represents a black box,鈥� said Michael Seltzer, an associate professor of education at UCLA. He said educators need to do a better job of measuring what鈥檚 actually going on in schools to produce such differences.
Researchers here also described several attempts to depict the quality of instructional practices, but acknowledged that none is ready for widespread use.
鈥淲e鈥檙e getting there, but we鈥檙e not there yet,鈥� said Hilda Borko, a professor of education at the University of Colorado.
