As concern over America鈥檚 competitiveness abroad intensifies, education officials in the U.S. are beginning to consider using individual states and districts鈥攏ot just the nation as a whole鈥攁s the units against which to measure their international peers.
In everything from population demographics to curriculum adoption, a country like Finland may be more comparable to an individual state like Minnesota than it is to the heterogeneous expanse of the United States鈥攍eading some policymakers and researchers to reason that such state-to-country comparisons can better highlight educational practices.
Yet education and testing experts warn that if such comparisons are to be useful, educators must go beyond basic test rankings to understand how countries鈥� specific policies and practices can make U.S. students more competitive. Some states, such as Massachusetts and Minnesota, are already comparing both their student achievement and educational practices to those of other countries via international tests and other studies.
International assessment expert Gary W. Phillips of the American Institutes for Research believes most states will eventually have to participate in tests like the high school-level Program for International Student Assessment and the 4th- and 8th-grade-level Trends in International Mathematics and Science Study in order to compete for global businesses. He likened it to the evolution of the , often known as 鈥渢he nation鈥檚 report card,鈥� which started primarily as a research tool and now is used as a pacesetter in state accountability report cards. 鈥淲hen NAEP first started, each state was living in its own little Lake Woebegone world, not knowing how it stacked up to other states,鈥� he says.
鈥淣ow the same is true internationally. Around the world a lot of these countries are eating our lunch. They are focusing on education in a way that we aren鈥檛,鈥� Phillips says. 鈥淲e have to know how we stack up. That鈥檚 why these studies are important; they allow us to benchmark what we do and know with what they know and do.鈥�
Andreas Schleicher, the head of education indicators and analysis programs for the Organization for Economic Cooperation and Development, administrator of the , said projects to compare students internationally can help America catch up to the global norm: Countries such as Germany, the United Kingdom, Belgium, and Canada, along with most other nations with federal forms of government, already compare their states or provinces to international testing benchmarks.
鈥淚f you go to Canada, the kinds of policies and practices that Alberta has put in place are very different from those that Newfoundland has put in place, or Ontario,鈥� Schleicher says. 鈥淚t produces much better policy insights to compare [state or provincial] education systems, than nations鈥�. That鈥檚 where systems differ, often significantly. In my country, Germany, the northern states and the southern states have quite different education systems. It makes much more sense to compare those systems [individually], than to mix them up in the aggregate.鈥�
Researchers such as Phillips and economist Eric A. Hanushek, of Stanford University, long have used statistical analyses to try to link a student鈥檚 proficiency on an American test such as NAEP to international tests, with varying degrees of success. All of these tests are given in different years, cover different topics within the same subject, and different groups of children鈥擯ISA tests by age, for example, while NAEP and test by academic grade.
鈥淭o date the only truly valid way to compare at [the state] level is you have to actually administer the international assessments in the states"鈥攚hich no states now do statewide with PISA, Phillips says. 鈥淚t鈥檚 more time consuming, but it is not based on an edifice of assumptions.鈥�
State as 鈥楥ountry鈥�
But how can a single state compare itself to a sovereign nation? In some ways, it鈥檚 easier to do, both in testing and analysis, than comparing nations. A state needs a smaller sample of students to get a representative group to study than America as a whole requires, and individual states can think more about how adding a new test will fit into their schools鈥� academic schedules.
As educators and policymakers seek to prepare students for a competitive global economy, U.S. states have looked to other nations to inform the development or revision of their own academic content standards. International comparisons are more common in mathematics and science, subjects often linked to economic competitiveness and technological innovation, than in English/language arts or social studies. In responses to a survey from the 91制片厂视频 Research Center, state education agency staff members most frequently cited standards from Singapore as models for current standards in mathematics and/or science.
SOURCE: EPE Research Center
鈥淵ou can do things with 3- or 4 million people that it鈥檚 very hard to do with 300 million,鈥� says Marshall S. 鈥淢ike鈥� Smith, a visiting professor at Harvard University in Cambridge, Mass. and a visiting scholar of the Carnegie Foundation for the Advancement of Teaching. 鈥淚f you make the assumption that we鈥檙e not going to change the [U.S.] Constitution 鈥� the states are responsible, and they are going to remain responsible, for education.鈥�
Because constitutionally, a state, rather than the federal government, has ultimate control over its education system, it can analyze its test results in a more similar context to a country with a national education system.
鈥淭he U.S. state-to-state diversity is so huge that when you think of PISA at the national level, the aggregate just doesn鈥檛 tell us very much,鈥� says V. Darleen Opfer, the director of the education division for the international research group RAND Corp. 鈥淎 comparison between Finland and Vermont or Finland and Wisconsin, that might be a better comparison in terms of size, population, homogeneity of the population, socio-economic status issues. Like everything else in education right now, the disaggregation is what matters鈥攁t least it tells us more about what鈥檚 going on and what we can do about it.鈥�
That鈥檚 why Massachusetts and Minnesota participated in the 2007 TIMSS as independent 鈥渃ountries.鈥�
Former Massachusetts education Commissioner David Driscoll pushed for the state, which regularly leads the nation in NAEP performance, to participate in the TIMSS after seeing America鈥檚 lackluster national performance.
鈥淲e can鈥檛 just pat ourselves on the back because we have the highest NAEP scores; there are other places out there that are clearly outperforming us, and we have to find out who they are and where and what we can learn from them,鈥� says Robert Lee, the chief analyst for student assessment services in Massachusetts. State education leaders weren鈥檛 鈥渟ure we were going to be at the top of the heap, knowing that the U.S. is not with the Singapores of the world, [and] wanted to get a realistic sense of where our students measured up against the rest of the world.鈥�
As it turned out, both Massachusetts and Minnesota scored well above the national and international average on the 2007 TIMSS; Massachusetts 4th graders led peers in all 59 participating countries and states but Hong Kong and Singapore in math, and Minnesota students outperformed all but Hong Kong, Singapore, Chinese Taipei, and Japan in the same subject. Hong Kong and Shanghai, like Massachusetts, participate in the TIMSS as separate 鈥渃ountries.鈥�
Yet the more detailed results available because Massachusetts had compared itself as a country, rather than as part of the national sample, turned up intriguing differences, Lee says. For example, the state found that while the average hours of math instruction across the United States went down between the 1999 TIMSS and 2007, it rose in top-performing countries, as well as in Massachusetts, from 141 hours per year in 1999 to 155 hours in 2007. Class size wasn鈥檛 a big factor in how the state鈥檚 students performed on TIMSS, but the state鈥檚 math and science curricula weren鈥檛 as rigorous as those used in countries like Singapore and Hong Kong, where 40 percent of students scored at the advanced level, compared with only about 15 percent in Massachusetts.
The state is doing an item-by-item comparison of performance on test questions in its math and science curricula. 鈥淚 believe the kids in Massachusetts are very bright, and they could meet that [advanced] benchmark; I feel we鈥檙e just not challenging those middle-level kids enough,鈥� Lee says. "[For] every item we look at, we鈥檙e seeing our state on a spectrum with many other nations. You get a chance to reflect on whether your curriculum is aptly pitched for students who are going to have to compete in a global job market.鈥�
Moving to full international testing takes serious commitment from states, though. Massachusetts spent about $450,000 per grade to administer the TIMSS in 2007; it added 72 minutes of additional testing per student in grade 4 and 90 minutes in grade 8. Yet more states seem to be considering the cost worth the rewards; in the 2011 test administration, eight states, in addition to Massachusetts, participated in the TIMSS as independent 鈥渃ountries": Alabama, California, Colorado, Connecticut, Florida, Indiana, Minnesota and North Carolina.
Moreover, the National Center for 91制片厂视频 Statistics is in the middle of a study that will link the NAEP and TIMSS with both a more detailed statistical analysis and the full state results. This year, states gave 4th and 8th grade students taking the NAEP some math problems linked to TIMSS, as well as an extra NAEP science assessment aligned to the international study. Then, during the later administration of TIMSS, the researchers did another linking study incorporating the same questions but using TIMSS testing procedures. This creates a pool of test questions solved by students taking both tests which can be used to align their scores.
鈥淚t鈥檚 a really strong design. In the past we鈥檝e and others have tried to link by using statistical techniques 鈥� but not literally giving the same test to the same kids, which is a much better way to do it,鈥� says Sean P. 鈥淛ack鈥� Buckley, the NCES commissioner.
Yet Buckley warns, 鈥淭hese kinds of studies are great at producing [ranking] tables, but they鈥檙e not very good at causality. International assessment data is useful at minimum for hypothesis generation, identifying top performers 鈥� [but] figuring out what works in education is really painstaking, and you鈥檙e not going to get it from rankings.鈥�
Closer Comparisons
Some school districts are starting to explore these finer-grained comparisons, too. For the last three years, the 4,700-student Scarsdale, N.Y., public schools has partnered with Teachers College, Columbia University, to compare its own teaching practices with those of countries that outperformed their students on the PISA. Teachers College researcher Ruth Vinz is conducting classroom observations, teacher interviews, and reviews of student work, both in the district and in Australia, Singapore, Finland, and Shanghai.
鈥淧ISA and others like it compare tests, but don鈥檛 give you a sense of the curriculum that built the capacity of the students being tested,鈥� says Lynne Shain, Scarsdale鈥檚 assistant superintendent for instruction. 鈥淲e hope to understand the components of the teaching and learning process that foster higher-order critical and creative thinking and problem-solving.鈥�
The district found that students were not actively questioning things they were taught in class鈥攁 high developmental skill included on the PISA. The district has since changed its local benchmarking assessments to include more questions that have more than one answer, or that require students to question the information they are given.
Smith, a former top education official in three separate U.S. Department of 91制片厂视频 administrations, says more states and districts should follow Scarsdale鈥檚 lead. 鈥淧ISA is specifically designed to be an assessment that requires transfer, and asks questions in the areas that you would not really pick up in the regular curriculum: They鈥檙e longer, take a whole page to write about; they鈥檙e the kind of questions different from the normal questions we expect on tests,鈥� he says. 鈥淚n my view, we ought to be as a country creating our own PISA-like assessments if we鈥檙e serious about the 21st-century skills.鈥�
