Home Articles Reader Opinion Editorial Book Reviews Discussion Writers Guide About TCRecord
transparent 13

The Development and Impact of High Stakes Testing - Part 4 - Interpreting the Movement for High Stakes Testing

by TC Record - November 15, 2000

(Editorís Note: This is a transcript of the lecture delivered. The text is not revised.)

Gary Natriello:

So this is a pretty ambitious change. Again, you wouldn't launch your Lexus this way, right? Because what's happening is, you know you're cruising for a real unfortunate set of circumstances. And I don't think that, you know, if you're a policy maker in those circumstances, that you can't foresee that; that that's going to surprise you. And it would surprise you even less if you actually had a system in place to report the data the way you should report it, which is very precisely at the individual level; which, of course, is the way they all collect it, but they don't want to report it that way. And there's a whole other lecture about how states systematically ignore their own data, but we'll talk about that at another time. And they collect -- and those of you who are in schools know, because you're always asked to report all this data 55 different ways, and slice it six ways this way, and you know, how many boys did this and how many girls did that, and how many?

Well, you know, you never see that actually reported that way. You see it in these huge aggregations which tend to, I think, cover things up. So, for example, in a lot of states, you'll see these school profile reports, but you'll never see the school profile reports with the same data on school resources. Or if you see it, you'll see it in a way that's so isolated that you can't make the connection yourself that, you know, in fact, schools without resources, schools with low proportions of certified teachers, schools with all these other deficiencies, in fact, have outcomes that are not very attractive, that most of us don't want to think about. Yeah?


Is this deliberate?

Gary Natriello:

I don't know that it's so much that the whole thing is deliberate, but I would suggest that, when given the chance to present data in a way that reveals problems, that there's an interest in not necessarily always presenting the data in that way. And I don't think that makes policy makers unique; I think most of us would figure out a way to do that. So, you know, when I'm reporting my course evaluations, I'm going to look down there and I'm going to say, well, gee, you know, yeah, there were these things that people didn't like, but let me show you what they really like. The seats were terrific! People thought they were comfortable. (Laughter) They could rock back and forth on them, and they said that I spoke loud enough. Now, in fact, they disagreed with everything I said, and they thought the content was trash, but no one had a problem hearing me in the room. So, you know, you tell the story that you want to tell. And we've already read enough papers about how this data gets interpreted from various perspectives. And keep in mind that what really happens in a lot of these reports -- these are, in most cases -- and I don't mean to beat up on any particular administration. Because I think if I was running the operation, I'd probably say, well, let's put the good face on it. But, you know, they're not dispassionate reports; these are self-reports of various administrations on their own performance, right? So, I mean, who's testing the policy makers? Well, no one's testing the policy makers except the electorate, and the electorate is going to test them based upon the information that is provided.

And the question, then, is how do you get that information? And, you know, we're making some progress, but it's tough. And in a lot of states where we go in to work on some of these court cases, you know, this data has to be requested as part of the court process. And I can tell you, it's usually collected and organized and presented in a way that doesn't make it particularly easy to reveal the patterns that you'd like to really reveal. It's connected at a very aggregated level, but it's very quickly aggregated to a level that doesn't allow certain kinds of comparisons, and yet facilitates other kinds of comparisons. And so, you know, what we're left with is the kind of displays I just showed you, which is really very aggregated kinds of data; we'd like to really have it in a different way, and a lot of states just don't present it that way. Keep in mind, it wasn't that long ago that New York City did not report student data by racial and ethnic background, because they thought that was pejorative, and they didn't want to do that. So they weren't really looking at how kids in different cultural and ethnic groups perform. You know, that's changed, but that's within the last 15 years that that kind of change took place. Yes?


(Inaudible) isn't it really a socioeconomic issue? I mean, don't those black and Hispanic students, once they move into more advantaged situations, they don't score differently than their white counterparts. I mean, I don't see white poverty rates up there. You know, I don't know why the data is teased out that way. Why isn't it strictly socioeconomic, if that's the real factor that makes student scores different?

Gary Natriello:

Well, I think, you know, what you might argue, first of all, you'd like to have a much better set of data to begin to tease some of these things out, and some of it certainly would be attributed to socioeconomic issues. But you might also want to raise issues about culture, and about the kind of cultural assumptions that are made, and whether those cultural assumptions advantage some students and disadvantage others; and whether the curriculum that we're constructing and offering as official knowledge that is, you know, given to kids and then is tested is, in fact, the same, received the same with the same advantage and disadvantage as others.

Now, again, I'm not arguing that it's all one side or the other; what I'm arguing is that the system ought to be reflecting on that, and that we really ought to be thinking about that, and that we ought to have the data to separate out the two issues that you raised. We ought to understand, for example, if there is an environment in the school which disadvantages racial and ethnic minority students, even if, in fact, they have the socioeconomic wherewithal comparable to majority students. We ought to want to know about that, because we ought to want to know if there's something that's being done in the system that is still problematic. And we ought to want to know the same thing about gender issues.

And then we probably also ought to want to know the same thing about learning issues. You know, one of things we spend -- again, as I mentioned last week -- relatively little time thinking about is, there are genuine differences in the way that people learn. And we ought to think about, how do we meet the needs of these very diverse kinds of learners? You know, the gender and the racial ones are easier, because we've got sort of big characteristics that we can focus on, you know, and we think we actually agree on what they mean, even though for some of them, it's not clear what we mean. But when you get to the learning style issues, there are lots of different issues combined in there. I mean, kids gravitate to certain kinds of learning and other kinds of learning. You know, for example, people have written a bit about how we essentially have a system that's based upon text, right; that most of our learning is text-based learning; and that not everyone learns best through systems that are dominated by text. And yet, we continue to perpetuate that system, right? So one of the things we know -- and not only do we know it, but we believe it's also good and right and pure -- is that those people who read better will do better, and you know that's right. (Laughter) And it's somehow wrong, if you don't have that skill. And you know, I think once, when you've finally figured this out, we're going to understand that, in fact, there are genuine differences in people's capacity to read. Some people are faster, some people are more efficient, some people are slower, some people read better for meaning, some people need to get material in other kinds of ways. But I don't think we've teased that out in any powerful way, and allowed the system to reflect that; we've just decided that's just. So what do I do? I assign lots of reading. And why can I do that? Because you wouldn't be here unless you were all great readers, because we wouldn't have let you in, because we've tested you, and other people have tested you. But that's not to say that there aren't a bunch of really interesting, creative, insightful, wonderful people who are tremendously talented out there, who were kept from making a contribution because of a reading barrier of one sort, that is primarily a reading barrier that we created. Now, we're not troubled by that. Well, you can't read. You know, go do something that doesn't require reading.

So, you know, it depends on how you want to think about it, how you want to structure the system and what your goal is in the end. Because it takes a lot of work to uncover, you know, these various differences and to find strategies to enable people to deal with them. But I think we've got, at this point, lots of case studies, and we've got lots of self-reports, of people who've had one kind of learning style issue or ability -- even call it ability or disability issue or another -- who figure out how to work around that, or who figure out other strategies, and all of a sudden become successful. And the question is, shouldn't, as a system, we be troubled by the fact that we're not doing more to help people figure that out, as opposed to simply branding them and labeling them and sorting them and selecting them, and saying, you know, I just don't really have the time, so you go over in this group? And maybe, you know, and here's another maybe, if this new testing regime, in fact, forces the system to confront these issues, maybe that could be useful.

But, of course, if you look at the testing strategies that we're using, they are not particularly reflective of the diversity of learning styles either. All right? I mean, even these new alternative assessments are, for the most part, pretty narrowly constructed in terms of what we take to be a reasonable demonstration of competence; and in many ways, constructed so that the competence that's demonstrated is not necessarily unambiguously related to success in other endeavors, which, of course, is one of the things we're supposed to be able to demonstrate. That, in fact, it leads to what we know is that success on tests leads to more success on schooling that, of course, is already configured to conform to the test. What it doesn't tell us very much about is the variety of human talents and how those might be developed, then, and how people might be helped to develop those talents. Other questions before I, I want to conclude, because it's getting late.

So I want to go back to the point that I raised at the beginning, and I think it's, again, worth stating. And I want to suggest that the LEP data that I put up is another way of thinking about that. And the debate over non-English speakers or English as a Second Language speakers that rippled through the New York state policy community is, again, another example. There is a huge debate over who should have to take the test in English, and when they should have to take it in English, and what would be a reasonable demonstration of competence. And I think, in the end -- unless someone wants this to be (Inaudible) progressed -- the policy makers held the line that, you know, you're going to take these tests, you're going to take them in English; that's the demonstration that we're looking for, and that your competence in another language is not particularly relevant.

Again, I think you can argue both sides of that, and, in fact, people have argued both sides of it. But what I'd like to suggest is that, you know, if we accept these assumptions regarding the positive consequences of high-stakes testing on all of the actors that we've talked about, then it seems to me that we can take a lot of comfort in the policies that we've seen enacted. Because we've got these challenging high-stakes standards for all students, these standards may, in fact, and these tests may lead to greater educational opportunities; and they certainly may do so more than the earlier tracked policies. And if we accept the legitimacy of these high-stakes tests as devices which appropriately measure student performance, and the effort of students which precedes that performance, then I think we can be satisfied that we're creating a system which justifiably rewards those who work hard and develop their talents of those (Inaudible) we'd like to see. If the managers of state educational systems use high-stakes tests as only one element of a broader rational examination of the processes and opportunities and outcomes that we've talked about, and if they organize the system to provide all students with these opportunities to get the desired level of performance, and if they do it well in advance of the testing, then we might see these tests as appropriate rational management tools placed on what is probably the least equitable educational system among modern nations.

However, I would argue that there's an equally plausible set of assumptions which we ought to be troubled by. And those assumptions go something like the following: if the motivational consequences of these tests are not positive, or if they're at least not universally positive across racial, ethnic and social class lines -- and I would suggest that there's some evidence that suggests that they're not; that, in fact, these tests have different motivational consequences for different groups in the population -- then, it seems to me, we should be concerned about the potential of these tests to further heighten the already substantial inequities in schooling outcomes.

If we have reason to question the legitimacy of the current testing technologies to support the purposes to which they're being put -- and I would argue that if you read the National Research Council report, that that's exactly what they're doing; they're questioning the legitimacy of the testing technologies and the way that they're being used -- then, it seems to me, we should be concerned that the science of testing -- and I use science in quotes -- is once again being abused to justify and achieve social purposes that would otherwise be indefensible. And remember Professor Sobol talking about bringing the kids into the gym and dividing them into two groups, and telling one group that you get good stuff, and the other group that you don't get good stuff. And the only difference between that and some of the policies that we've enacted is that we now have a scientific way of making that initial cut.

Now, if the managers of state educational systems use high-stakes tests in isolation for more comprehensive efforts to provide equitable and adequate resources, then we should question the true intent of testing. Now, I think this is one that, interestingly, is very cloudy. Because I think the case could be made by the proponents that, in fact, we've got these comprehensive standards, we've got the good curriculum, we've got dah-dah, dah-dah. So on paper we've got a lot of this comprehensive system, and the question is do we have any of it on the ground? Have we really implemented it? Have we provided the personnel? But I think it clouds the issue by the fact that we've at least got it on paper. Now, if the defenders of the current arrangements for schooling rely on the results of high-stakes tests to define any and all patterns of educational deficits as originating and residing in the backgrounds and individual capacities of students alone -- and it's all in the students -- then we should be concerned that these tests will be used to justify an educational system which only appears to provide fundamental educational rights, but, in fact, denies rights in defiance of state and federal constitutional provisions.

So I think the argument really comes down to a question of what perspective you want to apply. And I would suggest that it's not an easy decision to decide that you're going to reject either one of these perspectives. And I would argue that either perspective can be applied to look at the system. That, of course, is not what the proponents would suggest; that's not what the policy makers would suggest. In fact, they would deny vehemently the second perspective that I've suggested; that it's a social and political agenda that's designed to maintain control over the system and to advantage some at the expense of others. And in fact, they would say that's exactly what they're not trying to do; that's exactly what the old system tried to do. And if that's the case, it seems to me, to sustain that claim, then what they need to do is to move very quickly, when deficiencies are identified, and bring in the kinds of resources, whether those resources are technical or human or physical resources, that are going to actually be required to turn the situation around.

And I think that's where we are today. We now have a large number of states that have put these programs into effect; we have a rising cohort of students who are going to run smack-dab into these standards; and there are a couple of things that could happen that could be very troubling. Number one -- very troubling -- we can let the students run into the standards, deny them diplomas and send them out into the world disadvantaged. Number two -- equally troubling -- we could have state policy makers back off the standards, right? And in fact, some of the advocates for at-risk and disadvantaged youth find that to be the most troubling of all. That is, they critique the tests, and they critique the standards, but what they're most afraid of is that the policy-making community will say, you know what? This is just going to be too tough to deal with, we're just going to lower the scores, and we're going to lower the standards until we get just enough kids passing that, politically, we can sustain it. Because, in fact, that would knock us back to the old position of allowing this kind of two-tiered system.

So I think the thing to watch as we move ahead is, what happens when kids run into the standards? What happens when there's no other escape valve? Do they create one? Do they somehow run around the standards? And keep in mind, as I've pointed out before, these cut points and these standards are socially determined; they're not technically or scientifically determined in any meaningful way. So adjusting them is a consensus decision. And, you know, we could get together, if we were the testers and if we were the policy makers, and decide where we were going to set them; just as a teacher, you can decide how you're going to grade the test that you're giving, and how you're going to allow students to pass or fail and rise and fall on those tests. So, it seems to me, that's what we ought to keep our eyes on as we look forward. And finally think about how we would judge this part of the reform movement.

Cite This Article as: Teachers College Record, Date Published: November 15, 2000
https://www.tcrecord.org ID Number: 10645, Date Accessed: 1/22/2022 10:33:43 PM

Purchase Reprint Rights for this article or review
Member Center
In Print
This Month's Issue