Home Articles Reader Opinion Editorial Book Reviews Discussion Writers Guide About TCRecord
transparent 13
Topics
Discussion
Announcements
 

The Development and Impact of High Stakes Testing - Part 3 - Who is Advantaged and Who is Disadvantaged?


by TC Record - November 15, 2000

(Editorís Note: This is a transcript of the lecture delivered. The text is not revised.)

Gary Natriello:

So it just strikes me that, if you look at who's going to be advantaged and who's going to be disadvantaged by the imposition of this policy, there's at least a reasonable suspicion, in my mind, that this is about separating people into one group or another, and giving advantage to some and denying advantage to others. At least it seems to be a plausible interpretation. Now, of course, the proponents will argue against that; that, in fact, this is a great and wonderful thing which will make everyone equal.

Student:

(Inaudible) …but I think what's happening is, because we're emphasizing the test as opposed to the realignment of the curriculum, that's where we go wrong. I mean, what is wrong with realigning curriculum? If you take a look at the frameworks, they're quite good. I mean, you know, I teach social studies; I love the state curriculum. But what I don't love is that every child I teach has to be able to take a test, which I have no control over and I have to teach to, because all we rely on is the score. It's the test that's wrong; it's not the curriculum. And the intention is to expose children to a rich curriculum; who's going to argue against that? It's the test that's the problem. And people don't understand that that test means nothing. It means absolutely nothing.

And what's happening is, in the best of schools and in the worst of schools, all that's going to happen is kids are going to get drilled to a test constantly. They're going to get so turned off to learning; I don't care where they're starting from. The test is deadly. And until it becomes institutionalized the way the Regents have been, that's what education is going to be for the next five or ten years. It's deadly, and that's the problem. Curriculum is good; that's what we should focus on, not the test.

Gary Natriello:

Right. But the question is, was there a genuine intent backed up by genuine planning, to actually enable people to enact the curriculum? So that, you know, the best curriculum in the world, taught by someone who's unprepared in that area, is not going to be that great. And you know, how can you presume, you know, we can announce the curriculum or you can post it on the web; we can do whatever. And that's what the state has done. They've got all these great materials, you know, wonderful things and they do; they look great. I go in and I say, gosh, I'd love every kid to have this.

But what I'd really also love is every kid to have a teacher who would be competent to teach it. And that's a big challenge, as we learned a week or two ago; and it's a challenge where there's a distribution across communities that is very unequal and shows very little sign of getting addressed in any serious way. You know, who's going to go out and recruit people who can take the test, who can teach the curriculum that we agree is a good curriculum, in a way that's sophisticated and that makes it accessible to the kids who need to be taught? And not only is it a challenging curriculum to teach to kids who've traditionally taken that curriculum, it's a doubly challenging curriculum to teach to kids who've never been exposed, in their earlier years, to earlier experiences that would prepare them for that experience.

One of the things that's pretty clear is that the earlier generation of assessments was of a sort of nature that you could get a kid in eighth grade who really hadn't had adequate preparation, and you could drill them, and they could take that and probably score well enough to pass it. I think it's pretty clear, and when we interviewed curriculum specialists, and we interviewed teachers who've been in the system for a while, the consensus that we found was that people felt, you know, this is not something you start preparing for in eighth grade; this is something you start preparing for in kindergarten. And you need to reinforce it all the way up and down the line. Now, of course, what the testing folks will tell us is, in fact, they put in a whole set of tests all the way up and down the line, and in fact, we're going to have kids marching through each of those levels; and in fact, because we're not going to lose them at each of those levels, we're going to bring them into the system. But there's still this gap, right, because the tests have been put in place.

Again, let me just offer a rational strategy for enacting or implementing these tests would've been to move it from the elementary grades to the secondary grades; would've been to say, you know, we're going to bring everybody at third grade up to this level, and then we're going to ladder it up, and we're going to do it over a ten-year period. That's not the way the tests, in fact, were implemented. The tests were implemented at once across the board. So, you know, at the very least, you've got a whole group of kids, who went through the system with one set of expectations, at the very end exposed to a system with another set of expectations. So it raises questions in my mind as to what the real intent was. And keep in mind, there's a lot of discussion earlier on, before these things come to pass, about how disturbing it is that high school kids are presenting themselves for employment and they're not adequately prepared, so that employers can't use them properly in the workplace. And you know, there's discontent with people not fully appreciating and understanding the English language and basic communication skills. And if you listen to the rhetoric that goes with that, there's a lot of social and cultural assumptions built into that rhetoric, in terms of who's actually going to be qualified and deemed qualified to share in the advantages of society, and who will be deemed unqualified, again, for seemingly rational reasons.

Who would argue that, you know, one thing we argue in the US is that we're a meritocracy and that people who have, you know, the better scores, in fact, get to be advantaged over those who have the poorer scores. Most of you are here because you believe that people who have more education should have more opportunity. That's why you're all here, you know, you're not just here because you loved the décor of Teachers College; you're here because you think there's more opportunity at the other end, and you've worked hard, and we all believe in that. So there's a lot of emotional appeal with this supposedly merit-based system. Yeah?

Student:

(Inaudible) …that really makes me think about the idea of power and who has power. And those who have power never want to willingly give that up. And so it's a situation whereby the people who are making the rules are typically the ones who have the quote unquote power, on economic levels, on social levels, educational levels; and they want to pass that on to their children and the other people who share that power, thereby limiting other people, and taking away their power, because you don't want other people to have as much power as you. And it's just kind of a cyclical cycle, that you want to leave people out. And I'm not saying everybody's sitting there thinking, I want to leave people out; but it's a possible philosophy when you were talking about the social and political advantages.

Gary Natriello:

Yeah. I mean, think of it this way: we can all develop an examination or a test, and we could do it in one of three ways. So let's imagine the different ways that we -- I'm going to say there are three ways. There are probably a hundred ways, but let me just say there are three ways at which we could set standards. We could set a standard so that a hundred percent of the population achieves the standard; we can set a standard so that 50 percent of the population achieves the standard, and we can set a standard so that no one in the population achieves the standard. Well, most of the large-scale assessments that we establish are somewhere around here, right? So we're sorting. Now, that's not a scientific decision. I mean, there's no science to that, other than the fact that, if you want to sort one group of people above the line and another group of people below the line, here's where you set it, right?

And in fact, if you look at how a lot of these standards are set on these assessments, there's no compelling scientific rationale related to any kind of performance that's meaningful in any kind of adult context. And if you ask people how they're setting the standards, they will covertly tell you that we're going to wait and see what the responses look like, and then we're going to go in and we're going to set the standard based upon, you know, where we want to cut it. All right? So that it would be a problem if no one passed, and it would be a problem if everyone passed, because our goal is really to be able to differentiate different levels of performance, and make allocation decisions of resources and opportunities based upon some sense of the test. And all I'm going to suggest, in the next little segment, is that that's not going to fall equitably across different subgroups in the population; that there are going to be some patterns that are going to be set up that would make me suspicious as to how the whole testing regime gets constructed.

Student:

I was going to say, we can really (Inaudible) almost in the same way that education or provincialism (?) or whatever (Inaudible) how before a high school diploma meant something, and as soon as more and more people were getting a high school diploma, then suddenly everyone was trying to get a college degree. Now, obviously, a lot of (Inaudible) directed (Inaudible) undergraduate degree, because you need (Inaudible).

Gary Natriello:

Why? And you're hoping that a lot of people don't have a graduate degree. Right? That's what's going to give you the advantage in going through the door. So it's this social process of deciding how we're going to differentiate. And, you know, the policy community says, you know, how many kids are going to do well? Is our goal, a hundred percent, right? No child will be left behind. That's the rhetoric. But, in fact, that's not the distribution of test scores. And I would argue that, if that distribution ever came to pass, that would be a problem, and it would be quickly changed.

But they don't think that's (Inaudible) and if I'm wrong, I'd like someone to show me the evidence that shows there's a state where a hundred percent of the kids are passing the test, and everyone is happy. Or show me a state where no one's passing the test and everyone is happy. In fact, people want it somewhere in the middle. They want to make some decisions about who gets to have, you know, the new resources. And look at the distribution of resources at the higher ed level, right? State policy, in many states, says, gee, if you're below a certain line, you can't access the higher ed system, so you're not going to have those opportunities. Now, most of us aren't troubled by that, because we've accessed the higher ed system, and we think it's absolutely splendid that we have. And not only that, but, you know, we like palling around with the others who succeeded as well. That's why we're going to college, right, to get away from those other folks who couldn't get over this bar. And not only are we doing it, but we've gotten the state to pay for us to do it, to subsidize us, right?

So we're using general resources to advantage one part of the population over another very systematically, and we justify that as part of our policy, based upon merit. And then we turn around and we particularly justify it by saying that -- you'll all appreciate this, because I appreciate this -- you know what it is? We work hard! Right? We just really work (coin falls on floor). People are throwing money all over the place in this place. (Laughter) You know, we're here because we work really hard, and those other folks are just screwing around; they're not working hard. We're just really good, hard-working folks. Yeah.

Student:

(Inaudible) why would it not be advantageous to have a larger percentage of people passing the test if these results are going to be published?

Gary Natriello:

All I'm suggesting is, in fact, it might be advantageous to have more people pass the test. I'm just looking for a -

Student:

I mean politically advantageous.

Gary Natriello:

Well, I'm just looking for a system that will tolerate that. And what I see is state after state having systems where, in fact, making the differentiation seems to be important. So I'm waiting for the state to reach a level where they're happy, right, where everyone is passing the test.

Student:

Does that 50 percent mean that

Gary Natriello:

And 50 is just arbitrary, by the way. It could be 75 percent passing. You know, there's just some group that we want to eliminate from the competition. It might be 20 percent. Yeah. Who is it that we're not going to let have more opportunities, right?

Student:

(Inaudible)

Gary Natriello:

Right. And by the way, I'm only trying to raise the issue. Because I actually think that there is a rationale on both sides, all right; that you could argue that, in fact, it's an attempt to drive improvement. I don't think you can deny that. I think people that, in fact, the system will get better. I think there's evidence that, in fact, the system does get better. But I think there's also evidence that there are going to be associated costs with that, and that those costs are going to be disproportionately borne by certain segments of the population and not others, who will receive additional disadvantages, and those disadvantages will accrue over time.

And that at least should raise questions in our mind, particularly when we know that there are, in fact, support strategies and personnel development strategies and curriculum development strategies that could have been put in place in support of these kinds of high-stakes assessments. So now what we're saying -- and this is what I think is really important to understand in the policy -- that this assessment is going to have a lifelong consequence for the individuals who, in fact, cannot pass the assessment, cannot graduate from high school and not get other kinds of benefits that get associated with that. And, at the same time, we know that there's incredible differences in the access that people have to the resources that they're going to need to actually achieve a level of success on the assessment. So we're making it very important; everyone recognizes there are tremendous disparities in the ability of different schools and districts to prepare kids for the test, and we've decided, as a policy, that we have adopted that.

And there doesn't seem to be an enormous discontent with that, and in fact I don't know if you've seen the recent surveys. Since the presidential campaign is at full speed now, they're doing all kinds of surveys about who thinks that these testing programs are problematic or not problematic. And in fact, you find pretty good acceptance of these testing programs, the high-stakes testing programs. There are lots of people who think that, you know what? You can't pass the test, you shouldn't get a high school diploma. And not only that, but the interesting thing about some of these studies show that even the people who are being denied the diplomas think it's fair. Well, I guess I didn't work hard enough, or I guess I wasn't good enough. I mean, the people who believe in the meritocracy often times are the people who are most disadvantaged by it, and they're absolutely convinced. Yeah, you know, look at, for example, all of the national surveys that track dropouts, right? And let's argue that dropouts, in fact, are people disadvantaged by the system, at least for the moment. What do they all say? Well, you know, school was not for me; that's the most popular response. …it's the person who's doing all kinds of other tasks for you. You ask him, you know, how was school? Did you go to school? What happened? Well, it wasn't my thing. Well, how do you feel about that? Well, I can't help it, that's the way it is. So those folks often aren't very particularly upset by it. All right? The schooling myth and the meritocracy rationale has been very convincing to them.

Student:

(Inaudible)

Gary Natriello:

Can somebody tell me that? I don't think I really know that.

Student:

(Inaudible)

Gary Natriello:

Right. Which is kind of interesting, which means, you know, one way to sidestep the Regents is to use the private system, which, of course, in many cases, requires more resources.

Student:

(Inaudible)

Gary Natriello:

Right. Which, of course, we have private colleges that do the same thing, which move kids through higher education and get them the diploma under a certain set of circumstances. Charter schools probably do, I think, right?

Student:

(Inaudible)

Gary Natriello:

Right. Yes?

Student:

Inaudible)

Gary Natriello:

I think there is a test lobby. Anybody been involved with testing companies, or on the side of being lobbied by testing companies? They certainly compete for the contracts. And in fact, you know, I think the best thing we could all do is invest in testing companies, because I think there's no sign that anyone is going to be shying away from more differentiated kinds of tests. I mean there's just been this huge boom in testing for quite a while now. Yeah.

Student:

They're definitely testifying and quantifying that their tests actually test something, you know? (Laughs) I mean, that's the role I've seen their lobbyists, (?) is saying yes, this is a good test; it really does measure something; defending attacks from people who say, tests... (Inaudible).

Gary Natriello:

So I think, you know (Inaudible). I can tell you, you know, when you go to the research meetings, they always have the most lavish spreads of anyone. (Laughter) So they certainly want to entertain the research community, and I think they're certainly interested in maintaining the aura of science about their technology. But I think, and I won't go into this, because it's really a whole other talk, I think a really careful examination of testing techniques, testing methodology, the testing infrastructure would reveal there to be much less science than the general public typically believes.

I mean, people tend to believe that testing is scientific, that it's unbiased, that it is fair and equitable across the board. And I think, you know, testing is constructed from a certain social and cultural and political perspective, and it's inevitable that it's going to reflect a certain set of values. Testing, in many ways, turns out to be very arbitrary in the way it's constructed, in the way they're scored. And those of you who are teachers probably understand better than anyone else, that often times a set of performance results you get and get reflected in any set of assessments depends on how you construct the assessments and how you structure the conditions under which students are asked to perform the assessments. You can dramatically change the outcome in a whole variety of different ways, and teachers typically do reconstruct the assessment strategy to do that.

Student:

Just in hearing this whole discussion of the standards, and particularly of the performance standards is what you're basically talking about, it just came to me that, isn't it blatant hypocrisy to talk about performance standards if, in fact, not all children across the board are getting equitable opportunities to learn? It just seems ridiculous to sit here and talk about testing -- I mean, not all of you -- but, in a sense, to even carry on a discussion of performance standards if, in fact, the heart of the matter is the question, are kids getting equitable opportunities to learn? And if, in fact, they are, then we can talk about performance standards. Obviously the content standards are a given; the performance standards is the next issue; but probably before both of those items would be the equitable opportunities to learn.

Gary Natriello:

I think that's certainly a position that many people would support; that, again, to go back to this notion of rationality that I've been talking about; that if you want to build a system that's going to be able to do all of these things, you've got to put in the conditions that are going to allow people to acquire the skills and abilities and capacities that you want them eventually to demonstrate. And the fact that some people are systematically denied access to those opportunities is troubling; I mean, at the least it's troubling. I mean, it should raise questions about the entire policy apparatus, I think, in terms of how those decisions get made.

But there's been widespread denial of that as an issue and as a problem. And there's been a lot of discussion about, you know, once we put these standards in place, that's going to drive everything else. But one of the things it's not driving is a reallocation of resources; at least a reallocation that's led by those folks who are in policy-making positions. If anything, they get brought along by court action, and then very reluctantly. So I think that's part of the issue that I think is central to this whole thing. I think that's what Tom Sobol said a couple of weeks ago.

I think that's what we saw Karen Zumwalt talk about when she talked about the distribution of quality teaching, and how there's an enormous inequity in the distribution of -- and we're not talking about off-the-chart teachers, we're just talking about teachers who actually have something on paper that makes it look as if they had knowledge in the area that we're asking them to teach, all right? And now we're asking them to teach it at a very ambitious level, and potentially at a very ambitious level to children who, for much of their schooling, had not been exposed to adequate preparation.

So these are incredibly challenging tasks. And I'm not saying they're not terrific tasks, and that we shouldn't be ambitious, and that we shouldn't be very aggressive in trying to pursue them; but right now, most of our aggressive activity has been in imposing assessments rather than putting in place the resources. You know, if we were as aggressive in putting in adequate resources, we would have a very different-looking system. Instead what we've done is, we've been very aggressive about putting in testing. And, you know, that's why I have serious questions about whether this is a genuine attempt to drive reform, or whether it's really about something else.

And that something else is sort of separating people into different groups, some of whom will be given advantages and some of whom will be denied those advantages. And increasingly that's becoming a systematic process. And it's a process that I think most of us would argue is becoming increasingly important in society. Because as education becomes more and more fundamental to all other kinds of success in all other kinds of areas, it's not as it once was, something that would be nice to have. It really is a fundamental essential in society to have any kind of opportunities at all. And we are pretty systematic in the way, I mean, if you look at the material that Professor Sobol presented a couple of weeks ago, in terms of just the resources that people in various communities have to work with, you know, we're talking about multiples of three, four and five times in some communities over other communities. And he gave you the example where, you know, we're talking in two districts, hundreds of teachers denied to certain kids who would have had them if they had the same level of resources in the community as a neighboring community. They would've had several hundred more teachers to work on some of these tasks, and they probably would've had more well-prepared teachers. And you can argue that along a whole variety of resources, you know.

If you don't think teachers matter, pick some resource that you think does matter -- you know, whether it's better facilities or whether it's more technology or whatever you think it is -- and look across the distribution of those resources, and you're going to find the same disparities. You know, and then what we do -- and then I'll try to get back to my talk and off of my rant -- is we then put on remedial programs, using special allocations, right, and we deem them special. These are special allocations for kids who we think are going to have problems and they're going to need help. But those allocations never approach the level of resources that other kids get, just as a matter of course. And so those allocations are then used as an example of how it's just a failure, and how there's just no hope. And so there's a whole logic that gets built into this, about, well, should we invest in it or should we not invest? And, you know, at least one thing you can credit some of the testing reformers with is at least their rhetoric is that they intend everyone to have a shot, and at least everyone gets the same test; I guess that's progress. (Laughter)

Not everyone gets the same opportunity to do well on the test, but everyone at least gets the same test. And that probably is progress. I mean, that's a very different place than we were, you know, 20 years ago, where some folks should have this test and other folks should have that test. Now, at the same time, we've tightened the opportunities that get associated with the test, so there's that restriction that's growing over what it used to be.

Anything else before I? Let me try to lay out some of the New York data that we took a quick look at. A couple of more things about the New York standards, and again, there's lots to commend in the New York standards. They are more comprehensive; they have a holistic view of curriculum; when you look at them, you can be inspired by them. I mean, you know, if we thought that we can, in fact, actually enact them and provide the kind of instructional opportunities that would help kids reach them over a period of time, I think we'd be pleased with them. Now, you can quibble with certain pieces of them, you can quibble with any assessment, but I think if you looked at them across the board, in general you'd think, gee, you know, if kids could do this, this would be great.

What I want to do is talk a little bit about what we looked at in terms of the racial and ethnic disparities and what seems to be happening. And I should say that this is not often all that easy to do, and states differ a lot in the way they report this data. States often have a lot of data on performance by racial and ethnic groups, and they often don't report it or don't make it very easy to report. So you have to look for states where, in fact, you can look at some of these things. Some states are much better than others in making it available. One of the things we did was take a look at the racial and ethnic disparities in performance in New York state on the existing Regents tests before the new standards were put into effect, to give us a sense of what we might expect. And just to give you a sense of what that looks like, the top chart. It's hard to see, probably, in the back of the room; I'll make it a little bit bigger. This is just a graph. And these are school-level data. It would actually be much nicer to have student-level data, but these are school-level data. And at the time we did this, it was about 18 months ago. The latest data that they had was '94 and '95 data.

Some of you may have actually looked at some of the more recent data, but I think the patterns have not changed dramatically. Here we have Regents diplomas earned in '94 and '95, and this is the percentage of diplomas within a school. So these little marks here represent a single school in the state of New York. These are high schools. And so what we've got on one axis is the percentage of Regents diplomas going from 20 percent down here to a hundred percent -- or zero percent, sorry, to a hundred percent. And then the percentage of black students going from zero percent to a hundred percent. And what happens is is that, as the percentage of black students rises in a school, the number of schools that actually have high levels of Regents completion declines. When you look at that at a correlational level -- this is, again, between the proportion of black students in the school and proportion of students passing individual Regents exams -- this is all negative correlations on the English-math exams, biology, chemistry, earth sciences. They're all negative.

My sense is -- and again, I'm going to let you make some judgments about this -- my own judgment about this is that a society which has decided that these exams are going to be determining of life opportunities are going to be very troubled by this kind of distribution across racial and ethnic groups. Yeah?

Student:

(Inaudible)

Gary Natriello:

What this simply means is that there are no schools in this particular sample. The school that had the most highest percentage of students in the school passing the Regents was a school that had 80-some percent of the students taking the Regents. Now, this was before the Regents was required of everyone; this was when we still had the RCTs -- '94 and '95.

Student:

Okay. But then (Inaudible)

Gary Natriello:

Right. Well, there's none actually up there. The highest one we have is a school that had -- it's like about 92 percent black students.

Student:

(Inaudible)

Gary Natriello:

Right. So you can see here -- I mean, you look in here and you can basically say, any school that has more than 60 percent black students in its population never got more than 35 percent of the student population passing on the Regents exam. So there are real distributional issues in who's actually performing in a particular way on these exams. So it's not going to hit every group in the state equally. If you look down here, you've got the same data for Hispanic students, and you get a similar kind of pattern. As the percentage of Hispanic students in the school increases, the chance of having a school where the percentage of Regents diplomas is high decreases. We didn't run one -- and I'll give you the reference on this for the paper -- it looks the same for socioeconomic status. So it's really tied, in that case, to resources in the family and the community.

Here's one. This -- as a number of you have mentioned, LEP students -- this is what it looks like for LEP students. Again, this is school-level data, because we didn't have access to individual kid data, so it's not as precise as we might like it. But here we have the percentage of students in a school getting a Regents diploma -- this is actually '95, '96, a year later data. And here we have the percentage of students in a school who are LEP students. And what you see here is, as the percentage of students in a school who are LEP students crosses 20 percent -- this basically says that in all of these schools, there are more than 20 percent kids who are LEP students, once you cross that 20 percent threshold, you never get above 22 or 23 percent of the kids in the school getting a Regents diploma. So we've got real different distributions of who, under the old system, was getting a Regents diploma, and they're associated with, in this case, language status; they're associated with race and ethnicity; they're associated with income. So when you're putting in these new standards and you're not doing anything to change the nature of schools, the support structure and all of that, you're putting in place a whole series of processes which are going to deny diplomas potentially to certain groups of kids in disproportionate number to other groups of kids.

Now, keep in mind, what the proponents of testing would argue is that this is exactly why we need testing, and this is exactly why we need it across the board. Because, under this old system, when we didn't require it of everyone, look what was going on. Look, we gave people the choice, and look what those educators were doing. They were over here, probably pushing all these kids into the RCT track, and denying them the access to the diploma. So it's not as if the data can't be interpreted from both perspectives, but the thing that I wanted to point out to you is that it's not as if everyone is out there in kind of these equal settings, where everyone in all these various groups has the same sort of opportunity. And keep in mind, here we've got schools where, once we cross the 20 percent threshold, what does this also mean? This means that very small numbers of staff in these schools are actually even teaching a Regents curriculum. So we're just not asking people who've been teaching the Regents curriculum for 20 years to then, all of a sudden, pick up a few more students and get everyone to pass it; we're asking people who've never taught a Regents curriculum to be able to turn around and to teach it not to the kids who would typically do well on it historically, but the kids who've never even had exposure to it.



Cite This Article as: Teachers College Record, Date Published: November 15, 2000
https://www.tcrecord.org ID Number: 10644, Date Accessed: 1/19/2022 6:07:24 AM

Purchase Reprint Rights for this article or review
 
 
Member Center
In Print
This Month's Issue

Submit
EMAIL

Twitter

RSS