Falling Short of the Standards - Part 1: High Stakes Testing in American Education - The English Language Arts Test - The Reading Test

by Clifford Hill - December 07, 2000

A detailed examination of the English Language Arts Test - Reading Test

Gary Natriello:

Our speaker tonight is Professor Clifford Hill who will be talking about assessment, and he's here to talk to us about a lot of the current work that he's been doing in this area, and so we welcome you to Educational Policy.

Clifford Hill

Okay, thanks, Gary.

Before I begin to talk, I'd like to mention a seminar that I'll be doing next semester. I've never done this seminar before, and it's tied to a new research project that I'm just starting. We've received a grant from the US Department of Ed. They have a section called OERI that gives money to support research and universities, and I'll be working with the College Board, and they have a national program called Pacesetter, which is set up to support the academic preparation of promising students who are culturally diverse. They kind of recruit from large, urban areas. And this grant will be looking, in this research, will be looking at how they do with assessment-based or assessment tasks that are technology-based. And by technology-based, we're going to depart radically from the traditional kind of assessment, which is based, kind of, on a traditional model of print literacy, and we're going to be trying to build tasks that kind of get at the knowledge and skills that students need in a digital literacy age.

Just to give you a flavor of some of the kinds of things we'll probably be doing, we'll obviously be doing a lot of stuff with these high school students, being online, doing research, synthesizing what they find and then communicating it, so that, you know, we'll look a lot, not just at the capacity to search and find information, but also organize it and communicate it in good ways. I will be looking a lot, this curriculum is very good, the Pacesetter curriculum, it's a very rich curriculum, and this assessment will be very much embedded in that curriculum. I could give examples of some of the other types of tasks. There'll be a lot of multimedia stuff, and our hope is that as we diversify the sources of information, go from just a print literacy model of assessment, to one that's multimedia, that over time, we hope that we might be able to close some of the gap that exists in American education between so-called mainstream populations and culturally diverse populations.

Now, of course, everyone always says, there's such a gap in access to technology, and of course, we're going to worry about that a lot, but we want to show that if we moved the assessment paradigm in a new direction, and build new kinds of tasks, more within a constructivist paradigm, that in principle, our hope is that culturally diverse students who are kind of college-bound and in our society, there's an enormous increase, as you know, in the African American/Latino/Asian American populations who are college-bound. As those segments of our population grow, they're all so very much college-bound, and the projections are, in thirty years, that it will be approaching, you know, pretty much half of the American population will be people of color and going to be an enormous increase in those entering higher ed, and so we want to move the assessment in a new direction. Because right now, our current system of assessment, based on a traditional model of print literacy, is blocking access of a lot of these populations to, particularly, to institutions of high quality. Because we want these people not just to be in higher ed, we want them to have access to the very best in American higher ed. Now, these kinds of projects are just starting, and other people are doing this kind of work, too, but I'll run this through a seminar, and I thought it might possibly interest some of you.

And even though we'll be looking closely at this Pacesetter curriculum, and trying to build technology-based tasks that fit into that curriculum, some of you who may be working in other educational settings could join in,and experiment with building technology-based tasks that would fit into the educational setting you work in. Because our ultimate goal is kind of, to build good prototypes that can be used in different settings, and so, in some ways, we're interested in having a diverse group of students in the seminars. If any of you are interested, I put my e-mail address here, also, my telephone extension. You do need permission to enter, I'd just like to get a sense of people, before they enter this course, and I'm not looking for a huge group, I would like to have a diverse group, as we begin work on this project. We'll be working three years on this project.

Tonight, I'm going to talk about Falling Short of the Standards: High-Stakes Testing in American education. And Gary had suggested that it would be good if I can kind of do it in chunks, I think it kind of plays better in the medium, so Lisa, if you can turn to the next one. I'm going to do a kind of case study, and I'm going to look at how this high-stakes testing has played out in English language arts, at the fourth grade level in New York state. And we'll look first at the test itself, the English Language Arts Test, we'll just refer to it as ELAT, English Language Arts Test, and we're going to look at the two major components, the reading component and the writing component. And then, having done that, we'll move on, and we'll look at the effects of ELAT, and we'll look at two major kinds of effects. One will be the increased use of commercially based coaching, and then the diminished use of a lot of burgeoning movements in authentic assessment, which we're kind of getting off the ground. I'll describe one that I was involved in, with teachers in the metropolitan New York area, and it's not working very well now, because of the pressures of students to do a lot of test prep for this new high-stakes test. So what I'll do, I think, is, after we do the ELAT reading component, I'm actually going to ask you to do a little reading exercise. We'll kind of stop and have some discussion, and then we'll move over to the writing component, and then we'll talk about some coaching material, have some discussion on that, and then we'll kind of show, kind of, kind of how some of this authentic assessment was working, and how it's kind of disappearing under the pressure of the high-stakes testing.

Now, my approach in this area is probably a little different from what you've been getting in this policy course, because instead of kind of thinking about broad policy issues, I'm used, as a linguist, to look pretty closely at the tests themselves, and how they work, and what kinds of effects they have on classrooms. And I guess one critique I might have of policy studies is sometimes, I don't feel they're grounded enough. They don't look closely enough at the actual implementation of the policies, and really, look - get into the nitty gritty of the actual materials, how they're functioning, how kids are responding to them, what's happening in the classrooms, and so I kind of work from the ground up, a bit, in this area. And that's not to say, the other larger, macro considerations are important, certainly, Gary's work, which looks at whether or not we're putting enough resources into even support this high-stakes movement. It's very, very important, and obviously, that's a huge question which has to be asked, any time you're going to try to implement something so massive as this standards movement in American education, and obviously, the resources just haven't been there. And also, the phasing-in of these tests, it was just sprung on people, there wasn't a preparation, and so, those large-scale, macro considerations, what the costs would be, and whether or not people really thought that through, are very important. But tonight, we're going to be down in the trenches a little bit, and fussing a little bit with the test material itself.

So, let me just kind of give you a little bit of background on ELAT. It was developed by the California Test Bureau, which is a sub-division of McGraw-Hill, and I mention this because I've had a lot of battles with them over the years, because I've published quite a bit, where we actually look at the test material and critique it. And I've had a lot of conflict with test makers not wanting their materials closely examined, and I've had particular conflict with CTB. They've tried to suppress things that I've written at various points in time, and in doing research on testing, we do go to the test makers. I work within a framework that an anthropologist named Dell Hymes has set up, which is called Ethnography of Communication. And essentially, what you want to do, when you study communication, whether oral or written, is to try to get at what the various participants in that communication act think they're doing. And this social linguist, Dell Hymes, has a very elaborate model for studying this, and I won't take you into it, but for example, you want to know what the interpretive norms for understanding text are, both by the test makers, and by the test takers. And you try to get a sense of what both parties think they're doing. So that means that we often go to the test makers, and interview them, and dig around, and of course, they think we're snooping.

And I can tell a funny little story. The first time I went to CTB - they're out in Monterey, California, it's a beautiful place - and they didn't know my work at that point. They didn't know what I was up to, particularly. And so they had a huge sign up as I approached the big factory, you know, these are quite large complexes, you know, Welcome, Clifford Hill, Teachers College, Columbia University, you know. And they treat you, you know, very cordially, and they're nervous, often, and they aren't too anxious, sometimes, to have you snooping around. In fact, it's hard, often, to get the kinds of information we want. And we often have to infer what their intentions are, by what they constitute as correct responses on tests, and then we're able to kind of derive what their interpretive norms are, from looking closely at the test material, and we can kind of build a model of what they think literacy is, by really taking the tests apart.

At any rate, they're the ones who got the contract. It's big money, to develop this test for New York state. And as you've read in the press over the years, they've had troubles. They've had problems of scoring. There'd been some pretty big scandals around that, and you know, the first year, when the results came out, they were so dismal, at the state level. If you can imagine, at the New York state level, two thirds of the kids failed the first year, on this test. In the city, it was about 80%. And some decisions were made on those test scores. Actually, the really tough decisions, where a lot of superintendents got fired, were also made on one of the city-run tests. One of the horrors of testing in American education is you often have different test makers lobbying, and at the same time, we have this test at the state level, which city kids have to take. There's another city test that's been preserved, which is also very high-stakes. And this decision, to fire a lot of superintendents, was actually based on, it was a combination of these test scores, and another test score. But in both cases, the scores weren't reported accurately, and after firing a great majority of the superintendents in New York City, you know, after they corrected some of these decisions were obviously misguided because the students did, in fact, do better than had been reported by the test makers.

Anyway, that's just a little hit of background on the ELAT, it's not been around that long. The scores did come up this year. You know, obviously you see this all the time, when a new test comes in, and people are unfamiliar with it, you know, often the scores are way down, but they move up fairly rapidly you know, as kids begin to get socialized to what's expected on these things.

Now, on the reading component, rather than actually going into current E-L-A-T material, ELAT material, I've decided to take material from a book that I've just published, which goes very deeply into reading tests at this fourth grade level. And I've looked at the ELAT material, I've not done the deep analysis which I've done on this other material, which I'm just going to talk about a little bit, tonight. It's very much the same. One of the things that's very annoying about the high stakes tests was, we were told that they were going to be pretty much based on performance tasks. They were going to be more complex, but as far as I can see, it's not just New York State, it's most states, still use multiple choice for reading comprehension. It just hasn't budged. And the kinds of stuff I'm going to show you tonight, and we'll illustrate from the book, the same kinds of problems we can see.

One reason I haven't gone in to take the ELAT material apart is, I think I'll have problems publishing, because this material is copyrighted. And I hate, I'm on the blacklist at McGraw-Hill/CTB, and I just don't want to put all the work in, and battle, and not be able to get the stuff published. It's just very annoying. So there are some very juicy things there I'd like to get my hands on, that I've seen. But I'll show you material, it actually came from the Gates-MacGinitie reading test. A very good set of tests, but this whole genre of reading tests for children, based on the multiple choice, once you've looked at it and dealt with it, you see predictable configurations in it.

And I'm going to just show you just one of these configurations tonight, and I'd like to read something, this is actually, it's not taken from this book, it's taken from something else that I wrote recently, but the target response, which is the key, the right answer, often calls for a very low-level operation. Typically, just recycling a detail, sometimes defining some vocabulary item. But it's configured with a distractor, that's one of the choices that's wrong. And this distractor, actually, will call for a higher-level operation, the kind of thing you do when you read. Now, this higher-level operation will often, you'll make some kind of inference, which, though stimulated by passage information, cannot be strictly justified within the restricted model of literacy on which reading tests have been traditionally based.

And so, to get you into what we mean by this, Lisa, if you could, actually, just show this passage. And I think what we should do is actually read this, and - I've only shown you one, typically there's two tasks after these short passages and, and I'd like for us, just for a moment, to have a brief discussion about, you know, the way you process the information here. So, if you could just take a moment and read it, and we can just talk about it a bit

[pause]

Clifford Hill:

Does everyone have a sense, just to get the feel of how these things work. Are your palms getting sweaty? Bringing back painful memories, perhaps. Now, part of our research on testing material, too, it's not just to try to get a sense of talking to the test makers, and we talk a lot to the test takers, and we'll talk about that shortly. But we also often, try to find out where the material is drawn from. And typically, test makers will, and Arthur Gates, and MacGinitie, were good test makers, in the sense that they would try to sample the universe of reading material that children were expected to read. They'd had a certain proportion from literature, a certain proportion from social studies, etc., and they struggled to be very fair in selecting the material. They saw that as part of the validity of their enterprise.

Now, this particular test item, comes verbatim from this book, this is actually from the TC Library, What is that Alligator Saying? It's a beginning book on animal communication. And it's word for word, from the passage, no changes whatsoever. It's right here, this little paragraph. You can see the mother alligator there with her baby alligators coming out of the pile of mud and old leaves. Maybe those that are doing the videotaping would like to get a close-up of this.

Now, in that sense, they try to be authentic, you know, they try to get real material that kids read. But of course, it's excerpted, and it's very, very partial, and there's not much context for it. Now, apart from the very limited context for the passage, what I really want to concentrate on with you is the nature of the tasks. Now, in a sense, I've primed you to look at the task in a certain way, by what I said just previously, but perhaps someone would volunteer their way of handling this task. In a way, we're handling it in a very particular context, because I've primed you, and so forth, but still, you might talk about what you would like to do, and then what you'd actually do, because it's test material. If anybody wants to volunteer and just talk a little bit about how they handled this task. Anyone, care to. It's not easy to talk about your experience with test material publicly, because you're afraid that, you know, what you say may be held against you, because God knows it is, in American education, so we're a little shy about talking about our experience of tests. It's a little bit like this fiasco around the voting, everyone said, well, if people were confused, they should have asked for help, but when you're in that voting booth, it's very private. I mean, you don't have all those things, you shouldn't double punch, in your head, and you know, you have some notion, this is my private space, I'm exercising my right as a citizen. So it's a little bit like that with testing. It's an odd experience. So, anyone care to volunteer, here? If not, I can help us move forward on it. But, what would you naturally choose here, as you read the passage, the little grunts in the story are -

Student:

Well, what I did was, go back to the story to find the word "little grunts" and then try to use the clues. The clue that jumps out is "there." So it's something coming from the alligators, and really, "noises" is the obvious choice.

Clifford Hill:

You did exactly, and we did interview the test maker on this, and that's what he wanted the test takers to do. He wanted them to process the quotation marks as identifying quoted material in the passage, to go back to that point, and then take a look at how "little grunts" was functioning in the text. And so, since the word "there" is modifying "little grunts," and since "there" refers to baby alligators, your tendency to want to associate the "little grunts" with the baby alligators, which have to do with some interesting features in the passage, has to be suppressed. Otherwise, you would have a tautology: baby alligators, baby alligators. He wants that kind of very tight reading. And we interviewed him on this, this is Walter MacGinitie, and that's what he wanted for this particular task. And you did it beautifully. You took the quotation marks, you went back, you found the place in the text where that material was, you looked at the local context, you saw that "baby alligators" can't be substituted at that point, so you went to "noises."

Now, let me just say to you that when, most of the children do choose baby alligators here. And many adults do, as well. And there was one little girl, when she discovered that the right answer was "noises," she was really upset. She said, the mother alligator didn't hear "noises," she heard her "little grunts." And she felt very strongly that from the point of view of mother alligator, as she is represented in this anthropomorphized form in the passage, her experience of her babies, it couldn't be "noises," because of the relationship of the mother to a baby. That would never be thought of as noise. I wanted to tell her, you yourself have not had babies crying at night, because babies can sound like, when they cry, they can sound like noises, but she really got into the imaginative texture of the passage and rejected "noises" because it didn't fit.

Now, what I'd like to do - and you did a perfect job of identifying what the test maker wanted - I'd like to kind of take you through, if you could go to the next slide. We did a lot of interviewing with children, lots of experiments with them, to try to get at their processing. And what I'd like to do is go through some features in the passage structure and in the task structure, in a kind of cognitive schema which supports, actually, the choice of baby alligators here.

So, if you could just go to the next one, we'll look first, at the passage structure, and I mentioned to you that there's this anthropomorphized perspective, we might, just for fun, go to the next one and come back to this, because there's this neat New Yorker cartoon that I always love to show on this one. I'll read it, I'll assume you can read it, but, "Whenever Mother's Day rolls around, I regret having eaten my young." And you see the two mother alligators having their tea, and...so, to go back, the main point of this passage is a kind of contradiction, and a lot of writing for children about nature is like this. You may think X, but in fact Y, because our normal expectation would be that when these little baby alligators are born, the mother will be able to see them. But this whole passage is to really give the kids some new information, that the only way the mother alligator can make contact with her babies is through hearing, because they're buried under the pile of mud and old leaves.

And one of the things we do, when we do research with children, we often say, well, tell us what you read, and here's something we got from a child. Typically, in addition to words, you get nice pictures, too, because they have a kind of visual imagination. And this little girl shows the mother alligator here, saying, "If only I could see my babies." This little girl was nine. Here she shows the leaves kind of blocking her vision. Notice she brings the leaves almost up in front of the mother's eyes, and then you have all the babies coming out of the shells with their own grunt-grunt, grunt-grunt stuff going on. And then she wrote her, kind of summary of what she'd read, "the mother alligator can only" - and notice she initially puts "see", because, you know, that's the normal expectation you would have for the contact between a mother and the baby, and she marks it out, and goes to here, "the mother alligator can only hear her babies once they are born, their little grunts come from a pile of mud and leaves, where the mother alligator laid her eggs."

So, from the point of view of the passage structure, there's a very, very strong holistic focus on the mother experiencing her babies only as "little grunts." That's how she knows and makes contact with them. Now, there's a very interesting sentence in that passage. If you just go back to it, Lisa, we might look at it. And we discovered that this was really a very, very critical sentence in cueing children into an association of little grunts and baby alligators. And we actually, the kinds of experiments we would often do, before kids would read a passage, we would take something in the passage that, or from the task, that we felt was volatile, that might get the kids moving in a certain direction and play with it. So, a typical kind of experiment we would do is, we would isolate this sentence, pull it away - they hadn't read the passage at that point - and we would take out "grunts" here, and we would just have "their little blank are coming from a pile of mud and old leaves." And then we'd have kids, without any context, fill in the word for the subject there. And what they would fill in would, of course, be not a word that represents sound, but rather, a word that represents a real creature that can move. Because the word "little" typically modifies something that has bodily extension, and of course, "coming," typically, as a verb, is used with something that can move, and we think of something that, having volition and agency, and so forth.

So, this whole frame, at the local level, is primed very much for kids to think of the "little grunts" as actually referring to the baby alligators. If you look at those local linguistic cues, I can read some quotes from the book, which kids said. One child, when discovering that the target response was "noises," responded indignantly, "How could sounds come? They cannot walk or anything." Another kid said, "So baby alligators meant that the babies were coming up from a mud pile, and the mother alligator can't see them." Another child said, "Their little grunts are coming from a pile of mud, means, the baby alligators are coming from the mud." He rephrases it and focuses in on that.

If you can move forward, once you get to the task two, and a lot of our battle with test makers is that the task structure itself does not make sufficiently explicit what you're supposed to do. Now, you did it beautifully. You're probably a good test taker. But kids, typically - we did a lot of experiments on this - think of quotation marks as kind of names, nicknames, because often, that's how, in their own reading material, And we did some studies, in reading material for children quotes will often mean names, and nicknames, and so forth, so they associate that. And we went through the whole Gates-MacGinitie test too, and found that quotes were not, typically not, used to recycle material from the passage, even though they claimed that the quotes were performing that function.

You used it well, you went back and focused at the right place in the passage. But, typically, they don't, in the test material, and typically, in children's books, the quotes are kind of like names, often affectionate names, "Chicken Little," "Little Bo Peep". These diminutives are there to express affection in children's literature, so people have a very, very strong association between little and some fun word in quotes, for referring to real entities.

I can go on, our point is that when you kind of work with the task structure, get rid of those quotation marks, get rid of the word little, get rid of what we call deictic elements, which seem to refer to how grunts are functioning within the story itself, that what the test maker was really concerned with here, is whether or not children can identify the word grunts as belonging to the category of noises. I mean that's what they're really after here. Because in the context of the story, to redefine grunts as noises is not very sensitive to the overall style of the passage. It's a kind of meta-linguistic exercise, or a vocab, it's not quite vocabulary, because you're not asking for the meaning of grunts, you're asking for grunts to be categorized within some larger class. Now, if that's what you want to know, ask the kids that. Don't throw it in to a reading exercise where the whole stylistic texture pushes the children somewhere else.

When we look closely at this material, this happens all the time, these tensions between low-level operations and more constructivist responses which the distractors stimulate. And as the distractors interact with the passage material itself, they get very powerful interpretative frames going, which, when we interview the kids, it makes good sense, but they don't get any credit on the test.

Now, I have a question.

Student:

Yeah. I'm not familiar with elementary school, and kids learning different things at that level, but when did they learn how to diagram a sentence? Because that's the first thing I did. And I don't know if fourth grade was that, you know, that time, or, I can't remember.

Clifford Hill:

Yeah. Well, of course, a lot of, I certainly diagrammed sentences when I was in school. I find that my children didn't, going through New York City schools, things, it's not necessarily a technique that's always used. But, you know, what you're really focusing on is the looking at the structure of language carefully, and what are the techniques that are used to get kids thinking about how language is structured, and certainly, that kind of thing is around in the curriculum. There has been, over the last couple of decades, a lot of attempts to have more of a whole language environment in early childhood education, and that whole language environment has encouraged a style of reading which doesn't get rewarded on the kinds of tasks which are on tests for children. There's a huge tension between the model of literacy that the test makers are working with, which I do call a restricted one, kind of a strict information processing model, as opposed to a constructivist model. Because when we process language, whether in oral or written form, we're making very active inferences. We're drawing on our real world knowledge. It's just not that tight processing of language structure, it's making meaning, and these tests don't reward making meaning very much. They reward, really, kind of structural analyses, low-level operations. Now, you could build a test like that, but don't put in as the distractors, the multiple-choice, stuff that encourages these other operations.

Now, our own feeling is that historically, test makers have had pressure to build tasks that will divide the population. And over time, they've come up with certain types of configurations. This book has a number of configurations, I'm just giving you one. I can give you, I can just describe one or two, very quickly. A very typical thing, we'll have a passage that will start the kids moving in a certain direction, building a certain interpretive frame, and then the passage will shift, and they will have to go to another frame. But there will be a task that will be built in such a way that the target response will be the second frame, and the distractor will be built on the first frame. And these passages are very short, and the shift doesn't have much textural support.

Now, Piaget, in studying children's thinking, described a property called syncretism, and that is, that once a child builds a frame, they're very ingenious about absorbing new information into that frame, and kids just bomb out on these things. From a developmental point of view, I wouldn't make that kind of task. But it gets a lot of rewards for the test maker, who has to divide the population. Because if you build tasks that don't divide the population, they don't pull any weight, they don't make it onto the test. So, over the years, the test makers evolved. And I don't know how conscious they are of this. I'm not even accusing them of necessarily having, somewhere, a bag of tricks, and having all of this worked out, because when I interview them, I'm not struck that they're being quite as aware of what they're doing as I would hope they would be. And that's maybe a nasty comment to make, but I don't feel there's a highly sophisticated awareness of what they're doing always.

But this multiple choice format has been around quite some time now, and when the high-stakes movement started, the implication was, this kind of testing would kind of recede, and more authentic kinds of tasks would come forward. We're going to go to the writing component shortly, where we do see something more authentic. But on the reading components - and they're still around on all these language arts tests across the states - it seems to always be multiple choice, and nothing but multiple choice. No other type of task to get at reading comprehension.

So that's kind of my spiel on, oh, and one last thing. I promised you something on the cognitive schema. And this is, when we hear a sound, like if, all of a sudden, a fire engine were to go by on the street, we wouldn't think, aha, that's the sound of the fire engine. It's the fire engine. A jet goes overhead, you don't, aha, that's the sound a jet engine makes. It, you process it as the thing itself. The baby crying at night; it's the baby. So, there's a schema there whereby the sound and the creature emitting the sound are brought together, and there's so much in this passage that encourages that schema to work in the processing of the information.

This is just a nice little interview excerpt. This kid was ten years old, little girl. She speaks first, she's explaining why she chose "little grunts," and that "little grunts" in the story were baby alligators, because it says, "the mother alligator heard her babies but could not see them, the little grunts." They were grunt babies! She runs the two words together. They just meld in her mind. Pretty articulate. Mm? And why couldn't the answer be anything else? Well, it certainly couldn't be "eggs," very unusual for eggs to make grunts. "But they could have been "noises."" Well, why didn't you choose noises? "Well, baby alligators were much more to the point." If someone wished to choose noises, could you say they were wrong. "Well, I would think they didn't read very well, sorry."

The book is filled with the voices of children struggling with these tests. I, you know, it probably is not the kind of book you, unless you're into this kind of research, but I did bring something on the book, if any of you are interested, and if some of you teach in schools where people want ammunition against tests, and they want to take close looks at the tests themselves, this book does it. It goes in deep, it goes in deep to the material of the task. It goes in deep to the ways kids process the information in the text.

We might just end. The final chapter of this book shows a lot of what we call passage task configurations, because from our point of view, the text to be understood is not the passage. It's that configuration of all those texts with all those choices in relation to the passage information. There's a very complex built up discourse, because the kids have to work through all those choices, their virtual sentences. They have to make, they have to fill the choices in with the task stem, and really work with all that information in relation to a very brief passage, and this is a very common passage task configuration where there's a larger, kind of communicative, holistic pattern of reading that will lead to a choice. And we call that a communicative distractor. But then there's a very, kind of a communicative operation, some incongruent passage detail that will come down. And you do a technical operation of structural properties of language.

The book is filled with just tons of this kind of stuff. And I could find it on ELAT, but I'm not going to do the work, because I don't think I would be able to get it out easily. The problem is, when these test makers block you from showing the actual test material, you lose a lot of the power in your analysis, if you can't really show the material when you go to publish, particularly if you're trained like me, to look closely at language. And it just, it kind of enrages you that you see these problems in the language, and you can't get it out there. But there's a battle going on with test makers.

A lot of societies are much more open than our own. I work a lot in international education. In Japan, the morning after a high-stakes test, it's in the newspapers. The test makers have to defend the particular tasks they've set. The public can write in; there's public discourse around these high-stakes test materials. American society, as many of us are still learning, has, despite all of our claims to be such an open democracy, there are a lot of constraints on what goes on in our public discourse around stuff that's high-stakes.

Now, just to relate this to the current political mess, I think the kind of confusion that certain people obviously experienced when they were in that enclosed voting space, high-stakes, probably under time pressure, a little bit like on tests, because you know, often there are long lines. It's like when you're at the ATM machine, you know, there's a long line, you want to get on with what you're doing. And the processing error some of those people made, it's not that they can't read, but you know, if they came at that ballot with a traditional western model of literacy, they found in the upper left corner, the Democrat/Republican contrast. If they vote Republican, push the first hole, Democrat, the second hole, and then they walked out and left it. People, and a lot of the older people who made that mistake are not into digital literacy, they're not used to scanning a larger field of information where the orientational fields worked differently, because on that ballot, you had to do both left/right and right/left. But again, you see how, kind of a small thing has large consequences? That's what we see all the time on the tests. The little things just throw kids off. The framing they do, and ramify, and have large consequences in these kids' lives, if they're held back, because of how they perform on these tests. Okay. Let's leave the reading and move over to the writing component - yeah?

Student:

Well, one-word references, [unintelligible].

Clifford Hill:

Well, I personally am kind of a postmodernist when it comes to text. I think text is highly indeterminate, and I think it's very, very difficult to build tasks based on very tight ways of processing information, which are really fair to a very diverse population. I'm just very nervous about it. It's not the way I understand how text works. Text is very indeterminate, there's multiple ways of framing it, and I think it's very, very hard to build. What interests me in assessment, we'll go that way in our new work, I'm kind of interested in people who can process text from a critical point of view, and bring larger bodies of knowledge to bear upon what this particular text is saying. If you do that on these tests, you get in trouble, because if you bring in larger frameworks of knowledge, you're not sticking just with the passage information, and you get yourself in trouble. But for me, critical reading is being able to integrate what you know with what's on the page, often seeing multiple meanings for what's on the page, rather than some one meaning.

So, I have a whole different take on how text works, and this way of doing it doesn't interest me very much, and as we move forward in this lecture, I'm going to show that the effects on the classroom, I think, are very, very bad, because in order to coach to the test, you're teaching a way of reading which I don't think is the way we really read, or the way we should read. So I'm very nervous about the model of reading which is reified in this kind of material. But this is a big discussion, I just gave you a very quick take on where I come to it from. It's not that we're not going to build some assessment. We are. But it's going to be, it's going to allow for constructivist work from students. We aren't going to ask for just tight, tight, tight things. Now, in order to work with technology, they have to be able to do tight operations, you know, to handle technology, and there's a lot of knowledge and skills, which means that they have to know how to do things. It's not that there's not a lot of procedural knowledge, you have to know the structure of language to make meaning, but reading is so much more than that, and as we get into this digital literacy age, the complex knowledge and skills that kids need to find information, integrate it, communicate it, it's so much more complex. And unless we build tasks that are congruent with the knowledge and skills which kids have to have to participate in an age of digital literacy, there's no point in doing it. The assessment has to really, generally, be high standards stuff. And to me, this is not high standards type of stuff, these multiple choice tasks on reading.

Cite This Article as: Teachers College Record, Date Published: December 07, 2000 https://www.tcrecord.org ID Number: 10662, Date Accessed: 10/22/2021 10:17:07 PM Purchase Reprint Rights for this article or review