The Impact of Evaluative Research on Educational Programs for the Poor
by Albert E. Myers - 1970
Professor Myers' outspoken article is based on a talk be gave at the American Psychological Association in September, 1966. His description of the infinitesimal impact of scientific research on action programs is certainly as accurate today as when it was first written, as are his pictures of administrators in search of justifications and teachers fearful of evaluations. Research and evaluation have been welcomed, he says, only by those compelled to submit reports to the Feds. If research programs are ever to have local impact, researchers must cease to be outsiders and become involved with the systems they are attempting to serve.
It is frequently pointed out, usually by a superior-sounding researcher, that there is an extensive lag between the time researchers develop some usable information and the time that practitioners incorporate this information into their procedures. The cultural lag hypothesis, with its connotation of an inferior, less highly developed society stumbling behind a more sophisticated leader, finds favor with the researchers. There is an implementation gap and it is the fault of the implementers.
On the other side of the fence it is not so facetiously said that social scientists are like the weather: if you don't like the things they are offering now, just wait awhile, and they'll change. All points of view are represented at every point in time and most points of view are the dominant position at some point in time.
Both of these notions are, of course, overstatements. Both contain more than just a germ of truth. Together they portray a certain incompatibility. It is an incompatibility that is easily seen and widely noticed.
My paper is going to focus on those forces which make it difficult for top-grade scientific work to be of value in action programs. This is necessary because the impact of research on these programs, which is ostensibly my topic, seems to me to be small, microscopic, infinitesimal; there may be none.
The usual reason that people give for a service not being as effective as desired is that there isn't enough of it. "We need more staff" or "We need more money" are songs heard more frequently than The Star Spangled Banner. I had an opportunity this last year to hear a group of middle-level administrators discuss something like a dozen basic problems with which they were confronted. It was an appalling scene to see the representative of this group give a synopsis of the group's recommendations. Every issue was handled with either "That is not our problem" or "Give us more money." Naturally, the last thing that anyone would feel like doing, after hearing the way these administrators would tackle their problems, was to give them more money.
Somehow, you would rather burn it than give it to them, although they seem to be approximately equivalent actions.
It may be true that we need more research; everyone says we do. It is not at all obvious to me, however, that more research would mean more impact. More research obviously means more something—but more what? In order to answer that question we have to look and see what role research plays in this drama called the War on Poverty. Let us begin by looking at the people involved, the administrators and the program people.
Research and the Administration
The administrators include the central decision-making authorities like the executive director of the community action program, the superintendent of schools, and their staffs; the middle-management group, like the individual program directors and the principals of schools; and on the periphery in the operational sense, but centrally involved in all important decisions, is the political body.
How does research fit into their lives? (During this discussion, by the way, let's assume that these are capable people trying to do a good job. Let's not write off any shortcomings as representing personal deficiencies.) There is a school of thought which says that good programs are the by-product of good administrative decisions. This is the bureaucratic ethic. It is a point of view which is widely, if not universally, accepted by administrators. The role of research in this scheme is to provide information so that sound decisions can be made. Research is used to remove some of the guesswork from decision-making.
As a point of view I suppose it is fine, but it just doesn't describe how administrators behave. I have yet to see an instance of where a researcher has been asked to provide information for a decision. This is not to say that the administrators do not have any information which is relevant to the decision they must make. It simply means that the channels for getting whatever information they do use do not go through researchers. They expect different things from their research capability.
If the researcher is not asked to provide information for a decision, what is he asked to do? Well, he is asked to provide information about things for which decisions have already been made. In short, he is asked to justify the administrators' decisions.
It is not that the administrator looks the researcher in the eye and says, "Justify what I have done." Remember, we are talking about capable people trying to do a good job. The administrator knows quite a bit about the program in question. He probably knows its strengths and weaknesses. If he knows it's a good program, what else is research going to show except that it's a good program. If it's a poor program, he expects the research to show that. In other words, he anticipates the outcome of the research and, therefore, has no need to rely on the research act itself. What he wants is the respectability of a research report.
This is neither a particularly cynical nor naive assessment. It is quite in keeping with what researchers have given administrators in the past. Most of the outcomes of research projects which they have seen in the past have either been irrelevant or predictable or both. They have been treated, therefore, as being irrelevant and predictable. Unfortunately, imagination-free, trivial research only reinforces this vision of the researcher as a eunuch; he's useful to have around, but he's not one of the boys.*
The Program People
The other group of people that presumably could benefit from an evaluation capability would be the people who are actually running the program. Theoretically, research should help them to do their job more effectively. I'm afraid, however, that these are the forgotten people when it comes to putting the research to good use.
There are a variety of reasons why this is so. Let me mention just two. First, they are not so all-fired-up about this idea of research in the first place. They have the appropriate, socially-desirable attitude about research in general: "We need more research; we need to understand more, etc." However, they are not too keen on research in particular when the "in particular" means looking at them.
In one study I did which called, in part, for interviewing teachers, I included a couple of questions about research in the interview schedule. The biggest blank of the year had to be the answer to the question, "What can research do for you?" I have even had teachers tell me that there is nothing that they feel they have to know more about and the only obstacle was the size of the class. From this point of view, money spent in research and evaluation is a waste.
It is obvious that any evaluator, be it a test, a supervisor or a researcher, is a threat. As such, people are on their guard when the researcher is around. Until the research man has had time to demonstrate that he is there to help them and not to rat on them to superiors, he is something to be worried about; he is the inspector-general. If the researcher defines his job to be confined to doing research and writing reports, he will provide justification for this attitude.
This brings us to the second reason why the program people get forgotten. It is not generally considered part of the researcher's responsibility for him to be involved with the implementation of his finding. He is not an active agent of social change in either a grand or microscopic way. As a result, the researcher contributes to the impactlessness of his own work. Furthermore, I think it alters his view of what research questions should be asked and that the change is for the worse.
Research and evaluation have been incorporated into the bureaucratic system in such a way that no one is hurt—or even affected by it. On numerous occasions I have been asked by a member of the school system if I had data relevant to a certain issue or program. The reason this person needs the data is because he is writing a report. Frequently the recipient of this report is the local community action agency. The reason they want the report is because they have to report to the funding agencies (e.g., O.E.O.). It is the rule, not the exception, that no decisions are made in this process. These are exercises in report writing.
Evaluating for Washington
The intervention of the federal government into this scheme has intensified the difficulty on one hand while ameliorating it on the other. The mania for evaluating federally financed programs has generated a willingness for something called research and evaluation (just as the academicians distinguish between pure and applied research, non-researchers distinguish between research and evaluation). But it has turned the heads of the top administrators towards Washington. I have been asked only once to assist program people in understanding the nature of what they and the program have on school children.
I have had, on the other hand, many requests to evaluate programs because the federal or state government wanted an evaluation report. This is particularly true of those programs dealing with the poor. There seems to be a rather pervasive paranoia that exists in Washington and which is somewhat justified. The Feds, as they are lovingly called, are worried that their money will not be well spent. It is easy to be understanding and to sympathize with their position. They know that local politicians will substitute federal money for local money if they can get away with it. They know that local educators will repeat many of the old sins if they just give them the money. They are vulnerable to public controversy and criticism. They know they must shoulder the responsibility for mistakes made at the local level which are completely beyond their control. Remember what one dropped postcard in Africa did to Peace Corps officials a few years ago?
Yes, it is easy to understand the problems and tensions that federal administrators and legislators must face. But let's look at what happens at the local level. The federal dollar becomes preeminent. Its local administrators become super-conscious of satisfying. It changes the function of administrators from being unit leaders to being synapses in a communication link. It becomes their job to report the statistics of their program. That would not necessarily be so bad if somewhere in the process there was some decision making related to all this effort. Unfortunately, it seems to me that report making becomes an end in itself.
The Dangers of Legitimacy
But let me make a very important digression here to illustrate the process I'm concerned about. A colleague of mine at the Psych-Educational Clinic at Yale, Dr. Murray Levine, has been doing some historical research on the variety of social service movements that were taking place at the turn of the century. This includes the emergence of social work, visiting teachers, settlement houses, juvenile court and the first psychological and psychiatric clinics. Dr. Levine's work demonstrates a consistent, if not inevitable, trend. These movements were begun in an atmosphere of intellectual reform; their basic characteristic was that they imposed dramatic innovations in the way clients were dealt with and thought about. They were fundamentally radical.
The radical character of these groups began to subside as they became organized and moved towards having a professional legitimacy. The development of professionalism was accompanied by a reduction in their willingness to innovate and the loss of the aura of intellectual reform. Yesterday's radicals had indeed become today's reactionaries. Sadly, the effectiveness of the service suffered with this change.
This is not the time to discuss whether this is an inevitable change or not. It is sufficient to say that the betting odds are that successful community action programs will ultimately become conservative members of the professional and political establishment which will try to inhibit social innovation just as the dominant professional forces today try to inhibit innovation. Saul Alinsky's Back of the Yards movement is probably one of the more well known examples of how a radical movement can go middle class. It is almost naive to doubt that the War on Poverty and the community action programs which form the battle lines will become middle-class programs designed to meet the career needs of middle-class people. The War on Poverty has a fairly unique distinction, by the way, of being indicted on that point from the very beginning. Members of the political left and right have frequently accused the poverty program of being middle class.
This reminds me of the incident where a particular poverty program had both middle- and lower-class persons employed. A reduction in funds required that the program be curtailed somewhat. As a result, the job functions being fulfilled by the lower-class persons—job functions which were labeled non-professional—were abolished and the lower-class employees were laid off.
Administrative vs. Service
If left alone, community action programs will probably follow this trend. From where I sit, federal involvement is acting to hasten the process. It is providing an additional force in the direction of causing an administrative orientation to replace a service orientation.
The recommendations of the House of Representatives' committee dealing with anti-poverty legislation certainly must be viewed as distressing with respect to this, as well as other topics. According to accounts in the popular press, the committee has shown an increased desire to control the local programs by categorizing the way money should be spent. Predictably, the categories which seem most important to the Congressmen are those which are the safest politically.
The role of research in this arrangement is parallel. It is increasingly asked to fulfill the demands of justification. I am prepared to take the strongest possible stand on the value of this type of research. I know there are many who disagree with me violently, but I think that justification research has virtually no positive impact (not value, but impact). Furthermore, it is a brand of tokenism that discourages the development of something more meaningful.
The usual rebuttal to this strong position, which is probably mildly overstated, is that we will never be able to discard ineffective programs if we do not have evaluative systems. There are two major flaws in this argument. First is the conception that the basic unit for study and consideration is the program. Our interest must not be in accepting or rejecting programs in the form in which they exist. This leads to a confusion to which the history of educational research can certainly attest. Since every program that has the same name in different locales is run differently, research on programs with that name is ambiguous. Sometimes it's good and sometimes it's bad. Take the example of homogeneous versus heterogeneous grouping in classrooms; that certainly is clear-cut enough. But after forty years of research on the topic, I doubt that there is a single point on which researchers could reach a consensus.
For my own taste, the basic unit would be the decision that is made about the person being served. In my research in the schools, I have been impressed at how . . . slipshod . . . the decision making of program people is. There are many ways of slicing the pie though and other people will, of course, choose some other basic units.
The second flaw is the notion that programs are discarded because the research indicates they are no good. Programs are initiated and terminated because of the people and ideologies involved. The amount of emotion that can be aroused by a research effort is almost insignificant compared to enthusiasm or vehemence that an individual can have towards people or a program. A program is "phased out" at this point in the game because people are disgusted with it, not because research shows it to be wanting. Long before a program is dropped because the data demonstrate its inadequacies, the people involved will be in the process of diminishing its scope or importance. The research may help them justify their point of view, but it probably isn't going to change their mind in any basic way. On this point I would be interested in knowing the dynamics of why the Higher Horizons project in New York City was dropped. It was a famous program, emulated and described the country over. It also had a fat research report which could be summarized simply with "no measurable gains were found." The fact that it was dropped tells me that people in the system were unhappy with it, and not much more. What role did the evaluation play in the demise of the program? It would be instructive for us to know. In my experience, research does not turn people from being supportive to being antagonistic towards a program, or vice versa. It simply provides ammunition for that camp whose position the findings favored.
This raises immediately a basic difficulty faced by researchers. Every issue, every program has its two camps. There are the supporters and the detractors. There are always, of course, many people in the middle. The researcher can easily be dragged into the fray. He can become a participant, whether he likes it or not, in the internecine struggles. In my own case, I was identified with the central administration staff ("downtown") who has had a running battle with the "old guard" teachers and administrators around the city. In such an atmosphere it must be obvious that objectively done research will not be objectively interpreted, and perhaps more severely, that under this kind of pressure, over time, the researcher might stop being objective. I know I had an instance or two where my own sentiments began, to put it most kindly, to remove the appropriate benefit of doubt from my vision.
The effects of this problem were perhaps best dramatized by one project I did last year. My position as a result of research was:
Everyone in the world agrees that Project A is a good and necessary thing. I feel that Project A is good and necessary for the children in our city. I also feel there are certain shortcomings in the way we are administering Project A; it is inflexible. We must make certain changes.
It was uniformly concluded that I was in favor of removing Project A from the schools. My report was interpreted in terms of an existing struggle rather than by the terms of the content within it.
Researchers as Change Agents
It might seem that I am laying the basis for proposing that researchers withdraw completely from internal associations or, even further, that outside researchers be called in. Time doesn't permit me to develop the theme, but I would argue quite strongly that the only way for a research program to have real local impact is for it to be involved with the operational system it serves. Outsiders inevitably get a more superficial account of what is happening. In essence, I feel that there can only be minimal impact as long as research is viewed as an external, threatening face that the program people must contend with. It is not enough that researchers are paid and hold titles within the organization they are servicing, although that seems essential. They must see themselves as agents of social change who are in alliance with the persons running the program. Not threat-eners, but allies; not aloof, but involved.
I would like to conclude on a poetic note. Kingsley Amis has written a poem, After Goliath, which reads as if it were inspired by watching a new, radical innovative program tear into a school system. Goliath, representing the conservative professional establishment, has for his fans:
Aldermen, adjutants, aunts,
Administrators of grants
Advisors about careers
David's supporters include:
Academics, actors who lecture
Apostles of Architecture. . .
Angst-pushers, adherents of Zen
The poem concludes with this sobering thought:
. . . even the straightest
Of issues looks pretty oblique
When a movement turns into a clique,
The conqueror mused, as he stopped
By the sword his opponent had dropped.
Trophy, or means of attack
On the rapturous crowd at his back?
He shrugged and left it, resigned
To a new battle, fought in the mind,
For faith that his quarrel was just,
That the right man lay in the dust.
* I must, in all honesty, report that a colleague in the school system told me that I have had more impact than the comments suggest. I must also report that I think he has overestimated my impact.