Back to the Future: Implications of the Neopositivist Research Agenda for Adult Basic Education
by Alisa Belzer & Ralf St. Clair - 2005
Federal educational policy, funding, and legislation are currently forwarding a research agenda described by the current administration as "scientifically based." We characterize this agenda, described in multiple official documents as "the gold standard," as neopositivist. We consider the ways in which this research paradigm and the methods that accompany it (i.e., randomized field trials) are particularly problematic for knowledge generation in adult basic education, an important subfield of adult education. We focus on three areas of concern that exemplify in significant ways the limitations of scientifically based research: the contexts of adult basic education, the ways in which practitioners use research, and the state of the knowledge base.
Federal educational policy, funding, and legislation are currently forwarding a research agenda described by the current administration as scientifically based. We characterize this agenda, described in multiple official documents as the gold standard, as neopositivist. We consider the ways in which this research paradigm and the methods that accompany it (i.e., randomized field trials) are particularly problematic for knowledge generation in adult basic education, an important sub field of adult education. We focus on three areas of concern that exemplify in significant ways the limitations of scientifically based research: the contexts of adult basic education, the ways in which practitioners use research, and the state of the knowledge base.
The first U.S. doctorate in adult education was awarded by Columbia University in 1935, and by the mid-1960s, 323 had been awarded (Houle, 1964). At that time the Reports Committee of the Commission of Professors of Adult Education prepared two volumes to summarize and provide direction for what was characterized as a new field of knowledge production. In the second of these, Adult Education: Outlines of an Emerging Field of University Study (Jensen, Liveright, & Hallenbeck, 1964), also known as the Black Book, a number of academics came together to define a desirable research agenda, arguing that
Vast gaps exist in knowledge and theory about development and maturation beyond youth, adult learning processes, development of programs, and in other crucial areas. Research in these areas is still only a secondary interest of isolated social scientists and educators, and as yet there exists no mechanism for systematically planning, stimulating, and disseminating adult educational research. (P.viii)
Forty years later a similar statement could be made: Though the quantity of adult education research has increased, there is still little sense of a systematic plan. Yet the Black Book also argues that the nature of adult education as a practical discipline inevitably leads to enormous diversity within the research enterprise. This can be viewed as a weakness resulting from poor definition or embraced as strength of a pragmatically oriented and inclusive field of practice and research. However, it seems that if there are truly vast gaps in such a range of topics, only the most diverse range of methods is likely to address them.
In this discussion we examine recent developments in educational research that tend to reduce the diversity of empirical research activities by promoting one formexperimental randomized trialsabove all others. We characterize this as a return of neopositivism and discuss the implications for adult education research and practice in light of our belief that diversity in strategies for knowledge production is a fundamental strength, indeed an absolute necessity, of adult education. In particular, we focus on adult basic education (ABE) as an important sub-field of study in adult education. We define ABE as a broad array of educational services for adults who are seeking to improve their basic reading, writing, and math skills or prepare for the General Educational Development (GED) exam or high school alternative and for nonnative speakers who wish to increase their English language fluency. We further elaborate on these ideas by exploring the current policy and legislation as well as curriculum requirements indicating a contraction of what counts as legitimate research methodologies in federally funded education work, discuss some of the critiques made by educational researchers of this narrowing of method, and take a detailed look at what this means for research and practice in ABE.
THE RETURN OF NEOPOSITIVISM
Over the last few years there has been a concerted effort to shape the educational research agenda, perhaps more pervasivelyand more effectivelythan ever before. The aim appears to have been the establishment of neopositivism as the guiding philosophy for research and practice in education. The term neopositivism dates from the turn of the 20th century and signals the pre-eminence of a quasi-scientific model of knowledge. It rejects Metaphysics, and regards as basic the principle of verification, i.e. a process of definition of truth or untruth of a postulate by empirical proof (Baroni, n.d.). In other words, value-oriented, culturally based, and to a large extent, political concerns are not important to neopositivists; what matters is whether an argument can be proven by empirical data. Only that which can be shown empirically counts as knowledge, and only knowledge of that type, it is presumed, can really help to improve our lives. For example, the question of what education should or could achieve cannot be addressed through neopositivist approaches, since it cannot be tested; all that matters is what is achieved. Without careful balance, neopositivism can lead to a conservative approach to research.
The move to neopositivism reverses several decades of advances for alternative research approaches. By 1985, researchers were reading in one extremely influential textbook that they lived in a post positivist era in which data and theory were bound together by the understanding that all information is shaped by the eye of the beholder (Lincoln & Guba, 1985). The agenda assumes a need to increase the credibility of educational research and the success of education practice through a coordinated endeavor to generate a body of highly valid knowledge (using medical and natural science standards of validity.). This paradigm shift gained strength from postmodernism, post structuralism, critical, and feminist approaches to knowledge generation. In each of these approaches, the means by which knowledge is created is assumed to be as problematic as the knowledge itself, and no single research method guarantees truth. The lack of certainty involved in interpretivism can be disorienting and can lead to frustration for those who have a desire to know the best way to pursue a goal. Interpretivist approaches offer a more nuanced, potentially richer understanding of phenomena, whereas neopositivism offers apparently clearer answers with more sense of certainty.
Current attempts at the federal level to reform the research agenda within education are pervasive and extraordinarily powerful. The same agenda is being forwarded in policy and legislation on one hand and within curriculum requirements on the other. The agenda assumes a need for educational research to be a coordinated endeavor that can generate a body of highly valid knowledge (using medical and natural science standards of validity) to increase the credibility of educational research and the success of educational practice. In other words, neopositivism is being advanced as a way to answer, once and for all, the problems of practice and to ensure the best outcomes. Although considerable reservations have been expressed about this argument, its pervasiveness and consistent application throughout the rhetoric of government documents related to educational policy, research, practice, and funding carries sufficient weight to have a profound impact on educational research.
The most useful place to begin exploration of the neopositivist turn is the legislation shaping the educational field. The key term signaling the administrations research reform agenda is scientifically based research, which occurs dozens of times in recent legislation. This term is conceptualized in a particularand consistentway. The No Child Left Behind Act of 2001 (2002) states:
The term scientifically based reading research means research that
(A) applies rigorous, systematic, and objective procedures to obtain valid knowledge relevant to reading development, reading instruction, and reading difficulties; and
(B) includes research that
(i) employs systematic, empirical methods that draw on observation or experiment;
(ii) involves rigorous data analyses that are adequate to test the stated hypotheses and justify the general conclusions drawn;
(iii) relies on measurements or observational methods that provide valid data across evaluators and observers and across multiple measurements and observations; and
(iv) has been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective, and scientific review. (1208, part 6)
Though somewhat lengthy, this definition provides a clear insight into the standards for acceptable research. Section (A) is a general statement consistent with almost any approach to research, but section (B) has some significant implications. Throughout the definition there is an implicit assumption that good research follows a hypothetico-deductive model, and the use of wording such as rigorous data analyses that are adequate to test the stated hypotheses and valid data across evaluators and observers and across multiple measurements and observations clearly underlines the assumption that research is an objective, technical activity. The definition is extraordinarily limiting, especially for a field such as education, in which the notion of repeatable experiments with consistent general conclusions is often unsustainable.
The Workforce Reinvestment and Adult Education Act (2003), the latest legislation to authorize federal funding for ABE (still in legislative process at the time of this writing), refers to the No Child Left Behind Act for its definition of scientifically based research. Interestingly, the National Institute for Literacy, funded by the same act, is also obliged to adhere to this standard. So too is the Institute for Education Sciences (IES), created in late 2002 to replace the Office of Educational Research and Improvement. In the act creating the IES (Education Sciences Reform Act, 2002) scientifically based research is defined once more, slightly differently from the way it is defined in the No Child Left Behind Act. Yet a central point is reiterated: that research requires relying on measurements or observational methods that provide reliable data (Education Sciences Reform Act, 2002, 102, 18). Once again this definition, with its emphasis on reliability, tilts the field toward standardized instruments and control group methods.
Overall, the Bush administration has poured an enormous amount of energy into ensuring that replicable randomized trials delivering statistically analyzable data are seen as the gold standard of research. Policy and legislation consistently push for the universalization of scientifically based research in an apparently sincere belief that this will improve practice, and hence learner outcomes, in an unproblematic linear process.
The production and dissemination of research findings is only one front upon which neopositivist paradigms are being advanced. Federally funded programs are now required to show evidence that they are applying the findings of research that meets the gold standard to their practice. Thus curriculum is a second area in which the federal government has intervened to promote neopositivism, signaled by the recurring use of the term evidence-based practice. In discussing reading instruction, a document produced by the International Reading Association (IRA) argues that
evidence-based reading instruction means that a particular program or collection of instructional practices has a record of success. That is, there is reliable, trustworthy, and valid evidence to suggest that when the program is used with a particular group of children, the children can be expected to make adequate gains in reading achievement. (IRA, 2002, p. 2)
This definition makes a lot of sense: After all, who would want to implement practices without some assurance that they will make a difference? However, the document then goes on to identify those aspects of the research contributing to the credibility of the evidence. It should be objective (data that any evaluator would identify and interpret similarly), valid (data that adequately represent the tasks that children need to accomplish to be successful readers), reliable, systematic, and refereed (IRA, 2002, p. 2). Teachers are also encouraged to examine the generalizability, or fit, of the evidence (IRA, 2002, p. 3). The IRA also states that experimental and quasi-experimental designs provide the strongest evidence of the effects of a program.
This approach to practice manifests itself time and again in the No Child Left Behind Act and the Workforce Reinvestment and Adult Education Act, the two most influential legislative documents for federal adult education funding. On one hand, neopositivist, experimental research is defined as the gold standard for knowledge production, and on the other it is positioned as the only worthy standard of knowledge for knowledge application. It is clear that when researchers do conduct other kinds of research, practitioners are not encouraged to apply the results. In effect, this conceptualization of research may easily block other forms of empirically based knowledge production; in particular, the immense range of interpretivist approaches.
Neopositivism appears to be here to stay, at least for the foreseeable future. The direct implication for people who teach, conduct research, or do some of both, is that certain forms of information will be promoted to the exclusion of others. The remarkably seamless emphasis on neopositivist educational research will have profound implications, and it is critical to understand both what this means and the shaping of knowledge that is occurring. However, the neopositivist turn in educational research has been accompanied by many expressions of concerns, and we now examine these more closely.
REASON FOR CONCERN ABOUT THE REDEFINITION OF SCIENTIFIC RESEARCH
In a summary of the National Research Council report on the nature of scientific research in education (released in 2002), Feuer, Towne, and Shavelson (2002) endorse the notion of scientific research in education. However, they also acknowledge the danger of prescriptiveness and rigidity if a particular notion of research is codified into policy, funding decisions, and law. They argue instead for a broad definition of scientific research that does not specify method and outline a suggested set of guiding principles for research. These include the posing of significant questions that can be pursued empirically, a link between research and theory, methods that are directly tied to the questions they are designed to address, findings that can be replicated and generalized across studies, and full disclosure of data and methods for scrutiny and critique (p. 7). Although it is true that replicability and generalizability are terms usually applied to quantitative study, the overarching idea here is that high-quality research can be conducted using a variety of methods, yet still be defined as scientific.
In fact it is the definition of scientific that seems to be at issue here. The medical model of research (double-blind, experimental studies using randomized field tests) is being positioned by the current administration as the pinnacle of scientific research because of its apparent confidence that such an approach can answer questions with certainty. Only with such certainty, the administration seems to assert, can rational educational policy decisions be made and excellent instruction be implemented. Cunninghams (2001) analysis of the National Reading Panel (NRP; National Institute for Child Health and Human Development, 2000) indicates that the desire for certainty has led the positivists on the panel to reductionism: Their strategy has been to increase their comfort by reducing the questions one is permitted to ask, and reducing the ways that one is permitted to answer them (p. 328).
While the current administration advances its view of scientific research, others take issue with an apparent idealization of science. Some argue that real science is less about certainty than it is about uncertainty (Berliner, 2002; Erickson & Gutierrez, 2002) and that findings are always tentative. Cunningham (2001) asserts that philosophers of science cannot actually reach consensus on criteria for distinguishing science from non-science. In addition, a number of critiques of the medical model have been offered by educational researchers who question the gold standard Medical research is sometimes disputed and not always conclusiveeven here, certainty is elusive. Even if one can assume certainty of results, one can never assume consistency in application by practitioners because the familiar problem of disconnections between research and practice exists in medicine just as in teaching (Silver, 2003). In other words, outcomes are shaped not just by the intervention, but by the intervener. Controlling for this variable, not to mention all the others that come into play in classrooms, seriously limits the inferences it is possible to draw (Kilpatrick, 2001, cited in National Council for the Teaching of Mathematics [NCTM] Research Advisory Committee, 2003). An additional and obvious challenge to the medical model, as Gardner states simply, is that Minds are not the same as bodies (quoted in NCTM Research Advisory Committee, 2003, p. 187). Finally, Erickson and Gutierrez (2002) suggest that if the medical model is to be applied to education, researchers should concentrate their efforts on the potentially deleterious side effects that may result from implementing evidence-based reforms derived from gold standard research.
Overall, the major critique put forward by scholars who take issue with the contraction of what is valued (and indeed, allowed) in educational research and practice funded by the federal government is that it fails to provide a mechanism for addressing critical questions about teaching and learning, without which educators will be unable to fully succeed. For example, randomized field trials cannot address the contextual features in causal processes (Feuer et al., 2002). As Cochran-Smith (2002) puts it, experimentation works for answering one specific question: What works? However, what works? does not get us very far in improving the learning opportunities of all students and does not go very far toward explaining why some children have historically been left behind by the educational system (p. 189). Even before we can know something has worked, we have to know a considerable amount of information about what that something is, how it was implemented, the conditions under which it was implemented, who participated in its implementation, and a host of other contextual data that can go considerably further in explaining why, under what circumstances, and with whom something worksall questions that experimental research fails to address. As Erickson and Gutierrez (2002) suggest, even treatments are contextualized. Without documenting contexts, findings based on causal relationships are unwarranted.
Ultimately, no method is inherently good or bad. Science is agnostic on the issue of whether the data need to be quantitative or qualitative (Mayer, 2000, p. 39), and these two forms of data can complement each other. When applied appropriately, they can effectively answer unique and oftentimes equally important questions. When randomized experimental design studies and qualitative studies that pay close attention to context are combined in meaningful ways, they make for better scientific inferences than either can justify alone (Feuer et al., 2002). Recent critiques of the NRP (National Institute for Child Health and Human Development, 2000) attest to this. In their comments on the Panels report, Pressley (2002) and Cunningham (2001) point out that completing a systematic review of reading research that excluded any studies but those employing randomized field trials had discounted tremendously useful information about best practice and the conditions under which successful teaching and learning occur. In other words, the NRP failed to capture the big picture of how ideas, theories, and practices fit together.
Those critiquing laws, policies, and funding priorities that elevate the randomized trial method above all others argue that it cannot tell us enough to truly improve the educational outcomes for all learners. We believe this argument is valid in all realms of education. However, this plays out in distinct ways in particular educational contexts. We explore why in particular the gold standard is problematic for researchers, practitioners, and learners in ABE by focusing on the contexts of adult literacy education, the ways practitioners use research, and the current state of knowledge in ABE.
THE CONTEXTS OF ADULT LITERACY EDUCATION
ABE, as a field of practice, is highly diverse. Some of the dimensions of diversity include the institutional context, the population served, and the political orientation of the program. In addition, the ABE workforce tends not to share a universal experience of training or apprenticeship (Lytle, Belzer, & Reumann, 1992). Within the curriculum, ABE programs can vary by content, emphasis, or even pace of instruction. Whereas these dimensions of diversity would be unusual within the K12 system, simply because it is so systematized, the historical commitment of adult education to individualized, learner-oriented instruction (Knowles, 1980) makes variety within and among programs the norm rather than the exception.
Though our own experience as literacy instructors and researchers supports our belief in the diversity of ABE programs, there is little systematically gathered data on the topic. The form taken by both general adult education and ABE does seem to depend to a large extent on the particular cultural context in which it is practiced (Merriam & Brockett, 1997), but there are a great many factors bearing upon this form. One of the most striking factors is the relative lack of material resources associated with ABE. One recent study (Smith, Hofer, & Gillespie, 2001) found that 20% of ABE instructors had no computer access, 33% were concerned about the program facilities, 39% did not have their own classroom, and 29% lacked their own desk. Equally worrying were the limitations on educational resources for the instructors: 23% received no paid professional development time, and 32% received only 112 hr of paid professional development time each year. In these conditions an eclectic approach to practice may be a survival strategy.
A study conducted in Illinois 12 years ago (Leinicke, 2000) reports that over 90% of instructors in ABE were part-time, with few opportunities to get together with other instructors for training or support. The study also reports that classes varied by funding source, objective, skill level of new (and departing) learners, likelihood of learner success, student-to-teacher ratio (varying from 15:1 to 200:1), ethnicity, and computer accessibility and comfort. Another study (Sabatini et al., 2000) is more cautious, suggesting that around 41% of instructors are part-time and finding that around two thirds of all instructors have an unspecified credential. This second study did not address the identity of the instructors who responded or the context of their work. A more recent study (St. Clair, Chen, & Taylor, 2003) implies that the instructors are predominantly older White women with substantial experience but little training in adult literacy education. In summary, instructors are being asked to deal with remarkably diverse conditions of practice with little preparation or support beyond a possible initial credential.
It is important to pause before assuming that diversity in programming necessarily means that some contexts are better than others. There is a growing body of theory to suggest that learning, both general and literacy oriented, is not a universal process (Lave & Wenger, 1991; Vygotsky, 1978; Wenger, 1998). Rather, learning occurs through a learner-environment interaction in which the environment includes other learners, materials, instructors (if any), social context, and material conditions. One of the most remarkable manifestations of this contextuality is the degree to which use of languageboth spoken and writtenis a social act rather than an isolated skill.
One of the first studies of this phenomenon was concerned with showing that Black American English was as capable of communicating subtle and abstract meaning as Standard English (Labov, 1969). Until this work, most linguists had assumed that Black English was a degenerate form of Standard English, lacking power and clarity. In a similar, and perhaps even more interesting, vein, during the last decade the relatively new discipline of literacy studies has demonstrated that there are many varieties of social practice attached to literacy and that each is functional within a particular community (Barton, Hamilton, & Ivanic, 2000; Heath, 1983). These approaches press for recognition of the diversity inherent in literacy, both in its use and in its acquisition, and an acknowledgement of the way school literacy is given enhanced status within our society. One application of this argument is family literacy, where the need for great care has been strongly stated. It is all too easy to slip from offering school-centered literacy practices as an augmentation of family practices to insisting upon them as a replacement for those practices, resulting in impoverishment rather than enrichment of literacy within the family (Auerbach, 1995; Tett & St. Clair, 1997).
In terms of research, the overall effect of these layers of diversityof learner, of context, of literacy itselfis to make generalization of statistically derived results difficult to justify. The central principle of neopositivist research, as discussed earlier, is the existence of a single best answer to questions. To reach this answer using statistical tests requires that the test sample (and the control sample) represent the overall population on many dimensions, usually assured by selecting a reasonably large and well-randomized sample. This implies that learners can be averaged in some way and that the differences among them are difficulties to be ignored rather than sources of strength, motivation, and pride. We believe that responsible quantitative researchers aim to take this difficulty into account, and it is not our intent to undermine the value of their work. Our concern is the tendency for policymakers to overlook the complexities of statistical research and promote it as the single best way to address any question of practice, based on the assumption that it is more truthful and valuable than other research models. Within a field as diverse as ABE, all research results need to be treated with care and with respect for the individuality of learners and the diversity of the contexts in which they learn.
PRACTITIONERS USE OF RESEARCH
Practitioners enter the field of ABE through diverse routes and bring a wide range of knowledge and experience that can and often is used in teaching (Lytle et al., 1992). Sabatini et al. (2000) found that among full-time ABE staff surveyed, 61% had worked in the field for at least 6 years, and 28% for more than 10 years. With previous experience teaching in other areas, other professional experiences, and years of experience in the field, many practitioners have considerable knowledge to draw on.
It is true to say, therefore, that ABE practitioners have knowledge and resources to draw on that inform their work. However, studies do indicate that practitioners seek further knowledge about their field in several key areas. In addition, not only are they interested in drawing from the existent research base, but they have clearly articulated ideas for further research. Early in its first funding cycle, the National Center for the Study of Adult Learning and Literacy sought to include practitioners in setting its research agenda. It held focus groups for practitioner leaders in several states to determine what issues concern practitioners and how they would like to see these issues addressed through research (Practitioner Dissemination and Research Network, 1998). Student participation (recruitment, retention, and motivation), program and policy issues, curriculum and instruction, learner assessment and program performance measures, special populations, professional development, the impact of participation, and heterogeneous grouping were all identified as research topics that were of concern to practitioners. Practitioners were looking for research that has clear, practical implications for practice and, as one practitioner put it includes real students in real classrooms with real challenges (Practitioner Dissemination and Research Network, 1998, p. 16). Another priority among this group (granted, the sample was strongly biased in this direction) was a desire for practitioners to be included in the research process. In fact, practitioner inquiry and action research is found to be a highly effective form of staff development (Sabatini et al., 2000; Smith & Hofer, 2002).
This leads to the question of how practitioners make use of the research knowledge currently available to them. Depending on this use, it may be possible to identify some forms of research as offering a better fit with the final application and ensure that researchers at least recognize this factor. For example, if practitioners are strongly oriented toward rigorously tested notions with demonstrated effects, neopositivism may be the most appropriate means to generate the information they need, and other research approaches would appear insufficiently evidenced. However, if practitioners learn most effectively through classroom narrative, neopositivist approaches are likely to be seen as reductionist and lacking insight. It is critical to recognize that insight, arguing that practitioners should approach research in a particular way is not likely to be effective. Instead, researchers should begin by recognizingand acknowledging the legitimacy ofpractitioners use of research.
A recent study in one state (St. Clair et al., 2003) suggests that ABE practitioners approach research as a cognitive resource: a way to think about the everyday classroom world and hopefully make it better. This does not imply that research maps unproblematically onto practice, but that a process of translation is necessary between the abstract ideas of research reports and the lived reality of educational practice (Hammersley, 2002). The critical point here, one to which we return later, is that practitioners have final control over the application of research.
The study (St. Clair et al., 2003) identifies a number of factors making it more likely that ABE practitioners will use research to inform practice. These include longer experience in the field, education or experience in research (specificallynot education in general), and not surprisingly, higher consumption of research materials. Altogether, however, two thirds of practitioners self-identified as having made a change based on research, 84% said they found research somewhat or very interesting, and 85% said they found research somewhat or very relevant to their work. Research, most often communicated through newsletters or conferences, does have a significant role in the work of adult literacy instructors.
When the ways in which research was used were categorized, three main applications emerged. The first was the use of research as a way to validate a course of action, either to an internal body, such as an administrator, or to an external body, such as a funding agency. The second was research as a resource for program planning where a new service was being designed. The experience of others in similar situations would be drawn upon. The third use was program improvement, such as when instructors hear about a new approach at a conference and apply it in their classrooms. These three forms of application are broadly consistent with Weisss (1980) classic portrayal of research use by decision makers as knowledge creep. Rather than constantly skimming the literature looking for new ideas, individuals with decision-making power use it as a resource when needed, but not in a particularly systematic way. The findings of research simply become part of the decision makers general knowledge, assimilated and unattributable, but entirely pragmatic.
The pragmatic use of knowledge by ABE practitioners and their final responsibility for the translation process, imply that useful research knowledge will relate clearly to their practice setting. Otherwise, why would they put the effort into translating the findings and making them meaningful? This presents a problem for neopositivist research because, as addressed in previous sections, the whole point is to attain generalization or a delocalization of knowledgethe antithesis of what this study of the way practitioners use research suggests is necessary.
THE STATE OF KNOWLEDGE
Efforts to derive best practice only from research that uses experimental or quasi-experimental design are limited in the scope of what they can tell practitioners about adult learners, literacy, and learning. In ABE, one factor making this limitation particularly problematic is the paucity of available research of any kind. A second related challenge, although not distinctive to ABE, is amplified by the thin knowledge base and made more complicated by the nature of the workforce. This is the challenge of conducting research that truly speaks to the diversity of the field. In other words, we do not have enough research to begin with, and the nature of the field especially suggests the need for research that addresses itself of multiple contexts and perspectives.
The document Research-Based Principles for Adult Basic Education Reading Instruction (Kruidenier, 2002), produced by the Partnership for Reading illustrates these problems and challenges. As with the systematic review of reading research conducted by the NRP (National Institute for Child Health and Human Development, 2000), researchers who developed this document gave highest priority for review to studies that employed experimental and quasi-experimental designs. Based on this and other criteria for selection, the review panel was left with just 70 studies from which to derive principles for instruction to address six major topics (with many potential subtopics): assessment profiles, alphabetics, fluency, vocabulary, comprehension, and technology. This means that for each topic, reviewers had an average of less than 12 studies to review. From these, 18 principles were derived. By definition, these are based on the results of a minimum of just two experimental studies (Kruidenier, 2002). Meanwhile, so-called trends, ideas, and comments are based on fewer than two experimental studies, non-experimental studies, and findings from the K12-research literature. This is a surprisingly small sample of studies, involving a relatively small number of learners, from which to draw conclusions about reading instruction. Although packaged impressively, this document cannot make up for the fact that we know very little about how adults get better at reading. We know far less about how adults improve their writing (Gillespie, 2000) or their math skills (Tout & Schmitt, 2002).
This lack of knowledge does not in any way negate the potential to use experimental design to expand knowledge, but it is clear that there is much more to learn. The principles enumerated in this document raise more questions than they answer. Many of the unanswered questions can be addressed only through descriptive and ethnographic studies. For example, six of the principles (one third) address themselves to alphabetic or knowledge of the sounds of spoken language, and word analysis, or knowledge of the connection between written letters and sounds (Kruidenier, 2002, p. 35). The principles all point to the idea that alphabetic is an important concept, knowledge of letter-sound correspondence, yet do nothing to elucidate under what conditions readers use, knowledge of letter sound correspondence, what creates barriers to learning it, under what circumstances different kinds of learners acquire the concept, or how alphabetic functions in relationship with other components of reading. Nor do they inform practitioners of what best practices with real students in real classrooms with real challenges (Practitioner Dissemination and Research Network, 1998, p. 16) might look like given these principles. Although one might take issue with the principles themselves (Strauss, 2001), even taken at face value they are lacking in the kinds of information that practitioners say they need. ABE research that can address itself to this type of inquiry has yet to be fully pursued, and the current climate makes it increasingly less likely that it will be at any time in the near future.
A more specific example of what is missing from the knowledge base can be derived from one particular study cited in two research principles. Principles 3 and 4 assert that adult beginning readers have poor phonemic awareness and have difficulty applying letter-sound knowledge to decode unfamiliar words. Both reference a study conducted by Greenberg, Ehri, and Perrin (1997). Trend 1 summarizes a key finding of this study. On phonemic awareness tasks, adult beginning readers are not as good as reading-matched children (children progressing normally in their reading who are reading at the same level as the adults) (Kruidenier, 2002, p. 40). The study by Greenberg et al. suggests that there are important differences between developing readers of different ages. In a follow-up study (Greenberg, Ehri, & Perrin, 2002), the authors observe that children and adults tend to take on the task of decoding using different cognitive processes and approaches. What this work does not tell us, because it cannot, given the research question posed and research methods employed, is what aspects of adults experiences might influence the ways in which they approach the task of reading. Although the findings of this study indicate that adult developing readers are more like children who are struggling readers than normally developing readers, we cannot know whether this is because of ineffective strategies they have adopted based on their understanding of the task or because of other factors. Greenberg et al. (2002) acknowledge that their work raises questions that can be addressed only using qualitative methods. They state that it is important to understand the impact of instructional history on task performance. It is possible that a confound in this study is that instructional histories of the two groups differed and therefore the error patterns of both groups reflect their learning environments more than their innate skills or strategy preferences (p. 240). Research that addresses differences between adults and children regarding this so-called confound have tremendous potential for informing practice. But questions about instructional histories and learning environments are best answered through open-ended interviews and observations because they allow analyses of learner perspectives and learning contexts in ways that experimental designs cannot. Furthermore, research that gathers data on how adults (and children) actually solve the problem of reading in practice have the potential to fill in important gaps in the knowledge base that are unanswered by studies that utilize tests of reading nonsense words and syllables and other decontextualized applications of literacy skills. Such work could help practitioners focus on the features that adults bring to the instructional context by virtue of being adult, in addition to the components of reading elucidated by the NRP and the Research-Based Principles for Adult Basic Education Reading Instruction (Kruidenier, 2002).
Among the studies currently receiving federal funding is one that further illustrates our argument. It is aimed at determining the most effective instructional approach for teaching reading to low-literate adults. This study employs a quasi-experimental design that will allow the investigators to compare the effect of various instructional approaches, as well as specific reading characteristics of learners, on learner outcomes. It employs carefully controlled instructional contexts using five different approaches to teaching reading. Data are derived from pretests, midpoint tests, posttests, and 6-month follow-up tests and structured interviews with learners who participate. The main focus is on how instruction influences reading development. However, important questions about context cannot be addressed given the research questions and methods. For example, the study assumes that instruction remains relatively constant across cohorts and teachers and focuses on the diversity in learners primarily with regard to reading characteristics (although race, gender, country of origin, age, teacher, program site, and number of students enrolled in the class will be considered as variables). However, every practitioner knows that the same lesson on paper can often be very different in reality and that individual and group diversity with regard to race, class, gender, and previous school experiences can radically alter the same lesson presented to different groups. To really understand how instruction influences practice, research must address itself to the ways in which learners, instructors, and instructional contexts shape instruction and acknowledge, through the posing of questions and the matching appropriately of methods to questions, that instruction does not occur in an experimental vacuum.
We began this discussion by harkening back to the Black Book of 1964 and its concerns about gaps in the knowledge available to adult educators and commented that more than 40 years later, the same concerns apply. There is little doubt that we need to know more about the education of adults, as we have illustrated through the example of adult literacy education. We have characterized the current preoccupation with neopositivism, and the concomitant belief that there is a best answer, as a retrograde stepback to a future that is right where we left it in 1965. In concluding, we would like to offer a few last thoughts and tie up some loose ends.
One of the most limiting of all the concerns we have discussed is the impossibility of using experimental research to go beyond questions of what works? The inability to address what matters? or why? is a staggering limitation for educational research. For example, we know that some people consistently do better than others in educational settings, and understanding even some of the factors at play requires a broad-based approach and significant commitment of resources. Once we have that understanding it may be time for trials of ameliorative strategies. The effect of a premature move to experimental design is to privilege the current status quo, since experimental methods do not permit untestable explanations. Truly embracing neopositivism would mean losing Dewey, Freire, Brookfield, Auerbach, and much of the research of the last 20 years as legitimate influences on educational thinking and practices. It would potentially mean giving up the possibility of critical educational thought.
Previous research demonstrates how significant insights can be developed through interpretative research. It is not possible to prove that Auerbach (1995) is right to be concerned about the colonization of the family by school based literacy, but it remains a powerful and helpful insight into family literacy. Similarly, it cannot be proven using scientifically based research that workers and managers often have competing agendas for worksite literacy programs that are hard to unravel without gaining the perspectives of all the players. Yet Gowens (1992) work on the politics of workplace literacy offers us critically important guidance on planning and implementing programs in workplaces. Fingeret (1983) did work that helps us understand the ableness of adult literacy learners, and Reder (1987) helps us understand how adults with wide varieties of technical reading and writing skills participate in literacy acts in many ways. These studies have played an important role over the last 20 years by highlighting what adult literacy learners bring to learning and how this can and should influence practice. Experimental research could not have generated these insights or the possibilities for pedagogy and curriculum they hold.
The diversity of adult education is badly served by the narrowing of research. The knowledge base is already too narrow, and focusing resources into one fairly limited research paradigm is unlikely to lead to anything other than a loose set of decontextualized findings. We end with another reference from the 1960s. In The Republic of Knowledge, an extremely influential work in the philosophy of science, Polanyi (1962) reminds us that knowledge cannot advance through a coordinated effort led by a small group of controlling individuals, however well intentioned they are. Knowledge can advance only through the diversity and freedom of researchers to commit their working lives to the questions, and methods of addressing questions, they find truly fascinating. We would do well to listen.
Auerbach, E. R. (1995). Which way for family literacy: Intervention or empowerment? In L. M. Morrow (Ed.), Family literacy connections in schools and communities (pp. 1128). Newark, DE: International Reading Association.
Baroni, D. (Ed.). (n.d.). Voices. In Technologies of the Frontier Retrieved May 7, 2003, from http:// www.tdf.it/english/progetti/VOCI%20inglese.htm
Barton, D., Hamilton, M., & Ivanic, R. (Eds.) (2000). Situated literacies: Reading and writing in context. New York: Routledge.
Berliner, D. (2002). Educational research: The hardest science of all. Educational Researcher, 31(8), 1820.
Cochran-Smith, M. (2002). What a difference a definition makes: Highly qualified teachers, scientific research, and teacher education. Journal of Teacher Education, 53(3), 187189.
Cunningham, J. W. (2001). The National Reading Panel Report Reading Research Quarterly, 36(3), 326335.
Education Sciences Reform Act of 2002 Pub. L. No. 107-279, 116 Stat. 1940 (2002).
Erickson, F., & Gutierrez, K. (2002). Culture, rigor and science in educational research Educational Researcher 31(8), 2124.
Feuer, M. J., Towne, L., & Shavelson, R. J. (2002). Scientific culture and educational research. Educational Researcher, 31(8), 414.
Fingeret, A. (1983). Social network: A new perspective on independence and illiterate adults Adult Education Quarterly, 33, 133146.
Gillespie, M. (2000). Research in writing: Implications for adult literacy education. In J. Comings, B. Garner, & C. Smith (Eds.), Annual review of adult learning and literacy, (Vol. 2, pp. 63110). San Francisco: Jossey-Bass.
Gowen, S. (1992) The politics of workplace literacy: A case study. New York: Teachers College Press.
Greenberg, D., Ehri, L. C., & Perrin, D. (1997) Are word-reading processes the same or different in adult literacy students and thirdfifth graders matched for reading level? Journal of Educational Psychology, 89(2), 262275.
Greenberg, D., Ehri, L. C., & Perrin, D. (2002) Do adult literacy students make the same word reading and spelling errors as children matched for word-reading age? Scientific Studies of Reading, 6(3), 221243.
Hammersley, M. (2002). Educational research, policymaking, and practice London: Chapman.
Heath, S. B. (1983) Ways with words: Language, life and work in communities and classrooms. Cambridge, UK: Cambridge University Press.
Houle, C. O. (1964). The emergence of graduate study in adult education In G. Jensen, A A Liveright, & W. Hallenbeck (Eds.), Adult education: Outlines of an emerging field of university study (pp. 6983). Washington, DC: Adult Education Association of the United States.
International Reading Association (2002) What is evidence-based reading instruction? [Brochure]. Newark, DE: Author.
Jensen, G., Liveright, A. A., & Hallenbeck, W. (Eds.). (1964). Adult education: Outlines of an emerging field of university study. Washington, DC: Adult Education Association of the United States.
Knowles, M. (1980). The modern practice of adult education Englewood Cliffs, NJ: Prentice Hall.
Kruidenier, J. (2002). Research-based principles for adult basic education reading instruction Portsmouth, NH: RMC Research.
Labov, W. (1969). The logic of non-standard English In J. E. Alacis (Ed.), Report of the Twentieth Round Table Meeting on Linguistics and Language Studies, (Vol. 22, pp. 129). Washington, DC: Georgetown University Press.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York: Cambridge University Press.
Leinicke, W. (2000) An examination of the use of part-time instructors in adult basic education and GED programs in Illinois Unpublished doctoral dissertation, University of Iowa, Iowa City.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.
Lytle, S., Belzer, A., & Reumann, R. (1992) Invitations to inquiry: Rethinking staff development in adult education. Philadelphia: National Center on Adult Literacy.
Mayer, R. E. (2000). What is the place of science in educational research? Educational Researcher, 29(6), 3839.
Merriam, S. B., & Brockett, R. G. (1997). The profession and practice of adult education: An introduction. San Francisco: Jossey-Bass.
National Council for the Teaching of Mathematics Advisory Committee (2003) Educational research in the No Child Left Behind environment Journal for Research in Mathematics Education, (34) 3, 185190.
National Institute for Child Health and Human Development. (2000). Report of the National Reading Panel: Teaching children to read. An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction (NIH Publication No. 00-4769) Washington, D.C.: US Government Printing Office.
No Child Left Behind Act of 2001, Pub. L. No. 107-110 115 Stat. 1425 (2002).
Polanyi, M. (1962) The republic of science Minerva 1, 5473.
Practitioner Dissemination and Research Network. (1998). Practitioners speak: Contributing to a research agenda for adult basic education (NCSALL Report No. 4). Cambridge, MA: National Center for the Study of Adult Learning and Literacy.
Pressley, M. (2002). Effective beginning reading instruction Journal of Literacy Research 34 (2) 165188.
Reder, S. (1987). Comparative aspects of functional literacy development: Three ethnic American communities. In D. Wagner (Ed.), The future of literacy in a changing world (pp. 250270) New York: Pergamon.
Sabatini, J. P., Daniels, M., Ginsburg, L., Limeul, K., Russell, M., & Stites, R. (2000) Teacher perspectives on the adult education profession: National survey findings about an emerging profession (NCAL TR00-02). Philadelphia: National Center on Adult Literacy.
Shavelson, Richard J., & Lisa, Towne (Eds.). (2002) Scientific research in education Washington, DC: National Research Council, Committee on Scientific Principles in Education.
Silver, E. A. (2003) Improving education research Ideology or science? Journal for Research in Mathematics Education, 34(2), 106109.
Smith, C., & Hofer, J. (2002). Pathways to change: A summary of findings from NCSALLs staff development study. Focus on Basics, 5D, 1, 38.
Smith, C., Hofer, J., & Gillespie, M. (2001). The working conditions of adult literacy teachers: Preliminary findings from the NCSALL staff development study. Focus on Basics 4D, 17.
Strauss, S. L. (2001) An open letter to Reid Lyon Educational Researcher, 30(5), 2633.
St. Clair, R., Chen, C. Y., & Taylor, L. (2003). Abstract to action: How adult literacy and ESL educators make use of research. Perspectives: The New York Journal of Adult Learning, 2(1), 4961.
Tett, L., & St. Clair, R. (1997). Family literacy in the educational marketplace: A cultural perspective. International Journal of Lifelong Education, 16(2), 109120.
Tout, D., & Schmitt, J. J. (2002). The inclusion of numeracy in adult basic education In J. Comings, B. Garner, & C. Smith (Eds.), Annual review of adult learning and literacy, (Vol. 3, pp. 152202). San Francisco: Jossey-Bass.
Vygotsky, L. S. (1978). Interaction between learning and development In M. Cole, V. John Steiner, S. Scribner, & E. Souberman (Eds.), Mind in society (pp. 7991) Cambridge, MA: Harvard University Press.
Weiss, C. H. (1980) Knowledge creep and decision accretion. Knowledge: Creation, Diffusion, Utilization, 1(3) 381404.
Wenger, E. (1998) Communities of practice: Learning, meaning, and identity. New York: Cambridge University Press.
Workforce Reinvestment and Adult Education Act of 2003, H.R. 1261, 108th Cong. (2003).
ALISA BELZER is an assistant professor at Rutgers University. Her research has focused on professional development, policy, adult learner perspectives on literacy and learning, and reading. She has recently published a monograph for the National Center on Adult Literacy and Learning on policy implementation and has two articles (one forthcoming) on professional development systems in adult basic education. Together with Ralf St. Clair, she is the author of Opportunities and Limits: An Update on Adult Literacy Education (ERIC Information Series).
RALF ST. CLAIR is a Research Fellow at the Centre for Research and Development in Adult and Lifelong Learning at the University of Glasgow. His research interests include adult education and policy, socio-cultural approaches to learning, and the meaning of critique in late modernity. He has recently published on practitioner uses of research in Adult Education Quarterly and co-edited the New Directions in Adult and Continuing Education volume From Critical Theory to Critical Practice with Jennifer Sandlin.