Building a Scientific Community: The Need for Replication

Criticism directed toward educational research is not a new phenomenon. From the beginning of the 20th century through today, there have been numerous articles in scientific journals─ The Journal of Educational Sociology and Studies in Education of the early 1900s, and the Educational Researcher more recently─ that point out the lack of quality and irrelevance of topics pursued by educational researchers (Schneider, 2003). Education is not the only field to have been the target of criticism─ it is hard to think of a professional field that has not at one time or another suffered disapproval or reproach from the academy or policymakers (Abbott, 1988). What distinguishes current criticisms from earlier ones is that the legitimacy, integrity, and value of educational research are being seriously questioned, both internally within the field and externally among those who have supported its development (Educational Researcher ‘‘Theme Issue on Scientific Research in Education,’’ Jacob & White, 2002).

The field of educational research has become increasingly diverse and fragmented resulting in multiple understandings of what constitutes scientific inquiry, evidence, and interpretation. The quality problems in educational research stem from the complexity of the educational enterprise─ including the tenuous relationship between researchers and educational professionals and the numerous disciplinary perspectives of the researchers (Feuer, Towne, & Shavelson, 2002). It is unrealistic to expect all educational researchers to share identical perspectives on researchable problems given their dissimilar disciplinary training and methodological approaches (see Knorr-Cetina, 1999, and Bazerman, 1988, for further explanation on the epistemic cultures of science). However, the absence of a self-regulating community that appreciates diversity yet upholds a set of principles for quality research undermines both the work and the professionals in the field. The need for such a community is noted in Scientific Research in Education, a publication of the NRC Committee on Scientific Principles for Education Research. Shavelson and Towne (2002) enumerate five basic principles for conducting scientific research in education. The authors explain that these principles are not a set of rigid standards but an attempt to establish a set of norms that should be adopted and supported by the educational research community.

How does one create scientific norms that can unite a community of diverse paradigms and methods? The first step is finding common ground on which conversations can evolve. Because the possibility of replication is the key to knowledge accumulation in the sciences and social sciences, it presents us with an important unifying concept and starting point for constructive dialogue. Unless there is some shared appreciation for the importance of replication, the mechanisms for ensuring the replication of research are probably not worth pursuing. Even those with the most constructivist beliefs about science would agree that attempting replication is important. Replication necessitates the development of common languages and measures, data sharing, data analysis, and technological mechanisms that support these types of activities. It is the sharing of information about studies, including the actual data on which findings are based, that allows other researchers to verify, extend, and generalize findings. Data sharing is essential to replication and creates a vehicle through which investigators can build upon designs, create and revise measures, and study different populations for purposes of developing new theories.

‘‘REPLICATION WITH DIFFERENCE: BUILDING THE BEST TEST CASE FOR GENERALITY’’ (GOULD, 2003)

A major problem in educational research is that investigators find it difficult or are unable to replicate their work or that of their peers. Replication─ conducting an investigation repeatedly with comparable subjects and conditions so as to achieve what would be expected to be similar results─ is essential for being able to generalize to more people and settings than are represented in a single study (Cronbach, 1980; Cronbach & Shapiro, 1982; Shadish, Cook, & Campbell, 2002). This process involves both applying the same conditions to multiple cases or being able to make controllable changes as well as replicating the design to cases that are sufficiently different to justify the generalization of results and theories. It is not just replication of experiments that is rare in educational research; investigators infrequently reanalyze the findings of others using secondary data obtained from quasi-experiments or national and international data sets. The rarity of replication in education studies has unfortunately produced disparate results that undermine the community’s ability to accumulate knowledge. Without convergence of results from multiple studies, the objectivity, neutrality, and generalizability of research is questionable.

ENCOURAGING AND FACILITATING REPLICATION STUDIES

In some fields there is a tradition of replication and data sharing so that experiments can be repeated, perhaps with somewhat different methods. For example, in the natural and physical sciences norms exist regarding the distribution of findings. This is in part a response to changes in how research is conducted in these fields. Results are often made available on the World Wide Web as soon as analysis is completed. At one time, investigations in fields such as astronomy were based on natural observations. Today, these procedures have been partially transformed through the collection and processing of images of phenomena, with images and information that are available to multiple investigators simultaneously (Knorr-Cetina, 1999). Consortiums of scientists work on datasets housed in centralized databases. Similarly, in physics, there are projects where researchers, linked through various technologies, work on problems at the same time in locations throughout the world. Organized in teams and using the same databases, scientists race to be first in making new discoveries (such as in efforts to decipher the human genome). Attentive to advances in the field, researchers in the natural and physical sciences are quick to replicate experiments or reanalyze data from simulations reported at scientific meetings, on the web, and in the journals. In these fields, the more significant a study’s findings, the greater the likelihood that other investigators will attempt to reproduce, double-check, and extend the results.

Looking to the natural and physical sciences and some of the social sciences as a guide, several mechanisms appear especially useful for encouraging and facilitating replication. These involve constructing a dataset and detailing how the data were obtained; upholding ethical standards for replication and data sharing in the professional associations; reinforcing these standards by requiring researchers to abide by them when publishing findings in professional journals; and maintaining institutional infrastructures that assist researchers in data sharing and provide opportunities for study replication.

PROFESSIONAL STANDARDS

Matters of replication and data sharing are so fundamental to the social and physical sciences that many of the professional associations have enacted ethical codes regarding the disclosure of data and other pertinent documentation. For example, in the Ethical Standards for the American Sociological Association (ASA), there are six key elements regarding data sharing:

1. Sociologists make their data available after completion of the project or its major publications, except where proprietary agreements with employers, contractors, or clients preclude such accessibility or when it is impossible to share data and protect the confidentiality of the data or the anonymity of research participants (e.g., raw field notes or detailed information from ethnographic interviews).

2. Sociologists anticipate data sharing as an integral part of a research plan whenever data sharing is feasible.

3. Sociologists share data in a form that is consonant with research participants’ interests and protect the confidentiality of the information they have been given. They maintain the confidentiality of data, whether legally required or not; remove personal identifiers before data are shared; and if necessary use other disclosure avoidance techniques.

4. Sociologists who do not otherwise place data in public archives keep data available and retain documentation relating to the research for a reasonable period of time after publication or dissemination of results.

5. Sociologists may ask persons who request their data for further analysis to bear the associated incremental costs, if necessary.

6. Sociologists who use data from others for further analyses explicitly acknowledge the contribution of the initial researchers. (ASA Code of Ethics, 1997)

The ethical standards of the ASA were written in conjunction with those used by the American Psychological Association (APA). Similar ethics policies regarding data sharing are not included in the American Educational Research Association’s (AERA) ethics code. Last revised in 2000, the AERA ethics code covers fewer items than those listed in ASA and tends to focus more heavily on questions of authorship rather than the integrity of the research itself. Adequate attribution is certainly important, especially in dealing with issues of fairness and equity, and some attention is paid to issues of confidentiality. These considerations, however, are much less specific than those of the other professions and relatively brief, especially in light of changes in most Institutional Review Board policies and procedures for assuring confidentiality of subjects. The absence of standards in the AERA ethics code concerning the actual conduct of research, particularly regarding data sharing, represents one obstacle that makes it difficult to establish norms for replication.

REINFORCING NORMS FOR REPLICATION IN JOURNALS

In the physical and social sciences, the professional journals reinforce norms for replication. In journals such as Science and Nature, once an article is published the author has to make the data available to those who wish to replicate the results. What this actually means is that when a manuscript is accepted, the author is required to organize all their materials and methods in order to make them available to researchers for their own use. Both Science and Nature have specific procedures for data sharing that the author is expected to follow once a paper is accepted for publication.

It is important to underscore that these procedures are not simply contact web addresses. Rather, the data must be arrayed in well-identified files that directly correspond to results reported in the manuscript’s tables and figures. Nature requires that any supporting data sets for which there is no public repository must be made available to any interested reader on and after the publication date by the author. The author is expected to provide a URL to a specific website containing the data. Researchers who encounter a persistent refusal to comply with these guidelines are instructed to contact the editor with a materials complaint.

The social sciences also have procedures for ensuring the replication of research, although these procedures are less established than those in the physical sciences. As in the physical sciences, the replication issue is facilitated by journal guidelines. In sociology, a substantial body of research based on national datasets exists and is available to researchers for reanalysis. Demographers, for example, work primarily with secondary databases, although the use of other types of data is now more common. Demography, the leading journal of demographers, requires authors with accepted manuscripts to preserve the data used in their analysis and to make the data available to others at reasonable cost from six months after the publication date to a period of three years thereafter. Exemptions are possible for proprietary datasets, but such exemptions are made at the time of manuscript submission. One of the premier ASA journals, the American Sociological Review (ASR), has a similar policy, and the manuscript guidelines reiterate the ethics code for data sharing.

ESTABLISHING AN INFRASTRUCTURE FOR REPLICATION

Finally, there are institutional infrastructures that facilitate study replication and data sharing. The government has several different centers and institutes that are designed to make data accessible to researchers. One of the most helpful of these is the National Center for Education Statistics (for more information about data accessibility, see the NCES website, http://nces.ed.gov). In addition to these federal initiatives, there are independent centers that also make data available to researchers.

The Inter-university Consortium for Political and Social Research (ICPSR) at the University of Michigan, established in 1962, maintains and provides access to a vast archive of social science data for research and instruction and offers training in quantitative methods to facilitate more effective use of these data. Over 500 member colleges and universities around the world belong to the Consortium. Faculty and students in member institutions can browse the data sets, which are identified by thematic categories, one of which is education. Researchers and associations are encouraged to deposit data with ICPSR, and there are a series of established procedures for formatting and documenting data. Conferences, workshops, and special events are held for faculty and students interested in learning about the datasets and the latest statistical techniques for analyzing them.

Another type of institution that includes both survey data and qualitative data is the Murray Research Center at the Radcliffe Institute for Advanced Study at Harvard University. The Murray Center is dedicated to the study of lives over time and promotes the use of existing social science data to explore human development and social change. Founded in 1976, the center has established a national archive of materials from over 270 studies, which are available for new research. The objectives of the center include: (1) preserving social science data, especially qualitative and longitudinal data on women’s lives, human development, and the life span; (2) making archival data available for new research; and (3) contributing to knowledge of social science methods in the areas of longitudinal studies, qualitative data usage, and secondary analyses. The Murray Center data archive is unique in that it includes not only computer-accessible quantitative data, but also qualitative materials such as case histories, open-ended interviews, and responses to projective tests.

The Murray Center is also atypical in that it allows new researchers to contact the subjects of some of the existing data sets to obtain follow-up data. John Laub and Robert Sampson wrote the award-winning book, Crime in the Making: Pathways and Turning Points through Life (1993), which was based on a follow-up of Sheldon and Eleanor Glueck’s (1986) Crime Causation Study: Unraveling Juvenile Delinquency, 1940–1963. There have been several other notable follow-ups studies, including David McClelland and Carol Franz’s (1992) follow-up of the Sears, Maccoby and Levin (1951) Patterns of Child Rearing Study.

In the instance of data sharing in fields outside of education, there are incentives such as ethics codes and journal submission requirements that encourage researchers to be forthcoming with their data. Institutions like NCES, ICPSR, and the Murray Center also facilitate data sharing. These organizations do not have the authority to force researchers to share information. Rather the sharing of information needs to be voluntary and thus requires an incentive-based system.

DEVELOPING AN INCENTIVE SYSTEM FOR DATA SHARING

Data archiving is becoming increasingly a matter of course. At universities and research centers across the U.S. data are routinely archived. Computer technology and advances in software make the storage of data of all types, from extensive field notes to videotapes, considerably less cumbersome than it was a decade ago. However, archiving data for historical reference is quite different than producing data files that can be easily read and used by others for replication purposes. What is needed is some type of incentive that would encourage investigators to think at the onset of their work about data sharing and opportunities for replication. There has been some discussion about requiring investigators who receive awards from either federal or other sources to provide a public use data file at the completion of their work. Presently, the Alfred P. Sloan Foundation is requiring its six Working Family Centers to create and maintain datasets that can be made accessible to Sloan researchers and eventually the general public (for more information on the Sloan Centers, see www.sloan.org). This type of incentive for data sharing could be very useful for ensuring norms of replication.

REPLICATION STUDIES AND DATA SHARING: EXAMPLES OF QUANTITATIVE AND QUALITATIVE STUDIES

Replication studies occur fairly frequently in analyses of large-scale educational data sets. In 1965, James S. Coleman was asked to direct a study mandated in the Civil Rights Act of 1964 to examine the relationship between student achievement, school resources, and other contextual factors. The study surveyed students in grades 1, 3, 6, 9, and 12 as well as teachers and principals, yielding 639,650 fielded instruments. Released on the fourth of July, Equality of Educational Opportunity (1966), often referred to as the Coleman Report, found that school resources showed little relation to achievement in schools when students of similar backgrounds were compared. The report also found that students’ family background characteristics showed a significant relation to achievement and that minority children benefited academically by attending high schools with white students, while white students’ academic performance was not affected when they attended racially mixed schools. These findings were greeted with an onslaught of criticism by academics and politicians. One of the most important consequences of this report was that Harvard University organized a year-long faculty seminar to review the findings. This was not a critique without substance. Reanalysis of the data were published in a second volume, On Equality of Educational Opportunity, edited by Frederick Mosteller and Daniel Patrick Moynihan (1972).

There was a similar reaction when Coleman and colleagues released findings from their analyses of data from High School and Beyond (HS&B). HS&B is a national longitudinal study that was based on a randomly stratified sample of tenth and twelfth graders (for more information, see the NCES website on HS&B). Much like the EEOC study, the HS&B data set includes vast amounts of data and contains records on over 50,000 students, teachers, and school administrators. Findings showed that students, particularly those who were minority and living in urban areas, benefited educationally from attending private schools─ that is, they showed slight gains in performance and were more likely to complete high school than comparable minority students who attended public schools. Controversy ensued. Reanalyses took place in many universities─ one of the most significant at Stanford University led by Henry Levin which resulted in a two-volume book (Thomas & Levin, 1988).

More recently, two other studies, both of which are controversial, have been subject to reanalysis: the Tennessee STAR Experiment (Nye, Hedges, & Konstantopoulos, 2002) and the New York Voucher Experiment (Krueger, 2002; Peterson, & Meyers, 1998). Looking from Coleman to Peterson and Krueger, there appear to be several conditions that spur reanalysis and data sharing. First, in these instances the individuals involved in the original work and subsequent reanalysis had considerable intellectual and social capital in the field, their reputations as serious scholars were and are widely recognized. Second, the replication activities were public in that the intellectual and policy community were aware of the reanalysis. When the reanalysis was released, it was also subject to rigorous scholarly scrutiny. And finally, and perhaps most important, the topics these researchers chose to examine had significant consequences for educational practice; what they found mattered.

There has been some resistance to data sharing particularly among researchers who collect qualitative data. However, this trend seems to be changing. Currently researchers can obtain certain ethnographic materials through The Human Relations Area Files, Inc. (HRAF), a nonprofit research organization at Yale University (http://www.yale.edu/hraf/collections-body-development.htm). For over 50 years HRAF has assembled, indexed, and provided access to primary research materials relevant to the social sciences. Today over 300 colleges, universities, museums, and research institutions around the world have full or partial access to the Collection of Ethnography.

A new ethnographic study has dedicated itself not only to replication but to intensive data sharing and analysis. The Three-City Ethnography, funded by NICHD, is designed to conduct fine-grained assessments of how, over time, welfare reform policies influence the day-to-day lives of low-income African American, Latino, Hispanic, and non-Hispanic white families, including a subset of families who have a child with a moderate to severe mental or physical disability. Located in Boston, Chicago, and San Antonio, the study focuses on the interaction of welfare policies, family behaviors, and child development. Led by a team of senior researchers, Linda Burton and William Julius Wilson (who have directed a number of long-term ethnographic studies of urban, economically disadvantaged families and children residing in low-income neighborhoods), the ethnography team consists of over 210 research scientists, ethnographers, qualitative data analysts, system programmers, and staff, all of whom are committed to conducting high quality ethnographic research.

The Three-City Ethnography project (1999) is important because it underscores how research with qualitative data can be shared among multiple researchers. As Burton recently reported, the study has generated an extensive qualitative data set consisting of over 45,000 pages of field notes and supporting data (e.g., tapes, diagrams) that have been organized into a consistent data management system. Data transfer, tracking, and storage mechanisms are in place to protect the data from corruption and loss and to maintain respondent confidentiality. As Burton writes, ‘‘the qualitative data analysts read field notes about families and neighborhoods to which they are assigned, build organizational systems to classify the material, and look for emergent and recurring themes. The goal is to prepare materials for several layers of analysis─ a truly ongoing and collaborative process’’ (Burton & Lein, 2003).

There are several reasons for presenting these examples. One is that replication is not a comforting or comfortable process, as the controversy over the Coleman reports clearly illustrates. With replication comes the risk that someone will be shown to be wrong and that the researcher’s academic competence may be called into question. This risk is part of the work we undertake as scientists. However, we could lower the personal risks by strengthening the professionals themselves, particularly by providing for the continual professional development of researchers and improving the training of graduate students in research. It is hard to imagine training a first grade teacher in mathematics but not reading, nor is it plausible to think of training a graduate student in only one type of research method. Strong training in collaborative research, where the work of investigators is routinely scrutinized and graduate students are fully engaged in the project, as in the instance of the Three-City Project, helps to minimize personal risk and maximize the opportunity for replication.

ADVANTAGES OF USING STUDIES WITH PROBABILITY SAMPLES FOR SECONDARY ANALYSIS

One point that needs to be emphasized concerns the value of large-scale longitudinal databases. Such studies typically use random selection, which involves selecting cases by chance to represent a population, whereas experimental studies use random assignment, which involves assigning cases to multiple conditions. The national education datasets are designed to represent the population of the United States, or certain age groups within it. Data sets that are based on probability samples, such as ECLS, NELS:88-92, and Baccalaureate and Beyond, are potentially useful for testing and constructing measures, contextualizing the results of smaller-scale studies, and training graduate students.

These datasets typically include items from earlier surveys as well as new questions. Critics have argued that investigators do not incorporate advances in the field when developing new questions and that more attention needs to be paid to improving the types of questions that are asked. Another problem with large-scale datasets has been that investigators are often uninformed about statistical advances that can improve the analytic capacity of the datasets. For example, procedures have been developed for addressing problems of selection bias, and imputation techniques allow investigators to avoid problems associated with missing data. Advances in statistical techniques are increasing the accuracy and power of multivariate analyses of longitudinal data. Propensity analyses, which predict outcomes based on the probability that actors will behave in certain ways, are yielding results that are nearly as powerful as those achieved in carefully designed randomized experiments. Studies that fail to use recently improved statistical tools undermine their ability to produce robust findings.

Probability samples with extensive item pools are perhaps one of the best indicators of what our population looks like. In medical research, before conducting an experiment, researchers look for evidence of prevalence─ for example, how many people in the country have a particular disease, and who is likely to have the disease prior to treatment. National datasets provide us with extensive information on the population, which is essential to designing and implementing intervention studies.

Such datasets are particularly useful for creating common measures that can be used to document trends over time and allow for replication of controversial findings. Analyses conducted by Schneider and Stevenson (1999), which examined changes in educational expectations among U.S. high school seniors from 1955 to 1992, provide an example of the usefulness of national datasets. Working to create comparable measures across five datasets─ an NSF study in 1955, Project Talent in 1960, NLS-72, HS&B, and NELS:88-94─ the authors were able to show that the educational expectations and occupational aspirations of teenagers have risen dramatically over the last forty years. Coupling this information with college matriculation rates, it is clear that not all teenagers will meet their expectations. The question then becomes why does this happen and for whom is this situation the most likely to occur. Without upkeep of these massive datasets, it would have been impossible to make such conclusions about these topics regarding their change over time and to suggest alternatives.

Another benefit of national datasets is that they can be used to contextualize the results of smaller studies. A study based on a limited sample can be compared with nationally representative data to determine whether results for subjects with comparable characteristics are similar across data sets. We have conducted these types of analyses with the Sloan Study of Youth and Social Development and the 500 Family Study. Although these datasets are not entirely similar to national studies, the Sloan data sets include items from national studies such as NELS:88, CPS, and ECLS, making it possible to compare results and to verify certain findings.

Large-scale data sets also provide opportunities for training young scholars in quantitative analyses. These data sets can be analyzed with a variety of statistical packages, and comprehensive descriptions of variables and sample restrictions are provided. The CD-ROM, menu-driven data sets are a valuable tool for teaching young scholars data analysis techniques. As previously discussed, young investigators should be encouraged to learn how to deal with selection bias, to routinely report and distribute data, and to appropriately contend with outliers, missing data, and effect sizes. Training programs that teach quantitative techniques are another mechanism for establishing norms of replication.

Norms for replication need to be developed, and it is difficult in education because studies do not share a common language on what replication actually means; it is not merely doing the same study over and over again. It is also problematic because the infrastructure of the profession has only marginally supported it and there are few incentives for engaging in such a study. But if we are in the business of knowledge accumulation, as Stephen Gould (2003) has written, ‘‘replication with difference, builds the best case for generality.’’ And the way to begin this process is through data sharing.

This material is based upon work supported by the National Science Foundation under Grant No. 0129365. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

References

Abbott, A. (1988). The system of professions. An essay on the division of expert labor. Chicago: University of Chicago Press.

American Sociological Association. (1997). American Sociological Association Code of Ethics.  Retrieved April 26, 2004, from http://www.asanet.org/members/ecoderev.html

Bazerman, C. (1988). Shaping written knowledge: The genre and activity of the experimental article in science. Madison: University of Wisconsin Press.

Burton, L., & Lein, L. (2003, June). The walls of Jericho: Poverty, work, and mothering. Paper presented at the conference Workforce/Workplace Mismatch? Work, Family, Health, and Well-Being, Washington. DC.

Coleman, J. S., et al, (1966). Equality of educational opportunity. Washington DC: U.S. Government Printing Office.

Cronbach, L. J. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass.

Cronbach, L. J., & Shapiro, K. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.

Feuer, M. J., Towne, L., & Shavelson, R. J. (2002). Scientific culture and educational research. Educational Researcher, 31(8), 4–14.

Gluck, S., & Gluck, E. (1986). Crime causation study: Unraveling juvenile delinquency, 1940-1963. Retrieved April 26, 2004, afrom http://www.radcliffe.edu/murray/data/ds/0896Study

Gould, S. J. (2003). I have landed: The end of a beginning in natural history. New York: Three Rivers Press.

Jacob, E., & White, C. S. (2002). Theme issue on scientific research in education. Educational Researcher, 31(8).

Knorr-Cetina, K. (1999). Epistemic cultures: How the sciences make knowledge. Cambridge, MA: Harvard University Press.

Krueger, A. B., & Zhu, P. (2002). Another look at the New York City School voucher experiment. Cambridge, MA: National Bureau of Economic Research.

Laub, J., & Sampson, R. (1993). Crime in the making: Pathways and turning points through life. Cambridge, MA: Harvard University Press.

McClelland, D., & Franz, C. (1992). Life patterns project 1987-88 follow-up of patterns of child-rearing. Retrieved September, 2003, from http://www.radcliffe.edu/murray/data/ds/1012-StudyDescription

Mosteller, F. & Moynihan, D. P. (Eds.). (1972). On equality of educational opportunity. New York: Vintage Books.

National Center for Education Statistics (2003). National Center for Education Statistics home page. Retrieved September, 2003, from http://nces.ed.gov/.

National Center for Education Statistics. (1988). National Education Longitudinal Study of 1988. Retrieved September, 2003, from http://nces.ed.gov/surveys/nels88/

Nye, B., Hedges, L., & Konstantopoulos, S. (2002). Do low achieving students Benefit more from small classes? Evidence from the Tennessee Class Size Experiment. Educational Evaluation and Policy Analysis, 24, 201–217.

Peterson, P. E., Howell, W. G., & Myers, D. E. (1998). An evaluation of the New York City School Choice Scholarships Program: The first year. Cambridge MA: Harvard University Press.

Schneider, B., & Stevenson, D. L. 1999. Ambitious generation: America’s teenagers─ Motivated but directionless. New Haven, CT: Yale University Press.

Schneider, B. (2003). Sociology of education: An overview of the field at the turn of the twenty- first century. In M. T. Hallinan, A. Gamoran, A. Kubitschek, and T. Loveless (Eds.), Stability and change in American education. New York: E. Werner Publications.

Sears, R. R., Maccoby, E. E., & Levin, H. (1951). Patterns of child rearing: A study. Stanford, CA: Stanford University Press.

Shadish, W. R., Cook, T. D., & Campbell, T. D. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.

Shavelson, R. J., & Towne, L. (2002). Scientific research in education. Washington, DC: National Academy Press.

Thomas, J., Haertel, E., & Levin, H. M. (1989). Comparing public and private schools, (Vol. 2). New York: Falmer Press.

Thomas, J., & Levin, H. M. (1988). Comparing public and private schools, (Vol. 1). New York: Falmer Press.

Winston, P., Angel, R. J., Burton, L. M., Chase-Landsdale, P., Lindsay, C., Moffitt, A. J., & Wilson, W. J. (1999). Welfare, children, and families: A three-city study. Retrieved September, 2003, from http://www.jhu.edu/~welfare/overviewanddesign.pdf