The Relationship between Quantitative Inquiry, Scientific Collaboration, and Academic Formidability in the Social Sciences
by Neil Eckardt - January 23, 2008
The question Iím trying to raise has little to do with the value of qualitative or quantitative research. It has to do with the degree to which the amount of qualitative research in the field of education is related to the fieldís struggles at building a common knowledge domain.
I was pleased to learn that Aaron Cooley had written a response to my recent commentary, The Prevalence of Qualitative Methodology at AERAs Annual Meeting and the Potential Consequences, for the whole point of my commentary was to raise an important question for public discussion (Cooley, 2007). I was baffled, however, by the content of Cooleys essay. In it, Cooley makes several ungrounded assumptions about my views and he does little to contend with the primary question I attempted to highlight. I bear some responsibility for this, as the brevity of my original commentary might have left some readers wondering about where I was coming from. Here Id like to contend with Cooleys essay and spell out in greater detail what I was trying to illuminate in my commentary. The question Im trying to raise has little to do with the value of qualitative or quantitative research. It has to do with the degree to which the amount of qualitative research in the field of education is related to the fields struggles at building a common knowledge domain.
BEING CLEAR AND GETTING THE FACTS STRAIGHT
Cooleys opening paragraph contains the following excerpt:
The questions raised about ideology in educational research are important. I commend his effort at establishing a dialogue on the related issues. That being said, I would not be responding to the commentary, if I were not troubled by many of the statements, assertions, and conclusions that he makes in the piece.
This opening represents a misreading of my commentary on at least two levels. To begin with, my commentary has very little (if anything at all) to do with ideology, at least as that concept is understood in contemporary Political Science. (See Knight, 2006.) My commentary is clearly focused on research methodology. As a result, I dont understand why Cooley would say that my commentary is about ideology, just as I dont understand why he consistently refers to the relationship between ideology and research throughout his reaction.1 The second reason that these first few sentences are baffling is that in one breath Cooley refers to the questions I raise in my commentary and in the next he refers to the statements, assertions, and conclusions that I make in it. Aside from the fact that this seems to present a contradiction, for one cannot simultaneously aim to raise questions and set down conclusions, this was a surprising accusation in that I tried to be as straightforward as possible in my commentary. I presented some evidence, related it to other empirical realities in the field, and then put the two together in the form of a question.
As Cooleys essay proceeds, there are a number of other points where he misinterprets my essay. Allow me to deal with each separately:
Cooley spends a considerable portion of his essay addressing Frederick Hess work on education research. Indeed, Cooley devotes far more attention to Hess work than he does to my commentary. As a result, it is apparent that Cooley has qualms with Hess work on education research. It is less apparent that he has an opinion on the question I attempt to raise. Even so, Cooley is wrong to assume that I accept Hess views without criticism. Nowhere in my piece is there evidence of this. I say Hess work in this area is important, but certainly that does not constitute full affirmation. I use Hess writings on education research to introduce my commentary because they are highly relevant, well-known, provocative pieces of work. Moreover, Hess work on AERA at times addresses the question of research methodology, but it does so in insufficient detail. It therefore makes very good sense to use his commentaries as a launching pad for my own commentary: they are well known and they only slightly address the issue I want to emphasize. This is fairly standard academic practice; by no means does it denote an uncritical acceptance, just as it does not represent a superficial engagement.
Cooley assumes that raising a question about the amount of qualitative research in a field, and relating that question to the struggles a field experiences attempting to build a knowledge domain, constitutes a barely veiled attack on non-quantitative research. In my view this constitutes a knee-jerk, defensive reaction. All I did was raise a question, which is quite different from waging an attack.
Cooley thinks that the object of my piece is the value or merit of qualitative research. This constitutes a sore misreading. The object of my piece is a question about the sociology of education research. My specific question is about the distribution of research methods in education. It is about the degree to which the large amount of qualitative research in the field of education is related to the fields inability to build a common knowledge domain.
Cooley relates my remarks on federal efforts to improve the scientific quality of educational research to the current Executive administration of the Federal government without recognizing that the Education Science Reform Act of 2002 was passed by Congress with bipartisan support. This misreading allows for the introduction of a sorely misplaced political attack that has little to do with my commentary or education research.
Cooley interprets my observation that conceptual/theoretical papers present no original data as a full dismissal of such papers. Like other places in his essay, here Cooley shows an inability to separate an empirical observation from a value-laden position. In no way does my observation constitute dismissal. Further, my commentary was itself fairly conceptual, so I obviously do not dismiss such work.
At several points in his reaction, Cooley implies that my question stems from a concern for public policy. He juxtaposes this position with that of university professors. This constitutes a fundamental misreading of my commentary. Though my commentary is concerned with the role that education research plays in informing the practice of education, it is primarily concerned with the academic formidability of the field of education. Indeed, my ending question is concerned with the education knowledge domain itself.
Cooleys misinterpretation of my essay is particularly obvious in his final paragraph, where he finally attempts to address the question I end my commentary with. My question, again, is about the degree to which the seemingly high portion of qualitative work at AERA is related to the struggles the field experiences attempting to come together and build a core knowledge domain. Cooley responds to my question by defending qualitative research. He writes:
Instead, I would argue that the rise of qualitative research can be linked to the fact that strictly quantitative research is not infallible in answering all of the questions that emerge from the American educational system.
In no way is it clear that this response constitutes an answer to my question. Nowhere in this response is there evidence of Cooley grappling with the nature of qualitative research; with the degree to which it might tend to discourage co-authorship, collaboration, and citation; with (in light of its prevalence in the field) the role it might play in the struggles the field experiences attempting to build shared, common knowledge among its members. Instead, a defense of qualitative research is given, which is indicative of Cooleys entire response to my question and my commentary: Cooley sees my question as threatening to qualitative researchers, and he is quick to jump on the defensive. Cooley appears incapable of seeing that my question merely constitutes a bringing together of correlates, put together in the form of a question. He therefore appears incapable of addressing my question directly.
REARTICULATING A QUESTION
There are additional aspects of Cooleys reaction that I disagree with, but above Ive attempted to highlight what I see to be its most pressing problems. Id now like to rearticulate the question I was originally trying to illuminate. Because it is apparent that Cooley misunderstood my essay, it is far more important to be clear about this question than it is to belabor my concerns with his reaction.
In trying to be clear, perhaps imagery can help facilitate a useful transition.
The Disciplinary Structure of Social Science Journals: Co-citation ties among 1,657 Social Science Journals
Source: Moody, J. & Ryan Light. (2006). "A View From Above: The Evolving Sociological Landscape" The American Sociologist 38:67-86
These two images are cited with permission from James Moody, a Sociologist at Duke University who has been an active contributor to the Sociology of Science in recent years. They both represent the same data set the one on the left is a three-dimensional rendition of the two-dimensional image on the right. What these images show is the network of co-citation ties between 1,657 Social Science journals. Any two journals will have a strong edge connecting them to the extent that all other journals cite them similarly (Moody, 2006). Co-citation similarity essentially allows for estimating the degree to which two journals are substitutable. This is basically a way of identifying whether two journals do the same stuff.
In the images above, journals were placed close to one another based upon the strength of their connection. Taken as a whole, this helps to produce the variation in density shown in both images. With this completed, Moody placed descriptors on the most prominent journals in a variety of local regions. The more dense a local region, the more connected the academic discipline that it represents.
The field of education is not visible in the 3-D figure because it is hidden behind the Psychology mound. It is visible, however, in the 2-D figure it lies somewhere in the interstitial space between Psychology, Social Work, Nursing, and Health. Compared with other disciplines, education does not at all represent a connected area of study. This means that few education journals are cited similarly by other social science journals, and this implies that few education journals are seen as being engaged in the same work. In contrast, the disciplines that are the most connected are Economics, Law, Political Science, and Psychology. These disciplines, as a result, are well positioned to generate, legitimate, and sustain a strong foundation of basic core knowledge. Additionally, one must note that these disciplines (with the exception of Law) are quite likely to have more quantitative inquiry than education research. This raises the question of whether quantitative research plays an important role in providing the platform that allows each of these disciplines to come together for common knowledge production; just as it raises the related question of whether the absence of prevalent quantitative inquiry in education is related to the struggles the field experiences attempting to come together to build a core knowledge domain.
This set of questions is more than speculative. Some of Moodys broader work on scientific collaboration has shown that there is a clear relationship between co-authorship and quantitative research. In Moodys (2004) study of the discipline of Sociology, he found that quantitative studies were more likely to be coauthored. This meant that those writing quantitative work were more likely to be embedded in the center of a scientific collaboration network. Those working in largely qualitative areas were left out of the center of the network, which implies that they were less instrumental in bringing the discipline together for core knowledge production. This set of findings suggests that quantitative researchers play an instrumental role in bringing an academic discipline together. If we apply this to education research I think we have to ask ourselves whether the relatively small amount of quantitative research in the field is related to its struggles at generating and sustaining a formidable knowledge domain that is consequential in public life.
This is the question Ive been trying to bring to the fields attention the reader will note that in rearticulating this question I am again not vocalizing a concern with the merit of qualitative or quantitative research methodologies in themselves. I am simply providing some evidence and asking the field how it should be interpreted. It is not clear to me that Aaron Cooley has offered a serious engagement with my question, but I look forward to subsequent discourse on the matter, be it with him or otherwise.
1. I can only assume that Cooley meant one of two things: Perhaps Cooley read my bio in TCR and saw that my doctoral dissertation is concerned with the impact that political ideology has on the field of education and perhaps he just inferred that what I meant by ideology had something to do with research methods. This is not really a warranted assumption, for its far more of an empirical question, and its also inaccurate, for the body of empirical work that is my dissertation is at best tangentially related to my commentary on methods. It uses the same data set, but theres really not much more of a connection, at least at this point. To be fair, I suppose it is also possible that Cooley has in mind an abstract notion of ideology that encompasses most of what is ideational, which is far more than I. If this were true, he might have felt warranted relating my questions about research methods to the concept of ideology. That is fine, but it does not mean that my commentary is about ideology at all; it simply means that Cooley relates my commentary about research methods to some larger concept that he calls ideology.
Cooley, A. (2007). Epistemology, science, and the politics of educational research. Teachers College Record.
Knight, K. (2006). Transformations of the concept of ideology in the twentieth century. American Political Science Review, 100(4), 619-626.
Moody, J. & Ryan Light. (2006). A view from above: The evolving sociological landscape. The American Sociologist, 38, 67-86.
Moody, J. (2004). The structure of a social science collaboration network: Disciplinary cohesion from 1963 to 1999. American Sociological Review, 69, 213-238.