Multilevel Approaches to Documenting Change: Challenges in Community-Based Educational Research
by Margaret A. Gallego, Robert Rueda & Luis C. Moll - 2005
Increasing availability of funds for development, design, and evaluation of alternative learning environments has challenged educational researchers to develop and validate innovative and effective interventions. The focus on accountability has resulted in an accelerated effort to record events, activities, and participation in substantive ways that suggest significance, statistical and otherwise, and that warrant further program improvements and modification. Yet, relying on traditional individual standardized measuresones that are specifically designed to discriminate among students and that are better suited to the study of controlled experiments in laboratories rather than the sporadic and often spontaneous interactions common to learning settings in and out of schoolleaves educational researchers generally ill equipped. Even as alternative educational programs are financially supported, the sanctioned means with which researchers and program developers document success of all educational programs have progressively narrowed, favoring traditional experimental designs with an emphasis on whether it works rather than on understanding why the program is successful. In this article, we used a multimethod, multilevel analysis to document the underlying dynamics of specific alternative learning contexts to identify generalizable principles while allowing for local variation.
A concern about low academic achievement drives current school reform and restructuring efforts. Indeed, the long-standing pattern of educational underachievement among students from linguistically and ethnically diverse cultural groups is especially troubling because it often results in stigmatizing educational labels, placements in low-level educational tracks (including special education), diminished educational expectations, and other negative educational consequences. One common explanation for this pattern is rooted in deficit theories that place the blame for poor achievement on intrinsic characteristics of the child and the family or community. In the past, these have included factors such as low intelligence, poor motivation, and lack of linguistic or higher order thinking ability. Researchers (Gibson, 1997; Mehan, Hubbard, & Villanueva, 1994) have noted that these presumed deficits are often used to characterize members of entire cultural, linguistic, or economic groups (Harklau, 2000; Trueba, 1987; Valdes, 2001; Valencia, 1998), often relying on traditional individual standardized measures specifically designed to discriminate among students.
Although some work in this area has focused exclusively on the individual child and his or her presumed deficits, other approaches have begun to look at the nature of schooling as well. It has been argued, for example, that there is a great deal of diversity even among what is thought to be the normative classroom/school culture (Gallego & Cole, 2001), and much research therefore fails to document the varied ways that individual children experience schooling. Indeed, research has consistently reported that the standard curriculum is rarely neither delivered nor received in a standard manner (cf. Anyon, 1980; Eder, 1983; Gamoran, 1989; Moll, Diaz, Estrada, & Lopes, 1992; ODonnell, Wilson, & Tharp, 2002; Rist, 1970).
Alternatives to deficit-based explanations of achievement have drawn heavily on culturally grounded research approaches drawn from disciplines such as anthropology. Erickson (1986, 1987), for example, offered an alternative explanation for minority underachievement, drawing on an approach he described as interpretative. Anthropologically and socioculturally oriented educational research focuses upon individuals interaction within specific social contexts (e.g., schools and classrooms) and on how these interactions are consequential to students academic success. This shift in orientation toward the understanding of individuals within specific contexts has directed research toward the investigation of the features of the basic social organization, the underlying assumptions of traditional school, and the effects that these might have on students participation, and ultimately, on student achievement (cf. Gamoran, 1987; Spindler, 1982; Valdes,
Those who have conducted research from this perspective argue that learning and development occurs in settings beyond the classroom (Lave & Wenger, 1991; Moll et al., 1992; Rogoff, 1991; Rogoff, Turkanis, & Bartlett, 2001) and in ways not amenable to traditional measurement and assessment. Investigation of childrens participation in varied contexts (classroom/ school, home, community, churches) has been critical in offering alternative explanations for differential educational achievement among students (Gutierrez, 2002; Harklau, 2000; Heath, 1983; McDermott, 1993; Rueda & Mehan, 1986). Collectively, these research studies portray competence or incompetence as context bound; that is, the interactive situations and physical environments influence performance.
There is some evidence that some environments, such as after-school clubs and activities, provide an interesting alternative to the typical social organization of many classrooms. Indeed, free of typical constraints (e.g., mandatory curriculum, highly structured and hierarchical environments), these after-school environments, once assumed to be idle or recreational time, have the potential to support and enhance students school-based knowledge (Gallego & Blanton, 2002; Schauble & Glaser, 1996; Vasquez, 2003).
Research has indicated that those features, such as internal motivation, greater flexibility, and cooperative arrangements, are more frequently found in alternative settings than in school settings but are not guaranteed. Resnick (1991) cautioned that simply removing oneself physically from the classroom is insufficient because the majority of the supplemental learning environments outside schools only replicate the typical interaction and content provided in schools. Furthermore, piecemeal attempts at adopting nonschool characteristics into the educational setting have failed to be sustained with depressing predictability (Cuban, 1991; Sarason; 1991). We argue that a systemic approach to understanding individual performance within specific contexts is necessary.
Government support of alternative learning environments is clearly evident in the increasing availability of funds for development, design, and evaluation (cf. California Department of Education, 1994). Educational researchers are challenged as never before to develop and validate innovative and effective interventions. The focus on accountability has resulted in an accelerated effort to record events, activities, and participation in substantive ways that suggest significance─ statistical and otherwise─ and that warrant further program improvements and modification. Yet educational researchers are generally ill equipped, with traditional methods that are better suited to the study of controlled experiments in laboratories. Further, these traditional research methods, insufficient and inappropriate for the documentation of the learning that occurs in real classrooms, are the same research tools and designs available to researchers interested in examining learning as it occurs in nonschool settings─ that is, presumably, learning in the ‘‘real’’ world. In sum, even as alternative educational programs are financially supported, the sanctioned means with which researchers and program developers document success of all educational programs has progressively narrowed, favoring traditional experimental designs that emphasize whether it works rather than understanding why the program is successful (National Clearninghouse for Comprehensive School Reform, 2003). One serious limitation of traditional approaches to documenting student achievement is that the outcomes, or treatment effects, are thought to reside exclusively within the individual. Another is that traditional analysis seeks an understanding of individual difference/performance based on product scores (achievement) without attending to the circumstances that surround the generation of these outcomes (i.e., the process; see Rogoff, 2003 for discussion of these points). Last, traditional research approaches operate as if all students participation is equal. That is, there may be allowances made for assessing variability in interventions (fidelity of treatment), but not for how these may be differentially experienced by individuals in that setting.
It can be argued that the study and design of alternative learning environments such as after-school settings have much to offer in terms of school reform and change. However, there are unique challenges in studying learning and development in such settings. Given the potential of such settings to inform more traditional school practice, we believe that it is important to document and understand the conceptual and methodological challenges posed while conducting research in and out of school and community contexts.
In this article, we describe one such effort based on work in an after-school group of projects collectively known as the Fifth Dimension. In particular, we focus on the methodological challenges inherent in documenting childrens learning in this type of environment. Although a number of sites make up the Fifth Dimension network, each with its own characteristics (or what we have come to refer to as personalities), we elaborate here on the various methodological challenges we faced based upon our study of four of these sites in Southern California. The goal is to describe and explore some of the issues that have arisen in the course of documenting the features of the social organization and operating principles of specific sites, while documenting individual participation and outcomes, particularly those related to literacy and language proficiency. In addition, we describe some of the adaptations and approaches that we have used based on those considerations.
We begin our discussion with an overview of the Fifth Dimension project, our theoretical orientation toward the project, our research questions, and data collection methods for documenting the learning that took place in these nonschool environments. Second, we review how our theoretical orientation to research helped us to address methodological challenges posed in the evaluation of this project. Finally, we conclude by describing how our approach to field methods may be useful for reconsidering research conducted both in and out of the classroom/school context.
THE FIFTH DIMENSION PROJECT: A SYSTEM OF COMMUNITIES
The Fifth Dimension project is a collective of after-school programs located in a variety of community-based settings, such as Boys and Girls Clubs, YMCAs and YWCAs, recreation centers, and public schools. It was originally developed in California as a project at a single university and has since led to the development of affiliated sites statewide, nationally, and even internationally. Each site is aimed at improving the literacy of elementary school children; full participation in activity requires that everyone learn to use and communicate about many different types of educational activities, computer and noncomputer (a fuller description of this history and approach involved in this work is found in Cole, forthcoming).
Each Fifth Dimension site typically operates through a collaborative agreement of three partner entities. One partner is the university that provides various types of resources, including some that directly support the site (i.e., labor in the form of undergraduates or partial funding for site staff). The second community partner is the community agency or organization that provides the physical location of the site and other resources, to varying degrees, such as computers, and in some cases, staff members. Both of these entities typically embrace a host of various institutional and bureaucratic issues that impact the individual partnership and that vary from site to site. The third partner is the extended and loosely organized external community of participants of other Fifth Dimension sites (in the United States), which is collectively referred to as the Distributed Literacy Consortium (DLC). Relationships among the various sites are interactive in ways that differ for each site and reflect the dynamic changes of particular sites over time. Collectively, the partnership among these three communities aims to (a) create sustainable activity systems in different institutional settings, (b) foster cognitive and social development among participants, and (c) provide a context in which undergraduates from disciplines such as developmental psychology, communication, and teacher education have opportunities to connect theory with practice and at the same time deliver community service to children in the local community.
In service of the three overarching goals are the general principles that all Fifth Dimension sites hold in common. These principles are organized under the three categories: structure, participation, and learning.
There is an emphasis on interactive technologies, including, but not limited to, computers, telecommunications, and multimedia.
There is a mythical/virtual entity─ referred to at the various sites as the Wizard, El Maga, Golem, Proteo, or Volshebnik─ who stimulates, amuses, oversees, coordinates, and bemuses participants through the telecommunications system.
There is a focus on diversity: diversity of legitimate goals, diversity of ways of achieving goals, diversity of kinds of literacy promoted.
Intellectual resources and labor are distributed within and among Fifth Dimension sites.
Activity must be a mixture of play and education. The play element is needed for the children to participate; the education element is needed for the adults to justify support.
The mix of play and education must be such that the children come voluntarily.
Activities must allow children a substantial element of personal choice and self-direction within an overall structure designed to promote all participants development of levels of expertise.
Performance in the Fifth Dimension has no direct relation to any grading, testing, or evaluation in the childs regular school. It is an activity to be evaluated by criteria of success intrinsic to the community of practice that engages it.
Learning is an active process that is fostered by norms of interaction in which adults work alongside children as coparticipants and not as directors.
The environment should be dense in occasions for authentic problem solving and communication of the process and products of problem solving.
Children are encouraged to describe, through written and oral language and other modes of expression, how they accomplish tasks and other sense-making activities. It should be noted that these are not rigid prescriptions, but rather general organizing principles. Their implementation and the emphasis and focus of individual sites can and do result in a great deal of variation.
THE SETTING: THE FIFTH DIMENSION
There are noticeable critical elements that identify each site as a Fifth Dimension. At the core of the Fifth Dimension are routines, the patterned regular sequences of activities and behaviors that constitute the day-to-day life at each site (e.g., sign-in, snack time, homework station). Common artifacts include the maze, computers, games, cruddy creatures used to identify participants in board games, and other similar artifacts. The routines are mediated and supported by the artifacts (e.g., participants completed tasks are recorded in activity logs, and progress is indicated by ones position within the maze). Although computers and telecommunications networks are key artifacts, by and large, the level of technology is rather low-end, depending on microprocessors and off-the-shelf software of the kinds that communities are likely to provide through donations. Access to the Internet is also an important element of each system, serving as an important link to other sites and to the overall consortium.
The artifacts contained in each site are, in turn, embedded in various physical settings and locations where specific activities occur (e.g., computer stations, game board table, and maze display). The layouts of various sites are not standard, but rather reflect the unique distinguishing characteristics of each one. However distinct the physical environments, one common feature is that all Fifth Dimension activity takes place after school. This was a strategic design feature and a deliberate socioecological choice, because this period of the day represents unsupervised or unproductive time for many of the students in the communities targeted by the projects. This niche capitalized on the opportunities for intergenerational interactions among participants.
Participants at various sites include students, undergraduates, site coordinators, evaluation team members, parents, and visitors. The participants roles, length of participation, and levels of involvement vary. For example, childrens participation is completely voluntary, and they can leave any time. Undergraduates come and go every semester. Parents participation is variable and varies significantly by site. On the other hand, site coordinators attend site each session and typically remain staff members for multiple semesters, and in some cases, they have remained a part of the activity for years. Once inside, all participants typically follow the established routines at that site. The movement of evaluation team members (the authors) has been unrestricted, allowing them to follow activities, note changes in structure and routines over time, and document breakdowns and how they have been resolved.
A FOCUS ON LANGUAGE AND CULTURE
In an early phase of this work, the funding agency requested that three evaluation teams investigate and document different aspects of the implementation and effects of the projects. One team was designated the cognitive team, another the process team, and a third the language and culture team. The latter was the focus of the authors, and observations centered on four sites located in Southern California. Although each maintains a unique blend of local attributes and Fifth Dimension principles, a strategic selection criterion was that these sites served primarily children whose language background was diverse.
A brief description of each of the four sites is provided in the following paragraphs. In the remainder of this paper, we focus on select examples drawn from each of the sites to illustrate the methodological issues and challenges that research in such sites can engender.
SOLANA BEACH, CALIFORNIA
Two very distinct Fifth Dimension sites operate in the municipality of Solana Beach. The two sites both draw from the same university for student participants. However, their child participants are drawn from two distinct neighborhoods, one upper middle class and the other working class. La Clase Magica (the magical classroom), located at Saint Leos Church within the Eden Gardens neighborhood and serving working-class students, was the site selected by the language and culture team. All neighborhood participants walk to this site with younger and older siblings in tow. University students carpool to the site, about 10 miles from the university campus.
Bilingualism is evident in every aspect of life (e.g., schooling, church life, and neighborhood affiliations) within the neighborhood. The principal investigator of the La Clase Magica site is herself bilingual, biliterate, and bicultural. The explicit goal of the site was to encourage bilingualism and biliteracy. All bilingual communication attempts were encouraged and respected, including code-switching (mixing of English and Spanish in the same sentence), a practice commonly discouraged in schools.
SAN MARCOS, CALIFORNIA
The Escondido Boys and Girls Club served as the host for the Fifth Dimension site in this community. University students were enrolled at Cal State San Marcos and attended site sessions in partial fulfillment of coursework within either the psychology or teacher education departments. Many of the university students reported bilingual abilities in several languages. The children at this site had limited bilingual abilities even though many had exposure to multiple language backgrounds. Biliteracy was not a focus of research at this site, nor was it commonly used in daily operations. The principal investigator was a professor of psychology whose research interest was in cognitive development and computer literacy, and the activities at the site reflected this emphasis.
The local chapter of the Boys and Girls Club served as the host for the Fifth Dimension site in Whittier. University student participants were enrolled in the teacher education program of Whittier College, a short distance from the club. A majority of the university students who participated in the Fifth Dimension had partial to full bilingual Spanish/English proficiency. The children attending the site represented a broader range of ethnicity than the university students did, and many of the children had functional proficiency of Spanish/English bilingualism. This site also referred to itself as a Fifth Dimension. The site principal investigator was a professor of education at Whittier College.
SANTA BARBARA, CALIFORNIA
The host institution of the Santa Barbara site was the local chapter of the Boys and Girls Clubs. The children who attended this site were almost exclusively attendees of an elementary school that followed a dual-language (Spanish/English) immersion program. The university students who attended the site were enrolled at the College of Education at the University of California, Santa Barbara. In most cases, teacher education students were concurrently working with nonsite children in traditional academic settings (classrooms) during the same semester that they attended Fifth Dimension sessions. The principal investigator was a professor of education and bilingual/ biliterate. The Fifth Dimension activity was locally referred to as Club Proteo, a name selected to emphasize the multimedia representation of text and language that was supported and encouraged at the site.
Collectively, participants of the four focus sites represented a broad range of language abilities and language backgrounds. Although all sites served students who came from backgrounds in which English was not the primary language, the degree to which bilingualism and cultural factors were emphasized in daily operations differed. Another source of between-site variance had to do with location. For example, the Fifth Dimension sites that physically resided within a Boys and Girls Club environment constantly competed for participant attendance in the face of many other choices (soccer, basketball, crafts, and so on) available to children. To encourage attendance, these sites allowed a walk-in and try-out policy. On the other hand, participants at La Clase Magica were in a location that had no other activity available, and the Fifth Dimension sessions were the exclusive choices. This had the effect of sustaining a population of children who attended faithfully, while the walk-in attendees were few.
A THEORETICAL ORIENTATION
The task of the language and culture team was to investigate and document aspects of Fifth Dimension language and culture as they played out at different sites. Given the earlier discussion of the importance of sociocultural context and cultural process in learning and development, we adopted a sociocultural, or sociohistorical, approach (Moll, 1992; Rogoff, 2003; Wells & Claxton, 2002; Wenger, 1998; Wertsch, 1998) that focuses on these aspects of development and on the role of cultural tools in this process. In addition, we drew on the notion of communities of learners and transformation of participation (Wenger; Rogoff, 2003) that help conceptualize the ways that learners move from more peripheral to more central members of a given cultural community.
It quickly became apparent that the Fifth Dimension was organized differently than many intervention-oriented educational innovations. Rather than a rigid and inflexible adherence to core principles for the express purpose of sustaining and propagating the innovation, the Fifth Dimension sites adhered to a set of core organizing principles while simultaneously retaining the flexibility to adapt at the local level. Thus, the qualitative characteristics of each system were unique. The characteristics of individual sites led us to refer to themes or generalized personalities that differentiated the cultural features and themes at the various sites. Our analyses suggested that the thematic characterization, or personalities, of each site were heavily influenced by the research of the person running the site (principal investigators). For example, this led to a focus on computer literacy at the San Marcos site, an emphasis on a community service orientation at the Whittier site, an emphasis on primary language maintenance and bilingualism at the Solana Beach/La Clase Magica site, and community involvement and change at the Santa Barbara site. These varied research interests and their subsequent organizational consequences are examples of the flexible adaptation of a core set of organizing principles tailored to the unique context in which the site is located, and served the sites sustainability efforts.
OUR RESEARCH QUESTIONS
Building from a sociohistorical perspective, we have explored the following questions:
What are the unique and recognizable cultural features of the different Fifth Dimension sites?
How are these features developed and sustained?
How do participants become enculturated into the Fifth Dimension culture?
How is success and failure defined and marked within the Fifth Dimension culture?
What language and literacy outcomes are associated with participation in the Fifth Dimension?
COURSE SYLLABUS AND STUDENT APPLICATIONS
Although undergraduate students participation in the Fight Dimension was voluntary, they were required to apply to the course. This was not due to lack of space (although space at every site is an issue), but rather to make it clear to students at the outset the courses time and space demands and restrictions. Review of these applications provided a unique view into each sites representation of itself and into students understandings of the nature of the Fifth Dimension.
The research team developed a standard university student survey for distribution at each of the four focus sites. Students were prompted about their academic background, experience with children, community involvement, and experience with computers. The information gathered from the surveys highlighted the characteristics of students participating in the activity across the system. We anticipated that the students previous school-based experience, coursework, and out-of-school experience with communities and children would influence the field notes generated at each site.
UNDERGRADUATE STUDENT FIELD NOTES
As part of fulfilling the course requirements, students attended Fifth Dimension sites twice per week and generated written accounts of their interactions in field notes. Field notes were submitted to the university course instructor via electronic mail. Student field notes were designed to capture students recollections regarding their interactions with children during their site visits.
Members of the language and culture team conducted periodic interviews with key participants at each site. The interview sessions provided the participants with a venue for reflection regarding specific site issues and topics not often afforded to those in charge of site operation and course instruction. Information gained from these interviews offered the research team members (the authors) important perspectives on the site organization, implementation, and routine procedural activity.
RESEARCH TEAM FIELD NOTES
These written accounts were based on observations made by research team members. One member of the evaluation team was assigned to each of the four research sites. Each member routinely visited the research site (typically one to two times per week) and became familiar with the operations and routines at the site as well as with the site participants (children, university students, staff). After each visit, field notes were written to document the ongoing activities, features, and changes at the site, particularly those that related to the use of language and culture. These notes supplemented the accounts of site activity routinely generated by university students, site coordinators, and project directors. From the team members outsider perspective, these notes detailed information such as the enculturation of newcomers to the site, the management style of the site, and other typical interactions. In this way, the research team members were able to make the familiar strange. The outsider role allowed for the recognition and documentation of features of everyday activities and routines that might otherwise go unnoticed or be deemed unimportant by those participants intimately involved.
From these more general notes, patterns were identified regarding the regularities and differences in the nature and types of activity settings (Tharp & Gallimore, 1988) afforded children at each of the research sites. Review of the field notes indicated that activity settings across the sites represented both common features among the sites and unique characteristics of specific individual sites.
LANGUAGE ASSESSMENT SCALES (LAS)
The Language Assessment Scales (LAS; oral) were administered to children participating in the Fifth Dimension activity during the fall and spring. The instrument was used as a standardized measure of childrens language proficiency, rated on a 5-point scale based on performance on five separate subtest components, including (1) picture identification (objects); (2) picture identification (action words); (3) listening comprehension (a) yes/no response and (b) oral retelling; (4) minimal sound pairs (listening comprehension); and (5) phonemes (oral production).
Childrens language dominance as identified by the LAS was the basis for the selection of either the Spanish or English version of a writing task embedded in a computer-based game, Ace Reporter Reading & Writing. This game simulates a newsroom setting. Children selected game options according to those simulated in the game task card according to headline information. Children were engaged in a series of fact-finding activities to gain information about the news incident. Children were guided through the game options by their quest to answer who, what, when, how, and why questions. When children believed they had sufficient information, they requested an appointment with the editor, who then quizzed the children on the above-mentioned questions.
The language and culture team generated site profiles to accurately portray the complexity and synergy of the varied elements operating at each of the sites. Based upon the data generated from each of the above-described data sources, the team members drafted composite descriptions of each of the sites that characterized a typical session or routine. This was a way for us to reconcile our effort to foreground and background particular data sources while maintaining a focus on the overall picture and the overarching goals of each site. For example, understanding the beliefs and values held by those who participated at the site was essential in judgments that deemed particular artifacts essential and others not.
In attempting to capture the uniqueness of the sites, we confronted several methodological issues that required adaptations of the work of the research team. We will address each of the issues in the remainder of the paper. A summary of these is presented in Table 1.
THE UNIT OF ANALYSIS: WHAT IS AN APPROPRIATE THEORETICAL PERSPECTIVE?
As Rogoff (2003) noted, transmission and acquisition views of learning and development have been predominant in psychology and educational practice. Normally this orientation generates questions such as the following: Where is information stored? How is it retrieved? How is information generalized across situations? How is information translated into action? Often this framework is useful for traditional evaluation activities that focus on the outcomes of individuals as a result of participation in some activity or treatment. Although information gained through such questioning is
Theoretical Framework & Unit of Analysis
There was a strong press for both individual data and outcomes and for processes and outcomes involving a larger unit of analysis (dyads, groups of participants such as undergraduate students, whole sites, or even the entire consortium).
Most theoretical frameworks emphasize one aspect over the others or simply ignore other aspects.
Adopt a theoretical framework that permits backgrounding and foregrounding of specific planes or levels of analysis while taking all levels into account.
Insider vs. Outsider Perspective
Traditional outsider stance is too narrow and inadequate to capture the cultural features of the settings.
Although evaluation is traditionally seen as strictly an outsiders job, this perspective by itself was found to be inadequate to capture the essence of the settings in appropriate detail.
Adapt data collection such that both outsider and insider sources are drawn on.
Product vs. Process Data
There existed a push for both product and process outcomes.
The need for data on the effects of participation in the Fifth Dimension, such as language and literacy development, had to be balanced with data on how the sites functioned as cultural settings.
Adopt a multimethod data collection strategy, including collaboration with other evaluation teams.
Assumptions About the Treatment
Traditional assumptions about treatments violated principles of voluntary participation, self-selection of activities, inconsistent attendance, etc.
The nature of the organizing principles of the Fifth Dimension violate traditional considerations regarding reliability and validity in comparing sites.
Evaluate on a site-by-site basis, measure effects of treatments where possible, or describe development descriptively or on a case study basis.
Traditional designs were quickly found to be too narrow or otherwise inadequate.
It became clear that although there were common principles that organized the cultural features and behavior of the sites, there was marked diversity among them.
Pursue common questions and data collection strategies across sites but allow for flexibility and strive to capture common and unique features of each site search for a commonality in diversity.
Intrusiveness of Data Collection
Data collection was sometimes seen as a threat to the sensitive environments created at the sites.
Requiring students to complete certain tasks was seen as disruptive to these.
useful, it does not retain a holistic view of the dynamics of participants interactions.
As an alternative framework, Rogoff (1995) proposed a useful extension of sociocultural theory that assisted the language and culture team in addressing the evaluative task. Rogoff proposed a view of learning and development as a dynamic process of transformation of participation in a given community of practice. Rogoff s framework orients research to answer questions such as, What are the activities in which people participate? Why and with whom and with what? How do the activity, its purpose, and peoples roles in it transform? How do different activities relate to each other currently, historically, and prospectively?
Participation in any sociocultural activity, such as that in the Fifth Dimension community, was assumed to occur on many planes, or levels, of interaction. Rogoff has suggested that a complete account of learning and development must take into account three levels. The personal plane involves individual cognition, emotion, behavior, values, and beliefs. In educational research, this might correspond to studies of individual student or teacher actions, psychological characteristics, or competence. The interpersonal or social plane includes communication, role performances, dialogue, cooperation, conflict, assistance, and assessment. In educational research, this is often addressed in studies of teaching/learning interactions, such as a study of cooperative learning groups. The community or institutional plane involves shared history, languages, rules, values, beliefs, and identities. This is sometimes addressed in studies of entire schools, districts, professions, neighborhoods, tribes, or cultures.
Sociocultural theory in general emphasizes that these three planes are inseparable; moreover, language is the primary force that defines and connects these planes. While one plane might be foregrounded and the other planes backgrounded for a particular study or analysis, a complete account of learning and development needs to consider all three planes. In practice, the smallest unit of analysis that contains all three planes simultaneously is the activity setting, or the who, what, when, where, why, and how of the routines that constitute everyday life (Tharp & Gallimore, 1988). Several data sources were collected that were representative of participants activity within each of the three planes: individual plane (e.g., scores on Language Assessment Scales; interviews); interpersonal plane (e.g., field notes), and community plane (e.g., site profiles, field notes). Importantly, unlike traditional evaluation research, the unit of analysis here is greater than the individual; rather, it is the individual in interaction with others in a specific activity setting. This unit of analysis, along with the practice of foregrounding and backgrounding various planes of development for different purposes, formed the basic theoretical foundation for the study.
DATA COLLECTION: INSIDER AND OUTSIDER PERSPECTIVES
Often, evaluation is equated with a neutral, uninvolved stance with a high premium on the outsider point of view as a way to minimize bias. We found that such a perspective was limiting because it resulted in a less than valid picture of the sites. Early on in our work, it became clear that the Fifth Dimension sites followed a flexible model of implementation, thus making inflexible data collection strategies minimally useful. We therefore opted for a data collection strategy that relied on site-specific analysis, with an eye toward the consequences for the larger Fifth Dimension project.
A central part of this strategy involved combining data representative of both insider and outsider perspectives. Insider sources of data included course syllabi and student applications, course readings, student surveys, undergraduate field notes, and interviews with site participants. Outsider sources of data included field notes by language and culture team members, the Language Assessment Scales, the Ace Reporter Reading & Writing tasks, and videotape shared with the process evaluation team.
Our general strategy for data collection was to have a specific member of the research team conduct observations and interviews at each site on a regular basis, arrange visits by other team members on a less regular basis, use site-based data such as field notes, artifacts, and individual-child data, and collaborate with other evaluation teams to provide a combination of insider and outsider perspectives. Our approach to data collection required us to consider the interaction among direct and indirect participants. Although the insider and outsider labels appear to be distinct either/or categories, in reality, they were accurately conceived of as points along a continuum that collectively yielded a more complete portrayal of each sites dynamics. In our work, we have identified participants as either insiders or outsiders according to their investment and relationship to the site (e.g., the participants role and scope of responsibilities).
Members of the outsider group included evaluation team members, parents stopping by to pick up their children, and other children who were temporary visitors. Each participant had particular reasons and motivations for being at site. In the case of evaluation team members, the role did not typically include responsibility to be a direct participant with children, but rather to be involved directly, observing and documenting the language and cultural practices of each site.
Members of the insider group included the site coordinator, the principal investigator, and the undergraduate students responsible for daily operations at the site. However, their distinct roles and responsibilities within the activity marked each insider differently. In this way, undergraduate students were typically invested with children for an intense, yet brief (10 weeks), period, while site coordinators and site directors had less concentrated interactions with children but were engaged with children for potentially much longer periods of time. Expertise also varied among specific groups of insiders. For instance, site directors might know more about policy issues and funding, while day-to-day implementation issues were more familiar to site coordinators. These varied perspectives provided different views on similar topics, as illustrated in the following examples.
LANGUAGE AS A RESOURCE: LA CLASE MAGICA/SOLANA BEACH
With an explicit focus on language and bilingual development, participants at La Clase Magica were aware of the sites focus. However, not all participants were equally comfortable with their own linguistic backgrounds. Some university students, for example, were self-conscious about their Spanish abilities and expressed this concern in field notes. I dont know enough Spanish to help them . . . I mean I might say something wrong or in the wrong way. In contrast, the principal investigator of the site viewed the university student language issues as a positive occasion for children to act as language experts and serve as an asset to their university partners successful engagement in Spanish-based computer games. According to the principal investigator, this reversal of expertise was necessary for progress within the culture of the La Clase Magica site and was designed to afford children with genuine opportunities to teach the university students.
QUALITY V. QUANTITY: SAN MARCOS
Though children were offered many activity choices at this Boys and Girls Club, the Fifth Dimension had no trouble attracting a large number of regularly attending children. Balancing multiple goals that at times appeared to be at odds (e.g., pleasing children, collecting data) with each other was particularly difficult for the site coordinator, who stated, It is difficult to keep the kids out when we have just too many interested and they want to come . . . it is hard for me to tell them to come back later. Yet the principal investigator had a vested interest in maintaining a setting that was conducive to supporting research, as expressed in the following interview excerpt: I understand we have many children, but I want to provide optimal interactions among the children that are present . . . this helps too with keeping better records and accurate data . . . the optimal ratio is 2 children to 1 university student.
A MEANS TO AN END/AN END IN ITSELF: WHITTIER
Maintaining positive relationships with the host institution is paramount to the sustainability of all the various sites in the consortium. However, such relationships often require compromise, as the following statement made by the principal investigator of the site makes clear: One of the concessions weve made with our host institution [Boys and Girls Club] is that as part of the educational outreach we make sure that children first go to the homework club before they can come to the 5th D. In a way we serve as a carrot for the kids to get their homework done. Though I believe that often the kids who would benefit most from 5th D are those that can rarely come because they don’t finish their school tasks/homework.’’ Of course, the children had their own opinions of their prerequisite visits to the homework club: ‘‘I don’t get it, why can’t we just go in─ why do we have to do our homework first? . . . I come after school so I can play, not do more school.
VOLUNTARY ATTENDANCE/MANDATORY ACTIVITIES: SANTA BARBARA
As a voluntary program, the Fifth Dimension has little leverage for enticing childrens participation other than providing them with a unique learning environment. At times, the voluntary status of childrens participation poses a challenge for the university student, whose attendance is mandatory. In this case, a teacher education student reflects on this issue: Its a little crazy in here, I really try to keep the kids to their goals, you know the schedule of working on the computer games while they are in here, but . . . its just a lot different than how I work with kids at school [student teaching classroom field placement]. Attempting to keep everyone engaged, the site coordinator suggests, I tell the students that it is a balance of keeping the kids interested in their goals, but not so rigid that they dont want to come back at all.
Coupling insider and outsider accounts of site life afforded a balanced portrayal and allowed participants to address their specific area of expertise.
PROCESS VS. PRODUCT DATA
Although individual child outcomes, including language and literacy growth, were important products to be examined through discrete measures, unique cultural features of the sites required a different orientation. Following Rogoffs framework described earlier, we found the induction of novices into the Fifth Dimension to be a useful exemplar of the transformation of the participation process. This cultural ritual was a common feature of all of these sites, yet each site had its own unique traditions. We noted that that this transformation was not a unidirectional process; the participants change in status transformed his or her role and responsibilities within the Fifth Dimension and also served to simultaneously transform the Fifth Dimension learning community.
The transformation of participation requires a fundamental change in the individuals way of being within a given context. A few examples will demonstrate that change at this level can have far-reaching implications, indeed, causing changes in the transformation of the site. For example, a child participant who attends regularly and has completed required tasks may choose to continue attending the site but assuming a different role: that of a wizards assistant. In this capacity, the collective knowledge about games, computer idiosyncrasies, and the general normative culture is maintained within the system, yet their duties and status among the group have also changed. The cadre of young wizard assistants also transforms the site by providing additional resources for novice participants of all ages. The distributed knowledge represented increases the potential for all participants to be successful within the Fifth Dimension culture.
A second example of individual and site transformation pertains to the type of interactions between university students and their child partners. The success of a Fifth Dimension site requires that university participants may need to transform their interactions with children based on past experience to those that are likely to engender the specific skills that are valuable within the Fifth Dimension culture (e.g., collaboration, risk taking). For example, university students who have had experience with children as tutors need to adjust their approach with children and adopt a colearner/ follower position rather than the more familiar role of provider/leader, because childrens continual participation in the activity establishes them as the experts. If university students do not assume the role of colearner during their site visits, the Fifth Dimension culture will change and ultimately suffer the loss of one of its key core principles: culture of collaboration.
These transformations of individuals over time in turn yield changes within a site, and ultimately transform the multisite system research collective. Noting transformations in a system as complex as the Fifth Dimension requires an ecological approach to analysis that treats the data in relationship to one another, such that information and interactions are woven together. The data can be artificially held steady and examined long enough to understand their separate contribution to the system, then replaced into the context. Ironically, the potential for documenting transformations (and thus development) was enhanced in instances of breakdown, when an aspect of the site was not working and required modification. These discoordinations helped expose the goals, needs, and interests of those involved and therefore highlighted each sites personality.
The various kinds of data collected and the analysis methodologies used by the research team allowed each site to be viewed as a dynamic, rather than a linear, system. When changes occurred, they often stemmed from imbalances in the structure of the site and its participants. As new ways of mediating these crises were developed, they often led to changes in the structure of the site system, similar to the way that a rock thrown into a pond causes a series of ripples. The following example illustrates this complexity.
THE RIPPLE EFFECT: A SPECIFIC EXAMPLE OF SOCIOCULTURAL ACTIVITY IN CONTEXT.
The wait list, a phenomenon at one of the research sites, illustrates how the combination of agendas and goals interfaced─ and in some cases competed─ with one another in practice. At times, the relationships among existing routines, artifacts, and activity settings exposed one situation that led to a series of subsequent changes in a pattern akin to a ripple effect.
The optimal undergraduate/child ratio at the San Marcos site was 1 or 2 children to each undergraduate. The room size and the number of undergraduates enrolled in the course limited the number of children allowed entry. The ratio rule was established to ensure quality interaction and to reduce competition among the children. Prior to the wait list, children often physically jockeyed for position outside the door. If there were insufficient numbers of university students to serve them based upon the specified ratio, children were denied entry. Without an alternative, children periodically knocked on the door to review the possibility of a vacancy throughout the session, causing numerous interruptions.
The wait list was initiated to regulate stand-by status of child participants by formalizing their position on the list and therefore providing them the assurance of a fair entrance policy. The wait list also allowed children the freedom to participate in other activities offered at the Boys and Girls Club while waiting for Fifth Dimension admittance and therefore reduced the number of interruptions and inquires. However, the wait list resolution resulted in unforeseen problems. The wait list was typically checked at the beginning of each site session by the site coordinator, who compared the names on the list with those who lined up outside the Fifth Dimension door. Those physically present and on the list were allowed into the session first. Given childrens variable attendance and the temporary status of the university wizard assistants, the site coordinator often was the only Fifth Dimension participant who could reliably identify children by name. Ultimately, the wait list required the site coordinator to personally locate children on the list as vacancies occurred. Locating children often required looking for them throughout the club. In addition, because university students participation was a required activity for course credit, students looked to the site coordinator to find them children; they were eager to have something to do and someone to do it with.
Our multilevel approach and sociohistorical orientation to the activity suggested that we employ various perspectives to better understand the wait list phenomenon. For example, the physical list of names constituted an important cultural artifact designed to regulate activity. The physical space and associated interactions while the wait line formed were important activity settings that characterized this site. The set of negotiated practices established as the routines necessary for entrance into the activity further characterized daily life at this site. The wait list did regulate daily site attendance, although it required the organization of the physical artifacts, shifted the roles and responsibilities of participants, and had implications for the negotiations of university course requirements. Similarly, other activities and artifacts that changed in their use over time resulted in subsequent changes to the overall system and were therefore both causes of change and effects of change. These ripple effects were identified (a) during the original site development, including artifacts, activities, and the cultural practices surrounding them; (b) during the subsequent discoordinations or unforeseen problems as a result of progress; and (c) in the resulting modifications to the system and the ensuing creation of new iterations of artifacts, activities, and cultural practices. Change was made visible through tracking discoordinations (breakdowns) using a combination of product and process approaches to data analysis.
Initially, there was discussion among team members regarding the possibility of intersite comparisons across various dimensions typical of a traditional comparative approach. It soon became clear that although sites shared a common set of organizing principles, traditional comparisons on narrow dimensions would be of limited use. Rather, we opted for a case study approach. By generating a site profile for each site based upon our review of all the data sources, we were able to identify grounded research questions that could be pursued across all sites. We also allowed for common data collection strategies across the sites, while allowing for flexibility. In essence, we followed a strategy of trying to discover the commonality amid the diversity represented by the sites day-to-day routines, cultural practices, uses of artifacts, and overall themes or personalities.
ASSUMPTIONS ABOUT THE TREATMENT
The Fifth Dimension was designed to accomplish school-like goals (e.g., literacy development) in a non-school-like environment (e.g., voluntary participation). The organizing principles and structure that characterized the sites as a whole suggested that they differed in fundamental ways from most school activities. These unique features of the Fifth Dimension and other similar alternative environments may attract students whose attendance in traditional and formal learning settings is unlikely. We found that these features make it difficult, if not impossible, to achieve the level of control considered necessary to evaluate the effect of any treatment or program on childrens learning. For instance, childrens attendance in the Fifth Dimension is completely voluntary, often resulting in inconsistent attendance. Voluntary participation makes the investment in data collection intense, with no assurance of a return for this effort. Moreover, activities are normally self-selected so that the nature of a childs participation might be qualitatively different from that of any other student, making comparisons among children difficult.
We attempted to address these issues in various ways. For example, in analyzing the individual product level data, we attempted to relate student changes to the amount of participation by individual children. A second strategy was to employ a larger unit of analysis than the individual child. For example, in looking at language use, we noted clear patterns in the relative amounts of English and Spanish used in a site as a whole rather than in individual children. Another strategy was to collaborate, where possible, with other research teams to share data where appropriate and thus combine resources. Finally, as discussed earlier, we adopted a multilevel and multimethod strategy of data collection that allowed us to examine several aspects of the same dimension, such as language use over time.
INTRUSIVENESS OF DATA COLLECTION
Addressing the research questions required a range of data collection and analysis procedures. Insider information was most appropriate for answering the first query regarding the cultural features of the Fifth Dimension. Delineating the nature of these cultural features and how they developed was best addressed through the review of field notes generated by direct participants─ that is, insiders to the activity. These data included field notes written by undergraduate students who were directly engaged in tasks with children, and field notes written by the site coordinator, who recorded daily events at the site. Indeed, many of the field notes were sent via electronic mail directly to the accounts of the language and culture team and were therefore minimally intrusive to sites daily operations.
The questions related to the enculturation process in the Fifth Dimension and the local definitions of success were addressed by reviewing information gained in the direct interviews of each of the site coordinators and the site directors. Their distinct roles provided them the experience and distance from focused interaction with children useful to their reflection of and identification of overarching patterns.
The language and literacy outcomes associated with participation in the Fifth Dimension required the documentation of performance on more formal measures not directly connected to everyday site activities. Members of the research team thus administered tests and collected data. It is the last question that required the most school-like of data collection procedures: the Ace Reporter Reading & Writing tasks. The obtrusiveness of these measures was problematic because of the nature and cultural organization of the Fifth Dimension sites. Most activities, as mentioned earlier, were self-selected by students. In addition, sometimes the undergraduate-child or child-child collaborations were fragile, especially in early stages or for students who tended to have difficulty participating in such types of interactions. The need of the research team to have children engage in certain tasks, in a certain way, and at certain times bumped up against the cultural norm of the sites to maintain a particular type of environment for its participants. Although the use of rewards for participation and completion was considered, the team felt uncomfortable with this approach because it conflicted with the basic cultural norms of the sites.
One strategy that proved relatively successful was to embed the tasks in the maze, which was already used as a coordinating device at each site. In this way, the distinction between a research task and ongoing, more familiar activities was minimized. Another strategy was to have site staff help to collect data where possible, because they were more familiar to the students and were perceived as nonthreatening. Also, at each site, the team member who most consistently collected field notes was drawn on as a valuable resource and after extended time at site became a familiar figure to the participants, thus making data-collection activities easier.
Even as alternative educational programs are financially supported, the sanctioned means with which researchers and program developers document success of all educational programs has progressively narrowed. The current federal pressure for traditional experimental designs with an emphasis on what works is favored for the purposes of packaging and replicating, and ultimately for scaling up (National Clearinghouse for Comprehensive School Reform, 2003). In contrast, our alternative research design draws upon sociocultural theory as a model and sought to understand the underlying dynamics of specific contexts to identify generalizable principles while allowing for local context variation.
We have found that a multimethod, multilevel analysis approach was the most useful for studying the complexities of the Fifth Dimension for several reasons. First, the complexities of each site made them appear unstructured and even at times chaotic; however, our analysis revealed the systematic organization of each site. Second, although language analyses most often focus on individual students, we were able to document varied constellations of language and culture that characterized the site as a whole but that were not as visible in specific individuals. Third, there was an important tension related to sites attempts at producing a unique collective identity while maintaining core principles of the overall consortium. Finally, features that made the sites non-school-like (e.g., volunteer attendance and self-selection of activities) were problematic for conducting a clean analysis of the effects of childrens participation. Although these features made documentation and evaluation difficult, we believe that settings such as the Fifth Dimension warrant greater investigation both for the challenge they pose standard, traditional evaluation models and for their broader implications for childrens school success.
Anyon, J. (1980). Social class and the hidden curriculum of work. Journal of Education, 162, 6992.
California Department of Education. (1994). Kids time: A school age care program guide. Sacramento: Child development department, California Department of Education.
Cole, M. (Ed.) (in press). Creating and sustaining alternative educational activities: diversity as a tool for educational design. New York: Russell Sage.
Cuban, L. (1991). Reforming again and again and again. Educational Researcher, 19(2), 211.
Eder, D. (1983). Ability grouping and students self-concepts: A case study. Elementary School Journal, 84(2), 149161.
Erickson, F. (1986). Qualitative methods in research on teaching. In M. C. Wittrock (Ed.), Handbook of research on teaching (3rd ed., pp. 119161). New York: Macmillan.
Erickson, F. (1987). Transformation and school success: The politics and culture of educational achievement. Anthropology & Education Quarterly, 18, 335356.
Gallego, M. A., & Blanton, W. E. (2002). Cultural practices in Fifth Dimension sites around the world. In R. Garner, Y. Zhao, & M. Gillingham (Eds.), Hanging out: Community-based after-school programs for children (pp. 137147). Westport, CT: Bergin & Garvey.
Gallego, M. A., & Cole, M. (2001). Classroom culture and culture in the classroom. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 951997). Washington, DC: American Educational Research Association.
Gamoran, A. (1987). The stratification of high school learning opportunities. Sociology of Education, 60(3), 135155.
Gamoran, A. (1989). Measuring curriculum differentiation. American Journal of Education, 97, 129143.
Gibson, M. (1997). Exploring and explaining the variability: Cross national perspectives on the school performance of minority students. Anthropology & Education Quarterly, 28, 318329.
Gutierrez, K. D. (2002). Studying cultural practices in urban learning communities. Human Development, 45, 312321.
Harklau, L. (2000). From the good kids to the worst: Representations of English learners across educational settings. TESOL Quarterly, 34, 3567.
Heath, S. B. (1983). Ways with words: Language, life and work in communities and classrooms. Cambridge, England: Cambridge University Press.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge, England: Cambridge University Press.
McDermott, R. P. (1993). The acquisition of a child by a learning disability. In S. Chaiklin & J. Lave (Eds.), Understanding practice: Focus on activity and context (pp. 268305). Cambridge, England: Cambridge University Press.
Mehan, H., Hubbard, L., & Villanueva, I. (1994). Forming academic identities: Accommodation without assimilation among involuntary minorities. Anthropology & Education Quarterly, 25, 91117.
Moll, L. C. (1992). Vygotsky and education: Instructional implications and applications of sociohistorical psychology. New York: Cambridge University Press.
Moll, L. C., Diaz, S., Estrada, E., & Lopes, L. (1992). Making contexts: The social construction of lessons in two languages. In M. Saravia-Shore & S. F. Arvizu (Eds.), Cross-cultural literacy: Ethnographies of communication in multiethnic classrooms (pp. 339363). New York: Garland.
National Clearinghouse for Comprehensive School Reform (NCCSR). (2003). National Clearinghouse for Comprehensive School Reform Newsletter: Using scientifically based research in school reform, 5(3).
ODonnell, C. R., Wilson, K. K., & Tharp, R. G. (2002). The cross-cultural context: Lessons from community development projects. In G. B. Melton & R. A. Thompson (Eds.), Toward a child-centered, neighborhood-based child protection system: A report of the consortium on children, families, and the law (pp. 104114). Westport, CT: Praeger.
Resnick, L. B. (1991). Literacy in and out of school. In S. R. Graubard (Ed.), Literacy: An overview by 14 experts (pp. 169185). New York: Noonday Press.
Rist, R. C. (1970). Student social class and teacher expectations. The self-fulfilling prophecy in ghetto education. Harvard Educational Review, 40, 411451.
Rogoff, B. (1991). Apprenticeship in thinking: Cognitive in social context. New York: Oxford University Press.
Rogoff, B. (1995). Observing sociocultural activity on three planes: Participatory appropriation, guided participation, and apprenticeship. In J. V. Wertsch, P. Del Rio, & A. Alvarez (Eds.), Sociocultural studies of mind (pp. 139164). Cambridge, England: Cambridge University Press.
Rogoff, B. (2003). The cultural nature of human development. New York: Oxford University Press.
Rogoff, B., Turkanis, C. G., & Bartlett, L. (2001). Children and adults in a school community. New York: Oxford University Press.
Rueda, R., & Mehan, H. (1986). Metacognition and passing: Strategic interactions in the lives of students with learning disabilities. Anthropology & Education Quarterly, 17, 145165.
Sarason, S. (1991). The predictable failure of educational reform. San Francisco: Jossey-Bass.
Schauble, L., & Glaser, R. (1996). Innovations in learning: New environments for education. Mahwah, NJ: Erlbaum.
Spindler, G. (1982). Doing the ethnography of schooling: Educational anthropology in action. New York: Holt, Rinehart and Winston.
Tharp, R. G., & Gallimore, R. (1988). Rousing minds to life: Teaching, learning, and schooling in social context. New York: Cambridge University Press.
Trueba, H. (1987). Success or failure: Learning and the language minority student. Cambridge, England: Newbury House.
Valencia, R. R. (1998). The evolution of deficit thinking: Educational thought and practice. Bristol, PA: Falmer Press.
Valdes, G. (1998). The world outside and inside schools: Language and immigrant children. Educational Researcher, 27(6), 418.
Valdes, G. (2001). Learning and not learning English: Latino students in American schools. New York: Teachers College Press.
Vasquez, O. A. (2003). La clase magica: Imagining optimal possibilities in a bilingual community of learners. Mahwah, NJ: Erlbaum.
Wells, G., & Claxton, G. (2002). Learning for life in the 21st century: Sociocultural perspectives on the future of education. Malden, MA: Blackwell.
Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. New York: Cambridge University Press.
Wertsch, J. V. (1998). Mind as action. New York: Oxford University Press.