Changing the Context of Student Engagement: Using Facebook to Increase Community College Student Persistence and Success
by Loris Fagioli, Cecilia Rios-Aguilar & Regina Deil-Amen — 2015
Background: Community college leaders are now turning to social media/social networking sites for new avenues and opportunities to increase students’ interaction, engagement, and collaboration with peers, faculty, and staff. Social media may be a particularly attractive option because it can provide a potentially effective and exciting mechanism for catalyzing such connections for students.
Purpose/Objective: This study examines the use of social media/social networking sites and its relationship to academic outcomes in the context of community colleges.
Population/Participants: We used longitudinal data from about 17,000 students who joined a Facebook based online application (the Schools App) in seven community colleges across the country. We compared these members to students who did not join the app for an overall sample size of about 98,000.
Research Design: This study used a quasi-experimental design with propensity score matching and random effects regression to estimate the effects of online engagement on student outcomes.
Findings/Results: We find that there is, indeed, a relationship between social media use and academic outcomes. The most active users as well as passive users had the highest GPAs and chances of continuing the next semester compared to inactive members of the online community as well as compared to nonmembers.
Conclusions/Recommendations: We find that certain forms of online engagement have a distinct relationship with GPA and persistence. The results of this study also suggest that, although potentially valuable, it is not easy to build an online community. Sustaining continued use of the application was challenging and strongly dependent on the quality and relevance of the posted comments and discussion. Nevertheless, this study found that for those who continued to use the application, there were positive effects in terms of student outcomes. Our findings further support the notion that integration is relevant for community college student persistence, but the nature of that integration—more simultaneously social as well as academically oriented—is important to consider in both offline and online contexts. Recommendations focus on a more strategic use of social media, which puts specific emphasis on answering questions and getting involved in online communities and not using social media solely for marketing or dissemination of information purposes. On campuses where students realized that answers to pertinent questions were available (through administrators and peers), online engagement was of high quality. Sustaining high quality online interaction is therefore one element in ensuring a positive effect on student engagement and outcomes.
Online interactions are changing the way college students are communicating, getting information, and connecting with others (Junco & Cole-Avent, 2008). But despite the widespread use of social media among students and the increasing numbers of institutions using social media sites to reach out and connect with students, little is known about the benefits and challenges of using social media/social networking sites1 in certain educational contexts, such as community colleges (Davis, Deil-Amen, Rios-Aguilar, & Gonzáles-Canché, 2015). This paper presents analyses of students online activity, utilizing an internal college Facebook-based application (the Schools App2) in seven community colleges across several states. This paper is structured as follows: After a short literature review and presentation of the theoretical framework, we use descriptive statistics to describe the online interactions of students on the Schools App. Second, we use multiple and logistic regression analyses to examine how usage and online engagement are related to community college students academic outcomes as measured by GPA and persistence rates. In examining these relationships we took great care to address the selection bias that comes along with voluntary participation of online activities. Finally, we discuss the findings and their implications to policy and future research.
Community college enrollments have increased 741% since 1963, compared to increases of less than 200% within each of the public and private (not for profit) four-year sectors (Provasnik & Planty, 2008). This increase is due in part to the community colleges being a financially viable option for many students in the current economic context (Saenz et al., 2011). Another reason for this increase is the commitment of community colleges to providing multiple pathways of access, especially for first-generation, racial/ethnic minority, low-income, immigrant, and under-prepared students (Bragg, Kim, & Barnett, 2006; Teranishi, Suráez-Orozco, & Suárez-Orozco, 2011). But, despite enrollment gains among under-represented and marginalized groups of students, more than half of them have characteristics shown to reduce their likelihood of persistence and completion (Saenz et al., 2011). Thus, even with increased access, community colleges are struggling to graduate students. Recent data indicates that only 40% of the degree-seeking students who started at a community college completed a degree or certificate either at the starting institution or at a different institution within six years (Shapiro, Dundar, Ziskin, Yuan, & Harrell, 2013).
Many strategies provided by scholars to community colleges to improve college completion rates revolve around the notion of engagement: improve student engagement, enhance faculty engagement, and enhance engagement outside of the institution (Center for Community College Student Engagement [CCCSE], 2014; Kuh, Kinzie, Schuh, & Whitt, 2010; Zepke & Leach, 2010). Findings from the most recent research from the CCCSE indicate that higher levels of engagement (e.g., hands-on learning, faculty-student interaction, and participation in college-run student support programs and clubs and organizations) are associated with higher student outcomes. Given the low levels of academic achievement and college completion, and knowing their unique dynamics and complexities, community colleges must find effective ways to increase engagement by altering the context. We argue here that student engagement needs to be considered across multiple dimensions (e.g., real-time, multiple locations, student-driven, mobile, and highly visual), perhaps diverging from more traditional ways of conceptualizing engagement.
Many interventions that aim to promote community college student engagement (e.g., connecting students with faculty and other peers, mentoring, and orientation programs) have been implemented (see Brown, King, & Stanley, 2011, for a comprehensive review). Unfortunately, the usefulness of existing knowledge is limited in terms of generalizability of findings (most studies are based on single institutions), methodological limitations (e.g., use of self-reported data and failure to control for selection bias), and lack of theories and/or conceptual frameworks specific to community college students.
Community colleges are now turning to social media for new avenues and opportunities to increase students interaction, engagement, and collaboration with peers, faculty, and staff. Social media is a particularly attractive option because it can cost-effectively bridge the important yet severely limited services of counselors and advisors. Could social media provide a virtual space to build connections and a sense of campus community which is too often absent at such commuter-based campuses? This question is at the center of this ongoing project and the current paper is the first attempt to empirically test this hypothesis.
Social media/social networking sites such as Facebook have become an integral part of a students life inside and outside of college. These technologies are reshaping the way students communicate in general and within their college community (Davis et al., in press). The 2009 Community College Survey of Student Engagement found that the more students use social networking tools to communicate with other students, instructors, and college staff regarding coursework and other academic purposes, the higher their levels of engagement (CCCSE, 2009). Community colleges enroll underrepresented, commuting, nonresidential students whose busy lives are often filled with family and work obligations. As a result they tend to lack participation in the campus community of more traditional and high achieving students (Deil-Amen, 2011). We therefore believe it is critical to examine how community college students are incorporating the use of social media into their college pursuits, what relationship it has on their college experience, and finally their college success. There is no reason why postsecondary educators and researchers should not begin to explore the potential of social media to better serve the needs of students and enhance their success.
Despite the fact that the use of social media is near-universal for those under 30 years of age and actively utilized by nearly two-thirds of those under 50 (Barnes & Lescault, 2011; Lenhart, Purcell, Smith, & Zickuhr, 2010; Madden, 2010), we know almost nothing about the relevance of social media interactions for engagement, integration, belonging, college identity, or college community-building among community college students. In fact, only few studies have focused on the use and the effects of social media on college campuses in general (Ellison, Steinfield, & Lampe, 2011; Junco, 2012a, 2012b; Junco, Heiberger, & Loken, 2011; Wimmer & Lewis, 2010). The few studies that have looked at the usage and effects of social media are of great importance. They have helped educators and researchers understand how online interactions contribute to the formation of new connections among college students and how they can potentially improve academic outcomes as well. Some of the most intriguing findings of these studies include the following: (1) On average, students spent 26 minutes daily (or 21% of the total time spent on their computers) on Facebook (Junco, 2013); (2) certain uses of Facebook are associated with positive outcomes, including relationship building and maintenance (Ellison, Steinfield, & Lampe, 2007), civic engagement, and political participation (Valenzuela, Park, & Kee, 2009), and engagement behaviors (Junco, 2012a, 2012b); (3) both Twitter and Facebook have been found to have s positive association with academic performance; and (4) checking Facebook multiple times and spending time on Facebook is both positively and negatively related to students engagement depending on the type of usage.
These studies are however limited in several respects: First, most studies focus on traditional four-year college students and often ignore the community college context. This limits their capacity to offer solid explanations on how some of the most vulnerable student populations access social media to engage with and integrate into college communities facilitating social relations that are key to students success. Second, most studies rely on self-reported data and, thus, potentially include substantial measurement error. Third, the few studies that do use data generated directly from the social media platform are very limited in scope (e.g., only focus on racial/ethnic friendship ties between students and/or focus on the time spent on specific networking sites) and ignore other important outcomes such as student engagement and success. Our study attempts to address these limitations. We use institutional data matched with data from students use of an online community within Facebook (the Schools App) to answer the following research question: How do the use of the Schools App and students online engagement relate to students academic outcomes?
LITERATURE REVIEW AND CONCEPTUAL FRAMEWORK
Alexander Astin proposed a theory for college students that focused on the concept of involvement and engagement (Astin, 1984). He defined engagement as the amount of physical and psychological energy that the student devotes to the academic experience (Astin, 1984, p. 297). Student engagement can be seen as the efforts by students both in-class and out-of-class that are related to student success (Kuh, 2009). Decades of research have shown that productively connecting and engaging to a college community is key to student success (Pascarella & Terenzini, 2005). That is, how engaged college students are with their institutions (Kuh, 2001), how integrated they are into the social and academic life of the college (Tinto, 1993), and how involved they are in the campus community (Astin, 1984), are all associated with positive academic outcomes, retention, and degree attainment (Rios-Aguilar & Deil-Amen, 2012). However, these prior studies utilize frameworks for understanding college persistence that were developed with full-time, four-year, on-campus residential, predominantly White, 1823 year-old students in mind (Rios-Aguilar & Deil-Amen, 2012). Therefore, such frameworks disregard the experiences of community colleges and their heterogeneous student populations, leaving a void in our understanding of how engagement happens, especially for first-time college, low-income, underrepresented, commuting students who do not break former connections in order to forge new connections in some semi-isolated residential college world (Rios-Aguilar & Deil-Amen, 2012). Even for traditional age students, the experience of college-going in this context is fundamentally different from similar students in more traditional four-year colleges.
In the specific context of community colleges, engagement has also been broadly defined as the amount of time and energy that students invest in meaningful educational practices (McClenney, 2006, pp. 4748). Furthermore, the CCSSE project describes five areas of student engagement: active and collaborative learning, student effort, academic challenge, studentfaculty interaction, and support for learners (McClenney, 2006; Saenz et al., 2011). A recent study by Saenz et al. (2011) found several patterns of student engagement based on CCSSE data. They found that community college students who are well prepared for assignments and who reported that coursework is relevant and meaningful ranked highest on engagement. They also found that the defining characteristic between low engagers and high engagers is the number of student services used. That is, the most engaged students used more student services and used them more frequently than the least engaged students. The authors therefore suggested encouraging institutions and their students to seek out and utilize support services that increase engagement. Unfortunately, the existing literature on community college student engagement fails to contemplate the validity and relevance of engagement in an online space. How can social media be strategically used by community colleges as a tool to engage students with each other, with the college, and with the colleges faculty and staff to create a community within a community college? Furthermore, can social media give these more disadvantaged college students access to social networks (i.e., institutional connections and contact with other students, faculty, and sources of assistance and information) and increase engagement that research has shown to enhance college outcomes? Researchers and higher education practitioners are well aware that when students connect with available accurate information, services, resources, and advising/mentoring relationships, benefits accrue. Yet few understand how to get those students who need the most help to actually seek the help they need to strategize success.
COMMUNITY COLLEGE STUDENT ENGAGEMENT AND SOCIAL MEDIA
Building on Astins and Kuhs studies, the CCSEE defines engagement as participation in activities that are considered educationally meaningful and includes items that specifically measure student perceptions of the time and energy spent on these activities (McClenney, 2006). It appears that patterns of engagement are not universal, but rather, vary across student subgroups. Results from CCCSE studies focusing on specific student groups showed that high-risk students are more likely to be engaged than students in comparison groups (Lester, Leonard, & Mathias, 2013). At-risk students need to put forth more effort in attempting to compensate for a pervasive combination of academic and institutional barriers to education success (Greene, Marti, & McClenney, 2008, p. 508). For example, they need to travel a greater distance to achieve the same results as their lower risk peers. In addition, full-time and part-time students exhibit differing levels of engagement (Lester et al., 2013). Part-time students are significantly less likely to interact with instructors, make class presentations, or work with other students in or outside of class (Greene et al., 2008). Age also factors into the engagement of community college students. First-year students who are above the age of 25 engage in more purposeful activities than younger students, and the correlation between engagement and the quality of relationships is consequently higher for these older students (Gibson & Slate, 2010). Differences in student engagement also exist for community college students of color. Swigart and Murrell (2001) found that African American community college students get greater gains from efforts in class assignments and discussions and from using campus services. Furthermore, male students of color reported higher levels of student engagement (CCCSE, 2014).
Given these differences, new concepts have begun to emerge to better understand the dynamics of how connections to college happen for nontraditional, commuter, and community college students (Rios-Aguilar & Deil-Amen, 2012), and such understandings can now be applied to how engagement might occur in online environments for such populations. For example, aspects of traditional frameworks, such as the concept of integration, are still relevant but operate differently for community college students (Deil-Amen, 2011; Karp, Hughes, & OGara, 2010; Tinto, 1997). In particular, in contrast to prior studies of four-year residential students, Deil-Amen (2011) discusses alternative types of socio-academic behaviors among community college students that challenge the dichotomous notion of integration occurring along purely academic or social lines. These socio-academic integrative moments are events, activities, interactions, and relationships in which academic and social elements combine simultaneously to enhance learning, procedural knowledge, information-acquisition, intellectual competence, college identity/belonging, and connectedness. Routinely, these moments occur within and just beyond the classroom, often the most common place where commuting students meet other students and instructors, develop feelings of belonging and engagement, and learn success strategies (Karp et al., 2010). These fused socio-academic interactions play a prominent role in two-year students sense of connection and motivation to persist (Deil-Amen, 2011). Also, in contrast to more traditional residential four-year college students, the frequency and depth of relationships held less meaning for two-year college students. For them, it was the mere presence of a connection and how it signaled a welcoming climate that mattered. In fact, nontraditional college-goers tend to view purely social ties with college peers as distracting, preferring instead to reinforce their motivation and commitment to college goals through a clear sense of purpose/focus (Zell, 2010) and through a few key social ties related directly to their career pursuits (Deil-Amen, 2011). Social media presents opportunities for such socio-academic connections to occur, and interactions through social media may contribute to the feelings of belonging and/or goal commitment central to theories of persistence.
Furthermore, subjective college experiences that cultivate a college-going identity and validate the pursuit of college goals are also important for students who are not from social class communities with strong college-going norms (Collatos, Morrell, Nuno, & Lara, 2004; Saunders & Serna, 2004). This is consistent with what others have found to be of importance for community college, Latino/a, and lower SES students in feelings of care, community, and belonging (Braxton, Hirschy, & McClendon, 2004; Rendón, 1994; Rendón, Jalomo, & Nora, 2000). Social media presents a potential mechanism for these subjective identities and communities to develop, especially considering recent work suggesting that social media presents an effective means for college students to maintain relationships (Ellison et al., 2007) and engage in the types of communicative activities relevant to relationship-building (Junco, 2012a). Positive GPA outcomes were found to be related to the use of social media for communications related to the sharing of information, events, and the building of college community and relationships. These studies, although conducted on more traditional four-year college students, indicate the potential value of social media for enhancing college community among community college students.
STUDENT ENGAGEMENT IN AN ONLINE CONTEXT
Todays college students, the net generation, have integrated various forms of social media and technology into their everyday supply of communication and connection tools (Junco & Mastrodicasa, 2007). Since the explosion of social media/social networking sites, there is a great amount of professional and academic interest in the effects of social media on college student development, involvement, engagement, and success (Abramson, 2011; Heiberger & Harper, 2008; Junco, 2012a). The most recent data, collected by the EDUCAUSE Center for Applied Research (ECAR) from a sample of 36,950 students from 127 U.S. universities, showed that of the 90% of students who use social networking sites, 97% said they used Facebook (Smith & Caruso, 2010). These 97% also reported actively engaging on the site daily. While the percentage of college students who use social media is extremely high (Junco, 2012a), it is important to acknowledge that there are persistent differences along gender, racial, and socioeconomic lines in technology adoption and use (Junco, Merson, & Salter, 2010).
In an effort to understand the relationship between Facebook use and student engagement, Junco (2011) conceptualized student use and involvement with Facebook along Astins (1984) five tenets of engagement. In other words, Junco (2011) adapted existing premises of student engagement to an online context (quotes from p.164 in italics):
Engagement refers to the investment of physical and psychological energy. By this definition, according to Junco (2011), evidence abounds that students are highly involved with Facebook. For instance, as reported in Heiberger and Harper (2008), almost half of the 100 million active Facebook users are members of a college network.
Engagement occurs along a continuum. This tenet states that students will invest varying amounts of energy in different areas (Pascarella & Terenzini, 2005, p. 53). This concretely means that some students are more engaged online than others, while some dont use social media at all.
Engagement has both quantitative and qualitative features. This point references the fact that students can spend a great deal of time using Facebook (quantitative feature) and may have different levels of engagement and are involved in a wide variety of activities on the platform (qualitative features). Heibergers (2007) study showed that students spend an average of one to two hours a day on Facebook and logging in an average of three times per day. Qualitative features, on the other hand, have not been examined in depth by researchers but refer to how specific Facebook activities add to (or subtract from) college students lives and experiences.
The amount of student learning and development associated with an educational program is directly related to the quality and quantity of student engagement in that program. This tenant claims that students will develop in proportion to the amount of time spent and nature of their involvement in an activity (Astin, 1984). It is possible that Facebook use is correlated to college students social, cognitive, or academic development, as well as to real-world student engagement in some tangible ways. Such relationships might be revealed as either positive or negative.
The effectiveness of any educational practice is directly related to the ability of that practice to increase student engagement. Astin (1984) stated that programs and services that colleges offer should be assessed in terms of their ability to induce greater student involvement. In an online context this concretely means that if Facebook indeed increases engagement, it is possible for Facebook to be used in educationally-relevant ways to improve student academic outcomes.
In sum, Junco (2011) claims that Facebook offers college students ample opportunities for extracurricular activities, peer group interactions, social integration, and faculty-staff interactions. Involvement with Facebook may therefore help (or hinder) a students academic performance, integration, and connection with his/her college community. For this study, we will use Juncos adapted framework and focus specifically on tenets two and three. We hypothesize that there are differences between quantitative indicators (i.e., joining an online Facebook-based community) as well as qualitative indicators (i.e., active, passive, and inactive use of social media) that are particularly relevant to community college students success.
METHODS AND DATA
Beginning in Fall 2011 as part of a Bill and Melinda Gates grant-funded intervention, seven urban, small town, and rural U.S. community colleges (located in Arkansas, Arizona, California, New York, Ohio, Texas, Wisconsin, and Wyoming) were selected3 to adopt a Facebook-based application for use only by invited students, staff, faculty, and administrators. The Schools App is purposefully designed to host, manage, and facilitate social engagement for college students. As shown in Figure 1, the Schools App creates a private proprietary community for students attending a specific college, allowing them to make friends with other students at the college (i.e., by using the student directory), get involved by organizing social activities offline (i.e., using the meet-up feature), connect with other students with similar interests (i.e., using the community feature), and to access campus information and updates, ask questions, and seek advice and information (i.e., using the announcement and school feed feature).
For this paper, we examine data available from these community colleges (see Table 1) that participated in the funded longitudinal research collaboration. Data for this study came from two primary sources: (1) usage data collected since the adoption of the application and (2) institutional data.
Figure 1. Sample screenshots of the Schools App.
The variables used in this study and the source of the data are described in detail in Appendix A (Table A1). The two outcome variables for this paper were term Grade Point Average (GPA) across two terms (Fall 2011 and Spring 2012), and if a student would persist to the next semester (from Fall 2011 to Spring 2012 and from Spring 2012 to Fall 2013). We also included several control variables: Student background characteristics such as a students race/ethnicity, age, and gender and a students institutional status such as remedial status, full-time enrollment, financial aid status, educational goals, and failure to complete any course during the semester. The variables of interest are the social media usage indicator variables. We distinguish if students joined the online community (i.e., members vs. nonmembers) as well as the type of users once they joined the app (i.e., inactive, passive, or active). For those students who were users of the application, we also had more specific usage indicators such as time spent on site, number of subcommunities joined within the site, total Facebook friends, and number of accepted friend requests at school.
Selection bias is a concern in nonrandomized research designs (Campbell & Stanley, 1963). Because student participation in the application was voluntary, we were faced with the challenge of selection bias. That is, students who decided to join the app could be fundamentally different from those who decided not to join. One way of addressing this problem is with propensity scores (Rosenbaum & Rubin, 1983). In this particular study, we first calculated a student's likelihood or propensity of joining the application based on available background information. Then, the propensity scores (PS) were included in subsequent regression analyses and therefore adjusted in part for selection bias. We also included supplementary analyses to examine the robustness of our regression results. We performed four different types of propensity score matching and also performed sensitivity analyses to determine the potential bias of unmeasured variables. For the interested reader, we describe our approach and use of the propensity scores in detail in Appendix B. Overall, these adjustments do not entirely eliminate the issue of selection bias but do increase our confidence that the observed differences are not entirely due to unmeasured differences between the students who decided to join and those who did not.
Missing data were not a major concern with missingness less than 1% on the variables of interest. All analyses were thus run on complete data. The total analytic sample consisted of 143,307 students over two semesters across seven colleges.
Overall we engaged in three stages of analysis. The first stage involved the use of descriptive statistics to learn about the characteristics of the online communities in the Schools App and its users. The second stage included the use of multiple and logistic regressions to examine how usage and online student engagement was related to academic outcomes. More specifically, we used pooled data from seven colleges and ran the models on two samples: We calculated the effects of social media use on the entire analytic sample which included members of the application as well as nonmembers. In order to estimate the effects of more detailed usage statistics which were only available from students who joined the app, we also ran regressions only for the subsample of members. All regressions in this stage included a propensity control variable. The third stage included robustness and sensitivity analyses to gauge how confident we can be in the results observed during the second stage.
We used STATA for all our analyses. We ran random effects models with dummy variables for each college to control for some of the unobserved heterogeneity. According to Wooldridge (2012), including college/school dummy variables is one way of controlling for heterogeneity in a random effects model.5 These college fixed effects should also account for some of the differences observed between the colleges in terms of student recruitment and marketing for the application. We chose a random effects model over a fixed effects model because we were interested in the time invariant variables such as race and gender which are not estimated in a fixed effect model. All models for both outcome variables used robust standard errors to adjust for heteroscedasticity and included propensity scores as controls for selection bias.
WHO ARE THE STUDENTS?
The selection of colleges for participation in this study was done with careful consideration, attempting to enlist a diverse set of colleges. However, since the sample was not selected randomly we compared overall student demographics with the national averages reported for community colleges in 2011 in order to see the degree to which this sample is representative of the population.
As can be seen in Table 2, the students in our sample are remarkably similar compared to the national averages based on their demographics. Attendance status, gender, and age distribution closely mirrored the wider community college population nationally as reported by the Integrated Postsecondary Data System (IPEDS). In terms of race and ethnicity, our sample contained slightly smaller proportions of White and Asian students, and slightly fewer students received financial aid in the form of federal grants compared to the national averages. Overall, the colleges therefore represent a diverse set of institutions in terms of size, location, geography, and student body.
WHO IS USING THE SCHOOLS APP?
As of August 2012, there were approximately 17,000 community college students who joined the virtual community across all seven colleges (Table 3). From the descriptive analyses, we learned that, on average, 16% of students who were invited to join the Schools App actually became members of the online community.
There was however variation in joining rates across institutions. Some community colleges were more successful at marketing the app and at helping students understand the purpose of the online community. For instance, 33% of the student body joined the app at the community college in Arkansas, whereas only 8% of students joined at the college in Arizona. Furthermore, we learned that students engaged with the virtual community in different ways. For that reason, we developed the following typology of members who joined the app (see Figure 2). We defined inactive members as members who do not spend any time on the application. In contrast, we defined users as members who do spend time on the application. We further distinguished between active and passive users. Active users were defined as students who post and comment on the application, like other user's comments (click like button), or join offered meet-ups (users can invite other users to meet up for certain events such as parties, study groups, or any other user-defined activity). Passive users on the other hand are characterized by passively observing the activity on the application. This distinction of online users was inspired by previous research which suggests that the way students use technology is often more important than just binary indicators of using a technology or not or time spent online (Ellison et al., 2011; Gordon, Juang, & Syed, 2007; Junco, 2012a; Pempek, Yermolayeva, & Calvert, 2009). However, the specific categories used in this research were created based on our own observations of the online activity and general user statistics.
Figure 2. Distinct type of users of the Schools App.
Note: Time spent on site is total for one term. For new members, time spent on site calculated after 1 min cutoff for sign-up process.
Using the definitions from Figure 2, we disaggregated the descriptive analyses to better understand the patterns of usage of the app. The first thing we learned is that during the Fall 2011 semester, the overwhelming majority of students (69%) in all colleges were passive users of the app. This means that these users spend time observing what is happening on the colleges wall, but did not post or comment on anything in the virtual community (see Table 4). In Spring 2012, the majority of users (52%) were inactive, meaning that they joined the app in the Fall 2011, but did not log back into the virtual community at all the following semester. They simply joined the community the previous semester, but stopped using it thereafter. There was also an overall reduction of active users from 24% in the Fall to 18% in the Spring. The decrease in use of the app is also shown in Tables 5 and 6. For instance, we found a considerable reduction (50%) of time spent on the app from one term to the other.
We ran the regression analyses on all available data (i.e., members and nonmembers) as well as on the subsample of students who signed up for the application (members only). We restricted the sample in order to estimate application specific variables of usage which are not available in the larger sample. The complete results for both outcomesGPA and persistenceand all control variables are presented in Table 7. The following discussion will however only focus on the variables of interest to this study.
Notes: Robust standard errors in parentheses. Term GPA models estimated with 'xtreg' random effects and included propensity scores as control and college fixed effects, Persistence estimated with 'xtlogit' (Odd Ratios reported) and included by propensity scores as control and college fixed effects.
1 Effect coding was used for ethnicity, hence each group is compared to the overall mean.
2 For logistic regressions Pseudo R-Squared are reported.
*** p<0.001, ** p<0.01, * p<0.05
THE RELATIONSHIP BETWEEN BECOMING A MEMBER OF THE SCHOOLS APP AND STUDENTS OUTCOMES
After controlling for propensity of joining the application and controlling for several student background and institutional status characteristics, we observed several differences in terms of online activity. When comparing members to nonmembers in the overall sample, passive users had .02 higher GPAs, active users had .04 higher GPAs, but inactive members had about .04 lower GPAs compared to nonmembers. When restricting the sample to just members of the application, similar differences were observed with passive users (.06) and active users (.07) having higher GPAs compared to inactive users. These results suggest a small positive effect of using the application on a students GPA (see below for a more detailed discussion of these results).
When looking at rates of persistence we also observed differences between the groups. In the overall sample, passive users had 35% and active users had 28% higher likelihood of persistence compared to nonmembers. Inactive members were about 40% less likely to persist. When looking at just members, passive users were about 94% and active users were about 69% more likely to persist. These results suggest a considerable positive effect on persistence based on usage of the application.
The models which were restricted to just members allowed for further analysis of effects of online behavior. While several of the estimates such as the total time spent on the application, the number of friends at school, the number of Facebook friends, and the number of communities joined, showed statistically significant effects, the magnitude of these effects ranged from small to very small. Therefore, the practical significance of these results needs to be carefully interpreted.
ASSESSMENT OF CONSISTENCY AND ROBUSTNESS OF RESULTS
We performed several robustness checks of these regression results. Table 8 represents results across several propensity matching algorithms. All matching was performed on common support which ensures that matched students had a positive probability of being in either group of interest (Caliendo & Kopeinig, 2008). We performed these analyses with nonmembers as the reference category (dummy coding) as well as the overall average as the reference category (effect coding).
The results from these matching algorithms present a similar picture to the regression results with some important differences. When looking at GPA outcomes, three of four algorithms showed significant effects for active and passive users compared to nonmembers as well as compared to group averages. However, the results from one algorithmthe nearest neighbor with replacementsuggest that there were no significant differences between groups for GPA. Nevertheless, all other matching results suggest a small positive effect for GPA. This finding is comparable to the regression results even though the results from the matching tended to be slightly larger. The effects of passive users ranged from .08 to .17 higher GPAs and .12 to .15 higher GPAs for active users.
Unlike results from the regression, the matching algorithms suggest that there is no difference in GPA between inactive members and nonmembers. The observed differences in the regressions are therefore mostly likely due to bias. This is an important finding since there is no reason to expect any effect from just signing up for the application and using PS should adjust for differences in the likelihood of signing up for the application. The results from PS matching therefore suggest that there is indeed no difference between inactive and nonmembers.
When looking at the matching results for persistence, all algorithms suggested significant differences in rates of persistence between groups which mirror the results found in the regressions. Again, the magnitude of the effects from the matching algorithms were slightly larger than the ones from the regressions. Passive users were 26% to 63% more likely to persist and active users were 31% to 70% more likely to persist. However, even when matching students, there were some differences in persistence for inactive members compared nonmembers which could be a concern. Nevertheless, as discussed below, there is evidence that these differences are due to unobserved bias and can therefore be considered nonsignificant.
We also computed Rosenbaum bounds to estimate the sensitivity of these results to omitted bias (Rosenbaum, 2002). The gamma statistics indicate the amount of unobserved bias of a variable needed to change the conclusions of the study. For instance, a gamma of 1.2 indicates that the results are sensitive to a variable which would change the odds of being a member of the application by 20%, and this variable would have to be related to the outcome. Generally, the higher the gamma, the less sensitive to hidden bias the results are. Unfortunately, no clear guidelines exist in interpreting levels of gamma. DiPrete and Gangl (2004) suggest to compare the magnitude of gammas to variables in the model in order to interpret what can be considered a small or large effect. Others have reported gammas of 1.15 to 1.2 and showed moderate confidence in their results (Carbonaro & Covay, 2010). All researchers however stress that these measures are worst case scenarios and do not imply that there is indeed hidden bias present in the results (Caliendo & Kopeinig, 2008; DiPrete & Gangl, 2004).
In our study, gammas for active and inactive users ranged from 1.15 to 1.25 for GPA and 1.15 to 1.85 for persistence. This suggests that the results for GPA seem more sensitive to hidden bias, whereas the results for persistence are more robust. In addition, effects of inactive users were most sensitive to hidden bias with gammas ranging from 1.05 to 1.15. This would suggest that the observed effects of inactive users are most sensitive to hidden bias and hence decrease our confidence that there is indeed a difference between inactive members and nonmembers.
The results, on the whole, are an indication that there is a relationship between social media use and community college students academic outcomes. However, results also suggest that online engagement varies and can be difficult to sustain over time. While this study did find some indication of effects on GPA, the magnitude of the effect can be considered small and the estimates are sensitive to hidden bias (see also limitations below). Online engagement revealed a more robust and substantial relationship with persistence. More specifically, purposeful and active usage showed the largest effect and passive usage was also positively correlated with persistence. While some differences for inactive users were also observed, checks for robustness and sensitivity analyses suggest that these differences are mostly due to bias and can be considered nonsignificant.
Our analyses had several limitations of note. The outcome measures of GPA and persistence need to be interpreted with care especially concerning the GPA as we used an unadjusted measure of students grades. Much research has shown how grading standards differ between teachers and departments (Achen & Courant, 2009; Johnson, 1997). Indeed, results from our regressions suggest that similar differences are present in our dataset. For instance, students who indicated wanting to obtain a certificate had higher GPAs compared to students who indicated wanting an associate's degree. This finding is somewhat surprising and might go against expectations. For one college in our dataset (California) we were able to obtain course-level information for each student and were therefore able to investigate this difference in more detail. Analyzing these courses indicated discrepancies between the type of courses taken and student GPA. Table 9 indicates how the overall GPA was higher for students who wanted a certificate compared to students who wanted a degree. However, when limiting the GPA to courses which were transferrable and counted for credit at four-year public institutions in the state (University of California and California State Universities), students who wanted a degree had higher GPAs.
Note: UC=University of California, CSU=California State University. Standard errors in parentheses.
This indicates that depending on the courses taken, students' GPAs can vary and are not necessarily comparable across all students. Methods of adjusting GPAs between students based on courses or majors (Johnson, 2003; Young, 1990, 1993) were unfortunately not possible in this study as this information was not available for all colleges. This limitation of our GPA outcome measure needs to be taken into consideration when looking at the results.
Similarly, using an outcome such as persistence has limitations. Not all students take courses at community colleges for the same reason and have the intent of taking additional courses the next semester. We accounted for some of this variation by incorporating student goals. However, the validity of these student-reported goals can sometimes be questioned and can change over time. In addition, there was large variation in the ways colleges categorized student goals from four categories to 15 educational goal categories. These limitations should be considered when interpreting the results and we would also like to emphasize that student persistence might not be a desirable outcome for all students in community colleges. However, GPA and persistence are often used as important indicators of student and college outcomes and we believe the current analyses have tried to account for possible alternative explanations. But we acknowledge that our results are still limited.
Lastly, selection bias is still a concern in our results. Even with our quasi-experimental design, the observed effects still include some bias. We do believe however that the results do not just represent noise but include a significant and important signal that suggests a relationship between social media use and student outcomes. More research is certainly needed to further control for selection bias and make stronger claims of causality by using experimental designs.
DISCUSSION AND IMPLICATIONS
To our knowledge, this study is the first to use actual (nonself-reported) Facebook data to indicate that there is indeed a relationship between social media usage and community college students persistence and success. The overall goals of this study were to determine how community colleges can use social media to increase student engagement with the school, and if this engagement is related to academic outcomes. We found evidence of such a relationship between social media use and student engagement. One of the most important contributions of this study is that it actually provides a new way to operationalize online engagement for community college students by providing measures of the extensity (i.e., type of social media usage) and intensity (i.e., frequency of use) in the use of social media. We analyzed observational data of online activity to better understand how students use social media to engage with their colleges and peers in an online environment. By doing so, we learned that even passive users potentially benefit from being observing participants of the online community. This finding is contrary to the thinking that passive users are disengaged from the online community because they dont post on the app or because they dont have a large network of friends at school.
We find that certain forms or levels of online engagement have a distinct relationship to particular academic outcomes, including GPA and persistence. This is a major step forward in how social media platforms, like Facebook, should be considered as potential tools for engagement rather than mechanisms of distraction. However, the entirety of our study also points to the need for continued research to verify such dynamics, particularly research that attempts to dissect such relationships while accounting for selection bias in multiple ways. Nevertheless, our data are cutting edge, using tens of thousands of actual online behaviors rather than self-reported engagement measures. Coordinating such new forms of data with existing techniques to control for selection bias represents the future of such research.
The results of this study also suggest that, although potentially valuable, it is not easy to build an online community, particularly in the context of community college students. Some colleges were able to recruit more students for this specific application and were able to better explain the benefits of joining. However, even after recruitment, sustaining continued use of the application was challenging and strongly dependent on the quality and relevance of the posted comments and discussion. Nevertheless, this study found that for those who continued to use the application either passively or actively, there were positive effects in terms of student outcomes. These results support the tenets of online engagement as defined by Junco (2011). We found that online engagement does vary among students and colleges, and that usage has both quantitative as well as qualitative elements. Therefore, we found support for tenets two and three as described by Junco (2011). However, our study was not able to directly measure how online engagement is related to offline engagement or how online activity contributed to subjective feelings of engagement or belonging. Investigating these relationships would give additional support to Juncos (2011) conceptualization of online engagement and clarify how online tools could best be used to improve student engagement and outcomes.
Our findings that users benefit from the online communities students built using the Schools App suggests consistency with some of the past research using self-reported survey data on social media use among more traditional college student populations. Such prior research (Ellison et al., 2007; Junco et al., 2010; Junco, 2012a) showed some mixed results but supported the idea that its not just the frequency of the exchanges, but the nature of the interactions that matters. In some of these prior studies, increased social media use had a negative relationship to engagement and academic outcomes. In others, it had a positive relationship particularly when the type of use was specifically considered. For instance, Ellison et al. (2011) found that when an individual has too many Facebook friends, there are few benefits, and possibly even drawbacks of being socially overextended and underconnected at the expense of more meaningful online relationships that may be more valuable for more than just superficial social linkages. In the community colleges studied presently, the Schools App tended to be used less as a medium for purely social exchanges and more as a means to seek and acquire help and information from each other related to procedural and academic concerns relevant to navigating organizational complexities, financial aid matters, decisions relevant to course-taking, and the sharing of personal challenges and struggles (Fagioli, Rios-Aguilar, & Deil-Amen, in press). The fact that engagement in this context was associated with higher persistence and higher GPA resembles the findings of Ellison et al. (2007) who show that Facebook can be used to maintain relationships with college peers and enhance social capital relevant to college-going. Our findings also resemble prior research showing that use of social media is positively associated with increased GPA when used for communicative activities, such as the sharing of information (Junco, 2012a) and academically relevant exchanges (Junco et al., 2010).
Our findings suggest consistency with prior research regarding how community college student integration happens more generally offline. Similar to the prominence community college students attribute to socio-academic opportunities to connect with peers and college agents as an element in their ability to persist in college (Deil-Amen, 2011), it appears the Schools App might provide similar opportunities in an online space. Therefore, our findings further support the notion that integration is relevant for community college student persistence, but the nature of that integrationmore simultaneously social as well as academically oriented and perhaps procedurally oriented than purely socialis important to consider, both in offline and in online contexts. A community college students college identity and feelings of social belonging in college are perhaps best enhanced through their exposure to and opportunity to engage in fused socio-academic moments (Deil-Amen, 2011), and the Schools App appears to provide such a space for the acquisition of procedural knowledge, information, success strategies and advice as well as a strengthening or reinforcement of intellectual competence, college identity, and a sense of belonging and connectedness (Karp et al., 2010).
The fact that passive engagement with the app can be just as beneficial as active engagement shows that observing socio-academic exchanges and content might garner the same sense of belonging, connectedness, and information-sharing benefits. As Deil-Amen (2011) notes, the frequency and depth of relationships were not as important to community college students as was the mere presence of connecting moments and either their relevance to their career program goals or how they signaled a welcoming climate. The Schools App platform provides exchanges and relationships of such a nature. Relatedly, our findings regarding passive usage might also reinforce findings from prior research that emphasize the importance of feeling part of a caring and validating college community for lower SES, commuting, Latino/a and community college students generally (Braxton et al., 2004; Collatos et al., 2004; Rendón et al., 2000; Rendón, 1994; Saunders & Serna, 2004).
We believe that there are several policy implications based on the results of this study. While some administrators and faculty may think social media is a distraction, it can in fact be used to build a community of students with positive effects on academic outcomes. Especially at community colleges, which tend to have reduced opportunities to establish on-campus connections, an online platform could be a vital and important tool for students to stay connected to peers, teachers, and an academic environment. However, while our study shows that social media can be used to promote student success, there is still a need to assess the cost-effectiveness of these tools. We argue that social media can help create small-scale efficiencies to facilitate student engagement with peers and the campus community. But, these efficiencies should not be thought of as a silver-bullet solution, but as one element among other comprehensive efforts to improve student engagement with the social and academic aspects of college life. Furthermore, it is important to highlight the fact that building and sustaining an online community needs continued efforts to ensure that the posts and discussions online stay relevant and provide important information for students. If more high achieving students are associated with the application, and the quality of posts and information shared on the site are high, other students might be enticed to join and remain engaged.
There are also clear implications for the administration of colleges. Many student comments and discussions often centered on administrative questions of enrollment, financial aid, or rules and regulations. While peers can function as a vital resource, these questions and comments can serve as an indicator to administrators where more information is needed or what questions are most prevalent. If official answers from the administration are posted to some of these questions, the impact is also greater as it reaches a wider audience. In other words, colleges need to use social media more strategically, not only for marketing or dissemination of information purposes, but answering specific questions and getting involved in online communities. On campuses where students realized that answers to pertinent questions are available and administrators and peers indeed tried to help and were focused on creating a community, online engagement at these colleges was of high quality. Sustaining high quality online interaction is one element in ensuring a positive effect on student outcomes. However, colleges should also try to understand why some students spend time on the application without getting connected or involved with peers. These latter students, in particular, seem primed but hesitant or unsure of how to be engaged in the school community; consequently, they might benefit most from a guided use of the application. Teachers, mentors, and other students can function as guides in class, study halls, computer labs, and during other events to clarify the benefits and purpose of such an application.
While this study provides support and answers to some questions regarding social media and online engagement, several questions need to be addressed in future research in order to better understand the impact of social media use. It is important to recognize that the precise mechanisms driving the effects of social media can be difficult to assess due to students varied use and interpretation of behavior on social media generally and in the Schools App in particular. That is, being friends with or interacting with someone on Facebook might have significance to one student for reasons different than another student. Some students might take the initiative to join several types of social media but engage inconsistently or sparsely after an initial stage while other students engage infrequently but consistently.
For these reasons, more research is needed to use new forms of data to incorporate new research on social media engagement with existing traditional theories of student engagement (Kuh, 2001), integration (Tinto, 1993, 1997; Thomas, 2000), and involvement (Astin, 1984). This study, in combination with some recent research that extends Tintos concepts of integration to apply to two-year college students (Deil-Amen, 2011; Karp et al., 2010) and other recent research showing that how college students engage online matters, takes a step in that direction. An additional potentially fruitful direction may be to address how the concept of navigational capital (Yosso, 2005) applies to the current reality of social media as a social capital conduit, particularly for first-generation and underrepresented college students (Martínez-Alemán & Lynk-Wartman, 2009). Yosso (2005) describes how such populations often do activate navigational capital, which is constituted by the skills, resiliency, and network-building necessary for and developed by disadvantaged populations to maneuver through social institutions. Current preliminary analyses of the content of the Schools App exchanges suggest that the community college students studied are activating their navigational capital by pro-actively using the online space to create communities rich in mutually beneficial, socio-academic, financial, and procedurally relevant exchanges to enhance their success (Fagioli, Rios-Aguilar, Deil-Amen, ASHE conference paper, 2012). Martínez Alemán and Lynk-Wartman (2009) contend that online social networks are a medium where students can be empowered to produce culture and ritualize patterns of interaction regarding user-generated content that extends beyond what the institution provides. These social connections comprise social networks that provide a means for social capital processes to occur, specifically the communication and delivery of information as capital. We recognize that such social media opportunities provide fertile ground for navigational capital cultivation because our findings regarding the nature of students online community building and related academic outcomes supports this contention.
Future research should further analyze the quality and type of online engagement in more detail to better understand students efforts to connect and navigate. Also, the difficulty of sustaining engagement witnessed in some colleges is another reason to further analyze the content of comments and discussions and their effects on usage and student outcomes. Inevitably the content of students exchanges will contributes to a more in-depth understanding of the mechanisms operating to produce the findings weve revealed regarding the relationship between social media use and student outcomes.
This study is part of a longitudinal project funded by the Bill & Melinda Gates Foundation.
1. We use Ellison and Boyds (2007) definition of social networking sites as web-based services that allow individuals to (1) construct a public or semi-public profile within a bounded system, (2) articulate a list of other users with whom they share a connection, and (3) view and traverse their list of connections and those made by others within the system (Ellison & Boyd, 2007, para. 4).
2. For details about the Schools App go to: http://www.uversity.com
3. The selection process was conducted in two phases. First the researchers conducted a national poll to identify colleges that perceived social media had some value to increase community college students success. Second, a request for proposals was sent to 50 community colleges across the United States. After reviewing all proposals, the Co-PIs of the grant along with several partners developed criteria to select the participating community colleges (e.g., geographic location, leadership involvement, and plan of action).
4. Members of the Schools App were able to create unique groups of interest which other members were able to join.
5. We also ran fixed effect models to compare them with the random effects estimations. These fixed effect model results are available upon request to the authors.
Abramson, L. (2011). Can social networking keep students in school? NPR Morning Edition. Retrieved from http://www.npr.org/2011/02/09/133598049/can-social-networking-keep-students-in-school
Achen, A. C., & Courant, P. N. (2009). What are grades made of? The Journal of Economic Perspectives, 23(3), 7792.
Astin, A. W. (1984). Student involvement: A developmental theory for higher education. Journal of College Student Personnel, 25(4), 297308.
Barnes, N. G., & Lescault, A. M. (2011). Social media adoption soars as higher-ed experiments and reevaluates its use of new communications tools. Dartmouth, MA: University of Massachusetts Dartmouth Center for Marketing Research.
Becker, S. O., & Caliendo, M. (2007). Mhbounds-sensitivity analysis for average treatment effects. IZA Discussion Papers.
Becker, S. O., & Ichino, A. (2002). Estimation of average treatment effects based on propensity scores. The Stata Journal, 2(4), 358377.
Bragg, D. D., Kim, E., & Barnett, E. A. (2006). Creating access and success: Academic pathways reaching underserved students. New Directions for Community Colleges, 2006(135), 519.
Braxton, J. M., Hirschy, A. S., & McClendon, S. A. (2004). Understanding and reducing college student departure: ASHE-ERIC higher education report (Vol. 30/3). San Francisco, CA: Jossey Bass.
Brown, T., King, M. C., & Stanley, P. (2011). Fulfilling the Promise of the Community College: Increasing First-Year Student Engagement and Success. The First-Year Experience (No. 56). National Resource Center.
Caliendo, M., & Kopeinig, S. (2008). Some practical guidance for the implementation of propensity score matching. Journal of Economic Surveys, 22(1), 3172.
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Dallas, TX: Houghton Mifflin.
Carbonaro, W. J., & Covay, E. (2010). School sector and student achievement in the era of standards based reforms. Sociology of Education, 83(2), 160182.
Center for Community College Student Engagement. (2009). Making connections: Dimensions of student engagement (2009 CCSSE findings). Austin, TX: The University of Texas at Austin.
Center for Community College Student Engagement. (2014). Aspirations to achievement: Men of color and community colleges. Austin, TX: The University of Texas at Austin.
Collatos, A., Morrell, E., Nuno, A., & Lara, R. (2004). Critical sociology in K-16 early intervention: Remaking Latino pathways to higher education. Journal of Hispanic Higher Education, 3(2), 164179.
Davis, C., Deil-Amen, R., Rios-Aguilar, C., & González-Canché, M. (2015). The dynamics and effects of social media use in community colleges. Community College Journal of Research and Practice, 39(5), 409422.
Deil-Amen, R. (2011). Socio-academic integrative moments: Rethinking academic and social integration among two-year college students in career-related programs. The Journal of Higher Education, 82(1), 5491.
DiPrete, T. A., & Gangl, M. (2004). Assessing bias in the estimation of causal effects: Rosenbaum bounds on matching estimators and instrumental variables estimation with imperfect instruments. Sociological Methodology, 34(1), 271310.
Ellison, N. B., & Boyd, D. M. (2007). Social network sites: Definition, history, and scholarship. Journal of Computer-Mediated Communication, 13(1), 210230.
Ellison, N. B., Steinfield, C., & Lampe, C. (2007). The benefits of Facebook friends: Social capital and college students use of online social network sites. Journal of Computer-Mediated Communication, 12(4), 11431168.
Ellison, N. B., Steinfield, C., & Lampe, C. (2011). Connection strategies: Social capital implications of Facebook-enabled communication practices. New Media & Society, 13(6), 873892.
Gangl, M. (2004). RBOUNDS: Stata module to perform Rosenbaum sensitivity analysis for average treatment effects on the treated. Statistical Software Components.
Gibson, A. M., & Slate, J. R. (2010). Student engagement at two-year institutions: age and generational status differences. Community College Journal of Research and Practice, 34(5), 371385.
Gordon, C. F., Juang, L. P., & Syed, M. (2007). Internet use and well-being among college students: Beyond frequency of use. Journal of College Student Development, 48(6), 674688.
Greene, T. G., Marti, C. N., & McClenney, K. (2008). The effortoutcome gap: Differences for African American and Hispanic community college students in student engagement and academic achievement. The Journal of Higher Education, 79(5), 513539.
Heiberger, G. (2007). Have you Facebooked Astin lately? Masters thesis, South Dakota State University.
Heiberger, G., & Harper, R. (2008). Have you Facebooked Astin lately? Using technology to increase student involvement. New Directions for Student Services, 2008(124), 1935.
Johnson, V. E. (1997). An alternative to traditional GPA for evaluating student performance. Statistical Science, 12(4), 251269.
Johnson, V. E. (2003). Grade inflation: A crisis in college education. New York, NY: Springer.
Junco, R. (2011). The relationship between frequency of Facebook use, participation in Facebook activities, and student engagement. Computers & Education.
Junco, R. (2012a). The relationship between frequency of Facebook use, participation in Facebook activities, and student engagement. Computers & Education, 58(1), 162171.
Junco, R. (2012b). Too much face and not enough books: The relationship between multiple indices of Facebook use and academic performance. Computers in Human Behavior, 28(1), 187198.
Junco, R. (2013). Inequalities in Facebook use. Computers in Human Behavior, 29(6), 23282336.
Junco, R., & Cole-Avent, G. A. (2008). An introduction to technologies commonly used by college students. New Directions for Student Services, 2008(124), 317.
Junco, R., Heiberger, G., & Loken, E. (2011). The effect of Twitter on college student engagement and grades. Journal of Computer Assisted Learning, 27(2), 119132.
Junco, R., & Mastrodicasa, J. (2007). Connecting to the net. generation: What higher education professionals need to know about todays students. Washington, DC: National Association of Student Personnel Administrators.
Junco, R., Merson, D., & Salter, D. W. (2010). The effect of gender, ethnicity, and income on college students use of communication technologies. CyberPsychology, Behavior, and Social Networking, 13(6), 619627.
Karp, M. M., Hughes, K. L., & OGara, L. (2010). An exploration of Tintos integration framework for community college students. Journal of College Student Retention: Research, Theory and Practice, 12(1), 6986.
Kuh, G. D. (2001). Assessing what really matters to student learning: Inside the National Survey of Student Engagement. Change, 33(3), 1066.
Kuh, G. D. (2009). What student affairs professionals need to know about student engagement. Journal of College Student Development, 50(6), 24.
Kuh, G. D., Kinzie, J., Schuh, J. H., & Whitt, E. J. (2010). Student success in college: Creating conditions that matter. San Francisco, CA: Wiley.
Lenhart, A., Purcell, K., Smith, A., & Zickuhr, K. (2010). Social media and mobile internet use among teens and young adults: Millennials. Pew Internet & American Life Project.
Lester, J., Leonard, J. B., & Mathias, D. (2013). Transfer student engagement blurring of social and academic engagement. Community College Review, 41(3), 202222.
Leuven, E., & Sianesi, B. (2012). PSMATCH2: Stata module to perform full Mahalanobis and propensity score matching, common support graphing, and covariate imbalance testing. Statistical Software Components.
Madden, M. (2010). Older adults and social media. Washington, DC: Pew Research Center.
Martínez-Alemán, A. M., & Lynk-Wartman, K. (2009). Online social networking on campus: Understanding what matters in student culture. New York, NY: Routledge.
McClenney, K. M. (2006). Benchmarking effective educational practice. New Directions for Community Colleges, 2006(134), 4755.
Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: A third decade of research (2nd ed.). San Francisco, CA: Jossey Bass.
Pempek, T. A., Yermolayeva, Y. A., & Calvert, S. L. (2009). College students social networking experiences on Facebook. Journal of Applied Developmental Psychology, 30(3), 227238.
Provasnik, S., & Planty, M. (2008). Community colleges: Special supplement to The Condition of Education 2008 (No. (NCES 2008-033)). Washington, DC: National Center for Education Statistics.
Rendón, L. I. (1994). Validating culturally diverse students: Toward a new model of learning and student development. Innovative Higher Education, 19(1), 3351.
Rendón, L. I., Jalomo, R. E., & Nora, A. (2000). Theoretical considerations in the study of minority student retention in higher education. In J. M. Braxton (Ed.), Reworking the student departure puzzle (pp. 127156). Nashville, TN: Vanderbilt University Press.
Rios-Aguilar, C., & Deil-Amen, R. (2012). Beyond getting in and fitting in: An examination of social networks and professionally-relevant social capital among Latina/o university students. Journal of Hispanic Higher Education, 11(2), 179196.
Rosenbaum, P. R. (2002). Observational studies (2nd ed.). New York, NY: Springer.
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 4155.
Saenz, V. B., Hatch, D., Bukoski, B. E., Kim, S., Lee, K., & Valdez, P. (2011). Community college student engagement patterns a typology revealed through exploratory cluster analysis. Community College Review, 39(3), 235267.
Saunders, M., & Serna, I. (2004). Making college happen: The college experiences of first-generation Latino students. Journal of Hispanic Higher Education, 3(2), 146163.
Shapiro, D., Dundar, A., Ziskin, M., Yuan, X., & Harrell, A. (2013). Completing college: A national view of student attainment rates: Fall 2007 cohort (Signature Report No. 6). National Student Clearinghouse Center.
Smith, S. D., & Caruso, J. (2010). The ECAR Study of undergraduate students and information technology (Research Study No. 6). Boulder, CO: Center for Applied Research.
Swigart, T. E., & Murrell, P. H. (2001). Factors influencing estimates of gains made among African-American and Caucasian community college students. Community College Journal of Research & Practice, 25(4), 297312.
Teranishi, R. T., Suárez-Orozco, C., & Suárez-Orozco, M. (2011). Immigrants in community colleges. The Future of Children, 21(1), 153169.
Thomas, S. L. (2000). Ties that bind: A social network approach to understanding student integration and persistence. Journal of Higher Education, 71(5), 591615.
Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). Chicago, IL: The University of Chicago.
Tinto, V. (1997). Classrooms as communities: Exploring the educational character of student persistence. The Journal of Higher Education, 68(6), 599623.
Valenzuela, S., Park, N., & Kee, K. F. (2009). Is there social capital in a social network site? Facebook use and college students life satisfaction, trust, and participation. Journal of Computer-Mediated Communication, 14(4), 875901.
Wimmer, A., & Lewis, K. (2010). Beyond and below racial homophily: ERG models of a friendship network documented on Facebook. American Journal of Sociology, 116(2), 583642.
Wooldridge, J. (2012). Introductory econometrics: A modern approach. Mason, OH: Cengage Learning.
Yosso, T. J. (2005). Whose culture has capital? A critical race theory discussion of community cultural wealth. Race Ethnicity and Education, 8(1), 6991.
Young, J. W. (1990). Adjusting the cumulative GPA using item response theory. Journal of Educational Measurement, 27(2), 175186.
Young, J. W. (1993). Grade adjustment methods. Review of Educational Research, 63(2), 151165.
Zell, M. C. (2010). Achieving a college education: The psychological experiences of Latina/o community college students. Journal of Hispanic Higher Education, 9(2), 167186.
Zepke, N., & Leach, L. (2010). Improving student engagement: Ten proposals for action. Active Learning in Higher Education, 11(3), 167177.
In this appendix we explain how we addressed the issue of selection bias in our results in more detail. We understand that selection bias is difficult to control in nonrandomized research designs and selection bias could still be part of our observed results. However, we believe that using propensity scores (PS) as explained below increases our confidence that the observed results are not entirely resulting from selection bias but reflect something more.
As explained in the method section of the paper we adjusted the result with a propensity score methodology. The first step in estimating the propensity scores of membership was to regress the available background variables on the dummy variable indicating if a student was a member of the application or not. For this we used the STATA command pscore from Becker and Ichino . Because we cannot assume that the factors contributing to joining the application are the same on every campus, we estimated the propensity scores separately for each college.
For the estimation of the PS we included all available background information: gender, ethnicity, age, remedial status, financial aid status, full time status, credits attempted, credits completed, and percent of completed credits. Additionally, we added several interactions between credits attempted/completed and background information, as well as squared terms until the balancing property was achieved (Becker & Ichino, 2002). Although additional controls would be desirable, several variables included here can be seen as proxies for important determinants that could affect membership and are also related to outcomes: time constraints, economic constraints, and educational effort. A student's full time/part time status can be seen as a proxy for work and family obligations, especially in combination with number of credits attempted and completed. Students with high external demands are most likely not enrolled full time and are not able to enroll and complete larger numbers of credits. Additionally age is most likely also related with external obligations. Receiving financial aid can be seen as a proxy of a student's economic situation since determination of financial aid is based on family income and assets. Measures of student efforts were limited but the measures regarding term credits are related to student effort and commitment to education. Additionally, remedial status is a proxy for previous achievement.
After calculating the propensity scores we checked for the overlap or common support requirement (Caliendo & Kopeinig, 2008). As Figure 3 indicates, we did indeed have common support for many students. This suggests that most students had a positive probability of being in either group (treatment or control). This common support was also an important feature for the matching algorithms discussed below.
Figure 3. Common support for propensity score analysis.
Figure 4 shows the difference on these variables used in the first step before and after adjusting for selection bias. The full circles indicate the standardized differences between members and nonmembers before the adjustment. For instance, nonmembers took and completed about .4 SD fewer credits than students who did join. In fact, all differences between nonmembers and members were significant on observed measures except for Asian/PI students. Nonmembers were less likely to receive financial aid, attempted and completed fewer credits, were more likely to be part time students, more likely to be male, younger, and Hispanic, Black, or from Other/Missing ethnicity.
The empty circles represent the differences after adjusting for the propensity score. As can be seen in the graph, the differences between the groups disappeared and all differences were not significant in all colleges (p < .05). Using the PS in our analyses therefore corrects for the differences that exist between nonmembers and members on these measures. After calculating the propensity score for all students, we included these scores as an additional control in all our regressions.
Figure 4. Differences between members and nonmembers before and after propensity score adjustment.
Note: Standardized differences reported. PS = Propensity Score.
We also performed several matching algorithms. Specifically, we performed nearest neighbor matching with and without replacement, kernel matching, and radius matching (with caliper set to .02). We matched students on common support with psmatch2 from Leuven and Sianesi (2012). We also estimated the potential effects of hidden bias by calculating Rosenbaum bounds (Becker & Caliendo, 2007; Gangl, 2004).