
Multilevel Structural Equation Models for Investigating the Effects of ComputerBased Learning in Math Classrooms on Science Technology Engineering and Math (STEM) Major Selection in 4Year Postsecondary Institutionsby Ahlam Lee  2017 Background/Context: Because of the growing concern over the decline of bachelor degree recipients in the disciplines of science, technology, engineering, and math (STEM) in the U.S., several studies have been devoted to identifying the factors that affect students’ STEM major choices. A majority of these studies have focused on factors relevant to math and science. However, few studies have addressed the linkage between technology or engineeringrelated factors and students’ STEM major selection. Purpose: The purpose of this study was to investigate the extent to which 10thgrade students’ engagement in computerbased learning activities in math classrooms has an effect on student STEM major selection in 4year postsecondary institutions. After controlling for student and schoollevel variables, this study explored the effects of computerbased learning activities in math classrooms on students’ STEM major selections as mediated by either math selfefficacy or math performance. Research Design: The data from this study were collected from the Educational Longitudinal Study of 2002/2006 (ELS: 2002/2006) conducted by the National Center for Education Statistics of the Institute of Education Sciences in the U.S. Department of Education. ELS: 2002/2006 included a nationally representative sample of young adults who had enrolled in 4year colleges or universities by 2006. Considering the hierarchical structure of the ELS: 2002/2006, in which students are nested within schools, I used multilevel structural equation modeling (MLSEM) rather than traditional SEM. Findings: Students’ engagement in computerbased learning activities in math classrooms had a positive effect on their STEM major selection as mediated by math selfefficacy. Specifically, student computerbased learning activities had a larger effect on students’ STEM major selection than did individual and lecturebased learning activities. In addition, at the school level, a marginal, but significant, relationship emerged between math teachers’ motivation and students’ math performance. Conclusions: This study provided evidence that it is important to incorporate computerbased learning activities into math classrooms at the secondary level. Moreover, at the school level, the significant effect of teacher motivation on students’ math achievement scores suggests that motivating teachers is an important part of STEM education, particularly given the fact that teachers are responsible for designing and implementing progressive math curricula that are embedded in computerbased learning activities. Keywords: computerbased learning, STEM major choice, postsecondary institution, multilevel structural equation modeling Among the four STEM disciplines—science, technology, engineering, and math—math and science have been viewed as the major subjects in K–12 STEM education (Hernandez et al., 2013). Accordingly, a great deal of research on STEM education has been devoted to investigating the effects of math and sciencerelated learning factors on students’ STEM learning outcomes. Particularly, much of the research efforts have confirmed the positive roles of math and sciencerelated learning factors in students’ STEM major selection. However, a limited number of studies have examined the ways in which technology or engineeringrelated learning factors affect students’ STEM learning outcomes. Therefore, this study investigated the extent to which the technologyrelated learning factor—specifically, computerbased learning activities in math classrooms—is associated with students’ STEM major choices. A teacherrelated factor was considered along with the students’ learning activities in the math classroom because teachers play a major role in designing and implementing classroom curricula. As a teacherrelated factor, I chose math teacher motivation based on the literature, which indicates that motivation is the driving force in introducing a new initiative, such as designing and implementing a progressive curriculum (Graham & Weiner, 1996). Moreover, with respect to teacherrelated factors, an extensive body of literature has documented the relationship between student learning outcomes and teachers’ observed qualifications, which include a teacher’s pedagogical and subject content knowledge, certification status, and experience. All of these teacher qualifications are included in the definition of “a highly qualified teacher” in the federal No Child Left behind Act of 2001 (NCLB: Clotfelter, Ladd, & Vigdor, 2007; Goe, 2007; Vandevoort, AmreinBeardsley, & Berliner, 2004). However, relatively little attention has been paid to teachers’ unobserved qualifications, such as motivation. To fill the gaps in the literature, the following research questions guided this study. RESEARCH QUESTIONS 1) To what extent does the frequency of students’ engagement in computerbased learning activities in math classrooms, as opposed to the frequency of students’ engagement in selected traditional learning activities (i.e., lecture and individualbased learning activities), influence their STEM major choices in 4year postsecondary institutions, considering prior math performance and teacher motivation? Do math selfefficacy and math achievement scores mediate this effect? 2) To what extent does the frequency of students’ engagement in computerbased learning activities in math classrooms, as opposed to the frequency of their engagement in selected traditional learning activities (i.e., lecture and individualbased learning activities), influence their STEM major choices in 4year postsecondary institutions, considering prior math performance, teacher motivation, and gender? Do math selfefficacy and math achievement scores mediate this effect? 3) To what extent does the frequency of students’ engagement in computerbased learning activities in math classrooms, as opposed to the frequency of their engagement in selected traditional learning activities (i.e., lecture and individualbased learning activities), influence their STEM major choices in 4year postsecondary institutions, considering prior math performance, teacher motivation, gender, and socioeconomic status (SES)? Do math selfefficacy and math achievement scores mediate this effect? To investigate these research questions, three MLSEMs were proposed. The three research questions differ in the selected control variables. By comparing the three MLSEMs addressing each research question, I analyzed 1) whether the addition of the selected control variable changes the effects of computerbased learning activities in math classrooms and 2) the extent to which the overall model fits among the three MLSEMs differ. THEORETICAL FRAMEWORK The research questions are grounded in social cognitive career theory (SCCT: Lent, Brown, & Hackett, 1994). The following key components of the SCCT framed the selected variables in the study: a) personal input; b) background contextual affordances; c) learning experiences; d) selfefficacy expectations; e) goals; f) contextual influences, and g) actions. This section provides the rationale for selecting the variables associated with the key components of the SCCT. Figure 1. Social cognitive career theory (SCCT: Lent, Brown & Hackett, 1994) PERSONAL INPUTS, BACKGROUND, AND CONTEXTUAL AFFORDANCES IN SCCT As depicted in Figure 1, gender served as a control variable for the personal input component in this study. The underlying rationale of assigning gender as a control variable is that female students are traditionally underrepresented in STEM disciplines (National Science Foundation, 2013). Thus, it is recommended to remove the effect of gender when determining the effects of computerbased learning activities on student STEM learning outcomes (i.e., math selfefficacy, math achievement scores, and STEM major selection). The background component in the study was measured by students’ academic background. For student academic background, a student’s math item response theory (IRT) score was chosen, which was measured when s/he was a 10^{th} grade student in the base year of 2002. The math IRT score was used as a control variable because evidence suggests that a student’s math achievement score is a strong predictor of whether s/he will pursue a STEM degree. Thus, controlling for students’ prior math achievement score is critical in examining the extent to which computerbased learning activities in math classrooms contribute to students’ STEM learning outcomes. The description of the math IRT score is provided in the methods section. For the contextual affordance component, SES was chosen as a control variable because it is a critical contextual factor in a student’s decision to pursue a STEM career. The literature suggests that students from higher SES backgrounds are more likely to major in STEM disciplines compared to their lower SES counterparts (Ascher, 1985; Huang, Taddese, & Walter, 2000; Porter, 1989; Rotberg, 1990; Trusty, 2002; Wilson, 1990). As such, similar to gender, it is recommended to factor in the SES effect to measure its influence on STEM learning outcomes of computerbased learning activities in math classrooms. LEARNING EXPERIENCE IN SCCT The learning experience component in SCCT was articulated by the selected learning activities—specifically, computer, individual, and lecturebased learning activities—in 10^{th} graders’ math classrooms. The details of these learning activities are provided in the methods section. The rationale for choosing these three learning activities is supported by the pedagogical structure illustrated in the paper “How People Learn: Brain, Mind, Experience, and School” (HPL: Bransford, Brown, & Cocking, 1999, p. 22). As depicted in Figure 2, in HPL, six learning activities are described—technology, lecture, skills, inquiry, individual, and groupbased learning activities. HPL suggests that individuals’ learning outcomes may be maximized with a mixture of these designated different learning activities. Among the six learning activities, this study addressed technology, individual, and lecturebased learning activities for the following two reasons. First, technologybased learning activities should be a major focus of this study. As noted earlier, computerbased learning activities in math classrooms represent technologybased learning activities in this study. Second, individual and lecturebased learning activities in HPL, which represent traditional learning activities, should be compared with the effects of technologybased learning. Thus, because this study focused on the effects of technologybased learning activities as opposed to traditional learning activities, it did not consider other progressive learning activities (i.e., inquiry, skills, and groupbased learning activities) in HPL. Figure 2. How people learn: Brain, mind, experience, and school (HPL: Bransford, Brown, & Cocking, 1999, p. 22) SELFEFFICACY EXPECTATION IN SCCT Selfefficacy expectation in SCCT was articulated by math selfefficacy. Math selfefficacy is a latent construct that cannot be represented by a single variable; instead, it should be measured by more than two observed variables relevant to an individual’s confidence in mathrelated tasks. In this study, math selfefficacy comprised five observed variables, as follows: a) a student’s confidence in taking math tests; b) a student’s confidence in understanding the most difficult materials in his or her math textbooks; c) a student’s confidence in understanding the most complex materials presented by his or her math teacher; d) a student’s confidence in doing his or her math assignments, and e) a student’s confidence in mastering math skills. The details of the math selfefficacy construct are provided in the methods section. As shown in the hypothesized model (see Figure 3), math selfefficacy mediates the relationship between the selected exogenous and endogenous variables. In this study, the exogenous variables included selected student demographic characteristics and student learning activities in math classrooms. The endogenous variable is a student’s STEM major selection. In SEM, an exogenous variable refers to an independent variable while an endogenous variable is defined as a dependent variable. A mediator variable is considered a third type of variable that explains the relationship between exogenous and endogenous variable(s). GOALS IN SCCT The goals component of SCCT was aligned with an academic achievement goal. As an academic achievement goal, a student’s math IRT score, which was assessed in the first followup year of 2004 when the student was in 12^{th} grade, was selected. Together with math selfefficacy, a student’s math IRT score in the first followup year was another mediator variable in the hypothesized model shown in Figure 3. Placing both the math IRT score and math selfefficacy in the same position in the hypothesized model (see Figure 3) allowed me to determine which mediator variable was more critical in explaining the relationship between the selected learning activities and a student’s STEM major selection. The details of the math IRT score in the first followup year are explained in the methods section. CONTEXTUAL INFLUENCES IN SCCT The contextual influences component of SCCT was represented by math teacher motivation in this study. Math teacher motivation was considered a contextual learning factor that would affect the student learning process and outcomes as a schoollevel variable, since the existing literature has suggested that school organizational factors, which include the principal’s leadership and work conditions, largely affect teacher motivation (Barnett & McCormick, 2003; Davis & Wilson, 2000; Kelley, Heneman, & Milanowski, 2002; Leithwood, Jantzi, & Steinbach, 1999). As addressed briefly in the introduction, a teacherrelated factor needs to be coupled with a student’s engagement in the selected learning activities in math classrooms. The reason is that teachers design and implement those activities. This study used teacher motivation as an unobservable teacherrelated factor rather than observable teacherrelated factors such as teaching experience, certification status, and education level. The rationale for focusing on the teacher’s motivation emerged from the fact that there are relatively few studies on the relationship between unobservable teacher factors, such as psychological state, and student learning outcomes, compared to research on the relationship between observed teacher qualifications (listed above) and student learning outcomes. In this study, teacher motivation was measured by the math teachers’ perceptions of the importance of teachers’ attention, values, and enthusiasm in supporting their students’ success in math. The teacher survey questionnaires measuring teacher motivation can be found in Table 2. This measurement was framed conceptually by the modern expectancy value theory (Eccles & Wigfield, 2002). Eccles and Wigfield claimed that teacher motivation is driven by various cognitive actions that include teacher expectancy, efficacy, and value with reference to a student’s academic success. Teacher expectancy can be mirrored by math teachers’ expectation that their attention toward student success in math can contribute to the improvement of student math achievement (the survey questionnaire labeled as “BYTM44D” in Table 2). Teacher efficacy can be articulated by math teachers’ confidence that their effective pedagogy can improve student math achievement scores (the survey questionnaire labeled as “BYTM44E” in Table 2). Teacher value can be embodied by math teachers’ belief that their enthusiasm or perseverance is positively associated with students’ success in math (the survey questionnaire labeled as “BYTM44F” in Table 2). A mixture of these three cognitive actions could echo math teacher motivation. The details related to measuring math teacher motivation are provided in the methods section. ACTIONS IN SCCT The actions component of SCCT was related to the status of a student’s STEM major selection (hereinafter referred to as “STEM major selection”), which mirrors a student’s actions in pursuing a STEM career. The category of STEM majors in the study was adopted from the classification of STEM majors suggested in the paper entitled “Students Who Study Science, Technology, Engineering, and Mathematics (STEM) in Postsecondary Education” published by the National Center for Educational Statistics (Chen & Weko, 2009). STEM majors in this study included mathematics, agricultural/natural sciences, physical sciences, biological sciences, engineering/engineering technologies, and computer/information sciences, as depicted in Table 1. Table 1. The Categorization of STEM Majors in ELS 2002/06
Note. *STEM Categorization is adopted from the paper titled “Students Who Study Science, Technology, Engineering, and Mathematics (STEM) in Postsecondary Education” by Chen and Weko (2009) published by the National Center for Educational Statistics (NCES). Figure 3. Hypothetical multilevel structural equation model of STEM major selection LITERATURE REVIEW Due to rising concern about the continued decrease in the number of bachelor’s degree recipients awarded in the STEM fields over the past three decades (U.S. Census Bureau, 2010), a great deal of research has been conducted to determine the factors that contribute to student enrollment in STEM fields. Most of these studies have focused on investigating math and sciencerelated learning factors that affect a student’s STEM learning path, based on the dominant perception that math and science are the core subjects of K–12 STEM education. However, the learning contexts embedded in the disciplines of engineering and technology have not been well addressed in previous research on K–12 STEM education. Moreover, student STEM learning processes in classrooms should be investigated in relationship to a teacherrelated factor because students would not have technology or engineeringrelated learning opportunities without teachers’ efforts to implement progressive curricula. Several studies have documented the relationship between between observable teacher qualifications (e.g., pedagogical and content knowledge, teaching experience, and certification status) and student academic achievement. However, little is known about how a teacher’s cognitive actions affect student learning. Therefore, this study investigated the effects of math teacher motivation on student STEM learning outcomes, including student math achievement scores, math selfefficacy, and STEM major selection. The following section reviews the literature on two different areas and concludes with the summary of major findings. The first subsection provides a literature review of technology and engineeringrelated learning contexts at the secondary level. The second subsection reviews the literature on teacher motivation. Finally, following the summary of the literature review, the rationale for investigating the research questions is provided. TECHNOLOGY AND ENGINEERINGRELATED LEARNING CONTEXTS Because the competitive 21^{st}century workforce demands STEM literacy, there has been heightened awareness of the importance of increasing the STEM literacy among all students, regardless of whether they pursue a STEM career (National Research Council, 2011). Accordingly, a broad consensus has been reached among a range of stakeholders to promote an integrated STEM curriculum that is designed to connect at least two subjects among the four STEM disciplines (National Research Council, 2014). However, such a reformbased STEM curriculum has not yet been implemented fully in contemporary K–12 classrooms. Given the lack of practical implementation, relatively few studies have explored the influences of a reformbased STEM curriculum embedded in technology and engineeringbased learning contexts on students’ learning processes and outcomes (National Research Council, 2014). Among the few, a recent study by Gottfried, Bozick, and Srinivasan (2014) investigated the effects of applied STEM courses on student math achievement scores at the high school level using a nationally representative sample from the Educational Longitudinal Study of 2002 (ELS: 2002). The authors found that taking applied STEM courses was associated positively with student math achievement scores. In the study by Gottfried et al., applied STEM courses were designated as scientific research engineering (SRE) courses and information technology (IT) courses. The designated courses represent technology and engineeringrelated subjects. Another largescale study using the Programme for International Student Assessment (PISA) showed positive effects of information and communication technologies (ICT) on the math and science achievement scores of Turkish students (Delen & Bulut, 2011). In addition to largescale studies using national or international datasets, metaanalytical studies have provided empirical evidence of the effects of a reformbased STEM curriculum on student achievement scores. For example, 25 metaanalyses based on 1,055 studies (roughly 109,700 participants) suggested that the use of computer technology in classrooms had a significant and positive effect on student academic achievement in a range of subjects, including math and science, with a small to moderate effect size ranging from .30 to .35 (Tamim, Bernard, Borokhovski, Abrami, & Schmid, 2011). Becker and Park’s (2011) metaanalysis of 28 studies found that an integrative STEM curriculum, which is designed to connect at least two subjects among the four STEM disciplines, had a positive effect on student achievement in STEM subjects from K–12 to the college level. However, Becker and Park noted that the results of this metaanalysis were preliminary because of the dearth of empirical studies on the effects of an integrated STEM curriculum on student STEM learning outcomes. Furthermore, some studies have attempted to measure the effects of a nationally funded engineering education program on student learning outcomes. For example, the 1999 Infinity Project was designed to improve engineering and technology literacy among middle and high school students based on collaboration among multiple stakeholders in STEM, including the Institute for Engineering Education at Southern Methodist University, Texas Instruments, the U.S. Department of Education, and the National Science Foundation (Brophy, Klein, Portsmore, & Rogers, 2008). A qualitative study found that, according to the instructors of 85 students, the Infinity Project played a critical role in increasing students’ understanding in math and science and improving student academic achievement in other math and science courses (Douglas, 2006, as cited in Brophy et al., 2008). Additionally, Douglas reported that 94% of the student participants would recommend the Infinity Project to their friends, 83% were interested in pursuing engineeringrelated careers, and 95% learned a new math concept. Another example of an engineering education program is the Vanderbilt Instruction in Biomedical Engineering for Secondary Science (VIBES) that was initiated in 1999 with grants from the National Science Foundation (Brophy et al., 2008). Using a repeated pretestposttest design, the research showed that students who engaged in the VIBES modules acquired a better understanding of science concepts compared to their control counterparts, with moderate to large effect sizes (Klein & Geist, 2006). Klein and Geist noted further that the VIBES modules played a positive role in increasing student science knowledge in urban (N =240), suburban (N = 330), and rural classrooms (N = 1,233). However, some evidence suggests nonsignificant or negative effects of an integrated STEM curriculum on students’ math and science scores. Tran and Nathan (2010) investigated the relationship between an integrated STEM curriculum, called Project Lead the Way (PLTW), and student math and science scores at the high school level, drawing upon a sample of 140 students and 27 teachers in a Midwestern city. Using multilevel statistical modeling, the researchers found that students enrolled in PTLW achieved less in math and science from eighth to tenth grade compared to their counterparts who did not enroll in PLTW. Moreover, the Institute of Education Science (2010) reported no significant effects of a computer technology program on student math achievement scores among 1,723 high school students in 27 schools across seven districts based on four studies that used randomized controlled trials or quasiexperimental designs (Cabalo, Jaciw, & Vu, 2007; Campuzano, Dynarski, Agodini, & Rall, 2009; Shneyderman, 2001; Smith, 2001). In these studies, the computer technology program was referred to as the Carnegie Learning Curricula and Tutor^{® }software (CLC & CT^{®}S). These mixed results suggest that the effects of an integrated STEM curriculum, such as a computerbased math curriculum, on student academic achievement have not been determined yet. A recent report by Enyedy (2014) noted that these mixed results regarding the effectiveness of computermediated curriculum at the K–12 level, as well as the lack of largescale studies, call for more research on this subject. However, very few studies have explored the effects of an integrated STEM curriculum on a student’s STEM education path. TEACHER MOTIVATION As noted previously, student learning processes in classrooms are in juxtaposition with teachers’ curriculum design and implementation. Obviously, extra effort on the part of teachers is necessary to incorporate a reformbased curriculum, such as an integrated STEM curriculum, into contemporary traditional K–12 classrooms (which focus on teaching individual STEM subjects). Importantly, such extra effort emerges from teachers’ expectations of improving student learning outcomes, which is supported by modern expectancy value theories (e.g., Eccles, 1987; Feather, 1988; Wigfield & Eccles, 1992, 2001). Modern expectancy value theorists have indicated that motivation driven by an individual’s expectancy to accomplish a desired level of task performance engenders his/her expenditure of effort to perform the task. Furthermore, a study by Peter (1977) concluded that environmental factors influence individuals’ efforts in performing a task, which is driven by their motivation. Consistent with the findings from Peter’s study (1977), research has shown that teacher motivation is largely affected by school environmental or contextual factors (e.g., Barnett & McCormick, 2003; Davis & Wilson, 2000; Kelley et al., 2002; Leithwood et al., 1999). Specifically, teacher motivation rests on school organizational contexts, such as principals’ leadership styles and work conditions. For example, a study by Davis and Wilson (2000) found, using a sample of 660 elementary teachers and 44 principals, that the principals’ encouragement of teachers’ behaviors was associated positively with their motivation. Moreover, a literature review by Leithwood et al. (1999) showed that transformational leadership played an influential role in increasing teacher motivation to improve classroom practices and attitudes. Likewise, using a semistructured interview of four randomly selected principals and 11 teachers, Barnett and McCormick (2003) found that principals’ transformational leadership enhanced teacher motivation. In addition to the effects of principals’ leadership, work conditions are another school contextual factor linked to teacher motivation. For example, a review by Kelley et al. (2002) showed that schoolbased performance award programs played a positive role in motivating teachers. Moreover, a review by Firestone (2014) suggested that teachers are more motivated to show their competence in an orderly rather than overly punitive working environment. While it has been shown that the degree of teacher motivation is influenced largely by school organizational contexts, little is known about the relationship between teacher motivation and student learning outcomes. In response, this study offers evidence of the effects of teacher motivation on student STEM learning outcomes. In summary, a literature review of the STEM education path calls for more research on the relationship between an integrated STEM curriculum and student STEM learning outcomes. An extensive body of literature has documented the critical role of math and sciencerelated learning contexts in STEM learning outcomes. However, little research has been done on the effects of an integrated STEM curriculum embedded in technology or engineeringrelated learning contexts. Considering that STEM goes beyond the traditional core subjects of math and science, technology or engineeringrelated learning contexts should be integrated fully into STEM education at the secondary level. To address this issue, this study examined the extent to which computerbased learning activities in math classrooms, as opposed to traditional learning activities that represent individual and lecturebased learning activities, are associated with students’ STEM learning outcomes. Equally important, given that math teachers design and implement students’ learning activities in math classrooms, a teacherrelated factor was included in the study. Few studies have explored the effect of math teacher motivation, which is an unobservable psychological teacherrelated factor, on students’ learning outcomes. METHODS MLSEM analyses investigated the longitudinal effects of 10^{th}graders’ computerbased learning activities in math classrooms on their STEM major selections in 4year postsecondary institutions, as mediated by math selfefficacy and math performance. MLSEM is an appropriate statistical tool because this study focused on: (a) testing how well the proposed MLSEM models in the study fit into the conceptual framework of the SCCT, and (b) showing the direct and indirect effects of the selected variables within and between school levels. Furthermore, MLSEM generates more accurate and unbiased parameter estimates (coefficients) compared to the traditional SEM because it considers standard errors from the multilevel (hierarchical) model (Kaplan & Ferguson, 1999; Muthén & Muthén, 1998; Muthén & Satorra, 1989). The dataset used in this study has a hierarchical structure in which students are nested within schools. DATA SOURCE The data were extracted from the Educational Longitudinal Study of 2002/2006 (ELS: 2002/2006) conducted by the National Center for Education Statistics (NCES) of the Institute of Education Sciences (IES), US Department of Education (for detailed information see Ingels et al., 2007). The ELS: 2002/2006 was completed by multiple respondents (e.g., students and teachers) at three time points in 2002, 2004, and 2006. The survey gathered information that can articulate the learning process and outcomes of a nationally representative sample of young adults from 10^{th} grade to postsecondary education or employment status. In the base year of 2002, these young adults were 10^{th}graders and then 12^{th}graders in the first followup year of 2004. In the second followup year of 2006, they had become college students or adults beyond high school. The reliability and validity of the survey instruments in the ELS: 2002/2006 were ensured based on the following multilevel review and revision process among a range of stakeholders (Ingels et al., 2007). First, a draft of the survey questionnaires was shared with other government agencies, policy groups, and interested parties. Second, the draft was reviewed by the Technical Review Panel (TRP) composed of substantive, methodological, and technical experts. Third, interdivisional review at NCES was performed. Fourth, the survey questionnaires were revised based on feedback of the several stakeholders. Fifth, NCES wrote a justification of the data elements and any issues related to the survey questionnaires. Sixth, the Office of Management and Budget (OMB) reviewed the survey questionnaires. Seventh, the survey questionnaires were revised based on the feedback of the OMB. Finally, the survey questionnaires were tested at over 50 public and private schools in the five field test states (New York, North Carolina, Florida, Illinois, and Texas). Several test analyses of the field tests were conducted, including evaluation of item nonresponses, examination of testretest reliabilities, calculation of scale reliabilities, and examination of correlations between theoretically related measures. Final revisions of the survey questionnaires were made based on the results of field tests. The detailed field test report on the base year is found in the NCES report by Burns et al. (2003). Additionally, the NCES report by Ingels et al. (2007) provides a summary of the field test results on the first followup year and second followup year. SAMPLE From the ELS: 2002/06, I extracted 4,357 students from 711 high schools who later enrolled in 4year postsecondary institutions and disclosed their majors. Of the 4,357 student participants, 21.5% selected STEM majors and the remaining students (78.5%) enrolled in nonSTEM majors. As addressed previously, the definition of STEM majors in the study was determined based on the classification of majors suggested in a report by the U.S. Department of Education (Chen & Weko, 2009). The sample was weighted based on the weighting variable, which was labeled as “F2BYWT” in the ELS: 2002/06 to represent the nationwide student population. VARIABLES Exogenous Variables The three proposed MLSEMs included four common exogenous variables. Three learning activities were proposed at the student level: (a) computerbased learning activities, (b) individualbased learning activities, and (c) lecturebased learning activities. At the school level, teacher motivation was a common exogenous variable in the three models proposed. Table 2 shows the description of each variable. The exogenous variables at the student level were measured on a fivepoint Likert scale ranging from 1 = never and 5 = every day or almost every day. The computerbased learning activities variable was assessed based on students’ responses to the question: “How often do/did you use computers?” The mean of this variable was 1.65 with a standard deviation of 1.04 and a range of 1 to 5. The individualbased learning activities variable was assessed based on students’ response to the question: “How often do/did you review the work from the previous day?” This variable also ranged from 1 to 5 with a mean of 3.89 and a standard deviation of 1.30. The lecturebased learning activities variable was assessed via the survey question: “How often do/did you listen to the teacher’s lecture?” The mean of this variable was 4.49 with a standard deviation of .96 and a range of 1 to 5. As shown in Table 2, teacher motivation, which was the schoollevel exogenous variable as well as a latent variable, consisted of the composite value of the responses to the designated three teacher questionnaires. The rationale for selecting and combining the three teacher questionnaires is explained in the latent constructs section below. Each questionnaire was measured on a fourpoint Likert scale ranging from 1 = extremely important to 4 = not at all important. Thus, when combining the responses to the three teacher survey questionnaires, the teacher motivation variable ranged from 3 to 12. The mean of teacher motivation was 9.74 with a standard deviation of .85 and a range of 6.60 to 12.00. Table 2. Description of the Variables in the ELS: 2002
Note: Variable labels in the ELS: 2002 in parentheses 1. In SEM, exogenous variable is equivalent to independent variable. 2. In SEM, mediator variable is an explanatory variable that describe the relationship between independent and dependent variables. 3. In SEM, endogenous variable is equivalent to dependent variable. Control Variables The effects of the frequency of a student’s engagement in computerbased learning activities were examined after controlling for the following selected control variables: (a) prior math achievement scores, (b) gender, and (c) SES. The inclusion of the control variables in the proposed model enabled me to measure the effects of a student’s computerbased learning activities that were not confounded by the selected control variables. The difference among the three proposed models depended on which control variable was added. As addressed in the research question section, Research Question 1 had only one control variable: a student’s prior math achievement score. Research Question 2 included two of the control variables: (a) prior math achievement scores and (b) gender. To answer Research Question 3, all of the control variables, prior math achievement scores, gender, and SES, were included. There are two reasons for differentiating the type of control variables used to address each research question. First, the study took a close look at whether the effects of computerbased learning activities differed depending on the addition of the control variables in Research Questions 2 and 3. Second, from a methodological standpoint, the study analyzed the extent to which the overall model fit changed due to the addition of the control variables, given that in SEM, a model with fewer parameters (variables) would provide a better fit. As shown in Table 2, all of the selected control variables (i.e., prior math achievement scores, gender, and SES) were extracted from the base year study of 2002 in ELS: 2002/2006. Prior math achievement scores were represented by the base year math item response theory (IRT) scores. The gender of each student was collected from the base year student survey questionnaires. The base year parent questionnaire provided the SES of the student participants. The base year math IRT scores had a mean of 52.35 with a standard deviation of 11.84 and a range of 16.35 to 82.03. Gender, which was labeled as “BYSEX” in the ELS: 2002/06, was originally coded as 1 = male and 2 = female. However, this variable was recoded as 0 = male and 1 = female to transform it into a dummy variable. Overall, 56% of the study participants were female. When transformed into a dummy variable, the gender variable showed the effects of being female on the mediator and endogenous variables in the proposed MLSEM models (see Figure 3). SES, which was originally labeled as “BYSES1” in the ELS: 2002/06, ranged from 1 to 4. The average SES was 3.13, which suggested that the average study participant was upper middle class. Mediator Variables The three MLSEM models proposed commonly included math selfefficacy and math achievement scores. Like math teacher motivation, math selfefficacy has a latent structure that was identified in an exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). The detailed procedures used to identify the latent structure of math selfefficacy are described in the latent construct section. After the EFA and CFA, the math selfefficacy variable was justified as being composed of the following five indicators: (a) students’ confidence in their math tests, (b) students’ confidence in understanding the most difficult materials in their math textbooks; (c) students’ confidence in understanding the most complex materials presented by their math teachers; (d) students’ confidence in their math assignments, and (e) students’ confidence in mastering the skills taught in their math classes. The survey questions that refer to these five indicators are presented in Table 2. The mean of math selfefficacy was 13.76 with a standard deviation of 3.70 and a range from 5 to 20. The other mediator variable—math achievement score—was represented by the first followup math IRT scores, which was measured in 2004 in the ELS: 2002/06. The mean of the first followup math IRT scores was 58.81 with a standard deviation of 12.28 and a range from 18.73 to 88.09. Table 3. Descriptive Statistics and Correlation of the Variables
Note. ** p <.01; * p <.05 1. This variable was considered “prior math achievement scores” in the study. 2. This variable was considered “math achievement scores” in the study. 3. Gender was coded as 1 = female and 0 = male. Thus, the mean of the gender,.56 showed that female students comprised 56% of all students in 4year postsecondary institutions Endogenous Variable The endogenous variable in the three proposed models was the status of STEM major selection by students who enrolled in 4year postsecondary institutions. This was a dichotomous variable coded as 0 = nonSTEM major selection and 1 = STEM major selection. As shown in Table 3, the endogenous variable was created based on the variable “a student’s postsecondary major in 2006,” which was labeled as “F2MJR2_P” in the ELS: 2002/06. The F2MJR2_P variable had 33 categories of college majors. LATENT CONSTRUCTS In SEM, confirmatory factor analysis (CFA) is used to test the reliability and validity of any measurements in research (Doll, Raghunathan, Lim, & Gupta, 1995; Said, Badry, & Shahid, 2011). As shown in Figure 4, CFA confirmed a high reliability and validity of the latent constructs of math teacher motivation and math selfefficacy. The goodness of fit model indices indicated good fit of the latent constructs of both variables, with a CFI of .97 (higher than the cutoff value of .95). The observed indicators of both variables were also highly correlated based on factor loadings (i.e., .63 ~ .83). As addressed in the conceptual framework section, the modern expectancy value theory supported the three observed indicators that composed math teacher motivation (e.g., Eccles, 1987; Feather, 1988; Wigfield & Eccles, 1992, 2001). The teachers’ responses on the three questionnaires reflected their perceptions of the significant role of their attitude in student success in the following three domains: (a) attention to the unique interests and abilities of students, (b) use of effective methods of teaching, and (c) enthusiasm about student success (see Table 2). The combination of the teachers’ selfperceptions in these three domains mirrors the expectation, efficacy, and value of math teachers, which are consistent with the cognitive factors in the modern expectancy value theory. The five indicators observed measured student math selfefficacy based on the selfefficacy concept underlying Bandura’s social cognitive theory (Bandura, 1977), as shown in Figure 4. Selfefficacy is defined as one’s level of confidence in one’s ability to complete a specific task (Bandura, 1977). The five indicators of student math selfefficacy measured their confidence level in completing mathrelated tasks, which is consistent with the concept of selfefficacy. Figure 4. The latent constructs of math selfefficacy and math teacher motivation: Standardized regression coefficients MISSING VALUES Under the assumption of missing at random (MAR), the expectationmaximization (EM) algorithm imputed the missing values in the study with SPSS. MAR, which is most commonly assumed in a missing data structure (Robin, 1976), states that the missing values of a variable can be predicted by nonmissing values of other variables (Lu & Copas, 2004; Yuan & Bentler, 2001). Guided by the concepts of MAR, the missing values of the selected variables were imputed. For example, missing values of observed indicators of math selfefficacy were filled based on the first followup math IRT scores, given the literature that math achievement scores are associated positively with math selfefficacy. The data showed the approximate proportion of missing values as follows: (a) 4.18% of the first followup math IRT scores, (b) 5% of individualbased learning activities, (c) 5.5% of lecturebased learning activities, (d) 8% of computerbased learning activities, (e) 17% of math teacher motivation, and (f) 25.6% of math selfefficacy. Of note, Arbuckle (1996) recommended that if the percentage of missing values for a variable is more than roughly 20%, a researcher should use a modern missing data treatment, such as the EM algorithm, rather than listwise deletion. Following this recommendation, the missing values in the study were imputed based on the EM algorithm. DATA ANALYSIS Using Mplus 6.1 (Muthén & Muthén, 2010), MLSEM analyses were run based on the maximum likelihood robust (MLR) estimator. The MLR estimator was appropriate for the different nature of the variables selected in the study (i.e., continuous endogenous and mediator variables, dichotomous control variable, and dichotomous endogenous variable). The MLR estimator is generally recommended for a model on a nonnormality assumption that has different types of variables including binary, ordered categorical, and continuous (Kaplan, 2009). RESULTS Beginning with the descriptive statistics of the variables (see Table 3), this section reports the findings from the investigation of the research questions. Structural equation modeling (SEM) showed how well the proposed model for each research question fits into the SCCT. Also, the direct and indirect relationships among the variables were explored through SEM, which demonstrated the longitudinal STEM learning processes of a nationally representative sample of 10^{th}graders in 2002 who enrolled in STEM majors in 4year postsecondary institutions by 2006. Table 3 provides the descriptive statistics and correlation among the variables in the study. The descriptive statistics include the percentage of female students, the number of students who enrolled in STEM majors by 2006, and the means and standard deviations of the other selected variables. RESEARCH QUESTION 1 Research Question 1 was designed to examine the effects of a student’s frequency in engaging in computerbased learning activities in math classrooms (hereinafter referred to as “computerbased learning activities”), as opposed to the frequency of a student’s engagement in lecturebased and individualbased learning activities on his or her STEM major selection. The investigation of Research Question 1 showed the mediating effects of math selfefficacy and math performance on the relationship between the selected learning activities in the math classroom and a student’s STEM major choice. Consistent with the selected studentlevel variables, MLSEM measured the effects of math teacher motivation at the school level. A student’s prior math performance, which was measured by the math IRT scores in the base year of 2002, was treated as a control variable in the MLSEM analysis. Table 4 shows the direct effects of the selected learning activities on STEM major selection, as mediated by either math selfefficacy or math performance. The parameter estimate of computerbased learning activities (b = .693, p < .01) provided evidence that, compared to the individualbased (b = .030, p < .01) and lecturebased learning activities (b = .024, p < .10), computerbased learning activities had a larger effect on math selfefficacy that was linked positively with students’ STEM major selection. Table 5 shows the indirect effects of the selected learning activities on STEM major choices, as mediated either by math selfefficacy or math performance. Computerbased learning activities had significant indirect effects on STEM major choices through the mediator of math selfefficacy (b = .025, p < .05), while student math performance (i.e., math IRT scores in the first followup year of 2004) did not mediate this relationship. Moreover, there existed marginally significant indirect effects of individualbased learning activities on STEM major selection, as mediated by math selfefficacy (b = .001, p < .10). The significant mediating effects of student math performance emerged only in the relationship between individualbased learning activities and students’ STEM major selection (b = .006, p < .05). SEM analysis at the school level yielded a marginally significant relationship between math teacher motivation and student math performance (schoollevel math IRT scores in the first followup year of 2004). However, there existed no significant mediating effects of student math performance on the relationship between math teacher motivation and STEM major selection, suggesting that there was no significant indirect effect of math teacher motivation on STEM major selection across schools. The proposed model for Research Question 1 is considered a good fit, as evidenced by the following model fit indices: CFI = .993; TLI = .986; RMSEA = .026 (see Table 4). Figure 5 provides the diagram of the MLSEM model for Research Question 1. Figure 5. Multilevel model of STEM major choices in 4year colleges and universities: Standardized regression coefficients (Model 1) Note. *** p <.01; ** p <.05; * p <.10 RESEARCH QUESTION 2 Research Question 2 added gender to Research Question 1 and sought to identify whether the effects of computerbased learning activities differed when gender was added as a control variable. When controlling for gender and prior math performance, the relationship between the variables showed a pattern similar to the findings from Research Question 1 at both the student and school levels. As shown in Table 4, at the student level, compared to the individualbased (b = .032, p < .05) and lecturebased learning activities (b = .025. p < .10), computerbased learning activities showed a greater direct effect on increasing student math selfefficacy (b = .692, p < .01), which contributed substantially to a student’s STEM major decision. As shown in Table 5, a substantial mediating effect of math selfefficacy emerged in the relationship between computerbased learning activities and a student’s STEM major selection, as evidenced by the indirect effects of computerbased learning activities on STEM major selection through the mediator of math selfefficacy (b = .025, p < .05). However, student math performance, which was represented by math IRT scores in the first followup year, did not mediate this relationship significantly. As shown in Table 5, similar to the findings for Research Question 1, the significant mediating effects of student math performance existed only in the relationship between individualbased learning activities and a student’s STEM major selection (b = .007, p < .001). At the school level, controlling for math IRT scores in the base year of 2002, a marginally significant relationship existed between math teacher motivation and math IRT scores in the first followup year of 2004. However, there was no significant relationship between either mediator (i.e., math selfefficacy and math IRT scores in the first followup year), and a student’s STEM major selection. The proposed model for Research Question 2 is considered a good fit, as evidenced by the following model fit indices: CFI =.978; TLI =.959; RMSEA =.042 (see Table 4). However, the model fit indices for Research Question 2 were slightly worse than those for Research Question 1. The decrease in CFI and TLI between the model of Research Question 1 and that of Research Question 2 adds evidence that model fit indices in SEM tend to be penalized by the inclusion of more variables, although this is not always the case (Kenny & McCoach, 2003). Notably, because CFI and TLI measure the relative improvement in model fit of the proposed model over the baseline model, a value of 1 indicates the best fit and a value of 0 indicates the worst fit (Kaplan, 2009). In Figure 6, the diagram of the MLSEM model for Research Question 2 shows the relationships among the variables. Figure 6. Multilevel model of STEM major choices in 4year colleges and universities: Standardized regression coefficients (Model 2) Note. *** p <.01; ** p <.05; * p <.10 RESEARCH QUESTION 3 Research Question 3 was designed to show whether the effect of computerbased learning activities changed significantly when SES was added to the proposed model in Research Question 2. The MLSEM of Research Question 3 revealed that, similar to the findings of Research Questions 1–2, computerbased learning activities in math classrooms had a greater effect on increasing students’ math selfefficacy, which was the mediator between computerbased learning activities and students’ STEM major selection, as opposed to individual and lecturebased learning activities (see Model 3 in Table 4). Like the MLSEM for Research Questions 1–2, math selfefficacy played a significant role in predicting an increase in the proportion of students who chose STEM majors. To be specific, as shown in Table 4, at the student level, 10^{th}graders’ computerbased learning activities in the base year of 2002 had a significant and positive effect on student math selfefficacy in the first followup year of 2004 (b = .693, p < .01), controlling for prior student math performance, gender, and SES. Computerbased learning activities had a greater influence in increasing student math selfefficacy, compared to individualbased (b = .032, p < .05) and lecturebased learning activities (b = .024, p < .10). Among the three learning activities selected, the greatest direct effect of the computerbased learning activities on math selfefficacy did not differ from the findings of Research Questions 1–2. The pattern of the indirect effects was similar to that in Research Questions 1–2. Computerbased learning activities showed significant indirect effects on students’ STEM major selection, as mediated by math selfefficacy (b =.024, p <.01). As shown in Table 5, the indirect effect of computerbased learning activities was larger compared to those of both individual (b =.008, p <.01) and lecturebased learning activities (b =.000, p >.10). The significant and positive mediating effects of math performance were found only in the relationship between individualbased learning activities and STEM major selections (b =.007, p <.01). At the school level, however, no significant relationship existed between math teacher motivation and STEM major selection, which was different from the schoollevel findings in Research Questions 1–2. The proposed model for Research Question 3 is considered a good fit, with the following model fit indices: CFI = .975; TLI = .956; and RMSEA = .041 (see Table 3). However, the model fit indices for CFI and TLI in Research Question 3 were degraded slightly compared to those for Research Questions 1–2. However, the RMSEA of the model for Research Question 3 improved slightly compared to that of Research Question 2. A comparative analysis of the model fit indices between the MLSEM of Research Questions 2 and 3 was supported by the findings of Kenny and McCoach (2003), who indicated that adding more values would reduce CFI and TLI, but it could improve the value of RMSEA from .042 to .041. Of note, RMSEA measures the amount of error of approximation per model, suggesting that a value of 0 indicates the best fit and a higher value indicates a poorer fit (Kaplan, 2009). Figure 7 shows the diagram of the MLSEM model for Research Question 3 and gives the relationships among the variables. Figure 7. Multilevel model of STEM major choices in 4year colleges and universities: Standardized regression coefficients (Model 3) Note. *** p <.01; ** p <.05; * p <.10 Table 4. Multilevel Model of STEM Major Choices in 4Year Colleges and Universities: Standardized Regression Coefficients (Direct Effect)
Note. *** p <.01; ** p <.05; * p <.1 Standard error in parentheses. Table 5. Multilevel Model of STEM Major Choices in 4Year Colleges and Universities: Standardized Regression Coefficients (Indirect Effect)
Note. *** p <.01; ** p <.05; * p <.1 Standard error in parentheses. DISCUSSION As noted previously, this study was designed to test how well the proposed STEM college major choice models fit into the social cognitive career theory (SCCT). The MLSEM analyses suggested the proposed models fit the SCCT overall, except at the school level. Notably, math selfefficacy significantly mediated the relationship between computerbased learning activities in math classrooms and students’ STEM major selections. Students’ individualbased learning activities also contributed to their STEM major selection, as mediated by either math selfefficacy or math performance. In addition, a marginally significant positive relationship existed between a student’s lecturebased learning activities and math selfefficacy, which contributed to a student’s STEM major enrollment. Significant mediating effects of either math selfefficacy or math performance emerged in the relationship between all of the selected learning activities and a student’s STEM major selection. These results are consistent with the path from the learning experience to action component in SCCT. Looking at the demographic characteristics of the students selected and STEM major choices, female students were, not surprisingly, less likely than males to enroll in STEM majors, as mediated by math performance. Similarly, students from lower SES backgrounds showed lower math performance, which resulted in a lower enrollment in STEM majors compared to their peers from higher SES backgrounds. The significant association between student demographic characteristics and STEM major choices reflects the nexus between personal input components and actions in SCCT. At the school level, a marginally significant relationship between math teacher motivation and student math performance emerged. However, no significant mediating effect of schoollevel math performance existed in the relationship between math teacher motivation and an increase in the proportion of student STEM major selection across schools. This schoollevel finding suggests that a student’s STEM major selection is most likely to be determined at the student level. Nevertheless, math teacher motivation, which represents a contextual factor in SCCT, cannot be ruled out based on its marginally significant effects on math performance across schools. Taking a close look at the studentlevel results reveals that among the three learning activities selected, computerbased learning activities in math classrooms showed the most significant and positive effects on students’ STEM major selection. This result suggests that incorporating computerbased learning activities into math classrooms is a motivating factor for high school students to select a STEM career pathway. Of significant note, promoting computerbased curricula in math classrooms at the secondary level is becoming necessary, regardless of whether or not a student pursues a STEM career, because the 21^{st}century job outlook suggests that technological literacy, together with math knowledge and skills, are required in most professional occupations (National Research Council, 2014). Moreover, by 2024 proficiency in computing skills will be required for roughly three out of four projected STEM jobs (U.S. Bureau of Labor Statistics, 2014). Moreover, by 2024 proficiency in computing skills will be required for roughly three out of four projected STEM jobs (U.S. Bureau of Labor Statistics, 2014). Therefore, all students should be adept at computing skills to be prepared to enter the 21^{st}century workforce. As evidenced by the results of this study, incorporating computerbased learning into math classrooms will be a contributing factor, in that students who become familiar with STEMrelated tasks, such as computerbased activities, may ultimately perceive STEM jobs as desirable occupational choices. Consistent with the finding that the largest effects were shown by computerbased learning activities, MLSEM analyses also showed individual and lecturebased learning activities are associated positively with students’ STEM major selection. The synthesis of these results fits the theory of the pedagogical structure illustrated in the paper entitled: “How people learn: Brain, mind, experience, and school” (HPL: Bransford et al., 1999, p. 22). As addressed previously, Bransford et al. indicated that a combination of different learning activities could enhance an individual’s learning outcomes. The significant effects of all three learning activities reflect the importance of incorporating a mixture of diverse learning activities in classrooms, which is consistent with the theory of the pedagogical structure suggested by Bransford et al. (1999). This study contributed to the literature by providing evidence of the positive relationship between computerbased learning activities in math classrooms and STEM major selection. However, the study was unable to provide a full description of computerbased learning activities in math classrooms. The limitations of this study suggest several future studies. First, detailed contexts of computerbased math curricula should be provided. Second, the effects on STEM learning outcomes of a specific computerbased curriculum should be explored. Third, consistent with the national goal of broadening participation in STEM fields (National Academy of Sciences, 2011; National Research Council, 2011; President’s Council of Advisors on Science and Technology, 2010), a subsequent study should explore whether a specific computerbased curriculum meets the needs of diverse learners and increases the interest of students from diverse backgrounds in pursuing a STEM career. Finally, it is recommended that an interdisciplinary research team comprised of scholars in math education and computer science develop computerbased math software designed to increase a student’s interest in studying math with a computer, which will simultaneously enhance his/her computing skills. In addition to demonstrating the importance of academic support for students through computerbased math curricula, this study also recommends that students themselves make the effort to engage frequently in diverse learning activities. As noted earlier, computer, individual, and lecturebased learning activities refer to a student’s frequency of engaging in those activities. The definition of these learning activities implies that if students do not work hard by themselves, investment in academic resources for students will be useless. In summary, this study highlights the importance of cooperative efforts to enhance STEM learning outcomes among a range of stakeholders, including students, teachers, and policy makers. These mutual efforts will enable all students to have the chance to explore their talents and interest in STEM fields. References Arbuckle, J. L. (1996). Full information estimation in the presence of incomplete data. In G. A. Marcoulides & R. E. Schumacker (Eds.), Advanced structural equation modeling: Issues and techniques. Mahwah, NJ: Lawrence Erlbaum Associates. Ascher, C. (1985). Increasing science achievement for disadvantaged students (ERIC: Document Reproduction Service No. ED 253 623). Washington, DC: National Institute of Education. Bandura, A. (1977). Selfefficacy: Toward a unifying theory of behavioral change. Psychological Review, 84, 191–215. Barnett, K., & McCormick, J. (2003). Vision, relationships and teacher motivation: A case study. Journal of Educational Administration, 41(1), 55–73. Becker, K. & Park, K. (2011). Effects of integrative approaches among science, technology, engineering, and mathematics (STEM) subjects on students’ learning: A preliminary metaanalysis. Journal of STEM Education, 12(5), 23–37. Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (1999). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press. Brophy, S., Klein, S., Portsmore, M., & Rogers, C. (2008). Advancing engineering education in P–12 classrooms. Journal of Engineering Education, 97, 369–387. Burns, L. J., Heuer, R., Ingels, S. J., Pollack, J. M., Pratt, D. J., Rock, D., … Stutts, E. (2003, April). Educational Longitudinal Study of 2002: Base year field test report (NCES Working Paper No. 200303). Washington, DC: U.S. Department of Education, National Center for Education Statistics. Cabalo, J. V., Jaciw, A., & Vu, M. T. (2007). Comparative effectiveness of Carnegie Learning’s Cognitive Tutor Algebra I curriculum: A report of a randomized experiment in the Maui School District. Palo Alto, CA: Empirical Education, Inc. Campuzano, L., Dynarski, M., Agodini, R., & Rall, K. (2009). Effectiveness of reading and mathematics software products: Findings from two student cohorts. Washington, DC: U.S. Department of Education, Institute of Education Sciences. Chen, X., & Weko, T. (2009, July). Students who study science, technology, engineering, and mathematics (STEM) in postsecondary education (NCES Stats in Brief No. 2009161). Washington, DC: U.S. Department of Education, National Center for Education Statistics. Clotfelter, C., Ladd, H., & Vigdor, J. (2007). How and why do teacher credentials matter for student learning? (NBER Working Paper No. 12828). Cambridge, MA: National Bureau of Economic Research. Davis, J., & Wilson, S. M. (2000). Principals’ efforts to empower teachers: Effects on teacher motivation and job satisfaction and stress. Educational Research Information Clearinghouse, 73(6), 349–353. Delen, E., & Bulut, O. (2011). The relationship between students’ exposure to technology and their achievement in science and math. TOJET: The Turkish Online Journal of Educational Technology, 10(3), 311–317. Doll, W. J., Raghunathan, T. S., Lim, J., & Gupta, Y. P. (1995). A confirmatory factor analysis of the user information satisfaction instrument. Information Systems Research, 6(2), 177–188. Eccles, J. S. (1987). Gender roles and women’s achievementrelated decisions. Psychology of Women Quarterly, 11(2), 135–172. Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, 109–132. Retrieved from http://outreach.mines.edu/cont_ed/EngEdu/eccles.pdf Enyedy, N. (2014, November). Personalized instruction: New interest, old rhetoric, limited results, and the need for a new direction for computermediated learning. Boulder, CO: National Education Policy Center. Retrieved from http://nepc.colorado.edu/publication/personalizedinstruction Feather, N. T. (1988). Values, valences, and course enrolment: Testing the role of personal values within an expectancyvalue framework. Journal of Educational Psychology, 80(3), 381–391. Firestone, W. A. (2014). Teacher evaluation policy and conflicting theories of motivation. Educational Researcher, 43(2), 100–107. Goe, L. (2007, October). The link between teacher quality and student outcomes: A research synthesis. Washington, DC: National Comprehensive Center for Teacher Quality. Retrieved from http://www.gtlcenter.org/sites/default/files/docs/LinkBetweenTQandStudentOutcomes.pdf Gottfried, M. A., Bozick, R., & Srinivasan, S. V. (2014). Beyond academic math: The role of applied STEM course taking in high school. Teachers College Record, 116(7), 1–35. Graham, S., & Weiner, B. (1996). Theories and principles of motivation. In D. C. Berliner & R. C. Calfee (Eds.), Handbook of educational psychology (pp. 63–84). New York, NY: Macmillan. Hernandez, P. R., Bodin, R., Elliott, J. W., Ibrahim, B., RamboHernandez, K. E., Chen, T. W., & de Miranda, M. A. (2013). Connecting the STEM dots: Measuring the effect of an integrated engineering design intervention. International Journal of Technology and Design Education, 24(1), 107–120. doi:10.1007/s1079801392410 Huang, G., Taddese, N., & Walter, E. (2000). Entry and persistence of women and minorities in college science and engineering education (NCES 2000601). Washington, DC: U.S. Department of Education, National Center for Education Statistics. Retrieved from http://www.nces.ed.gov/pubs2000/2000601.pdf Ingels, S. J., Pratt, D. J., Wilson, D., Burns, L. J., Currivan, D., Rogers, J. E., & HubbardBednasz, S. (2007, October). Education Longitudinal Study of 2002: Baseyear to second followup data file documentation (NCES 2008347). Washington, DC: U.S. Department of Education, National Center for Education Statistics. Retrieved form http://nces.ed.gov/pubs2008/2008347.pdf Institute of Education Science. (2010). Carnegie learning curricula and cognitive tutor® software: What Works Clearinghouse Intervention Report. Washington, DC: US Department of Education. Kaplan, D. (2009). Structural equation modeling: Foundations and extensions. Los Angeles, CA: SAGE. Kaplan, D., & Ferguson, A. (1999). On the utilization of sample weights in latent variable models. Structural Equation Modeling, 6(4), 305–321. Kelley, C., Heneman, H. G., III., & Milanowski, A. (2002). Schoolbased performance rewards: Research findings and future directions. Educational Administration Quarterly, 38(3), 372–401. Kenny, D. A., & McCoach, D. B. (2003). Effects of the number of variables on measures of fit in structural equation modeling. Structural Equation Modeling, 10(3), 333–351. Klein, S. S., & Geist, M. J. (2006). The effect of a bioengineering unit across high school contexts: An investigation in urban, suburban, and rural domains. New Directions in Teaching and Learning, 108, 93–106. Leithwood, K., Jantzi, D., & Steinbach, R. (1999). Changing leadership for changing times. Buckingham, UK: Open University Press. Lu, G., & Copas, J. (2004). Missing at random, likelihood ignorability and model completeness. Annals of Statistics, 32(2), 754–765. Muthén, L. K., & Muthén, B. O. (2010). Mplus user’s guide. Los Angeles, CA: Muthén & Muthén. Muthén, B. O., & Satorra, A. (1989). Multilevel aspects of varying parameters in structural models. In R. D. Bock (Ed.), Multilevel analysis of educational data. San Diego, CA: Academic Press. National Academy of Sciences. (2011). Expanding underrepresented minority participation: America’s science and technology talent at the crossroads. Washington, DC: National Academies Press. National Research Council. (2011). Successful K–12 STEM education: Identifying effective approaches in science, technology, engineering, and mathematics. Washington, DC: National Academies Press. Retrieved from http://www.stemreports.com/wpcontent/uploads/2011/06/NRC_STEM_2.pdf National Research Council. (2014). STEM integration in K–12 education: Status, prospects, and an agenda for research. Washington, DC: National Academies Press. Retrieved from https://www.nap.edu/catalog/18612/stemintegrationink12educationstatusprospectsandan National Science Foundation. (2013). Women, minorities, and persons with disabilities in science and engineering. Retrieved from http://www.nsf.gov/statistics/wmpd/2013/pdf/nsf13304_digest.pdf Peter, L. H. (1977). Cognitive models of motivation, expectancy theory and effort: An analysis and empirical test. Organizational Behavior and Human Performance, 20, 129–148. Porter, O. F. (1989). Undergraduate completion and persistence at fouryear colleges and universities: Completers, persisters, stopouts, and dropouts. Washington, DC: National Institute for Independent Colleges and Universities. President’s Council of Advisors on Science and Technology. (2010, September). Prepare and inspire: K–12 education in science, technology, engineering, and mathematics (STEM) for America’s future [Report to the President]. Retrieved from http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcaststemedreport.pdf Robin, D. (1976). Inference and missing data. Biometrika, 63(3), 581–592. Rotberg, I. C. (1990). Sources and reality: The participation of minorities in science and engineering education. Phi Delta Kappan, 71, 672–679. Said, H., Badry, B. B., & Shahid. M. (2011). Confirmatory factor analysis (CFA) for testing validity and reliability instrument in the study of education. Australian Journal of Basic and Applied Sciences, 5(12), 1098–1103. Shneyderman, A. (2001). Evaluation of the cognitive tutor Algebra I program. Unpublished manuscript. Miami, FL: MiamiDade County Public Schools, Office of Evaluation and Research. Smith, J. E. (2001). The effect of the Carnegie Algebra tutor on student achievement and attitude in introductory high school algebra (Unpublished doctoral dissertation). Virginia Polytechnic Institute and State University, Blacksburg. Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: A secondorder metaanalysis and validation study. Review of Educational Research, 81(1), 4–28. Tran, N. A., & Nathan, M. J. (2010). Precollege engineering studies: An investigation of the relationship between precollege engineering studies and student achievement in science and mathematics. Journal of Engineering Education, 99(2), 143–157. Trusty, J. (2002). Effects of high school coursetaking and other variables on choice of science and mathematics college majors. Journal of Counseling and Development, 80(4), 464–474. U.S. Bureau of Labor Statistics (2014). Employment by detailed occupation, 2014 and projected 2024. Retrieved from http://www.bls.gov/emp/ep_table_102.htm U.S. Census Bureau. (2010). The 2010 statistical abstract: Bachelor’s degree earned by field. Retrieved from https://www.census.gov/prod/2009pubs/10statab/educ.pdf Vandevoort, L. G., AmreinBeardsley, A., & Berliner, D. C. (2004). National board certified teachers and their students’ achievement. Education Policy Analysis Archives, 12(46), 1–117. Wigfield, A., & Eccles, J. (1992). The development of achievement task values: A theoretical analysis. Developmental Review, 12(3), 265–310. Wilson, R. (1990, February 21). Only fifteen percent of students graduate, a new study finds. Chronicle of Higher Education, pp. A1, 1–42. Yuan, K., & Bentler, P. (2001). A unified approach to multigroup structural equation modeling with nonstandard samples. In G. Marcoulides & R. Schumacker (Eds.), New developments and techniques in structural equation modeling (pp. 35–56). Mahwah, NJ: Lawrence Erlbaum Associates.


