Multilevel Structural Equation Models for Investigating the Effects of Computer-Based Learning in Math Classrooms on Science Technology Engineering and Math (STEM) Major Selection in 4-Year Postsecondary Institutions
by Ahlam Lee - 2017
Background/Context: Because of the growing concern over the decline of bachelor degree recipients in the disciplines of science, technology, engineering, and math (STEM) in the U.S., several studies have been devoted to identifying the factors that affect students’ STEM major choices. A majority of these studies have focused on factors relevant to math and science. However, few studies have addressed the linkage between technology- or engineering-related factors and students’ STEM major selection.
Purpose: The purpose of this study was to investigate the extent to which 10th-grade students’ engagement in computer-based learning activities in math classrooms has an effect on student STEM major selection in 4-year postsecondary institutions. After controlling for student- and school-level variables, this study explored the effects of computer-based learning activities in math classrooms on students’ STEM major selections as mediated by either math self-efficacy or math performance.
Research Design: The data from this study were collected from the Educational Longitudinal Study of 2002/2006 (ELS: 2002/2006) conducted by the National Center for Education Statistics of the Institute of Education Sciences in the U.S. Department of Education. ELS: 2002/2006 included a nationally representative sample of young adults who had enrolled in 4-year colleges or universities by 2006. Considering the hierarchical structure of the ELS: 2002/2006, in which students are nested within schools, I used multilevel structural equation modeling (ML-SEM) rather than traditional SEM.
Findings: Students’ engagement in computer-based learning activities in math classrooms had a positive effect on their STEM major selection as mediated by math self-efficacy. Specifically, student computer-based learning activities had a larger effect on students’ STEM major selection than did individual- and lecture-based learning activities. In addition, at the school level, a marginal, but significant, relationship emerged between math teachers’ motivation and students’ math performance.
Conclusions: This study provided evidence that it is important to incorporate computer-based learning activities into math classrooms at the secondary level. Moreover, at the school level, the significant effect of teacher motivation on students’ math achievement scores suggests that motivating teachers is an important part of STEM education, particularly given the fact that teachers are responsible for designing and implementing progressive math curricula that are embedded in computer-based learning activities.
Keywords: computer-based learning, STEM major choice, postsecondary institution, multilevel structural equation modeling
Among the four STEM disciplinesscience, technology, engineering, and mathmath and science have been viewed as the major subjects in K12 STEM education (Hernandez et al., 2013). Accordingly, a great deal of research on STEM education has been devoted to investigating the effects of math- and science-related learning factors on students STEM learning outcomes. Particularly, much of the research efforts have confirmed the positive roles of math- and science-related learning factors in students STEM major selection. However, a limited number of studies have examined the ways in which technology- or engineering-related learning factors affect students STEM learning outcomes. Therefore, this study investigated the extent to which the technology-related learning factorspecifically, computer-based learning activities in math classroomsis associated with students STEM major choices. A teacher-related factor was considered along with the students learning activities in the math classroom because teachers play a major role in designing and implementing classroom curricula. As a teacher-related factor, I chose math teacher motivation based on the literature, which indicates that motivation is the driving force in introducing a new initiative, such as designing and implementing a progressive curriculum (Graham & Weiner, 1996). Moreover, with respect to teacher-related factors, an extensive body of literature has documented the relationship between student learning outcomes and teachers observed qualifications, which include a teachers pedagogical and subject content knowledge, certification status, and experience. All of these teacher qualifications are included in the definition of a highly qualified teacher in the federal No Child Left behind Act of 2001 (NCLB: Clotfelter, Ladd, & Vigdor, 2007; Goe, 2007; Vandevoort, Amrein-Beardsley, & Berliner, 2004). However, relatively little attention has been paid to teachers unobserved qualifications, such as motivation. To fill the gaps in the literature, the following research questions guided this study.
To what extent does the frequency of students engagement in computer-based learning activities in math classrooms, as opposed to the frequency of students engagement in selected traditional learning activities (i.e., lecture- and individual-based learning activities), influence their STEM major choices in 4-year postsecondary institutions, considering prior math performance and teacher motivation? Do math self-efficacy and math achievement scores mediate this effect?
To what extent does the frequency of students engagement in computer-based learning activities in math classrooms, as opposed to the frequency of their engagement in selected traditional learning activities (i.e., lecture- and individual-based learning activities), influence their STEM major choices in 4-year postsecondary institutions, considering prior math performance, teacher motivation, and gender? Do math self-efficacy and math achievement scores mediate this effect?
To what extent does the frequency of students engagement in computer-based learning activities in math classrooms, as opposed to the frequency of their engagement in selected traditional learning activities (i.e., lecture- and individual-based learning activities), influence their STEM major choices in 4-year postsecondary institutions, considering prior math performance, teacher motivation, gender, and socioeconomic status (SES)? Do math self-efficacy and math achievement scores mediate this effect?
To investigate these research questions, three ML-SEMs were proposed. The three research questions differ in the selected control variables. By comparing the three ML-SEMs addressing each research question, I analyzed 1) whether the addition of the selected control variable changes the effects of computer-based learning activities in math classrooms and 2) the extent to which the overall model fits among the three ML-SEMs differ.
The research questions are grounded in social cognitive career theory (SCCT: Lent, Brown, & Hackett, 1994). The following key components of the SCCT framed the selected variables in the study: a) personal input; b) background contextual affordances; c) learning experiences; d) self-efficacy expectations; e) goals; f) contextual influences, and g) actions. This section provides the rationale for selecting the variables associated with the key components of the SCCT.
Figure 1. Social cognitive career theory (SCCT: Lent, Brown & Hackett, 1994)
PERSONAL INPUTS, BACKGROUND, AND CONTEXTUAL AFFORDANCES IN SCCT
As depicted in Figure 1, gender served as a control variable for the personal input component in this study. The underlying rationale of assigning gender as a control variable is that female students are traditionally underrepresented in STEM disciplines (National Science Foundation, 2013). Thus, it is recommended to remove the effect of gender when determining the effects of computer-based learning activities on student STEM learning outcomes (i.e., math self-efficacy, math achievement scores, and STEM major selection).
The background component in the study was measured by students academic background. For student academic background, a students math item response theory (IRT) score was chosen, which was measured when s/he was a 10th grade student in the base year of 2002. The math IRT score was used as a control variable because evidence suggests that a students math achievement score is a strong predictor of whether s/he will pursue a STEM degree. Thus, controlling for students prior math achievement score is critical in examining the extent to which computer-based learning activities in math classrooms contribute to students STEM learning outcomes. The description of the math IRT score is provided in the methods section.
For the contextual affordance component, SES was chosen as a control variable because it is a critical contextual factor in a students decision to pursue a STEM career. The literature suggests that students from higher SES backgrounds are more likely to major in STEM disciplines compared to their lower SES counterparts (Ascher, 1985; Huang, Taddese, & Walter, 2000; Porter, 1989; Rotberg, 1990; Trusty, 2002; Wilson, 1990). As such, similar to gender, it is recommended to factor in the SES effect to measure its influence on STEM learning outcomes of computer-based learning activities in math classrooms.
LEARNING EXPERIENCE IN SCCT
The learning experience component in SCCT was articulated by the selected learning activitiesspecifically, computer-, individual-, and lecture-based learning activitiesin 10th graders math classrooms. The details of these learning activities are provided in the methods section. The rationale for choosing these three learning activities is supported by the pedagogical structure illustrated in the paper How People Learn: Brain, Mind, Experience, and School (HPL: Bransford, Brown, & Cocking, 1999, p. 22). As depicted in Figure 2, in HPL, six learning activities are describedtechnology-, lecture-, skills-, inquiry-, individual-, and group-based learning activities. HPL suggests that individuals learning outcomes may be maximized with a mixture of these designated different learning activities.
Among the six learning activities, this study addressed technology-, individual-, and lecture-based learning activities for the following two reasons. First, technology-based learning activities should be a major focus of this study. As noted earlier, computer-based learning activities in math classrooms represent technology-based learning activities in this study. Second, individual- and lecture-based learning activities in HPL, which represent traditional learning activities, should be compared with the effects of technology-based learning. Thus, because this study focused on the effects of technology-based learning activities as opposed to traditional learning activities, it did not consider other progressive learning activities (i.e., inquiry-, skills-, and group-based learning activities) in HPL.
Figure 2. How people learn: Brain, mind, experience, and school (HPL: Bransford, Brown, & Cocking, 1999, p. 22)
SELF-EFFICACY EXPECTATION IN SCCT
Self-efficacy expectation in SCCT was articulated by math self-efficacy. Math self-efficacy is a latent construct that cannot be represented by a single variable; instead, it should be measured by more than two observed variables relevant to an individuals confidence in math-related tasks. In this study, math self-efficacy comprised five observed variables, as follows: a) a students confidence in taking math tests; b) a students confidence in understanding the most difficult materials in his or her math textbooks; c) a students confidence in understanding the most complex materials presented by his or her math teacher; d) a students confidence in doing his or her math assignments, and e) a students confidence in mastering math skills. The details of the math self-efficacy construct are provided in the methods section.
As shown in the hypothesized model (see Figure 3), math self-efficacy mediates the relationship between the selected exogenous and endogenous variables. In this study, the exogenous variables included selected student demographic characteristics and student learning activities in math classrooms. The endogenous variable is a students STEM major selection. In SEM, an exogenous variable refers to an independent variable while an endogenous variable is defined as a dependent variable. A mediator variable is considered a third type of variable that explains the relationship between exogenous and endogenous variable(s).
GOALS IN SCCT
The goals component of SCCT was aligned with an academic achievement goal. As an academic achievement goal, a students math IRT score, which was assessed in the first follow-up year of 2004 when the student was in 12th grade, was selected. Together with math self-efficacy, a students math IRT score in the first follow-up year was another mediator variable in the hypothesized model shown in Figure 3. Placing both the math IRT score and math self-efficacy in the same position in the hypothesized model (see Figure 3) allowed me to determine which mediator variable was more critical in explaining the relationship between the selected learning activities and a students STEM major selection. The details of the math IRT score in the first follow-up year are explained in the methods section.
CONTEXTUAL INFLUENCES IN SCCT
The contextual influences component of SCCT was represented by math teacher motivation in this study. Math teacher motivation was considered a contextual learning factor that would affect the student learning process and outcomes as a school-level variable, since the existing literature has suggested that school organizational factors, which include the principals leadership and work conditions, largely affect teacher motivation (Barnett & McCormick, 2003; Davis & Wilson, 2000; Kelley, Heneman, & Milanowski, 2002; Leithwood, Jantzi, & Steinbach, 1999).
As addressed briefly in the introduction, a teacher-related factor needs to be coupled with a students engagement in the selected learning activities in math classrooms. The reason is that teachers design and implement those activities. This study used teacher motivation as an unobservable teacher-related factor rather than observable teacher-related factors such as teaching experience, certification status, and education level. The rationale for focusing on the teachers motivation emerged from the fact that there are relatively few studies on the relationship between unobservable teacher factors, such as psychological state, and student learning outcomes, compared to research on the relationship between observed teacher qualifications (listed above) and student learning outcomes.
In this study, teacher motivation was measured by the math teachers perceptions of the importance of teachers attention, values, and enthusiasm in supporting their students success in math. The teacher survey questionnaires measuring teacher motivation can be found in Table 2. This measurement was framed conceptually by the modern expectancy value theory (Eccles & Wigfield, 2002). Eccles and Wigfield claimed that teacher motivation is driven by various cognitive actions that include teacher expectancy, efficacy, and value with reference to a students academic success. Teacher expectancy can be mirrored by math teachers expectation that their attention toward student success in math can contribute to the improvement of student math achievement (the survey questionnaire labeled as BYTM44D in Table 2). Teacher efficacy can be articulated by math teachers confidence that their effective pedagogy can improve student math achievement scores (the survey questionnaire labeled as BYTM44E in Table 2). Teacher value can be embodied by math teachers belief that their enthusiasm or perseverance is positively associated with students success in math (the survey questionnaire labeled as BYTM44F in Table 2). A mixture of these three cognitive actions could echo math teacher motivation. The details related to measuring math teacher motivation are provided in the methods section.
ACTIONS IN SCCT
The actions component of SCCT was related to the status of a students STEM major selection (hereinafter referred to as STEM major selection), which mirrors a students actions in pursuing a STEM career. The category of STEM majors in the study was adopted from the classification of STEM majors suggested in the paper entitled Students Who Study Science, Technology, Engineering, and Mathematics (STEM) in Postsecondary Education published by the National Center for Educational Statistics (Chen & Weko, 2009). STEM majors in this study included mathematics, agricultural/natural sciences, physical sciences, biological sciences, engineering/engineering technologies, and computer/information sciences, as depicted in Table 1.
Table 1. The Categorization of STEM Majors in ELS 2002/06
Note. *STEM Categorization is adopted from the paper titled Students Who Study Science, Technology, Engineering, and Mathematics (STEM) in Postsecondary Education by Chen and Weko (2009) published by the National Center for Educational Statistics (NCES).
Figure 3. Hypothetical multilevel structural equation model of STEM major selection
Due to rising concern about the continued decrease in the number of bachelors degree recipients awarded in the STEM fields over the past three decades (U.S. Census Bureau, 2010), a great deal of research has been conducted to determine the factors that contribute to student enrollment in STEM fields. Most of these studies have focused on investigating math- and science-related learning factors that affect a students STEM learning path, based on the dominant perception that math and science are the core subjects of K12 STEM education. However, the learning contexts embedded in the disciplines of engineering and technology have not been well addressed in previous research on K12 STEM education. Moreover, student STEM learning processes in classrooms should be investigated in relationship to a teacher-related factor because students would not have technology- or engineering-related learning opportunities without teachers efforts to implement progressive curricula. Several studies have documented the relationship between between observable teacher qualifications (e.g., pedagogical and content knowledge, teaching experience, and certification status) and student academic achievement. However, little is known about how a teachers cognitive actions affect student learning. Therefore, this study investigated the effects of math teacher motivation on student STEM learning outcomes, including student math achievement scores, math self-efficacy, and STEM major selection.
The following section reviews the literature on two different areas and concludes with the summary of major findings. The first subsection provides a literature review of technology- and engineering-related learning contexts at the secondary level. The second subsection reviews the literature on teacher motivation. Finally, following the summary of the literature review, the rationale for investigating the research questions is provided.
TECHNOLOGY- AND ENGINEERING-RELATED LEARNING CONTEXTS
Because the competitive 21st-century workforce demands STEM literacy, there has been heightened awareness of the importance of increasing the STEM literacy among all students, regardless of whether they pursue a STEM career (National Research Council, 2011). Accordingly, a broad consensus has been reached among a range of stakeholders to promote an integrated STEM curriculum that is designed to connect at least two subjects among the four STEM disciplines (National Research Council, 2014). However, such a reform-based STEM curriculum has not yet been implemented fully in contemporary K12 classrooms. Given the lack of practical implementation, relatively few studies have explored the influences of a reform-based STEM curriculum embedded in technology- and engineering-based learning contexts on students learning processes and outcomes (National Research Council, 2014).
Among the few, a recent study by Gottfried, Bozick, and Srinivasan (2014) investigated the effects of applied STEM courses on student math achievement scores at the high school level using a nationally representative sample from the Educational Longitudinal Study of 2002 (ELS: 2002). The authors found that taking applied STEM courses was associated positively with student math achievement scores. In the study by Gottfried et al., applied STEM courses were designated as scientific research engineering (SRE) courses and information technology (IT) courses. The designated courses represent technology- and engineering-related subjects. Another large-scale study using the Programme for International Student Assessment (PISA) showed positive effects of information and communication technologies (ICT) on the math and science achievement scores of Turkish students (Delen & Bulut, 2011). In addition to large-scale studies using national or international datasets, meta-analytical studies have provided empirical evidence of the effects of a reform-based STEM curriculum on student achievement scores. For example, 25 meta-analyses based on 1,055 studies (roughly 109,700 participants) suggested that the use of computer technology in classrooms had a significant and positive effect on student academic achievement in a range of subjects, including math and science, with a small to moderate effect size ranging from .30 to .35 (Tamim, Bernard, Borokhovski, Abrami, & Schmid, 2011). Becker and Parks (2011) meta-analysis of 28 studies found that an integrative STEM curriculum, which is designed to connect at least two subjects among the four STEM disciplines, had a positive effect on student achievement in STEM subjects from K12 to the college level. However, Becker and Park noted that the results of this meta-analysis were preliminary because of the dearth of empirical studies on the effects of an integrated STEM curriculum on student STEM learning outcomes.
Furthermore, some studies have attempted to measure the effects of a nationally funded engineering education program on student learning outcomes. For example, the 1999 Infinity Project was designed to improve engineering and technology literacy among middle and high school students based on collaboration among multiple stakeholders in STEM, including the Institute for Engineering Education at Southern Methodist University, Texas Instruments, the U.S. Department of Education, and the National Science Foundation (Brophy, Klein, Portsmore, & Rogers, 2008). A qualitative study found that, according to the instructors of 85 students, the Infinity Project played a critical role in increasing students understanding in math and science and improving student academic achievement in other math and science courses (Douglas, 2006, as cited in Brophy et al., 2008). Additionally, Douglas reported that 94% of the student participants would recommend the Infinity Project to their friends, 83% were interested in pursuing engineering-related careers, and 95% learned a new math concept. Another example of an engineering education program is the Vanderbilt Instruction in Biomedical Engineering for Secondary Science (VIBES) that was initiated in 1999 with grants from the National Science Foundation (Brophy et al., 2008). Using a repeated pretest-posttest design, the research showed that students who engaged in the VIBES modules acquired a better understanding of science concepts compared to their control counterparts, with moderate to large effect sizes (Klein & Geist, 2006). Klein and Geist noted further that the VIBES modules played a positive role in increasing student science knowledge in urban (N =240), suburban (N = 330), and rural classrooms (N = 1,233).
However, some evidence suggests nonsignificant or negative effects of an integrated STEM curriculum on students math and science scores. Tran and Nathan (2010) investigated the relationship between an integrated STEM curriculum, called Project Lead the Way (PLTW), and student math and science scores at the high school level, drawing upon a sample of 140 students and 27 teachers in a Midwestern city. Using multilevel statistical modeling, the researchers found that students enrolled in PTLW achieved less in math and science from eighth to tenth grade compared to their counterparts who did not enroll in PLTW. Moreover, the Institute of Education Science (2010) reported no significant effects of a computer technology program on student math achievement scores among 1,723 high school students in 27 schools across seven districts based on four studies that used randomized controlled trials or quasi-experimental designs (Cabalo, Jaciw, & Vu, 2007; Campuzano, Dynarski, Agodini, & Rall, 2009; Shneyderman, 2001; Smith, 2001). In these studies, the computer technology program was referred to as the Carnegie Learning Curricula and Tutor® software (CLC & CT®S).
These mixed results suggest that the effects of an integrated STEM curriculum, such as a computer-based math curriculum, on student academic achievement have not been determined yet. A recent report by Enyedy (2014) noted that these mixed results regarding the effectiveness of computer-mediated curriculum at the K12 level, as well as the lack of large-scale studies, call for more research on this subject. However, very few studies have explored the effects of an integrated STEM curriculum on a students STEM education path.
As noted previously, student learning processes in classrooms are in juxtaposition with teachers curriculum design and implementation. Obviously, extra effort on the part of teachers is necessary to incorporate a reform-based curriculum, such as an integrated STEM curriculum, into contemporary traditional K12 classrooms (which focus on teaching individual STEM subjects). Importantly, such extra effort emerges from teachers expectations of improving student learning outcomes, which is supported by modern expectancy value theories (e.g., Eccles, 1987; Feather, 1988; Wigfield & Eccles, 1992, 2001). Modern expectancy value theorists have indicated that motivation driven by an individuals expectancy to accomplish a desired level of task performance engenders his/her expenditure of effort to perform the task. Furthermore, a study by Peter (1977) concluded that environmental factors influence individuals efforts in performing a task, which is driven by their motivation.
Consistent with the findings from Peters study (1977), research has shown that teacher motivation is largely affected by school environmental or contextual factors (e.g., Barnett & McCormick, 2003; Davis & Wilson, 2000; Kelley et al., 2002; Leithwood et al., 1999). Specifically, teacher motivation rests on school organizational contexts, such as principals leadership styles and work conditions. For example, a study by Davis and Wilson (2000) found, using a sample of 660 elementary teachers and 44 principals, that the principals encouragement of teachers behaviors was associated positively with their motivation. Moreover, a literature review by Leithwood et al. (1999) showed that transformational leadership played an influential role in increasing teacher motivation to improve classroom practices and attitudes. Likewise, using a semistructured interview of four randomly selected principals and 11 teachers, Barnett and McCormick (2003) found that principals transformational leadership enhanced teacher motivation. In addition to the effects of principals leadership, work conditions are another school contextual factor linked to teacher motivation. For example, a review by Kelley et al. (2002) showed that school-based performance award programs played a positive role in motivating teachers. Moreover, a review by Firestone (2014) suggested that teachers are more motivated to show their competence in an orderly rather than overly punitive working environment.
While it has been shown that the degree of teacher motivation is influenced largely by school organizational contexts, little is known about the relationship between teacher motivation and student learning outcomes. In response, this study offers evidence of the effects of teacher motivation on student STEM learning outcomes.
In summary, a literature review of the STEM education path calls for more research on the relationship between an integrated STEM curriculum and student STEM learning outcomes. An extensive body of literature has documented the critical role of math- and science-related learning contexts in STEM learning outcomes. However, little research has been done on the effects of an integrated STEM curriculum embedded in technology- or engineering-related learning contexts. Considering that STEM goes beyond the traditional core subjects of math and science, technology- or engineering-related learning contexts should be integrated fully into STEM education at the secondary level. To address this issue, this study examined the extent to which computer-based learning activities in math classrooms, as opposed to traditional learning activities that represent individual- and lecture-based learning activities, are associated with students STEM learning outcomes. Equally important, given that math teachers design and implement students learning activities in math classrooms, a teacher-related factor was included in the study. Few studies have explored the effect of math teacher motivation, which is an unobservable psychological teacher-related factor, on students learning outcomes.
ML-SEM analyses investigated the longitudinal effects of 10th-graders computer-based learning activities in math classrooms on their STEM major selections in 4-year postsecondary institutions, as mediated by math self-efficacy and math performance. ML-SEM is an appropriate statistical tool because this study focused on: (a) testing how well the proposed ML-SEM models in the study fit into the conceptual framework of the SCCT, and (b) showing the direct and indirect effects of the selected variables within and between school levels. Furthermore, ML-SEM generates more accurate and unbiased parameter estimates (coefficients) compared to the traditional SEM because it considers standard errors from the multilevel (hierarchical) model (Kaplan & Ferguson, 1999; Muthén & Muthén, 1998; Muthén & Satorra, 1989). The dataset used in this study has a hierarchical structure in which students are nested within schools.
The data were extracted from the Educational Longitudinal Study of 2002/2006 (ELS: 2002/2006) conducted by the National Center for Education Statistics (NCES) of the Institute of Education Sciences (IES), US Department of Education (for detailed information see Ingels et al., 2007). The ELS: 2002/2006 was completed by multiple respondents (e.g., students and teachers) at three time points in 2002, 2004, and 2006. The survey gathered information that can articulate the learning process and outcomes of a nationally representative sample of young adults from 10th grade to postsecondary education or employment status. In the base year of 2002, these young adults were 10th-graders and then 12th-graders in the first follow-up year of 2004. In the second follow-up year of 2006, they had become college students or adults beyond high school.
The reliability and validity of the survey instruments in the ELS: 2002/2006 were ensured based on the following multilevel review and revision process among a range of stakeholders (Ingels et al., 2007). First, a draft of the survey questionnaires was shared with other government agencies, policy groups, and interested parties. Second, the draft was reviewed by the Technical Review Panel (TRP) composed of substantive, methodological, and technical experts. Third, interdivisional review at NCES was performed. Fourth, the survey questionnaires were revised based on feedback of the several stakeholders. Fifth, NCES wrote a justification of the data elements and any issues related to the survey questionnaires. Sixth, the Office of Management and Budget (OMB) reviewed the survey questionnaires. Seventh, the survey questionnaires were revised based on the feedback of the OMB. Finally, the survey questionnaires were tested at over 50 public and private schools in the five field test states (New York, North Carolina, Florida, Illinois, and Texas). Several test analyses of the field tests were conducted, including evaluation of item nonresponses, examination of test-retest reliabilities, calculation of scale reliabilities, and examination of correlations between theoretically related measures. Final revisions of the survey questionnaires were made based on the results of field tests. The detailed field test report on the base year is found in the NCES report by Burns et al. (2003). Additionally, the NCES report by Ingels et al. (2007) provides a summary of the field test results on the first follow-up year and second follow-up year.
From the ELS: 2002/06, I extracted 4,357 students from 711 high schools who later enrolled in 4-year postsecondary institutions and disclosed their majors. Of the 4,357 student participants, 21.5% selected STEM majors and the remaining students (78.5%) enrolled in non-STEM majors. As addressed previously, the definition of STEM majors in the study was determined based on the classification of majors suggested in a report by the U.S. Department of Education (Chen & Weko, 2009). The sample was weighted based on the weighting variable, which was labeled as F2BYWT in the ELS: 2002/06 to represent the nationwide student population.
The three proposed ML-SEMs included four common exogenous variables. Three learning activities were proposed at the student level: (a) computer-based learning activities, (b) individual-based learning activities, and (c) lecture-based learning activities. At the school level, teacher motivation was a common exogenous variable in the three models proposed. Table 2 shows the description of each variable.
The exogenous variables at the student level were measured on a five-point Likert scale ranging from 1 = never and 5 = every day or almost every day. The computer-based learning activities variable was assessed based on students responses to the question: How often do/did you use computers? The mean of this variable was 1.65 with a standard deviation of 1.04 and a range of 1 to 5. The individual-based learning activities variable was assessed based on students response to the question: How often do/did you review the work from the previous day? This variable also ranged from 1 to 5 with a mean of 3.89 and a standard deviation of 1.30. The lecture-based learning activities variable was assessed via the survey question: How often do/did you listen to the teachers lecture? The mean of this variable was 4.49 with a standard deviation of .96 and a range of 1 to 5.
As shown in Table 2, teacher motivation, which was the school-level exogenous variable as well as a latent variable, consisted of the composite value of the responses to the designated three teacher questionnaires. The rationale for selecting and combining the three teacher questionnaires is explained in the latent constructs section below. Each questionnaire was measured on a four-point Likert scale ranging from 1 = extremely important to 4 = not at all important. Thus, when combining the responses to the three teacher survey questionnaires, the teacher motivation variable ranged from 3 to 12. The mean of teacher motivation was 9.74 with a standard deviation of .85 and a range of 6.60 to 12.00.
Table 2. Description of the Variables in the ELS: 2002
Note: Variable labels in the ELS: 2002 in parentheses
In SEM, exogenous variable is equivalent to independent variable.
In SEM, mediator variable is an explanatory variable that describe the relationship between independent and dependent variables.
In SEM, endogenous variable is equivalent to dependent variable.
The effects of the frequency of a students engagement in computer-based learning activities were examined after controlling for the following selected control variables: (a) prior math achievement scores, (b) gender, and (c) SES. The inclusion of the control variables in the proposed model enabled me to measure the effects of a students computer-based learning activities that were not confounded by the selected control variables. The difference among the three proposed models depended on which control variable was added. As addressed in the research question section, Research Question 1 had only one control variable: a students prior math achievement score. Research Question 2 included two of the control variables: (a) prior math achievement scores and (b) gender. To answer Research Question 3, all of the control variables, prior math achievement scores, gender, and SES, were included. There are two reasons for differentiating the type of control variables used to address each research question. First, the study took a close look at whether the effects of computer-based learning activities differed depending on the addition of the control variables in Research Questions 2 and 3. Second, from a methodological standpoint, the study analyzed the extent to which the overall model fit changed due to the addition of the control variables, given that in SEM, a model with fewer parameters (variables) would provide a better fit.
As shown in Table 2, all of the selected control variables (i.e., prior math achievement scores, gender, and SES) were extracted from the base year study of 2002 in ELS: 2002/2006. Prior math achievement scores were represented by the base year math item response theory (IRT) scores. The gender of each student was collected from the base year student survey questionnaires. The base year parent questionnaire provided the SES of the student participants.
The base year math IRT scores had a mean of 52.35 with a standard deviation of 11.84 and a range of 16.35 to 82.03. Gender, which was labeled as BYSEX in the ELS: 2002/06, was originally coded as 1 = male and 2 = female. However, this variable was recoded as 0 = male and 1 = female to transform it into a dummy variable. Overall, 56% of the study participants were female. When transformed into a dummy variable, the gender variable showed the effects of being female on the mediator and endogenous variables in the proposed ML-SEM models (see Figure 3). SES, which was originally labeled as BYSES1 in the ELS: 2002/06, ranged from 1 to 4. The average SES was 3.13, which suggested that the average study participant was upper middle class.
The three ML-SEM models proposed commonly included math self-efficacy and math achievement scores. Like math teacher motivation, math self-efficacy has a latent structure that was identified in an exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). The detailed procedures used to identify the latent structure of math self-efficacy are described in the latent construct section. After the EFA and CFA, the math self-efficacy variable was justified as being composed of the following five indicators: (a) students confidence in their math tests, (b) students confidence in understanding the most difficult materials in their math textbooks; (c) students confidence in understanding the most complex materials presented by their math teachers; (d) students confidence in their math assignments, and (e) students confidence in mastering the skills taught in their math classes. The survey questions that refer to these five indicators are presented in Table 2. The mean of math self-efficacy was 13.76 with a standard deviation of 3.70 and a range from 5 to 20. The other mediator variablemath achievement scorewas represented by the first follow-up math IRT scores, which was measured in 2004 in the ELS: 2002/06. The mean of the first follow-up math IRT scores was 58.81 with a standard deviation of 12.28 and a range from 18.73 to 88.09.
Table 3. Descriptive Statistics and Correlation of the Variables
Note. ** p <.01; * p <.05
This variable was considered prior math achievement scores in the study.
This variable was considered math achievement scores in the study.
Gender was coded as 1 = female and 0 = male. Thus, the mean of the gender,.56 showed that female students comprised 56% of all students in 4-year postsecondary institutions
The endogenous variable in the three proposed models was the status of STEM major selection by students who enrolled in 4-year postsecondary institutions. This was a dichotomous variable coded as 0 = non-STEM major selection and 1 = STEM major selection. As shown in Table 3, the endogenous variable was created based on the variable a students post-secondary major in 2006, which was labeled as F2MJR2_P in the ELS: 2002/06. The F2MJR2_P variable had 33 categories of college majors.
In SEM, confirmatory factor analysis (CFA) is used to test the reliability and validity of any measurements in research (Doll, Raghunathan, Lim, & Gupta, 1995; Said, Badry, & Shahid, 2011). As shown in Figure 4, CFA confirmed a high reliability and validity of the latent constructs of math teacher motivation and math self-efficacy. The goodness of fit model indices indicated good fit of the latent constructs of both variables, with a CFI of .97 (higher than the cutoff value of .95). The observed indicators of both variables were also highly correlated based on factor loadings (i.e., .63 ~ .83).
As addressed in the conceptual framework section, the modern expectancy value theory supported the three observed indicators that composed math teacher motivation (e.g., Eccles, 1987; Feather, 1988; Wigfield & Eccles, 1992, 2001). The teachers responses on the three questionnaires reflected their perceptions of the significant role of their attitude in student success in the following three domains: (a) attention to the unique interests and abilities of students, (b) use of effective methods of teaching, and (c) enthusiasm about student success (see Table 2). The combination of the teachers self-perceptions in these three domains mirrors the expectation, efficacy, and value of math teachers, which are consistent with the cognitive factors in the modern expectancy value theory.
The five indicators observed measured student math self-efficacy based on the self-efficacy concept underlying Banduras social cognitive theory (Bandura, 1977), as shown in Figure 4. Self-efficacy is defined as ones level of confidence in ones ability to complete a specific task (Bandura, 1977). The five indicators of student math self-efficacy measured their confidence level in completing math-related tasks, which is consistent with the concept of self-efficacy.
Figure 4. The latent constructs of math self-efficacy and math teacher motivation: Standardized regression coefficients
Under the assumption of missing at random (MAR), the expectation-maximization (EM) algorithm imputed the missing values in the study with SPSS. MAR, which is most commonly assumed in a missing data structure (Robin, 1976), states that the missing values of a variable can be predicted by nonmissing values of other variables (Lu & Copas, 2004; Yuan & Bentler, 2001). Guided by the concepts of MAR, the missing values of the selected variables were imputed. For example, missing values of observed indicators of math self-efficacy were filled based on the first follow-up math IRT scores, given the literature that math achievement scores are associated positively with math self-efficacy. The data showed the approximate proportion of missing values as follows: (a) 4.18% of the first follow-up math IRT scores, (b) 5% of individual-based learning activities, (c) 5.5% of lecture-based learning activities, (d) 8% of computer-based learning activities, (e) 17% of math teacher motivation, and (f) 25.6% of math self-efficacy. Of note, Arbuckle (1996) recommended that if the percentage of missing values for a variable is more than roughly 20%, a researcher should use a modern missing data treatment, such as the EM algorithm, rather than listwise deletion. Following this recommendation, the missing values in the study were imputed based on the EM algorithm.
Using Mplus 6.1 (Muthén & Muthén, 2010), ML-SEM analyses were run based on the maximum likelihood robust (MLR) estimator. The MLR estimator was appropriate for the different nature of the variables selected in the study (i.e., continuous endogenous and mediator variables, dichotomous control variable, and dichotomous endogenous variable). The MLR estimator is generally recommended for a model on a non-normality assumption that has different types of variables including binary, ordered categorical, and continuous (Kaplan, 2009).
Beginning with the descriptive statistics of the variables (see Table 3), this section reports the findings from the investigation of the research questions. Structural equation modeling (SEM) showed how well the proposed model for each research question fits into the SCCT. Also, the direct and indirect relationships among the variables were explored through SEM, which demonstrated the longitudinal STEM learning processes of a nationally representative sample of 10th-graders in 2002 who enrolled in STEM majors in 4-year postsecondary institutions by 2006.
Table 3 provides the descriptive statistics and correlation among the variables in the study. The descriptive statistics include the percentage of female students, the number of students who enrolled in STEM majors by 2006, and the means and standard deviations of the other selected variables.
RESEARCH QUESTION 1
Research Question 1 was designed to examine the effects of a students frequency in engaging in computer-based learning activities in math classrooms (hereinafter referred to as computer-based learning activities), as opposed to the frequency of a students engagement in lecture-based and individual-based learning activities on his or her STEM major selection. The investigation of Research Question 1 showed the mediating effects of math self-efficacy and math performance on the relationship between the selected learning activities in the math classroom and a students STEM major choice. Consistent with the selected student-level variables, ML-SEM measured the effects of math teacher motivation at the school level. A students prior math performance, which was measured by the math IRT scores in the base year of 2002, was treated as a control variable in the ML-SEM analysis.
Table 4 shows the direct effects of the selected learning activities on STEM major selection, as mediated by either math self-efficacy or math performance. The parameter estimate of computer-based learning activities (b = .693, p < .01) provided evidence that, compared to the individual-based (b = .030, p < .01) and lecture-based learning activities (b = .024, p < .10), computer-based learning activities had a larger effect on math self-efficacy that was linked positively with students STEM major selection.
Table 5 shows the indirect effects of the selected learning activities on STEM major choices, as mediated either by math self-efficacy or math performance. Computer-based learning activities had significant indirect effects on STEM major choices through the mediator of math self-efficacy (b = .025, p < .05), while student math performance (i.e., math IRT scores in the first follow-up year of 2004) did not mediate this relationship. Moreover, there existed marginally significant indirect effects of individual-based learning activities on STEM major selection, as mediated by math self-efficacy (b = .001, p < .10). The significant mediating effects of student math performance emerged only in the relationship between individual-based learning activities and students STEM major selection (b = .006, p < .05).
SEM analysis at the school level yielded a marginally significant relationship between math teacher motivation and student math performance (school-level math IRT scores in the first follow-up year of 2004). However, there existed no significant mediating effects of student math performance on the relationship between math teacher motivation and STEM major selection, suggesting that there was no significant indirect effect of math teacher motivation on STEM major selection across schools.
The proposed model for Research Question 1 is considered a good fit, as evidenced by the following model fit indices: CFI = .993; TLI = .986; RMSEA = .026 (see Table 4). Figure 5 provides the diagram of the ML-SEM model for Research Question 1.
Figure 5. Multilevel model of STEM major choices in 4-year colleges and universities: Standardized regression coefficients (Model 1)
Note. *** p <.01; ** p <.05; * p <.10
RESEARCH QUESTION 2
Research Question 2 added gender to Research Question 1 and sought to identify whether the effects of computer-based learning activities differed when gender was added as a control variable. When controlling for gender and prior math performance, the relationship between the variables showed a pattern similar to the findings from Research Question 1 at both the student and school levels.
As shown in Table 4, at the student level, compared to the individual-based (b = .032, p < .05) and lecture-based learning activities (b = .025. p < .10), computer-based learning activities showed a greater direct effect on increasing student math self-efficacy (b = .692, p < .01), which contributed substantially to a students STEM major decision. As shown in Table 5, a substantial mediating effect of math self-efficacy emerged in the relationship between computer-based learning activities and a students STEM major selection, as evidenced by the indirect effects of computer-based learning activities on STEM major selection through the mediator of math self-efficacy (b = .025, p < .05). However, student math performance, which was represented by math IRT scores in the first follow-up year, did not mediate this relationship significantly. As shown in Table 5, similar to the findings for Research Question 1, the significant mediating effects of student math performance existed only in the relationship between individual-based learning activities and a students STEM major selection (b = .007, p < .001).
At the school level, controlling for math IRT scores in the base year of 2002, a marginally significant relationship existed between math teacher motivation and math IRT scores in the first follow-up year of 2004. However, there was no significant relationship between either mediator (i.e., math self-efficacy and math IRT scores in the first follow-up year), and a students STEM major selection.
The proposed model for Research Question 2 is considered a good fit, as evidenced by the following model fit indices: CFI =.978; TLI =.959; RMSEA =.042 (see Table 4). However, the model fit indices for Research Question 2 were slightly worse than those for Research Question 1. The decrease in CFI and TLI between the model of Research Question 1 and that of Research Question 2 adds evidence that model fit indices in SEM tend to be penalized by the inclusion of more variables, although this is not always the case (Kenny & McCoach, 2003). Notably, because CFI and TLI measure the relative improvement in model fit of the proposed model over the baseline model, a value of 1 indicates the best fit and a value of 0 indicates the worst fit (Kaplan, 2009). In Figure 6, the diagram of the ML-SEM model for Research Question 2 shows the relationships among the variables.
Figure 6. Multilevel model of STEM major choices in 4-year colleges and universities: Standardized regression coefficients (Model 2)
Note. *** p <.01; ** p <.05; * p <.10
RESEARCH QUESTION 3
Research Question 3 was designed to show whether the effect of computer-based learning activities changed significantly when SES was added to the proposed model in Research Question 2. The ML-SEM of Research Question 3 revealed that, similar to the findings of Research Questions 12, computer-based learning activities in math classrooms had a greater effect on increasing students math self-efficacy, which was the mediator between computer-based learning activities and students STEM major selection, as opposed to individual- and lecture-based learning activities (see Model 3 in Table 4). Like the ML-SEM for Research Questions 12, math self-efficacy played a significant role in predicting an increase in the proportion of students who chose STEM majors.
To be specific, as shown in Table 4, at the student level, 10th-graders computer-based learning activities in the base year of 2002 had a significant and positive effect on student math self-efficacy in the first follow-up year of 2004 (b = .693, p < .01), controlling for prior student math performance, gender, and SES. Computer-based learning activities had a greater influence in increasing student math self-efficacy, compared to individual-based (b = .032, p < .05) and lecture-based learning activities (b = .024, p < .10). Among the three learning activities selected, the greatest direct effect of the computer-based learning activities on math self-efficacy did not differ from the findings of Research Questions 12.
The pattern of the indirect effects was similar to that in Research Questions 12. Computer-based learning activities showed significant indirect effects on students STEM major selection, as mediated by math self-efficacy (b =.024, p <.01). As shown in Table 5, the indirect effect of computer-based learning activities was larger compared to those of both individual- (b =.008, p <.01) and lecture-based learning activities (b =.000, p >.10). The significant and positive mediating effects of math performance were found only in the relationship between individual-based learning activities and STEM major selections (b =.007, p <.01).
At the school level, however, no significant relationship existed between math teacher motivation and STEM major selection, which was different from the school-level findings in Research Questions 12.
The proposed model for Research Question 3 is considered a good fit, with the following model fit indices: CFI = .975; TLI = .956; and RMSEA = .041 (see Table 3). However, the model fit indices for CFI and TLI in Research Question 3 were degraded slightly compared to those for Research Questions 12. However, the RMSEA of the model for Research Question 3 improved slightly compared to that of Research Question 2. A comparative analysis of the model fit indices between the ML-SEM of Research Questions 2 and 3 was supported by the findings of Kenny and McCoach (2003), who indicated that adding more values would reduce CFI and TLI, but it could improve the value of RMSEA from .042 to .041. Of note, RMSEA measures the amount of error of approximation per model, suggesting that a value of 0 indicates the best fit and a higher value indicates a poorer fit (Kaplan, 2009). Figure 7 shows the diagram of the ML-SEM model for Research Question 3 and gives the relationships among the variables.
Figure 7. Multilevel model of STEM major choices in 4-year colleges and universities: Standardized regression coefficients (Model 3)
Note. *** p <.01; ** p <.05; * p <.10
Table 4. Multilevel Model of STEM Major Choices in 4-Year Colleges and Universities: Standardized Regression Coefficients (Direct Effect)
Note. *** p <.01; ** p <.05; * p <.1
Standard error in parentheses.
Table 5. Multilevel Model of STEM Major Choices in 4-Year Colleges and Universities: Standardized Regression Coefficients (Indirect Effect)
Note. *** p <.01; ** p <.05; * p <.1
Standard error in parentheses.
As noted previously, this study was designed to test how well the proposed STEM college major choice models fit into the social cognitive career theory (SCCT). The ML-SEM analyses suggested the proposed models fit the SCCT overall, except at the school level. Notably, math self-efficacy significantly mediated the relationship between computer-based learning activities in math classrooms and students STEM major selections. Students individual-based learning activities also contributed to their STEM major selection, as mediated by either math self-efficacy or math performance. In addition, a marginally significant positive relationship existed between a students lecture-based learning activities and math self-efficacy, which contributed to a students STEM major enrollment. Significant mediating effects of either math self-efficacy or math performance emerged in the relationship between all of the selected learning activities and a students STEM major selection. These results are consistent with the path from the learning experience to action component in SCCT.
Looking at the demographic characteristics of the students selected and STEM major choices, female students were, not surprisingly, less likely than males to enroll in STEM majors, as mediated by math performance. Similarly, students from lower SES backgrounds showed lower math performance, which resulted in a lower enrollment in STEM majors compared to their peers from higher SES backgrounds. The significant association between student demographic characteristics and STEM major choices reflects the nexus between personal input components and actions in SCCT.
At the school level, a marginally significant relationship between math teacher motivation and student math performance emerged. However, no significant mediating effect of school-level math performance existed in the relationship between math teacher motivation and an increase in the proportion of student STEM major selection across schools. This school-level finding suggests that a students STEM major selection is most likely to be determined at the student level. Nevertheless, math teacher motivation, which represents a contextual factor in SCCT, cannot be ruled out based on its marginally significant effects on math performance across schools.
Taking a close look at the student-level results reveals that among the three learning activities selected, computer-based learning activities in math classrooms showed the most significant and positive effects on students STEM major selection. This result suggests that incorporating computer-based learning activities into math classrooms is a motivating factor for high school students to select a STEM career pathway. Of significant note, promoting computer-based curricula in math classrooms at the secondary level is becoming necessary, regardless of whether or not a student pursues a STEM career, because the 21st-century job outlook suggests that technological literacy, together with math knowledge and skills, are required in most professional occupations (National Research Council, 2014). Moreover, by 2024 proficiency in computing skills will be required for roughly three out of four projected STEM jobs (U.S. Bureau of Labor Statistics, 2014). Moreover, by 2024 proficiency in computing skills will be required for roughly three out of four projected STEM jobs (U.S. Bureau of Labor Statistics, 2014). Therefore, all students should be adept at computing skills to be prepared to enter the 21st-century workforce. As evidenced by the results of this study, incorporating computer-based learning into math classrooms will be a contributing factor, in that students who become familiar with STEM-related tasks, such as computer-based activities, may ultimately perceive STEM jobs as desirable occupational choices.
Consistent with the finding that the largest effects were shown by computer-based learning activities, ML-SEM analyses also showed individual- and lecture-based learning activities are associated positively with students STEM major selection. The synthesis of these results fits the theory of the pedagogical structure illustrated in the paper entitled: How people learn: Brain, mind, experience, and school (HPL: Bransford et al., 1999, p. 22). As addressed previously, Bransford et al. indicated that a combination of different learning activities could enhance an individuals learning outcomes. The significant effects of all three learning activities reflect the importance of incorporating a mixture of diverse learning activities in classrooms, which is consistent with the theory of the pedagogical structure suggested by Bransford et al. (1999).
This study contributed to the literature by providing evidence of the positive relationship between computer-based learning activities in math classrooms and STEM major selection. However, the study was unable to provide a full description of computer-based learning activities in math classrooms. The limitations of this study suggest several future studies. First, detailed contexts of computer-based math curricula should be provided. Second, the effects on STEM learning outcomes of a specific computer-based curriculum should be explored. Third, consistent with the national goal of broadening participation in STEM fields (National Academy of Sciences, 2011; National Research Council, 2011; Presidents Council of Advisors on Science and Technology, 2010), a subsequent study should explore whether a specific computer-based curriculum meets the needs of diverse learners and increases the interest of students from diverse backgrounds in pursuing a STEM career. Finally, it is recommended that an interdisciplinary research team comprised of scholars in math education and computer science develop computer-based math software designed to increase a students interest in studying math with a computer, which will simultaneously enhance his/her computing skills.
In addition to demonstrating the importance of academic support for students through computer-based math curricula, this study also recommends that students themselves make the effort to engage frequently in diverse learning activities. As noted earlier, computer-, individual-, and lecture-based learning activities refer to a students frequency of engaging in those activities. The definition of these learning activities implies that if students do not work hard by themselves, investment in academic resources for students will be useless. In summary, this study highlights the importance of cooperative efforts to enhance STEM learning outcomes among a range of stakeholders, including students, teachers, and policy makers. These mutual efforts will enable all students to have the chance to explore their talents and interest in STEM fields.
Arbuckle, J. L. (1996). Full information estimation in the presence of incomplete data. In G. A. Marcoulides & R. E. Schumacker (Eds.), Advanced structural equation modeling: Issues and techniques. Mahwah, NJ: Lawrence Erlbaum Associates.
Ascher, C. (1985). Increasing science achievement for disadvantaged students (ERIC: Document Reproduction Service No. ED 253 623). Washington, DC: National Institute of Education.
Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84, 191215.
Barnett, K., & McCormick, J. (2003). Vision, relationships and teacher motivation: A case study. Journal of Educational Administration, 41(1), 5573.
Becker, K. & Park, K. (2011). Effects of integrative approaches among science, technology, engineering, and mathematics (STEM) subjects on students learning: A preliminary meta-analysis. Journal of STEM Education, 12(5), 2337.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (1999). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press.
Brophy, S., Klein, S., Portsmore, M., & Rogers, C. (2008). Advancing engineering education in P12 classrooms. Journal of Engineering Education, 97, 369387.
Burns, L. J., Heuer, R., Ingels, S. J., Pollack, J. M., Pratt, D. J., Rock, D., Stutts, E. (2003, April). Educational Longitudinal Study of 2002: Base year field test report (NCES Working Paper No. 2003-03). Washington, DC: U.S. Department of Education, National Center for Education Statistics.
Cabalo, J. V., Jaciw, A., & Vu, M. T. (2007). Comparative effectiveness of Carnegie Learnings Cognitive Tutor Algebra I curriculum: A report of a randomized experiment in the Maui School District. Palo Alto, CA: Empirical Education, Inc.
Campuzano, L., Dynarski, M., Agodini, R., & Rall, K. (2009). Effectiveness of reading and mathematics software products: Findings from two student cohorts. Washington, DC: U.S. Department of Education, Institute of Education Sciences.
Chen, X., & Weko, T. (2009, July). Students who study science, technology, engineering, and mathematics (STEM) in postsecondary education (NCES Stats in Brief No. 2009-161). Washington, DC: U.S. Department of Education, National Center for Education Statistics.
Clotfelter, C., Ladd, H., & Vigdor, J. (2007). How and why do teacher credentials matter for student learning? (NBER Working Paper No. 12828). Cambridge, MA: National Bureau of Economic Research.
Davis, J., & Wilson, S. M. (2000). Principals efforts to empower teachers: Effects on teacher motivation and job satisfaction and stress. Educational Research Information Clearinghouse, 73(6), 349353.
Delen, E., & Bulut, O. (2011). The relationship between students exposure to technology and their achievement in science and math. TOJET: The Turkish Online Journal of Educational Technology, 10(3), 311317.
Doll, W. J., Raghunathan, T. S., Lim, J., & Gupta, Y. P. (1995). A confirmatory factor analysis of the user information satisfaction instrument. Information Systems Research, 6(2), 177188.
Eccles, J. S. (1987). Gender roles and womens achievement-related decisions. Psychology of Women Quarterly, 11(2), 135172.
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, 109132. Retrieved from http://outreach.mines.edu/cont_ed/Eng-Edu/eccles.pdf
Enyedy, N. (2014, November). Personalized instruction: New interest, old rhetoric, limited results, and the need for a new direction for computer-mediated learning. Boulder, CO: National Education Policy Center. Retrieved from http://nepc.colorado.edu/publication/personalized-instruction
Feather, N. T. (1988). Values, valences, and course enrolment: Testing the role of personal values within an expectancy-value framework. Journal of Educational Psychology, 80(3), 381391.
Firestone, W. A. (2014). Teacher evaluation policy and conflicting theories of motivation. Educational Researcher, 43(2), 100107.
Goe, L. (2007, October). The link between teacher quality and student outcomes: A research synthesis. Washington, DC: National Comprehensive Center for Teacher Quality. Retrieved from http://www.gtlcenter.org/sites/default/files/docs/LinkBetweenTQandStudentOutcomes.pdf
Gottfried, M. A., Bozick, R., & Srinivasan, S. V. (2014). Beyond academic math: The role of applied STEM course taking in high school. Teachers College Record, 116(7), 135.
Graham, S., & Weiner, B. (1996). Theories and principles of motivation. In D. C. Berliner & R. C. Calfee (Eds.), Handbook of educational psychology (pp. 6384). New York, NY: Macmillan.
Hernandez, P. R., Bodin, R., Elliott, J. W., Ibrahim, B., Rambo-Hernandez, K. E., Chen, T. W., & de Miranda, M. A. (2013). Connecting the STEM dots: Measuring the effect of an integrated engineering design intervention. International Journal of Technology and Design Education, 24(1), 107120. doi:10.1007/s10798-013-9241-0
Huang, G., Taddese, N., & Walter, E. (2000). Entry and persistence of women and minorities in college science and engineering education (NCES 2000-601). Washington, DC: U.S. Department of Education, National Center for Education Statistics. Retrieved from http://www.nces.ed.gov/pubs2000/2000601.pdf
Ingels, S. J., Pratt, D. J., Wilson, D., Burns, L. J., Currivan, D., Rogers, J. E., & Hubbard-Bednasz, S. (2007, October). Education Longitudinal Study of 2002: Base-year to second follow-up data file documentation (NCES 2008-347). Washington, DC: U.S. Department of Education, National Center for Education Statistics. Retrieved form http://nces.ed.gov/pubs2008/2008347.pdf
Institute of Education Science. (2010). Carnegie learning curricula and cognitive tutor® software: What Works Clearinghouse Intervention Report. Washington, DC: US Department of Education.
Kaplan, D. (2009). Structural equation modeling: Foundations and extensions. Los Angeles, CA: SAGE.
Kaplan, D., & Ferguson, A. (1999). On the utilization of sample weights in latent variable models. Structural Equation Modeling, 6(4), 305321.
Kelley, C., Heneman, H. G., III., & Milanowski, A. (2002). School-based performance rewards: Research findings and future directions. Educational Administration Quarterly, 38(3), 372401.
Kenny, D. A., & McCoach, D. B. (2003). Effects of the number of variables on measures of fit in structural equation modeling. Structural Equation Modeling, 10(3), 333351.
Klein, S. S., & Geist, M. J. (2006). The effect of a bioengineering unit across high school contexts: An investigation in urban, suburban, and rural domains. New Directions in Teaching and Learning, 108, 93106.
Leithwood, K., Jantzi, D., & Steinbach, R. (1999). Changing leadership for changing times. Buckingham, UK: Open University Press.
Lu, G., & Copas, J. (2004). Missing at random, likelihood ignorability and model completeness. Annals of Statistics, 32(2), 754765.
Muthén, L. K., & Muthén, B. O. (2010). Mplus users guide. Los Angeles, CA: Muthén & Muthén.
Muthén, B. O., & Satorra, A. (1989). Multilevel aspects of varying parameters in structural models. In R. D. Bock (Ed.), Multilevel analysis of educational data. San Diego, CA: Academic Press.
National Academy of Sciences. (2011). Expanding underrepresented minority participation: Americas science and technology talent at the crossroads. Washington, DC: National Academies Press.
National Research Council. (2011). Successful K12 STEM education: Identifying effective approaches in science, technology, engineering, and mathematics. Washington, DC: National Academies Press. Retrieved from http://www.stemreports.com/wp-content/uploads/2011/06/NRC_STEM_2.pdf
National Research Council. (2014). STEM integration in K12 education: Status, prospects, and an agenda for research. Washington, DC: National Academies Press. Retrieved from https://www.nap.edu/catalog/18612/stem-integration-in-k-12-education-status-prospects-and-an
National Science Foundation. (2013). Women, minorities, and persons with disabilities in science and engineering. Retrieved from http://www.nsf.gov/statistics/wmpd/2013/pdf/nsf13304_digest.pdf
Peter, L. H. (1977). Cognitive models of motivation, expectancy theory and effort: An analysis and empirical test. Organizational Behavior and Human Performance, 20, 129148.
Porter, O. F. (1989). Undergraduate completion and persistence at four-year colleges and universities: Completers, persisters, stopouts, and dropouts. Washington, DC: National Institute for Independent Colleges and Universities.
Presidents Council of Advisors on Science and Technology. (2010, September). Prepare and inspire: K12 education in science, technology, engineering, and mathematics (STEM) for Americas future [Report to the President]. Retrieved from http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-stemed-report.pdf
Robin, D. (1976). Inference and missing data. Biometrika, 63(3), 581592.
Rotberg, I. C. (1990). Sources and reality: The participation of minorities in science and engineering education. Phi Delta Kappan, 71, 672679.
Said, H., Badry, B. B., & Shahid. M. (2011). Confirmatory factor analysis (CFA) for testing validity and reliability instrument in the study of education. Australian Journal of Basic and Applied Sciences, 5(12), 10981103.
Shneyderman, A. (2001). Evaluation of the cognitive tutor Algebra I program. Unpublished manuscript. Miami, FL: Miami-Dade County Public Schools, Office of Evaluation and Research.
Smith, J. E. (2001). The effect of the Carnegie Algebra tutor on student achievement and attitude in introductory high school algebra (Unpublished doctoral dissertation). Virginia Polytechnic Institute and State University, Blacksburg.
Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81(1), 428.
Tran, N. A., & Nathan, M. J. (2010). Pre-college engineering studies: An investigation of the relationship between pre-college engineering studies and student achievement in science and mathematics. Journal of Engineering Education, 99(2), 143157.
Trusty, J. (2002). Effects of high school course-taking and other variables on choice of science and mathematics college majors. Journal of Counseling and Development, 80(4), 464474.
U.S. Bureau of Labor Statistics (2014). Employment by detailed occupation, 2014 and projected 2024. Retrieved from http://www.bls.gov/emp/ep_table_102.htm
U.S. Census Bureau. (2010). The 2010 statistical abstract: Bachelors degree earned by field. Retrieved from https://www.census.gov/prod/2009pubs/10statab/educ.pdf
Vandevoort, L. G., Amrein-Beardsley, A., & Berliner, D. C. (2004). National board certified teachers and their students achievement. Education Policy Analysis Archives, 12(46), 1117.
Wigfield, A., & Eccles, J. (1992). The development of achievement task values: A theoretical analysis. Developmental Review, 12(3), 265310.
Wilson, R. (1990, February 21). Only fifteen percent of students graduate, a new study finds. Chronicle of Higher Education, pp. A-1, 142.
Yuan, K., & Bentler, P. (2001). A unified approach to multigroup structural equation modeling with nonstandard samples. In G. Marcoulides & R. Schumacker (Eds.), New developments and techniques in structural equation modeling (pp. 3556). Mahwah, NJ: Lawrence Erlbaum Associates.