Mathematics Course Placement Using Holistic Measures: Possibilities for Community College Students
by Federick Ngo, W. Edward Chi & Elizabeth So Yun Park - 2018
Background/Context: Most community colleges across the country use a placement test to determine studentsí readiness for college-level coursework, yet these tests are admittedly imperfect instruments. Researchers have documented significant problems stemming from overreliance on placement testing, including placement error and misdiagnosis of remediation needs. They have also described significant consequences of misplacement, which can hinder the educational progression and attainment of community college students.
Purpose/Objective/Research Question/Focus of Study: We explore possibilities for placing community college students in mathematics courses using a holistic approach that considers measures beyond placement test scores. This includes academic background measures, such as high school GPA and math courses taken, and indicators of noncognitive constructs, such as motivation, time use, and social support.
Setting: The study draws upon administrative data from a large urban community college district in California that serves over 100,000 students each semester. The data enable us to link studentsí placement testing results, survey data, background information, and transcript records.
Research Design: We first use the supplemental survey data gathered during routine placement testing to conduct predictive exercises that identify severe placement errors under existing placement practices. We then move beyond prediction and evaluate student outcomes in two colleges where noncognitive indicators were directly factored into placement algorithms.
Findings/Results: Using high school background information and noncognitive indicators to predict success reveals as many as one quarter of students may be misassigned to their math courses by status quo practices. In our subsequent analysis we find that students placed under a holistic approach that considered noncognitive indicators in addition to placement test scores performed no differently from higher scoring peers in the same course.
Conclusions/Recommendations: The findings suggest a holistic approach to mathematics course placement may improve placement accuracy and provide access to higher level mathematics courses for community college students without compromising their likelihood of success.
What does it mean to be ready for college? And how do colleges know? In the community college setting, the answers to these questions are usually informed by a placement test that students take when they begin or restart their educational careers. Over 90%of all community colleges in the country use a placement test to determine students readiness for college-level coursework (Fields & Parsad, 2012). At the same time, nearly 60% of all incoming community college students in the nation enroll in a remedial course, most likely because they are deemed underprepared for college-level coursework (Bailey, Jeong, & Cho, 2010; NCPPHE & SREB, 2010).
Yet placement tests are admittedly imperfect instruments. Recent research has estimated that nearly 25% of students may be misplaced into their math courses by commonly used placement tests (Mattern & Packman, 2009; Scott-Clayton, Crosta, & Belfield, 2014), with potentially serious consequences for educational attainment (Melguizo, Bos, Ngo, Mills, & Prather, 2016). Researchers examining these issues have found that using additional information such as those available in high school transcripts and math diagnostics could improve placement accuracy and reduce the rate of placement errors (Ngo & Kwon, 2015; Ngo & Melguizo, 2016; Scott-Clayton et al., 2014). In tandem with these findings, a number of states (e.g., Florida, North Carolina, Texas) have enacted legislation promoting the incorporation of multiple indicators of students academic readiness into community college placement policies (Burdman, 2012). California, the setting for the study, has mandated the use of these multiple measures since the early 1990s, but there is wide variation in and few evaluations of these practices (Perry, Bahr, Rosin, & Woodward, 2010).
What remains unknown is whether using a holistic placement approach that includes noncognitive1 measures can improve placement accuracy in community colleges. Noncognitive measures are those not specifically related to academic content knowledge or skills, such as, but not limited to, students college plans (e.g., use of time) and their beliefs about the importance of math or college (Sedlacek, 2004). While indicators of some of these constructs are implicitly and sometimes explicitly used in the selection and sorting processes in four-year institutions (e.g., via college admissions essays and letters of recommendation), they largely remain untested and unused in the community college setting. Further, noncognitive attributes have been shown to be predictive of students postsecondary success (Sedlacek, 2004) but have rarely been examined in the context of community colleges which serve large numbers of students of color, low-income students, and first-generation college students (Horn, Nevill, & Griffith, 2006). We therefore ask two research questions: (1) What possibilities are there for using a more holistic placement approach that includes noncognitive measures to better identify college readiness among community college students? (2) Do noncognitive measures improve placement accuracy in developmental math?
We focus on the use of indicators of noncognitive constructs for course placement in a large urban community college district (LUCCD) in California, and we conduct two sets of analyses to answer these questions. The first is a predictive exercise that examines possibilities for using indicators of noncognitive constructs in placement decisions. To do so, we use methods outlined by Scott-Clayton et al. (2014) to estimate severe placement errors, but capitalize on the availability of noncognitive questionnaire data that LUCCD colleges simultaneously collected during the time of placement testing. We calculate placement error rates for all colleges under existing placement policies (e.g., placement test scores and other academic background measures) and compare these to estimates of placement error when indicators of noncognitive measures are included in the prediction equation. We emphasize here that the noncognitive questionnaire items are proxy indicators of students noncognitive attributes and not necessarily measures validated in prior literature. In the second set of analyses, we examine actual placement algorithms in two colleges that factor in these noncognitive indicators, such as those of motivation and college plans, into placement decisions. We examine the outcomes of students who were able to take a higher level course due to additional points they earned based on noncognitive indicators, comparing them to peers placed in the same level but who scored higher on placement tests.
Our study adds insight to the broader question of whether a more holistic approach to mathematics course placement that includes indicators of noncognitive attributes can be useful within the open-access setting of community colleges. In contrast to four-year colleges with selective admissions, selection and sorting in the community college setting primarily happens during remedial screening where students who are deemed not college-ready may be referred to remedial coursework (Hughes & Scott-Clayton, 2011). Since this typically hinges on the result of a placement test, incorporating noncognitive information into the screening process may provide opportunity and access for students who do not appear to be academically prepared based on their placement test results alone. Indeed, our analyses first show that a substantial portion of students, as many as one quarter, may be considered as misplaced under current test-based placement practices, and that high school background and noncognitive indicators may offer an improvement over status quo practices. When we test this hypothesis in two colleges that actually factor noncognitive information into placement rules, we find that this holistic approach can increase access to higher level math courses without compromising the likelihood of success in those courses.
The paper proceeds as follows. We first discuss the role of placement testing and selection processes in community colleges. We highlight research on noncognitive measures, and then draw upon expanding conceptions of college readiness and validation theory to frame the study. We then describe the LUCCD and our two analytical approachesone that investigates possibilities for using noncognitive indicators and one that evaluates existing placement practices already using such indicators. We present the findings and discuss how our work can add insight to current reforms in assessment and placement in community colleges.
PLACEMENT TESTS AND ACADEMIC MEASURES OF COLLEGE READINESS
Given that community colleges are open-access institutions serving a diverse body of students, they need some means of identifying academic preparedness and directing students towards appropriate course work. Placement tests are commonly used in community colleges for this purpose, and two tests, the College Boards ACCUPLACER and the ACT Inc.s COMPASS, have dominated the market (Conley, 2010).2 These tests are multiple choice, adaptive, computer administered, and are used to assess subjects like math, English, and reading. Regarded as a cost-efficient way to assess students academic abilities, the placement test score is the primary measure that determines where students start in their educational trajectory (Hughes & Scott-Clayton, 2011).
Despite the near ubiquity of placement testing, studies investigating the predictive validity of placement tests have found that correlations between test scores and student achievement are weaker than those between student background variables and achievement (Armstrong, 2000). In fact, Jenkins, Jaggars, and Roksa (2009), examining data from Virginia community colleges, found no significant relationship between reading and writing placement tests and whether students passed gatekeeper English courses, though they did find a relationship between math placement tests and whether students passed gatekeeper math courses.
These findings, along with concerns about the accuracy of placement tests, have fostered growing interest in using multiple measures and a more holistic approach to improve placement decisions (Burdman, 2012; Smith, 2016). Measures such as the level of prior math courses taken and high school GPA are known to be strong predictors of college course completion and success and can be used to identify readiness for college-level work (e.g., Armstrong, 2000; Noble & Sawyer, 2004). Adding to this evidence, Ngo and Kwon (2015) found that community college students who were placed using academic background measures (e.g., prior math and GPA) in addition to test scores performed no differently from their peers who earned higher test scores. This study and others (e.g., Fong & Melguizo, 2016; Marwick, 2004) suggest that using multiple measures may increase access to higher level math without compromising students likelihood of success in those courses.
RESEARCH ON NONCOGNITIVE MEASURES
One understudied question is whether noncognitive measures can also improve placement decisions. The logic for incorporating these measures stems from research in educational psychology, which demonstrates that an array of noncognitive attributes beyond cognitive skills are predictive of college success and future outcomes (Duckworth, Peterson, Matthews, & Kelly, 2007; Duckworth & Yeager, 2015; Noonan, Sedlacek, & Veerasamy, 2005; Porcea, Allen, Robbins, & Phelps, 2010; Robbins, Allen, Casillas, Peterson, & Le, 2006). In lieu of providing a comprehensive review of all noncognitive factors and measures associated with college student success, we focus only on those we believe to be related to the measures used by LUCCD colleges to assess and place students in developmental math sequences. These include use of time, motivation, and social support. To our knowledge, the measures used by LUCCD are not directly tied to particular constructs or scales in the literature. We therefore discuss the general literature on each of these noncognitive areas.
One such area is students use of time. Researchers examining college students time use found that in certain populations, working concurrently in college predicts weaker academic outcomes (Ehrenberg & Sherman, 1987; Pascarella, Edison, Nora, Hagedorn, & Terenzini, 1998; Stinebrickner & Stinebrickner, 2003) and that time studying predicts improved academic outcomes (Michaels & Miethe, 1989; Rau & Durand, 2000; Stinebrickner & Stinebrickner, 2004; 2008). These studies suggest that employment while attending school is associated with decreased likelihood of persistence and lower academic outcomes. Therefore, measuring students intended use of time may be an important consideration in the remedial screening process, as community college students in particular are more likely to work while attending college (Horn et al., 2006).
Motivation is another well-studied noncognitive construct that is predictive of college success (Pintrich & Shunk, 2002; Robbins et al., 2006). Theories of motivation can explain an individuals choices, effort, and persistence in various tasks (Covington, 2000; Pintrich, 2003). For example, the concept of expectancy value within motivation research suggests that individuals make certain decisions or enact certain behaviors because they are motivated by the expected results of those behaviors (Eccles & Wigfield, 2002). Relatedly, motivation may stem from utility or task value, which refers to how and whether individuals perceive tasks as having positive value or utility because they facilitate important future goals (Husman & Lens, 1999). These values can therefore directly influence performance, persistence, and task choice. A students motivation, as understood through these values, may encourage persistence in the face of challenging or boring academic learning contexts and therefore be predictive of success in those contexts (Miller & Brickman, 2004).
Ones sense of social support may also influence college outcomes (Noonan et al., 2005). This may be related to the concept of mattering, defined as the feeling one is personally important to someone else (Cooper, 1997; Gossett, Cuyjex, & Cockriel, 1996; Marshall, 2001; Rosenberg & McCullough, 1981; Schlossberg, 1989; Tovar, Simon, & Lee, 2009). In studies of college students, a stronger sense of mattering is linked to pro-academic behaviors and affects (Dixon & Kurpius, 2008; Dixon Rayle & Chung, 2007; France & Finney, 2010). In another study, Dennis, Phinney, and Chuateco (2005) assessed the extent to which motivation to attend college and the availability of social support from family and peers influenced academic success in ethnic minority college students. They included survey items such as how supportive family and peers were of students college aspirations and students beliefs about attending college. The researchers found that personal interest, desire to attain a rewarding career, and intellectual curiosity were all related to successful adjustment in college. Finally, Sedlacek (2004) demonstrated that noncognitive measures of adjustment, motivation, and leadership are predictors of postsecondary success, particularly for underrepresented minority students.
USE OF NONCOGNITIVE MEASURES
While studies find positive associations between noncognitive measures and college outcomes, whether it is beneficial to use noncognitive measures or indicators of them to inform selection processes remains an outstanding question. Some evidence from four-year institutions has suggested that incorporating noncognitive measures into college admissions can be favorable (Noonan et al., 2005; Sedlacek, 2004; Sternberg, Gabora, & Bonney, 2012). However, despite calls in the literature for the use of holistic assessments of student readiness for college coursework (Hughes & Scott-Clayton, 2011), noncognitive measures are rarely used to make course placement decisions in community colleges (Gerlaugh, Thompson, Boylan, & Davis, 2007). We found just one completed study examining placement using noncognitive measures in the community college setting.3 The Educational Testing Service (ETS) conducted a study to investigate the usefulness of SuccessNavigator, a commercial product that incorporates psychosocial/noncognitive measures, including personality, motivation, study skills, intrapersonal and interpersonal skills, and other factors beyond cognitive ability (Markle, Olivera-Aguilar, Jackson, Noeth, & Robbins, 2013). Examining a set of community colleges, Rikoon, Liebtag, Olivera-Aguilar, Robbins, and Jackson (2014) compared mathematics course passing rates between students placed in math courses using standard institutional practice (i.e., the COMPASS placement test) and those placed using the ETS SuccessNavigator mathematics course placement index in conjunction with test scores. They found no statistically significant differences in course passing rates between the two groups. Since students placed using the ETS instrument and test scores were just as successful as their peers in a higher level course, this suggests that the noncognitive measures may be useful for course placement. The goal of the present study is to complement this work by examining noncognitive measure use in another community college setting. We also analyze survey data rather than data gathered from a proprietary instrument, which may be a more viable option for resource-strapped colleges. We frame the study using expanding conceptions of college readiness and modern approaches to validation, which we discuss next.
The policy interest in using both high school background measures and noncognitive measures is in accordance with expanding notions of college-readiness. Historically, college-readiness has been measured by students academic ability and cognitive skills but it has also expanded to include noncognitive attributes and college knowledge that are thought to be essential for success in college (Almeida, 2015; Duncheon, 2015; Roderick, Nagaoka, & Coca, 2009; Sedlacek, 2004). The underlying logic is that what determines whether students will be successful in college is broader than cognitive skill or academic background. The research described in the previous section demonstrates that noncognitive attributes play an important role in explaining persistence and success in college. In line with expanding notions of college readiness, we test whether these broader concepts of college readiness can be imported into assessment and placement processes in community colleges. Could selection into developmental courses be improved by expanding concepts of college-readiness beyond academic and cognitive attributes of students?
Modern validation theory makes it possible to identify and evaluate the usefulness of noncognitive measures of college readiness. In modern validation theory, a validation argument considers the interpretation, purposes, and uses of a measure in addition to its predictive properties; it emphasizes the examination of outcomes that result from the uses of the measure (Kane, 2006). While past conceptions of validation relied mainly on determining the correlation between a measure such as tests scores and college outcomes, this theory suggests that the intended use and purpose should guide how the analysis is conducted to determine predictive properties (Kane, 2001).
Therefore, the validity of a measure such as a placement measure is based on the actual decisions or proposed decisions made using the measure, and not simply the correlation between a measure such as test scores and subsequent outcomes. In the current context, the measures used to make course placement decisions in developmental math would therefore be evaluated in terms of the relevant student outcomesplacement and success in the highest level course possible (Kane, 2006), and the frequency with which these accurate placements occur (Sawyer, 1996, 2007; Scott-Clayton, 2012). If a measure places students into higher level courses and they are successful in those courses, then using those measures improved placement accuracy. If the measures led to placements resulting in worse outcomes, then using those measures did not improve accuracy.
In order to validate noncognitive measures in the context of community colleges, there must be a context where measures of noncognitive attributes, or in this case indicators of them, are actually used to make placement decisions. Our study takes advantage of the fact that some colleges factor in what we are argue are indicators of noncognitive attributes into their placement process. This enables us to assess the validity of these placement measures (Kane, 2006; Sawyer, 1996, 2007). In a related analysis, we also predict the potential usefulness of these indicators for identifying and avoiding placement error in colleges that mainly rely on placement tests.
SETTING & DATA
The setting for the study is a Large Urban Community College District (LUCCD) in California that enrolls over 100,000 students each semester.4 Being open-access institutions, the nine colleges in the district serve a widely diverse body of students, with more than a quarter of students over 35 years of age, and over 40% indicating that their native language is not English. Close to 90% of students report having completed a high school level education.5 This student population is different from the national community college student population since about two-thirds of all students in these California colleges identify as African-American or Latina/o. In contrast, the majority of students who enter a community college in the U.S. are White, and just over one-third identify as African-American or Latina/o (NCES, 2014).
Each college has considerable autonomy over choice and use of placement tests. The colleges are also required by California law to utilize some combination of multiple measures to inform placement decisions (Perry et al., 2010). In the LUCCD, colleges chose to consider items from self-reported background questionnaires as multiple measures. Table 1 shows the placement tests and additional measures used to make course assignments in developmental math in each college. Table 2 shows the types of self-reported background information from the Educational Background Questionnaire (EBQ) also collected at the time of assessment.
The data for the study consist of all student-level assessment and enrollment records for students assessed between 2005 and 2012, tracked through fall 2013. We focus on the sample of students who took math placement assessments, had not already earned a college degree, were not concurrently enrolled in high school, and were under the age of 65. Since we are interested in noncognitive measures, we focus on six colleges that collect indicators of this information: B, C, D, F, G, and J.6 The data available enable us to examine important short- and long-term student outcomes such as passing the math course in which they are placed and earning 30 degree-applicable units. Table 3 presents a demographic profile of each college included in our analyses. The table shows that each college is unique in its student composition. However, the overall pattern is that Latinas/os and African Americans are the two largest racial groups served by all the colleges in the study. In addition, the table delineates how students were placed in each level of the developmental math sequence by college. While placement distribution varies among different colleges, the majority of students were placed in the three lowest levelseither elementary algebra, pre-algebra, and arithmeticand few students placed into intermediate algebra or above.
WHICH STUDENTS MAY HAVE BEEN PLACED IN ERROR?
We examined the possibility of using noncognitive measures for placement in developmental math by capitalizing on indicators of this information collected via the EBQ in each college. Our first methodological approach enabled us to assess whether utilizing indicators of noncognitive attributes can help to identify and thus avoid placement errors. Based on our analysis of each colleges EBQs, three areas of noncognitive attributes were common across sets of colleges: motivation, college plans (e.g., use of time), and social supports (see Table 2).
The analytical approach we used to understand how noncognitive measures can identify placement errors follows the sequence described in Scott-Clayton et al. (2014). The procedure estimates the overall proportions of students placed successfully and students placed in error for each level of math in a developmental sequence. Students placed successfully are those who were either placed into a math class level they were predicted to pass or those who were placed into one level below a math class they were predicted to fail. Students placed in error are those who were either: overplaced, placed into a math class that they were predicted to fail, or underplaced, placed into one level below a math class that they were predicted to pass.
We identified placement error for every level of the developmental math sequence, from arithmetic to college-level math. We also performed the procedure using different combinations of cognitive and noncognitive indicators (described further below). In the case of college-level math (CM), the respective logit models are:
Here is the test score, and and are proposed additional measures used for placement, and is a vector of student level demographic characteristics, including age, race, gender, language, and residence status, added as controls for factors that may be associated with college success. The obtained coefficients are extrapolated to students placed in the course below (e.g., intermediate algebra is one level below college math) to predict each students probabilities of success and failure in intermediate algebra. We used the probabilities to identify students placed successfully and students placed in error at each level. Specifically, we identified severe placement errors, defined by two criteria: (1) students predicted to fail the upper level course they were placed into, or (2) students predicted to pass the upper level course with a B or better, but were placed into a course one level below.7 We estimated the proportion of severe placement errors at each level of math in the developmental sequence for each college.
Since we are interested in comparing various placement scenarios, we calculated the percent of severe placement errors using different combinations of measures:
With placement test scores alone, ,8
with additional academic background measures (e.g., HS GPA or prior math experience), ,
with noncognitive indicators obtained from colleges EBQs (e.g., motivation or college plans), .
This analysis enabled us to determine whether high school background and noncognitive measuresa more holistic profilecould improve upon placement results based on cognitive measures (placement tests) alone.9 The calculated rate of severe placement errors for each set of alternative criteria is the proportion of students that can be considered as having been placed in error by status quo practices, and therefore, the set of students for whom placement errors could be avoided. The rate of severe placement errors is not a measure of the placement accuracy of each set of measures. Instead, it estimates the amount of error in existing placement policy.
DO NONCOGNITIVE MEASURES ACTUALLY IMPROVE PLACEMENT ACCURACY?
We then took advantage of the placement decision rules in two colleges (Colleges F and J) where proxy indicators of noncognitive constructs were actually factored into placement algorithms. College F awarded up to two additional points to students based on their college plans (units enrolled and expected employment), how important a college education was to them, and how long they have been out of school.10 College F used the COMPASS, with 100 test points possible. College J awarded one additional point to students who indicated that math was important, and four other possible points for cognitive measures (high school math background and receipt of a diploma or GED). College J used the ACCUPLACER, with 120 points possible. Since we could identify student responses on each colleges EBQ along with raw placement test scores, we could therefore examine the success outcomes of students whose final placements were due to the additional points earned from these noncognitive indicators. Although the additional points may seem nominal relative to the placement tests, they did provide a benefit to some students. About 1.8% and 26.4%11 of students in Colleges F and J, respectively, were placed in a higher level course based on their combined score.
Modern validation theory would suggest that measures used for placement in developmental courses are valid if they increase access to higher-level math courses without compromising the likelihood of success in those courses (Kane, 2006; Sawyer, 1996). Therefore, we compared the outcomes of students whose resulting placements were boosted up due to noncognitive indicators with the outcomes of students in the same level of math whose placements were the result of their placement test scores. Specified as a linear regression model, the main variable of interest () is a dummy indicator that equals one for students whose responses to the noncognitive oriented questions on the EBQ resulted in a multiple measure boost to a higher level course. We tested the relationship between earning this multiple measure boost and the outcome of interest (), passing the course in which the student was placed. We also examined the relationship between receiving a placement boost and the outcome of earning 30 degree-applicable credits, which are half the units required for an associates degree. The linear probability model is:
captures the difference in average outcomes between students who were assigned to the course due to additional points from noncognitive indicators and students who had higher raw test scores. We include , the number of multiple measure points,12 the raw placement test score (normalized), and all covariates as before. We also estimated models controlling for math level and cohort. For each model we compared boosted students to other students just above the cutoff, as well as to all students in the same math level.
Unlike College F, which used only noncognitive indicators to augment its placement algorithm, College J awarded points for both academic and noncognitive indicators. Thus, for College J we could differentiate between students whose boost was solely from academic measures from those who obtained points for both academic and noncognitive indicators.13 We identified the differences between these students by including an interaction term with a dummy variable . The variable is an indicator of the students motivation and equals one when students marked that math was very important to their goals. It equals zero for students who responded not important or somewhat important. The model with this interaction is:
The interaction term enabled us to determine whether there were differential effects for those who earned points for both cognitive and noncognitive indicators relative to those whose boost was from cognitive measures alone.
POSSIBILITIES FOR NONCOGNITIVE MEASURES
We first investigated possibilities for the use of noncognitive indicators either in addition to or in place of common placement tests. We did so by estimating the percent of severe placement errors at each placement cutoff within each college. As described above, the procedure involves using different sets of alternative placement criteria (e.g., placement tests, high school background measures, and noncognitive indicators) to model successful placement of students, which we defined as: students predicted to pass the math course of interest with a B or better and placed into that math course, and students predicted to fail the math course of interest and placed into one level below that math course. Severe placement errors are those where students predicted to fail the math course of interest were placed into that math course, and students predicted to pass the math course of interest with a B or better were placed into one level below that math course.14
The full results are presented in Tables 4 and 5. Table 4 shows the results of the analysis when the entire sample of students within each pair of math levels is included, e.g., all students in intermediate algebra (IA) or elementary algebra (EA), and Table 5 shows the results for students within a five-point bandwidth around the cutoff. The first column of each table shows the estimated percent of severe placement errors under the status quo practice (which is essentially placement test score alone). There is a range of estimates across colleges, from nearly zero error to over 30% of students identified as placed in error. In comparing error rates across placement scenarios, a higher rate of errors does not suggest the scenarios placement criteria would produce more placement errors. Rather, a higher estimate suggests the alternative placement criteria identify a larger fraction of placement errors in existing practice that can possibly be avoided. We also decomposed the total estimated placement error into its two respective partsthe percent of students considered as under-placed and the percent considered as overplaced. This also reveals variation across levels and colleges. This is expected given variation in the level of the placement cutoff set at each college and institutional differences between colleges.
Notes: Intermediate algebra (IA; elementary algebra (EA); pre-algebra (PA); arithmetic (AR); high school background (HSB); noncognitive (NC).
Notes: Intermediate algebra (IA; elementary algebra (EA); pre-algebra (PA); arithmetic (AR); high school background (HSB); noncognitive (NC).
Nevertheless, a common pattern emerges. Comparing Columns 14 with Columns 57 reveals that the proportion of placement error identified in existing practice rises when high school background and noncognitive indicators are utilized to model student success. This is the finding in nearly all colleges and levels, and it suggests that more students can be considered as having been placed in error by existing practice when these alternative criteria are used to model the likelihood that students would fail or pass the higher level course, compared to when test scores are incorporated in the model. Patterns are consistent across the full and narrow bandwidth specifications, though the severe placement error estimates are larger when all students are considered. Moving forward, we focus on the narrower bandwidth of five points right around each placement cutoff to better understand sorting and accuracy around each cutoff.
Given the similarity in patterns across colleges and levels, we also calculated the average severe errors for each column to better summarize these trends. Figures 1 and 2 visually present these average results. The first bar of Figure 1 shows the severe placement error of existing placement practices, which typically consider a students placement test score and additional points from multiple measures (academic) when making placement decisions. On average, the estimate is that 8.7% of students within a five-point bandwidth around the cutoff can be considered as severe placement errors (11% for the full bandwidth). Figure 2 decomposes errors into the proportion of under- and overplacement, and these results indicate that more students appear to be underplaced when high school background and noncognitive indicators are used to examine placement errors.15 Students who are identified as under-placed presumably could have passed the higher level course.
Figure 1. Severe Placement Errors Identified in Status Quo Practice (Bandwidth=5 points Around Placement Cutoff)
Note: *Placement Test is representative of status quo placement practice, which in some cases is placement test plus additional multiple measures (see Table 1).
Figure 2. Type of Severe Placement Error Identified (Bandwidth = 5 Points Around Placement Cutoff)
Note: *Placement Test is representative of status quo placement practice, which in some cases is placement test plus additional multiple measures (see Table 1).
The primary goal of the exercise is to understand whether or not alternative measures identify more or less placement error than the status quo practice. Overall, the results indicate that substantially more students can be considered as having been placed in error when high school background and noncognitive indicators are used to identify placement errors, compared to when placement test scores are used. For example, if the estimating equation for success/failure in the upper level course consists of high school background measures (e.g., HS GPA, prior math, etc.) or noncognitive measures (e.g., motivation), then it appears that 22%25% percent of students in the full bandwidth and 17%21% percent in the five-point bandwidth have been placed in error by current practices. Since these students that can be considered as errors are those who could have passed the upper level course if placed there or would have fared better in the lower level course, then it follows that these alternative schemes may offer an improvement over current practices.
The answer to whether noncognitive indicators of motivation, social support, and college plans might improve placement accuracy is less clear. We see that indeed the use of HSB and noncognitive indicators identified the most amount of error, suggesting that these measures may be suitable alternatives to current practices. Comparing the HSB-only results with HSB and noncognitive indicators, we see that noncognitive indicators may make marginal improvements, but our supplementary analysis also revealed that the HSB measures were generally positively correlated with the noncognitive indicators.
NONCOGNITIVE MEASURES IN USE
We therefore complement this analysis with evidence from two colleges in the district, Colleges F and J, which actually do factor in indicators of noncognitive constructs into their placement algorithms. These colleges award supplemental points which are added (or in some cases subtracted) from the raw placement test scores to determine the final score used to determine math placement. To this end, students placement results may be directly related to some noncognitive attributes. In fact, the policy of using these noncognitive indicators increased access to higher level courses for students who otherwise would have been in a lower level math course based on placement test scores alone. As mentioned above, 1.8% and 26.4% of students in Colleges F and J, respectively, were placed in a higher level course based on their final scores after the multiple measures were considered.
The question of interest is whether students who received a multiple measure boost generally performed differently from their higher scoring peers in terms of passing the placed math course and completing 30 degree-applicable units.16 Tables 6 and 8 present the results from this analysis. Based on our estimation of Equation 3 we found no evidence of differences for students around the placement cutoffs for College F (Table 6) or J (Table 8). We show that these are robust to model specifications that include placement level dummies, cohort fixed effects, and student background variables. The null results in Colleges F and J suggest that the use of noncognitive indicators increased access to higher level courses without compromising the likelihood of success in those courses.
Since the multiple measure boost in these two colleges consisted of additional points drawn from a number of survey questions (e.g., college enrollment and employment plans, importance of math, and time since last enrollment in College F; highest math, HS diploma/GED, and importance of math in College J) we also investigated heterogeneity by boost type. That is, we attempted to disentangle those multiple measure boosts that were largely due to points from academic measures from those that included points from the noncognitive indicators. This was only possible in College J, where there was enough variation in answer choice and a large enough sample size to identify a group of students who, although they earned a multiple measure boost into a higher level course, indicated that they consider math to not be important or only somewhat important personally towards their educational goals (see Table 7, N = 547).
We therefore included a dummy variable that equaled one for each student that said math was important, and interacted this with the boost variable as shown in Equation 4. The results in Table 8 show no differential relationship between boost and student outcomes for the interaction term. This suggests that boosted students who indicated they did believe that math was important, along with those who did not indicate that math was important, did not exhibit differences in their outcomes. The two groups had statistically equivalent probabilities of passing the course. While this can be interpreted as evidence of the irrelevance of the motivation measure, we remind the reader that this result can still be considered as an improvement in placement accuracy. The students who received this boost due to a noncognitive indicator were able to access a higher level course and their likelihood of success in the course was the same as their peers.
The study contributes to concerns about selection and sorting processes at the start of community college, which have been of increasing policy interest in recent years. A number of states (e.g., Colorado, Florida, Texas) have begun to consider multiple measures, including noncognitive measures, for developmental student advising, assessment, and placement practices (Bracco et al., 2014). However, there has been scant evaluation of these practices to inform policy, and our study attempts to fill this gap.
We drew upon expanding conceptions of college readiness to frame our investigation of possibilities for a more holistic approach to placement that includes the use of noncognitive indicators in contexts where they are currently not used. We found that using high school background and noncognitive indicators may help to identify and thus reduce and avoid some placement errors associated with test-based placement. This is key descriptive evidence indicating that alternative placement criteria may offer a means of improving upon status quo practices, and further research should continue to test this hypothesis.
We were able to validate the use of noncognitive indicators in two colleges. The results from Colleges F and J, which essentially incorporate noncognitive characteristics in the identification of low-scoring students who could be moved to a higher level course, reveal that students placed under this approach performed no differently from their higher scoring peers. Interestingly, they also performed no differently from their peers whose boost was based primarily on academic background measures. The study therefore contributes to the burgeoning literature on using multiple measures such as high school transcript information to optimally place students (Fong & Melguizo, 2016; Ngo & Kwon, 2015; Scott-Clayton et al., 2014), but adds the important component of evaluating the use of indicators of noncognitive attributes. Similar to the aforementioned studies of academic measures, we find that noncognitive indicators may make marginal improvements in placement accuracy over high school background factors alone or test scores alone.
A limitation of these analyses is that we were unable to observe instructor characteristics, which may be important determinants of developmental math student outcomes (Chingos, 2016). We also relied on instructors grades as a metric of success. If math instructors adjust their instruction or grading in order to meet the needs of students, this would bias the estimate of the relationship between academic background measures, such as placement test scores, and course success. We reasoned that because math is a fairly hierarchical subject, instructors grading practices may not vary as much as it may in other subjects (e.g., English). To mitigate this bias, we also chose to focus on the B or better criterion, which is a more cautious approach to identifying error than using a C or better criterion (Scott-Clayton et al., 2014). There likely is less variation in grading practices related to awarding an A or B grade than a C grade.
A second limitation is that results may be more emblematic of where placement cutoffs are set rather than the accuracy of measures themselves. The boost analysis, for example, may not be reflecting the validity of the additional measures used, but rather, the measurement error inherent in placement testing. Test scores are noisy measures and students a few points apart may be very similar. To address this concern, we ran models where we compared students to similar-scoring peers right above the cutoff, as well as to all students in a given level. We found the results to be consistent across model specification for College F, but we found differences between the around and entire groups in College J (see Table 8). However, it is likely that the significant negative coefficients for entire in College J may be related to the fact that there is as much as a 30-point range of placement scores that result in assignment to the same courses. Therefore students in the entire regressions in College J may be substantially different along unobservable characteristics.
While the results from both sets of analyses point towards a potential advantage to using high school background measures and, to a lesser extent, noncognitive measures in the assessment and placement process, a study in which students were experimentally placed using these measures would provide stronger evidence on the usefulness of these measures. Indeed, the results from Colleges F and J provide slightly more convincing evidence that noncognitive indicators improve placement accuracy, but again, these students are placed using a combination of placement test scores, high school background, and noncognitive indicators. Further research should examine student success under conditions where noncognitive indicators are the sole placement measure in order to determine the actual usefulness of these measures as an alternative to academic placement measures.
Further, while we chose colleges existing items from questionnaires and classified some as cognitive and others as noncognitive, we are unsure of their psychometric properties and validity in the traditional sense. It stands to reason that different cognitive and noncognitive measures would yield different results than what we obtained. We encourage more work in investigating differences in measures that can be potentially used for placement. To this end we encourage a thorough exploration of the burgeoning literature on noncognitive constructs and their scales in predicting college outcomes for their applicability in placement policy at the community college level.
Finally, we also note that these may be noisy measures of high school background and noncognitive attributes since they are gathered from a self-reported student questionnaire administered at the time of placement testing. A placement policy that incorporated such additional self-reported information could be susceptible to misinterpretation, reference bias, faking, or gaming (Duckworth & Yeager, 2015). Colleges therefore need to take caution and consider evaluation of the measures and the accessibility of placement policy information. Incorporating high school transcript information to automate these decisions may be more efficient and accurate, but this would necessitate data-sharing agreements between K12 and community college districts.
Ultimately, these findings concerning noncognitive indicators are related to the fact that psychosocial attributes such as motivation, and nonacademic characteristics such as students use of time and their degree of social support, are useful for explaining why some students do well in school while others fall behind (Pintrich, 2003). Our findings suggest that gauging these various measures at the start of each students college career may assist colleges as they sort students into community college coursework. Although evaluation and selection on noncognitive skills may unfairly focus attention on student characteristics rather than the role of institutions, using noncognitive measures may nevertheless promote equity. Holistic placement practices that increase access to upper level courses without compromising the likelihood of student success in those courses provide opportunities for community college students to progress faster and further in their college careers.
We recognize that there is debate over terminology, with some scholars preferring terms such as character skills, social and emotional competencies, dispositions, personality, temperament, 21st century skills, and personal qualities (Duckworth & Yeager, 2015). We choose the term noncognitive because it provides a contrast to the academic/cognitive measures discussed and because these terms refer to the same conceptual space, (Duckworth & Yeager, 2015, pg. 239).
2. The ACT, Inc. has recently decided to phase out the use of the COMPASS (Fain, 2015).
3. The Multiple Measures Assessment Project reported collecting noncognitive data from one college for analysis (Bahr et al., 2014).
4. About one quarter of all community college students in the United States are enrolled in California community colleges, many of which are in located in urban centers (Foundation for Community Colleges, n.d.).
5. Source: California Community College Chancellors Office DataMart (http://datamart.cccco.edu/datamart.aspx)
7. Students are considered likely to pass if their predicted probability of success in the upper level course is 50% or greater, and considered likely to fail if the predicted probability of failure is 50% or greater.
8. We use the adjusted score, which includes multiple measure points for each college.
9. We also run pooled college models by standardizing student test scores. For these pooled analyses, we identify items on colleges EBQs that measure common constructs, such as use of time, motivation, and social support.
10. College F subtracted points for high hours of schooling and work (use of time), low importance of education (motivation), and returning students who had been out of education for more than 5 years.
11. This is 221 out of 12,224 students in College F and 2,054 students out of 7,782 students in College J.
12. BOOST and MMPOINTS are correlated but this should not be a multicollinearity issue. We ran the models without the multiple measure points and did not observe significant differences in the magnitude, direction, or statistical significance of the boost variable.
13. There were 157 cases of boosts solely determined by noncognitive measures in College J and 547 with cognitive measures alone.
14. We also conducted the same analyses with a Pass at all outcome. These results are available from the authors upon request.
15. We caution against interpreting the results as indicators of accuracy of the current system. This is because accuracy is related to the predictive validity of the placement instrument and we do not perform an analysis that measures predictive validity per se. Other methods may be used to determine whether placement tools are accurate and, relatedly, whether cutoffs are set correctly (Melguizo et al., 2016).
16. The regression results for the longer term outcomes are available in the Appendix.
Armstrong, W. B. (2000). The association among student success in courses, placement test scores, student background data, and instructor grading practices. Community College Journal of Research and Practice, 24(8), 681695.
Almeida, D. (2015). The roots of college readiness. In W. G. Tierney & J. Duncheon (Eds.), The problem of college readiness (pp. 344). SUNY Press: Albany.
Bahr, P. R., Hayward, C., Hetts, J., Lamoree, D., Newell, M., Pellegrin, N., & Willett, T. (2014). Multiple Measures for Assessment and Placement (White Paper). Retrieved from http://www.rpgroup.org/system/files/MMAP_WhitePaper_Final_September2014.pdf
Bailey, T., Jeong, D. W., & Cho, S.-W. (2010). Referral, enrollment, and completion in developmental education sequences in community colleges. Economics of Education Review, 29(2), 255270. http://doi.org/10.1016/j.econedurev.2009.09.002
Bracco, K. R., Dadgar, M., Austin, K., Klarin, B., Broek, M., Finkelstein, N. . . . & Bugler, D. (2014). Exploring the use of multiple measures for placement into college level courses: Seeking alternatives or improvements to the use of a single standardized test. San Francisco, CA: WestEd.
Burdman, P. (2012). Where to begin? The evolving role of placement exams for students starting college. Boston, MA: Jobs for the Future.
Chingos, M. M. (2016). Instructional quality and student learning in higher education: Evidence from developmental algebra courses. The Journal of Higher Education, 87(1), 84114.
Conley, D. (2010). Replacing remediation with readiness. An NCPR Working Paper. Retrieved from http://files.eric.ed.gov/fulltext/ED533868.pdf
Cooper, J. (1997). Marginality, mattering, and the African American student: Creating an inclusive college environment. College Student Affairs Journal, 16(2), 1520.
Covington, M. V. (2000). Goal theory, motivation, and school achievement: An integrative view. Annual Review of Psychology, 51, 171200.
Dennis, J. M., Phinney, J. S., & Chuateco, L. I. (2005). The role of motivation, parental support, and peer support in the academic success of ethnic minority first generation college students. Journal of College Student Development, 46(3), 223236.
Dixon Rayle, A., & Chung, K.-Y. (2007). Revisiting first-year college students mattering: Social support, academic stress, and the mattering experience. Journal of College Student Retention: Research, Theory and Practice, 9(1), 2137.
Dixon, S. K., & Kurpius, S. E. R. (2008). Depression and college stress among university undergraduates: Do mattering and self-esteem make a difference? Journal of College Student Development, 49(5), 412424.
Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92(6), 10871101.
Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 237251.
Duncheon, J. (2015). The problem of college readiness. In W. G. Tierney & J. Duncheon (Eds.), The problem of college readiness (pp. 344). Albany: SUNY Press.
Ehrenberg, R. G., & Sherman, D. R. (1987). Employment while in college, academic achievement, and postcollege outcomes: A summary of results. Journal of Human Resources, 22(1), 1.
Eccles, J., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, 109132.
Fields, R., & Parsad, B. (2012). Tests and cut scores used for student placement in postsecondary education: Fall 2011. Washington, DC: National Assessment Governing Board.
Fong, K., & Melguizo, T. (2015). Utilizing additional measures to buffer against students lack of math confidence and improve placement accuracy in developmental math. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
France, M. K., & Finney, S. J. (2010). Conceptualization and utility of university mattering: A construct validity study. Measurement and Evaluation in Counseling and Development, 43(1), 4865.
Gossett, B. J., Cuyjex, M. J., & Cockriel, I. (1996). African Americans and Non-African Americans sense of mattering and marginality at public, predominantly white institutions. Equity and Excellence in Education, 29(3), 3742.
Gerlaugh, K., Thompson, L., Boylan, H., & Davis, H. (2007). National study of developmental education II: Baseline data for community colleges. Research in Developmental Education, 20(4), 1-4.
Horn, L., Nevill, S., & Griffith, J. (2006). Profile of undergraduates in US postsecondary education institutions, 2003-04: With a special analysis of community college students. Statistical Analysis Report. NCES 2006-184.National Center for Education Statistics.
Hughes, K. L., & Scott-Clayton, J. (2011). Assessing developmental assessment in community colleges. Community College Review, 39(4), 327351.
Husman, J., & Lens, W. (1999). The role of the future in student motivation. Educational Psychologist, 34, 113125.
Jenkins, D., Jaggars, S. S., & Roksa, J. (2009). Promoting gatekeeper course success among community college students needing remediation: Findings and recommendations from a Virginia Study. New York, NY: Community College Research Center. Retrieved from http://eric.ed.gov/?id=ED507824
Kane, M. T. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38(4), 319342.
Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational Measurement (4th ed., pp. 1764). Westport, CT: ACE/Praeger Publishers.
Markle, R., Olivera-Aguilar, M., Jackson, R., Noeth, R., & Robbins, S. (2013). Examining evidence of reliability, validity and fairness for the SuccessNavigator assessment (Research Report No. RR-13-12). Princeton, NJ: Educational Testing Service. http://dx.doi.org/ 10.1002/j.2333-8504.2013.tb02319.x
Marshall, S. K. (2001). Do I matter? Construct validation of adolescents perceived mattering to parents and friends. Journal of Adolescence, 24(4), 473490.
Marwick, J. D. (2004). Charting a path to success: The association between institutional placement policies and the academic success of Latino students. Community College Journal of Research and Practice, 28(3), 263280.
Mattern, K. D., & Packman, S. (2009). Predictive validity of ACCUPLACER scores for course placement: A meta-analysis (Research Report No. 2009-2). New York, NY: College Board.
Melguizo, T., Bos, J., Ngo, F., Mills, N., & Prather, G. (2016). Using a regression discontinuity design to estimate the impact of placement decisions in developmental math. Research in Higher Education, 57(2), 123151.
Michaels, J. W., & Miethe, T. D. (1989). Academic effort and college grades. Social Forces, 68(1), 309.
Miller, R. B., & Brickman, S. J. A model of future-oriented motivation and self-regulation. Educational Psychology Review, 16, 933.
National Center for Public Policy and Higher Education & Southern Regional Education Board (NCPPHE & SREB). (2010). Beyond the rhetoric: Improving college readiness through coherent state policy. Atlanta, GA: NCPPHE. Retrieved from http://publications.sreb.org/2010/Beyond%20the%20Rhetoric.pdf
Ngo, F., & Kwon, W. (2015). Using multiple measures to make math placement decisions: Implications for access and success in community colleges. Research in Higher Education, 56(5), 442470.
Ngo, F., & Melguizo, T. (2016). How can placement policy improve math remediation outcomes? Evidence from community college experimentation. Educational Evaluation and Policy Analysis, 38(1), 171196.
Noble, J. P., & Sawyer, R. L. (2004). Is high school GPA better than admission test scores for predicting academic success in college? College and University Journal, 79(4), 1722.
Noonan, B. M., Sedlacek, W. E., & Veerasamy, S. (2005). Employing noncognitive variables in admitting and advising community college students. Community College Journal of Research and Practice, 29(6), 463469.
Pajares, F., & Rayner, S. (2001). Self-beliefs and school success: Self-efficacy, self-concept, and school achievement. Perception, 239266.
Pascarella, E. T., Edison, M. I., Nora, A., Hagedorn, L. S., & Terenzini, P. T. (1998). Does work inhibit cognitive development during College? Educational Evaluation and Policy Analysis, 20(2), 75.
Perry, M., Bahr, P. M., Rosin, M., & Woodward, K. M. (2010). Course-taking patterns, policies, and practices in developmental education in the California Community Colleges. Mountain View, CA: EdSource. Retrieved from https://edsource.org/wp-content/publications/FULL-CC-DevelopmentalCoursetaking.pdf
Pintrich, P. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology, 95(4), 667686.
Pintrich, P., & Schunk, D. (2002). Motivation in education. Englewood Cliffs, NJ: Merrill.
Porcea, S. F., Allen, J, Robbins, S., & Phelps, R. P. (2010). Predictors of long-term enrollment and degree outcomes for community college students: Integrating academic, psychosocial, socio-demographic, and situational factors. The Journal of Higher Education, 81(6), 750778.
Rau, W., & Durand, A. (2000). The academic ethic and college grades: Does hard work help students to make the grade? Sociology of Education, 73(1), 19.
Rikoon, S., Liebtag, T., Olivera-Aguilar, M., Robbins, S., & Jackson, T. (2014). A pilot study of holistic assessment and course placement in community college: Findings and recommendations. Retrieved from https://www.ets.org/Media/Research/pdf/RM-14-10.pdf
Robbins, S., Allen, J., Casillas, A., Peterson, C. H., & Le, H. (2006). Unraveling the differential effects of motivational and skills, social, and self-management measures from traditional predictors of college outcomes. Journal of Educational Psychology, 98(3), 598616.
Roderick, M., Nagaoka, J., & Coca, V. (2009). College readiness for all: The challenge for urban high schools. The Future of Children, 19(1), 185210.
Rosenberg, M., & McCullough, B. C. (1981). Mattering: Inferred significance and mental health among adolescents. Research in Community and Mental Health, 2, 163182.
Sawyer, R. (1996). Decision theory models for validating course placement tests. Journal of Educational Measurement, 33(3), 271290.
Sawyer, R. (2007). Indicators of usefulness of test scores. Applied Measurement in Education, 20(3), 255271.
Sedlacek, W. E. (2004). Beyond the big test: Noncognitive assessment in higher education. San Francisco, CA: Jossey-Bass.
Schlossberg, N. K. (1989). Marginality and mattering: Key issues in building community. New Directions for Student Services, 1989(48), 515.
Scott-Clayton, J. E. (2012). Do high-stakes placement exams predict college success? (Working Paper No. 41). New York, NY: Community College Research Center. Retrieved from http://academiccommons.columbia.edu/catalog/ac:146482
Scott-Clayton, J. E., Crosta, P. M., & Belfield, C. R. (2014). Improving the targeting of treatment: Evidence From college remediation. Educational Evaluation and Policy Analysis, 36(3), 371393. http://doi.org/10.3102/0162373713517935
Smith, A. (2016, May 26). Determining a students place. Inside Higher Ed. Retrieved from http://www.insidehighered.com/news/2016/05/26/growing-number-community-colleges-use-multiple-measures-place-students
Stinebrickner, R., & Stinebrickner, T. R. (2003). Working during school and academic performance. Journal of Labor Economics, 21(2), 473491.
Stinebrickner, R., & Stinebrickner, T. R. (2004). Time-use and college outcomes. Journal of Econometrics, 121(12), 243269.
Sternberg, R. J., Gabora, L., & Bonney, C. R. (2012). Introduction to the special issue on college and university admissions. Educational Psychologist, 47(1), 14.
Tovar, E., Simon, M. A., & Lee, H. B. (2009). Development and validation of the college mattering inventory with diverse urban college students. Measurement and Evaluation in Counseling and Development, 42(3), 154178.