College Completion, the Texas Way: An Examination of the Development of College Completion Policy in a Distinctive Political Culture
by Denisa Gándara & James C. Hearn - 2019
Background: College-completion policies dominate state higher education policy agendas. Yet we know little about how policy actors make decisions—and what sources of evidence they use—within this policy domain.
Focus of Study: This study explores the use of evidence in college-completion policymaking in depth, focusing on Texas. In addition to exploring policymakers’ use of different types of information, this study examines the role played by intermediaries.
Research Design: We employed a qualitative case study design drawing on interviews with 32 policy actors engaged in college-completion policy in Texas. Our analysis consisted of both deductive coding (based on our a priori coding scheme) and inductive coding (based on emerging themes) to arrive at our four major findings.
Findings/Results: The analysis revealed four primary findings. The first theme suggests an insular culture of college-completion policymaking: Policymakers at various levels preferred Texas-based data and rejected the notion that external groups contributed to setting the college completion agenda in Texas. Second, business groups and a business ethos permeated college-completion policymaking in Texas. Third, research evidence was seldom employed in this policy process, partly because policymakers prefer concise and timely information. Finally, the study uncovered a new tactic for supplying research employed by certain intermediaries: punchy messaging, which was effective at garnering attention but also yielded unintended consequences.
Conclusions/Recommendations: Overwhelmingly, higher education policy actors tended to prefer Texas-based data. Respondents cited three major reasons for this preference: the high quality of the state higher education coordinating board’s data, Texas’s unique demographics, and the accessibility of statewide data. These findings reflect the mediating role that is played not only by state structural characteristics, but also by culture. Perceptions of Texas’s distinctive inward-looking nature permeated our interviews and set the stage for the role that intermediaries played in the state and the preferences for information. Intermediaries wishing to inform college-completion policy activity at the state level should consider the uniqueness of the state context in supplying information. For states that are more insular, like Texas, working through internal (in-state) intermediaries may be an effective strategy. In light of our findings of preferred types of information, those intending to influence policymaking should consider making information—especially research evidence—concise and easily accessible and establish relationships with policymakers and their staff members.
In 2010, the College Board published a report in collaboration with the National Conference of State Legislatures (NCSL) titled The College Completion Agenda: State Policy Guide. In the first lines of the report, the authors described the urgency of addressing college completion in the United States. They called attention to the crisis across the educational landscape and in particular, the fact that the United States is facing an alarming education deficit that threatens our global competitiveness and economic future (College Board, 2010, p. 3).
Consistent with the College Board and NCSLs call to action, numerous national organizations, including the National Governors Association (2010) and philanthropic groups (e.g., Lumina Foundation, 2013), have advocated for increased attention to college completion. Despite strong advocacy by individuals and organizations for college-completion policy activity in the states (Perna, Finney, & Callan, 2014), we know little about how state policymakers have approached the college-completion agenda. The present study explores one critical component of state-level college-completion pursuits: the extent to which higher education decision-makers use various types of information, including research evidence, to inform policy activity around the college-completion agenda. To date, policymakers utilization of information in higher education policymaking has gone virtually unexplored (Ness, 2010). This dearth in the higher education literature is especially problematic given the rapid diffusion of certain policies like performance-based funding for higher education (Snyder, 2015), which has occurred despite limited evidence of the policys effectiveness (e.g., Dougherty & Reddy, 2013; Rutherford & Rabovsky, 2014; Tandberg & Hillman, 2014). As such, it is critically important to examine the process leading to decisions regarding these policy instruments and the extent to which information, including research evidence, is used.
In parallel to exploring the degree to which policymakers use different types of information, this study examined the role played by intermediariesgroups that, among other roles, have the potential to serve as research brokers. Education policy scholars have begun to call attention to the distinct role intermediaries play in fostering the use of research and other forms of information in policymaking (DeBray, Scott, Lubienski, & Jabbar, 2014; Jabbar, LaLonde, DeBray, Scott, & Lubienski, 2014; Ness, 2010; Scott, Jabbar, LaLonde, DeBray, & Lubienski, 2015). Ness (2010) defines intermediary organizations as entities that serve a translating function between two principals with different values and perspectives (p. 37). While scholars have begun to document intermediaries influence over primary and secondary education policy in recent years (DeBray et al., 2014; Jabbar et al., 2014; Scott et al., 2015; Scott, Lubienski, DeBray, & Jabbar, 2014), their role in higher education, including in state-level college-completion activity, remains largely unexplored (Ness et al., 2015).
This study of information use and the role of intermediaries in higher education policymaking focuses on college-completion policy activity in Texas. As we discuss in greater detail in a subsequent section, we chose to examine this state because the Texas legislature (1) has robust resources, including professional staff, suggesting fertile ground for evidence use, and (2) was engaged in substantial college-completion policy activity prior to data collection, as revealed by preliminary document reviews. Together, these factors facilitated an in-depth analysis of the use of evidence in college-completion policymaking within a bounded context.
For this analysis, we drew upon a dataset that included interviews with 32 actors involved in higher education policymaking in Texas to examine the development of college-completion policy in the state. We focused on two primary research questions:
(1) How was informationespecially research evidenceused in discussions and decisions related to college completion in Texas?
(2) What unique role did intermediaries play in enhancing or restricting the use of information in college-completion policy discussions and decisions in Texas?
Recently, Ness and colleagues (2015) urged scholars to pay greater attention to the roles played by less obvious actors (p. 158)including researchers, business groups, and other intermediariesin higher education policymaking. Employing a case study methodology, this study seeks to respond to that call by exploring in-depth the extent to which intermediaries help shape college-completion policy in Texas. Further, the study provides valuable insights into the policymaking process by examining whether and how policy actors use various types of information in decision-making processes in higher education.
We anchored this analysis in the established literatures on policymakers research use and on intermediary organizations. Our analysis attended to the use of information more broadly, and not to research use exclusively. However, theoretical perspectives on policymakers use of research evidence were instructive since they provided insights on the types of information policymakers prefer and how they use this information. In addition to garnering insights from perspectives on research use and intermediaries separately, we drew on a developing body of research examining the distinct role of intermediaries in promoting research use in policymaking.
USE OF RESEARCH EVIDENCE
The early literature on research use in the policymaking process examined the extent to which policy actors (including elected officials and agency officials) relied upon research evidence in designing policy. Many early studies of research utilization suggested that researchers and policymakers belonged to two communities, and the two had separate priorities and cultures (Birnbaum, 2000; Caplan, 1979; Dunn, 1980; Snow, 1961). There is evidence of this phenomenon within higher education policymaking (Hearn, 1997). Education professionals working in state higher education agencies have also documented the disconnect between researchers agendas and state needs, noting that many research studies are either not relevant or timely enough to be used in policymaking (Conaway, Keesler, & Schwartz, 2015).
Studies of the two communities phenomenon (e.g., Caplan, 1979) have tended to emphasize the problem with the supply side of research, suggesting scholars can do more to enhance research uptake by policymakers (Ness, 2010). In contrast, studies that have explored the demand side of information use (e.g., Weiss, 1979) have focused on policymakers preferred information sources and the ways in which policymakers use evidence. For instance, studies have revealed legislators preference for information from constituents (Mooney, 1991; Webber, 1987), nonpartisan legislative research offices (Hird, 2006), and state government agencies (Hamann & Lane, 2004; Shakespeare, 2008). The present analysis focuses on both the demand for information by policymakers and the supply of information, including by intermediaries.
In addition to the supply and demand dynamics of information flow, this study garners insights from literature on how policymakers use information. Weisss (1979) landmark study was especially instructive. In that paper, she identified seven meanings of research use, which she extracted from the extant policy research on research utilization. The first is the knowledge-driven meaning of research use, through which basic research reveals an opportunity for applied research. Second, the interactive meaning involves using research as only one component of a complex decision-making process involving a variety of actors and considerations, including political calculations and pressure. Weisss third meaning, tactical use of research, captures policymakers use of the research process itself, rather than the content of the research, for practical purposes. For instance, policymakers can use the research process to delay decision-making, such as by requesting a research study to avoid taking immediate action on a policy proposal. In the fourth meaning, research as a process interacting with policy, not only does research inform policy, but policy also influences and motivates research.
The final three meanings have been the subject of most studies of research use. In the instrumental, or problem-solving, meaning, policymakers use research to inform the solution to a specific policy problem. This is the meaning we typically associate with research use and hope to observe in policymaking. Conversely, the political meaning refers to the use of research as ammunition to support predetermined positions (Weiss, 1979, p. 429). Discussing their findings of evidence use in district central offices, Coburn, Toure, and Yamashita (2009) captured the duality of our ideal for evidence use (i.e., the instrumental meaning) on one hand, and the reality we sometimes observe (i.e., the political meaning) on the other. They wrote, Research and evidence often become further tools in the very political processes that they are meant to circumvent (p. 1145). Finally, according to the conceptual or enlightenment meaning, concepts that have emanated from research infuse the policy process over time. Previous research has found that this is the most common use of evidence reported by policymakers (Amara, Ouimet, & Landry, 2004; Coburn, 2005; Weiss, 1977).
In addition to theoretical perspectives on the use of research and other forms of knowledge in policymaking, this study built on literature related to intermediary organizations. Ness (2010) urged higher education scholars to examine the unique role of these boundary-spanning groups in studying the extent and ways in which policymakers use information. In the present study, we used Nesss (2010) inclusive definition of intermediaries (i.e., entities that serve a translating function between two principals with different values and perspectives), although others define these groups more narrowly (e.g., as research brokers). Our definition also adhered to the taxonomy used in Scott and colleagues (2014) chapter on incentivist K12 policies. In their classification, think tanks, foundations, advocacy groups, civil rights organizations, and the press are examples of intermediaries.
Philanthropic foundations, another type of intermediary, warrant further scrutiny given their ascending role in education policy in the United States and globally (Ball, 2008; Ball & Junemann, 2012). Researchers have documented these organizations significant role in setting K12 education policy agendas (Reckhow, 2012; Reckhow & Snyder, 2014). Furthermore, through funding, they have played crucial roles in supporting data gathering, analysis, and distribution (Scott et al., 2014). In higher education, they have engaged in college-completion policy activity, primarily by funding other intermediaries (Gándara, Rippner, & Ness, 2017).
Beyond philanthropic foundations, numerous intermediary organizationsincluding the American Council on Educations Commission on Education Attainment, the National Governors Associations Complete to Compete, the Education Commission of the States, the Southern Regional Education Board, and the College Boardhave been active in the college-completion policy arena. These types of groups can be classified as intermediary organizations based primarily on structural featuressince they serve both policymakers and campus representativesand functional characteristics, since they translate research and make policy recommendations (e.g., via policy reports and briefs and through legislative testimony).
Finally, one other type of organization was central to this analysis: state higher education agencies. These groups are government bodies, but can also serve as intermediaries (i.e., brokers between state higher education systems and institutions on one hand and the legislative and executive branches on the other). State agencies serve at least two important functions: mediator between federal and local policy (Hamann & Lane, 2004) and data source (Shakespeare, 2008). Studies have found mixed results with respect to state agencies influence on policymaking as compared to other suppliers of information, such as constituents and research offices (Gray & Lowery, 2000; Guston, Jones, & Branscomb, 1997; Webber, 1987). Ness (2010) has advocated examining these unique organizations in the context of higher education policymaking. Thus, within this analysis, we paid particular attention to the unique boundary-spanning function of the state higher education agency, the Texas Higher Education Coordinating Board (THECB).
RESEARCH USE AND INTERMEDIARIES
At the intersection of research use and intermediaries lies a small literature examining intermediaries roles in promoting evidence use. In primary and secondary education, Honig (2004) illustrated the role that intermediary organizations at the district level play in promoting the use of evidence-based curricula and pedagogical practices. Scott and colleagues (2014), for their part, proposed that intermediary functions, as they relate to information use, include producing, funding, interpreting, and disseminating information. According to this taxonomy, intermediaries serve as both suppliers and demanders (or consumers) of information, including research (Scott et al., 2014). The supplier role of intermediaries, particularly in producing and disseminating research, is most commonly acknowledged, especially since policymakers increasingly rely on these organizations to interpret research evidence (Finnigan & Daly, 2014).
This analysis of college-completion policy activity in Texas employed a case study design, which allowed for an intensive, holistic description and analysis (Merriam, 2009, p. 46) of the use of various types of information in state-level college-completion policy activity. In particular, this analysis focused on a unique or extreme case (Yin, 2009), given Texass distinctive level of legislative professionalism, as measured by legislative staff capacity, and high college-completion policy activity, as suggested by document reviews. By bounding the analysis geographically, we held constant various state-level characteristics, including political, socioeconomic, organizational, and higher education features of Texas policymaking that might influence college-completion policy activity.
This study stemmed from a multiyear, multi-case project, in which we examined college-completion policy activity in five states: Georgia, North Carolina, South Carolina, Tennessee, and Texas. We selected these states for two major reasons. First, they shared relevant state characteristics for which we sought to control (e.g., location in the South). These states also had low educational attainment levels, which would suggest amplified attention to college completion relative to states with higher levels of educational attainment. Further, these states varied along dimensions that were of interest for the broader study, including type of higher education governing structure and levels of engagement in a national college-completion initiative. Specifically, three of the five states were members of the Complete College America (CCA) Alliance of States. Because we were interested in the role of CCA in college-completion policy activity for the larger project, we sought variability in membership in this organization.
From this group of states, we chose to focus on Texas for this analysis of the use of evidence and the role of intermediary organizations for the following reasons. First, given our interest in the use of research, we chose a state with considerable legislative staff capacity, since a legislature with sizeable legislative staff resources may lead to better-informed legislators (Squire, 2007, p. 214). According to the National Conference of State Legislatures, in 2015, Texas ranked second (after New York) in total staff employed during legislative sessions. Accordingly, we focused this analysis on a state with higher potential for evidence use to better glean preferences for various types of evidence and the mechanisms of evidence use.
Second, we focused on Texas for this analysis because of its robust college-completion policy activity, which enhanced opportunities for examining the use of evidence in college-completion policymaking. Document reviews preceding data collection revealed that Texas legislators were considering the following policies: a shift toward outcomes-based funding, expansion of dual credit, changes to developmental education, and modifications of the statewide need-based aid program. In addition, a task force was developing a replacement to the states access-centered master plan for higher education. The new plan, 60x30TX, emphasized completion by calling for an increase in postsecondary degree or certificate attainment rates for adults between the ages of 25 and 34 from 38% to 60% by the year 2030 (THECB, 2015).
Beyond its uniqueness in both legislative staff capacity and college-completion policy activity, Texas constituted an apt case for this analysis given the states broader national significance. First, Texas is the second-largest state in the country in land size and population. It also ranks second (after California) in number of students enrolled in a degree-granting postsecondary institution (National Center for Education Statistics, 2014). Thus, college-completion policy activity in this state affects a considerable portion of the nations current and prospective college students. Beyond its size, the demographic composition of Texas epitomizes demographic trends across the United States, particularly given the fast-growing Hispanic/Latinx population. This demographic shift is especially important in the context of postsecondary attainment since for the past four years, Hispanics/Latinxs have been the largest minority group on college campuses. In addition, in 2010, Texas was one of four majorityminority states, meaning there were more non-White residents than there were White residents living in Texas (Murdock, Cline, Zey, Jeanty, & Perez, 2014). As such, understanding education policymaking within this state context is relevant beyond Texas.
Data for this study are from the aforementioned multiyear project, in which we examined college-completion policy discussions, decisions, and recommendations in five states. The data collection process entailed conducting individual interviews with actors involved in college-completion policy initiatives in Texas. For the broader project, we also collected documents and videos and engaged in participant observation of legislative hearings in which policymakers discussed college completion. These data enriched our learning throughout the study, helping us identify interview participants and construct the interview protocols, and supplementing our findings from the interviews.
Specifically, we conducted interviews with 25 policy actors in Texas and with seven intermediary (national or regional) officials. We used purposive sampling to identify participants. In particular, we drew on media accounts and on actors formal roles in government and higher education settings to identify actors who were involved with the supply or demand of information in college-completion policy processes in Texas. We targeted the following potential interview participants: (1) legislators (and their staff members) who had sponsored college-completion legislation; (2) leaders of Texas-based intermediaries who had been active in college-completion activity, per document reviews; (3) governors education staff members; (4) high-ranking higher education agency (i.e., coordinating board) officials; and (5) higher education leaders, including chancellors, vice-chancellors, and presidents of Texas colleges and universities who were active in college-completion policymaking, per our document reviews.
At the end of each interview, we asked participants to recommend other actors familiar with college-completion policymaking in Texas. In addition to snowball sampling (Patton & Patton, 2002), after the first round of interviews, we relied on document analysis and participant observation to identify additional interview participants. We also contacted journalists who covered higher education in Texas. Table 1 lists the number of interview participants included in this study by sector. Data collection ceased after we reached saturation, the phase at which minimal new insights were gleaned from the interviews (Lincoln & Guba, 1985).
Table 1. Number of Interview Participants by Sector
We used semi-structured protocols of open-ended questions to guide the interviews (Rubin & Rubin, 1995). Two sample interview protocols (one for intermediary officials and another for elected officials and their staff members) appear in Appendix A. The protocols included questions about the supply and demand dynamics of information acquisition and use in college-completion policymaking. The interviews also specifically addressed the role played by national, regional, and state-level intermediaries in college-completion policy activity in the state. With participants permission, the interviews were all recorded and transcribed.
In addition to the interview data, we drew on an archival inventory of college-completion policy activity in Texas. The corpus of documents (n = 58) comprise newspaper articles, meeting materials, legislative statutes and codes, intermediary documents and website content referencing Texas college completion, and higher education agency website content. We also collected over 12 hours of archived video footage documenting THECB meetings.
The documents helped inform the selection of the first round of interview participants and the creation of the interview protocols. For instance, following reviews of college-completion legislation, we asked about specific policies and the information that was used to inform their development. We also relied on our review of publications, including those by intermediaries, to inquire about specific decisions made by information suppliers. For example, we asked CCA officials about their decision to include relatively few footnotes in their reports, compared to those produced by similar organizations. On the demand side of information use, we asked policymakers about their engagement with specific reports or other publications. Document reviews also helped us identify interview participants (e.g., legislators to interview based on their sponsorship of college-completion legislation).
In addition to interviews and documents, we drew on participant observations of four legislative hearingswhere legislators discussed and heard legislative testimony on higher education policies and on 60x30TXand one THECB Strategic Planning Committee Meeting. Of the legislative hearings, two were in the Senate Higher Education Committee, one in the House Higher Education Committee, and one in the House Public Education Committee, where legislators discussed a college-readiness policy. In the THECB Strategic Planning Committee Meeting, committee members deliberated the development of the states new strategic plan for higher education (60x30TX). This studys authors observed the proceedings in the audience and took notes, including notes on what types of evidence actors referenced in discussions and in legislative testimony. We also made note of who the key actors, beyond policymakers, were in these conversations. Following each event, we debriefed on our notes and identified additional interview participants based on the events.
DATA ANALYSIS AND TRUSTWORTHINESS
We coded the interview data using both descriptive and conceptual codes (listed in Appendix B). Descriptive codes included categories such as key policy actors and college-completion policies. The conceptual coding scheme comprised both a priori codes based on the conceptual framework on evidence use and intermediaries, as well as post hoc codes based on themes that emerged during the analysis. The a priori coding scheme included codes such as: the role played by an intermediary (e.g., information provider and policy advocate); characteristics of the supply of information; features of the demand for information; and state organizational and political characteristics (e.g., research capacity). Emergent codes included preferred types and sources of information, the legitimizing role of intermediaries (under Role Played by Intermediary), and state uniqueness. To perform the analysis, we used Dedoose, a mixed-methods data-analysis application. Examples of the coding process appear in Appendix C.
We arrived at the four primary themes that emerged from the analysis by using an axial coding technique to evaluate how our various categories crosscut and link (Strauss & Corbin, 1998). For instance, we found significant overlap in our themes on preferred sources of information, the role of intermediaries, and state characteristics. This analysis led to our first theme of insularity in college-completion policymaking in Texas. That theme captures the findings that policy actors prefer Texas-based data (partly because they view Texas as incomparable to other states) and that out-of-state, including national, intermediaries did not set the completion agenda in Texas.
Our second theme on the percolation of a strong business ethos into college-completion policymaking in Texas emerged from our analysis of the intersection between policymakers demand for information, the role played by intermediaries (including business groups), and state characteristics (including workforce/economic development). For our third theme, our analysis of actors preferences for different types of information, uses of the information, and differences by actors revealed a dearth of research (analytic) evidence in the policy process and dissimilarities in uses of evidence between policymakers and their staff. Finally, for our fourth theme, our analysis of the relationship between intermediaries roles and their preferences for supplying information uncovered a unique motivation for supplying information: to shame colleges and universities.
A team of five researchers coded the data. Each transcript was coded twice, by two separate researchers. During the second round of coding, the secondary coders took notes of any discrepancies between their coding decisions and those of the first coder. We discussed these differences and arrived at coding decisions with input from the full coding team during regular coding meetings.
To optimize the degree to which our findings represented the phenomena we examined, we followed a number of recommended methods (Merriam, 2009). First, we triangulated our findings by drawing on interviews with both Texas policy actors and intermediary representatives. Concurrence in interpretations of phenomena across actors in various roles enhanced our confidence in internal validity. In addition, a peer review process helped us compare individual interpretations of findings across researchers on the team and facilitated testing of rival hypotheses. The final technique used to enhance internal validity was adequate engagement with data collection (i.e., saturation after 32 interviews over a 19-month span).
The structural and policy contexts in Texas are unique and important for understanding the states use of information and the role of intermediaries in college-completion policy decisions. Before presenting the themes that materialized in this analysis, we first outline contextual elements that emerged from this study that are especially critical for understanding college-completion policy activity in Texas.
Organizationally, the state is home to a complex network of higher education institutions. The public higher education institutional landscape, on which we focus in this analysis, comprises 50 community college districts and the Texas State Technical College System (with four technical colleges), 10 university systems, and 46 universities (including medical and health science centers). This landscape represents more public higher education institutions than in any other single state.
The higher education agency, the THECB, oversees all public colleges and universities in the state. This agencys formal authority is primarily limited to approving degree programs and off-campus activities, providing recommendations on state appropriations to public colleges and universities, and approving most major construction projects. In addition to having limited formal powers, during the 83rd legislative session, the THECB lost authority during Sunset Review, the process by which legislators evaluate state agencies in Texas. In particular, legislators required the coordinating board to involve colleges and universities in its decision-making processes on the front end rather than later on . . . [and] stripped the agency of the authority to close low-performing degree programs (Hamilton & Ramshaw, 2013, para. 11). These dynamics constitute important context for this analysis, since relations with the coordinating board might affect legislative officials use of information from this agency.
Notwithstanding political dynamics surrounding the THECB, it is clear that the agency collects, produces, and distributes a plethora of information, much of which relates to college completion. As the first state to use a formula to allocate funds to public higher education institutions, Texas has had a robust data collection system for decades (McKeown-Moak, 1999). Three state agencies, including the THECB, worked collaboratively to create a longitudinal data system that tracks students from pre-kindergarten through the workforce.
Texas policymakers have long been interested in issues related to college student access and success in the state, partly due to concerns with the states economy and partly as an acknowledgment of substantial demographic shifts (Malandra, 2012). Although the national college-completion movement began to gain momentum circa 2009, policymakers in Texas began focusing on completion well before 2000. Texass Closing the Gaps statewide master plan, implemented in 2000, included student success as one of four major goals. Another goal, which received the most attention, was increasing participation (or access), and closing the gaps in access and attainment between different demographic groups.
For over a decade, policymakers in Texas have been attuned to gaps, or differences in participation and attainment, particularly between racial/ethnic groups. This focus is reflected in the data the coordinating board collects and reports, which are often disaggregated by race and ethnicity, and sometimes by gender and by students socioeconomic profiles. Unfortunately, state policymakers have also taken steps that scholars have considered detrimental to educational attainment, especially for racial/ethnic minorities. For instance, tuition deregulation for public institutions in Texas resulted in decreased participation by Hispanic students (Flores & Shepherd, 2014). This state, like many others, has also divested in higher education; declines in financial support have been particularly harmful for community colleges (Perna et al., 2014).
Finally, one important feature of the Texas policy context is the substantial business influence in the state. Commenting on the business communitys prominence in Texas, political scientist Cal Jillson (2014) observed, Groups that represent business come in many shapes and sizes, but together they are the dominant force in Texas politics (p. 85). Some business leaders have played a crucial role in both the extent to which policymakers have focused on higher education policy and their preferred areas of emphasis. Malandra (2012) has reported that industry representatives, motivated by concerns with an adequately skilled workforce, were largely responsible for leading higher education reform. The present study examines how this context shapes debates on college completion specifically; this issue lies prominently at the intersection of multiple stakeholders and actors in the state (e.g., business, government, educators, students and families, and the media).
Our analysis led us to identify four themes. The firstand most prominenttheme reveals the inward-looking nature of college-completion policymaking in Texas. This theme of insularity encompasses decision-makers preferences for Texas-based data and the perceptions that out-of-state influences on the college-completion agenda in Texas have been limited. The second theme confirms that the strong business ethos in Texas permeates college-completion policymaking in the state, and is especially apparent in business groups direct engagement in this process. Our third theme captures the dearth of analytical research in this policy process and discerns differences in evidence use by policymakers on one hand and their staffers and higher education agency officials on the other. Our final theme centers on the supply side: Some stakeholders deployed information tactically to shame colleges and universities.
THEME 1: LONE-STAR THINKING IN THE LONE-STAR STATE
The first theme captures two major phenomena: (1) decision-makers at various levels prefer Texas-based data to other sources of information, and (2) state-level policy actors rejected the notion that external actors helped set the college-completion agenda in Texas.
Preference for Texas-specific Data
Overwhelmingly, Texas policymakersat the state legislative and higher-education system levelsexpressed a preference for Texas-based (internal) data, including reports produced or repackaged by the coordinating board. Numerous reasons were given for this preference, including the quality of Texass longitudinal data system, the absence of states that are comparable to Texas, and the accessibility of Texass data. Identifying the coordinating board as her first stop, one senators aide responded, when asked about her preferred sources for information relating to college-completion policy:
I can talk to you generally about how I do research when issues come up during the interim, and . . . about how we go about researching other bills that come in. . . . I would say . . . my first stop, generally, is to contact the higher education coordinating board. They just have a wealth of data and theyre able to pull it pretty easily, just do pretty quick queries, and so as . . . an analyst at the state, theyre just an extraordinary resource. My next stop, usually, is . . . if its an issue related to a university, to call the government relations staff at different universities to get their specific data or experience or perspective, etc. If Im . . . looking at questions related to community colleges, Ill call specific community colleges or Ill call . . . the Texas Association of Community Colleges. Oftentimes theyll provide me with either . . . primary source in-house data, or . . . whatever reports have been developed. So by the time Im actually getting to the literature, its usually kind of guided by who I perceive to be the experts. You know, we dowe use the CCAs Game Changers publication, especially as it relates to like, outcomes-based issues and structured pathways research, and so thatsthats been helpful.
This respondent noted two clear examples of internal (Texas-based) data as preferred sources: data supplied by the THECB and in-house data from the Texas Association of Community Colleges. Beyond internal data, this aide, who worked for a legislator who served on the Senate Higher Education Committee, identified stakeholder input as a critical information source. After consulting those resources, she would refer to the literature, including intermediaries reports.
Another legislative aide, who also worked in the Senate Higher Education Committee, likewise identified Texas-specific data as his preference, pointing to Texass unique demographic landscape:
When youve got a state of 25 million people, 1.6 million college students, 37 public universities, 50 community college districts . . . it is always preferential to have Texas-based data. . . . Because if I have research that . . . is limited to private schools in the northeast, that is not especially relevant to a policy decision made that will impact . . . our own institutions. . . .We have a different population than Wyoming or Iowa or . . . even Florida, or Louisiana and . . . the more specific whatever it is can be to Texas, the better.
At the system level, a vice chancellor at one of the flagship university systems described his own data acquisition process around college-completion policy initiatives, identifying first system-level data, followed by national data for comparisons:
First of all, we look at four-year graduation rates . . . and we also look at semesters completed to assess how long the person take[s] to graduate from high school. And thats how we devised our metrics, thats how we devised our goal. And we typically have a group of peers thatthat we bring together from national data and thats how we set targets and goals. . . . So thats how we do it.
One THECB official confirmed policymakers preference for internal data, claiming that the coordinating boards data carries the day, and rightly so. This participant credited the quality of the data for this preference, since THECB data are subject to audits (as opposed to data submitted by institutions to the U.S. Department of Education to be included in the Integrated Postsecondary Education Data System). Beyond the quality of the coordinating boards data, one legislative aide echoed the aforementioned legislative aides sentiment in pointing to the accessibility of the THECBs data as his reason for preferring it:
I will just Google THECB accountability and follow the link, and just go from there. . . . You know, somebody called me the other day [asking] how many students are enrolled in our public four-year institutions and thats a 45-second look at on the . . . Internet. I mean, I dont have to keep things memorized because itsit is literally less than a minute away.
As illustrated previously, respondents also credited the unique demographic and structural state characteristics in the state for their preference for Texas-specific data. In parallel to expressing their partiality for Texas-specific data, some respondents cited the uniqueness of Texass demographics to explain their beliefs that cross-state comparisons were of limited value. One intermediary official, for example, observed that Texas is
. . . what the nations going to look like, and so were kind of like, on the cusp of what will be the reality as a nation, years to come, and so who can we look to? We look to ourselves and look to the people within those regions that are smart, that havehave a sensitivity to the population and know it best. That being said, yes, we can learn a lot from similar states like Florida and California, you know, perhaps New Mexico and maybe Colorado in terms of its Hispanic population, but we are . . . such a large state that, I mean, we can go from El Paso, and thats very different from whats going on in Houston, and then the [Rio Grande] Valley.
Likewise, an official in the governors office noted: We look outside but I mean, a lotbut therestheres just not a lot of states that are like us. So its hard to compare . . . New Yorks not like us even though theyre the same size.
Texas-driven Completion Agenda
Following this theme of insularity in college-completion policymaking, the analysis also revealed policymakers perceptions that out-of-state intermediaries did not influence college-completion agenda setting. That said, we found ample evidence of national intermediaries engagement in local (e.g., campus- and system-level) college-completion initiatives, especially by providing technical assistance. While some state-level policymakers recognized CCAa single-issue group focused on elevating college completion on state policy agendasand some were familiar with their reports, they rejected the notion that any external groups, including CCA, helped set the college-completion agenda. One legislator highlighted the states uniqueness in denying external groups role in agenda setting:
Our agendasand maybe its the size of Texas, maybe its the number of institutions and flagship institutions and things like thatwere driven . . . by mostly what goes on in the state of Texas. That doesnt mean that our graduates arent going all over the country and things like that, but our system is so largethats most of our interaction. . . . If I talked to all the people around here that want to talk about higher education, quite frankly we dont have time for [them]. . . . These organizations specifically, like I say, I have no bias against them, buthave they been providing us with a lot? No. Are they welcome? Yeah, to the degree that I [and staff] have time, sure, but . . . they shouldnt expect us to . . . call them.
When asked about CCAs role, this legislator replied, I dont know who the hell they are. Another legislative staff member also was not familiar with this group. Only one of the three legislative staffers recognized the group; as mentioned previously, she was familiar with some of the Game Changers, CCAs proposed policy solutions for improving college-completion rates.
Consistent with our review of documents, respondents noted that the completion agenda in Texas preceded the national momentum around this policy issue. One coordinating board representative described CCAs non-role in initiating college-completion policy discussions in Texas by noting: I dont think . . . Complete College America has done much to sort of raise . . . the profile of the issue within the state, in large part because I think its already there.
The interviews with intermediary officials serve to triangulate these findings from state policymakers about the challenges CCA has faced exerting influence at the state policy level. One senior CCA official, after discussing the organizations influence in Georgia, credited state structural and political characteristics for CCAs limited influence in agenda setting in Texas:
Texas has been more problematic. I think we probably went to Texas early, but its such a big state, its really hard to get a handle on how to start or to get a critical mass of people. The governor, this was not a priority of his, so we really couldnt get much traction with the governor. We had a lot of support from the business groups, who have been our supporters there . . . and weve been to Texas quite a few times, but its been difficult.
Despite CCAs challenges at the state policy level, this organization has been more successful working directly with the coordinating boardwhich, as previously mentioned, has limited formal and informal authorityand with business groups, as described in the previous quote. The THECB commissioner structured the 2013 State of Higher Education address around CCAs Game Changers, the five college-completion policies for which CCA advocates. Regarding CCAs relationship with business groups, one university system official remarked:
Thats how they have aligned. Less with legislators, more with Texas Association of Business to get to the legislators. . . . I think probably theas youve noted, the lack of success [in influencing state-level policy agendas] probably has to do more with thethe vastness of the state, its regions, its economy, the whole host of issues. . . . It gets very complicating from a political perspective to get them all on thesome of the same page.
Indeed, the Texas Association of Business (TAB) has hosted CCA representatives to present at their higher education convenings. One CCA official confirmed that they will pursue a more regional approach, including working with systems and campuses, considering the challenges they have faced at the state policy level.
In addition to their work with the TAB and the coordinating board, CCA and other national intermediaries have engaged directly with some campuses and multi-campus systems, for example, by providing technical support to implement college-completion initiatives. One respondent described her perception that college-completion policy activity in Texas is less centralized than in Tennessee, another state with which she is familiar. Indeed, we found considerable evidence of local college-completion efforts and substantial intermediary influence at the localized level. CCA, for instance, has engaged with individual institutions, including the University of Texas Rio Grande Valley, and with multi-campus systems. For example, CCA is collaborating with the University of Houston System on an initiative called Guided Pathways to Success. Beyond CCA, respondents identified other national intermediaries active at the campus and system levels. The Texas State University System has worked closely with both the Education Advisory Board and Degree Compass, national groups that provide technical assistance on college-completion initiatives. In addition, community college districts in the state have collaborated with Achieving the Dream, also a national intermediary group, and the New Mathways Project, a Texas-based group with a national presence.
Not all multi-campus systems engaged with intermediaries, however. A senior official at a flagship university system described the key players in college completion, in his view:
In Texas, I mean itsto be honest with you, its going to be the coordinating board . . . and every system by itself. . . . We read about college completion and all that but you know, I havent worked with any [intermediaries] and weve been at it for a while . . . working on that . . . agenda . . . [W]e [at the system] created an accountability program a long time ago . . . where we started talking about four-year graduation rates and how do we . . . get them up to a much better rate than what we have and . . . how do we make use of community colleges and all that so that we increase the production of graduates? But we were not driven by, you know, sort of the intermediary organizations at all.
By noting that they were not driven by intermediaries, this official seems to be referring to intermediaries lack of influence over agenda setting specifically. A representative from the other flagship university system in Texas only mentioned one national intermediary, the Gardner Institute, and the Lumina and Gates Foundations, but could not speak to their impact. In light of these findings, it is conceivable that intermediaries are less influential with flagship institutions, perhaps because flagships internal capacity is more robust than other institutions given their higher levels of fiscal resources.
THEME 2: PRO-BUSINESS, PRO-TEXAS
Our second theme captures the strong business ethos in Texasreflected in the slogan on the TABs business cards (Figure 1), Pro-Business, Pro-Texasand, specifically, its influence over college-completion policy. First, the composition of the Higher Education Strategic Planning Committee demonstrates the central role of the business community in higher education policymaking in Texas. This committee, charged with replacing the states higher education strategic plan, was chaired by Woody Hunt, CEO of a real estate conglomerate.
The response of one legislator, when asked to identify the major players in this policy area, also illustrates the prevalent role played by the business community in college-completion policymaking in Texas: [The] Texas Higher [Education] Coordinating Board, obviously. Us, because were the appropriating group. The business community in the State of Texas. Essentially, were producing a product for them, it could be said. They certainly think so.
As further evidence of the business communitys role in higher education policy activity in Texas, the TAB has hosted annual higher education conferences to discuss college completion and productivity in higher education. In addition, on their website, College Completion appeared as one of 11 Advocacy issues. Others include Telemedicine, Title Insurance Reform, and the Trans-Pacific Partnership. Explaining the TABs rationale for engaging heavily in higher education policy activity, the organizations president said, Workforce is why were in the game. I mean, because itit also happens to be the right thing to do. Likewise, in 2014, leaders from the TAB went on a jobs agenda tour to more than 13 cities across the state, where they met with leaders of chambers of commerce. On the tour, they discussed the TABs agenda, which includes higher education proposals, such as performance funding for higher education.
In addition to business groups engagement in higher education policymaking, a strong business ethos influenced college-completion policymaking, including the supply and demand for information around this policy issue. On the supply side, at a legislative hearing we attended, Mr. Hunt commented on data comparing Texass educational attainment rates to other countries, I just translate those to dollars. Specifically, he observed:
Were much closer to Mexico than we are to Canada as far as educational attainment of our younger population, the 25- to 34-year-olds, and once again what I see there is income. In other words, when you measure our 25- to 34-year-olds against Massachusetts or Canada and you look at a gap thats over 20%, I just translate those to dollars. . . . So thats our challenge as a state, at least in my view, and why I was willing to participate and provide leadership to this committee. . . .
On the demand side, an official at the coordinating board noted that most of the information that policymakers or campus officials have requested from the coordinating board are either about dual credit or workforce outcomes, adding that, education is an economic factor. In response to decision-makers preference for these types of data, in the 2015 version of the Almanaca document produced by the coordinating board that contains a plethora of information related to higher educationthe coordinating board added a new section on graduates workforce outcomes. As another example, at the last few Strategic Planning Committee meetings, presenters referenced the connection between education and the economy. The presenters included representatives from the Georgetown Center for Education and the Workforce and the Organisation for Economic Co-operation and Development (OECD).
THEME 3: ANALYTIC EVIDENCE ON THE SIDELINES
The third theme captures the relative absence of research evidence in college-completion policymaking and policymakers attitudes toward research. Related to this finding, we present evidence regarding legislators general preferences for information that is concise and relevant to their agendas. This third theme also gleans differences between legislators on one hand and their staffers and coordinating board officials on the other in uses of information, with staff more likely to use information conceptually.
As noted in the first theme, policymakers preferred descriptive information, especially Texas-based data, to other types of information, including research evidence, as a guide to college-completion policymaking. Research evidence, for its part, was largely absent from college-completion policy deliberations in state-level policymaking. The only exceptions were references to reports by the Community College Research Center in Columbia University and by the Georgetown Center on Education and the Workforce.
When we asked one journalist who covered higher education policy about whether research played a role in college-completion discussions, he replied:
. . . not particularly, as far as I can tell. I mean, Im sure the presidents of the universities and thewell, the chancellors are not necessarily academics. Theyre mostly politicians. But the presidents, Im sure, can have ways of making their feelings about [completion] known. But my impression is that we dont see a lot of likeyoull see professors give presentations at Coordinating Board meetings, but not at legislative hearings and you wont see them cited in a lot of things. . . . But they also might beyou know, I might just be zoning out and not hearing it because theyre so dull that Im missing them. [laughs]
In addition to the dull way in which it is presented, one possible reason for the dearth of research evidence in college-completion policymaking is policymakers preference for concise and relevant information. Exemplifying this preference, one legislator observed:
You can give me all the research youve done since youve gotten your doctorate and it probably is this thick [holds fingers two inches apart]. . . . Im actually going to read the executive summary, which is this thick [holds fingers two millimeters apart]. . . . And Im going to sit down with [the policy analyst] and say, what does this mean related to our legislative initiative? . . . Its not meant to imply that people like me are either not diligent or intellectually capable, but we only have so much time and that research is your job, not mine. Its my job to use it for some purpose in the milieu of public policy.
In addition to illuminating a preference for user-friendly information, this quote indicates the policymakers interest in examining evidence that relates to an existing legislative initiative, rather than information on an exploratory topic. Indeed, due to the time and resource constraints to which this legislator alludes, policymakers were less likely to use information conceptually than their staff members were. The three legislative aides interviewed all cited examples of conceptual use of evidence. One legislative aide observed that he looks at academic work but does not share it with elected officials for similar reasons to those cited by the legislator:
White papers probably are more utilized than the traditional academic articles. You know, I look at academic articles. I look at things written by faculty members, but its hard to translate that. Its not hardyou cant just give that to a member. You know, if I gave a member an R squared, thats something that . . . would need to be translated. I mean, we can talk about the predictive nature of X . . . but I think, you know, anything beyond this is predictive of college success can get a little in the weeds.
In contrast to the legislator who preferred information to inform a contemporary issue, staff members provided numerous examples of conceptual uses of evidence (i.e., for enlightenment, rather than for direct application). Illustrating conceptual use of information, one legislative aide described that in his office, staff dont like to limitat least initiallythe person who gives us the information. I mean ourour jobI often say to people, too much information is my problem, not yours. Describing what they do with information once they receive it, he noted, I will review it and/or staff will review it, and then kind of, you know, put it in our black box in our files, so that we can take a look at it later. This respondent contrasted his role as a policy analyst on one of the higher education committees to that of staff who have not been assigned to these committees, suggesting that legislative staff members who are not on higher education committees may not be as diligent about learning about higher education issues. He noted his sense of . . . duty to be the most informed people in the . . . building with regard to higher education . . . that is our job. . . . People call us when they have questions and we should have the answers.
Like legislative staff members, coordinating board officials also indicated familiarity with best practices reports by national intermediaries, including the Lumina Foundation and College Board. Further depicting conceptual uses of information, one coordinating board official claimed, Most of the Texas stakeholders, at least in the [higher education] agency, are very well plugged in to what the national literature is saying.
THEME 4: SHAMING AS A TACTIC FOR SUPPLYING INFORMATION
Our final theme from this analysis identifies a distinctive tactic for supplying information related to college completion: shaming public higher education institutions or states into improving completion rates. One particularly striking example, to which participants referred on multiple occasions, involved billboards. Specifically, the TAB, the most prominent business lobbying group in the state, erected billboards off major highways in Texas. One such billboard appeared near Austin Community Colleges (ACC) main campus in 2011. The sign (presented in Figure 2) comprised pure text (no graphics), and the message was simple: 4% OF ACC STUDENTS GRADUATE IN 3 YEARS. IS THAT A GOOD USE OF TAX $? TX ASSOCIATION OF BUSINESS. The president of the TAB, Bill Hammond, proudly noted in one of the interviews that those were the best $300 his organization had ever spent.
In response to the question of why the billboards were so impactful relative to other types of information (e.g., research studies or reports released by Lumina or other national intermediaries), Hammond credited the shocking nature of the data and the resulting press coverage for the billboards effectiveness in garnering attention:
Because [legislators] dont read [studies or reports], and none of that stuff made the newspapers . . . youve got to create a fight or something . . . youve got to say something shocking, unfortunately, like we did with ACC. I mean, 4%. Youre just trying to . . . get the public going and get the establishment upset. . . . I mean, the billboard was a kind of radical ideayou know, became a discussion point.
The billboard example also illustrates an intermediarys preferred method for supplying information: blunt messaging. As another example of employing blunt messaging with the intention of drawing attention to inadequate completion rates, CCA staff members title many of the organizations reports using bold statements such as Remediation: Higher Educations Bridge to Nowhere; Four-Year Myth, which reports on postsecondary students time to degree completion; and Time Is the Enemy of Graduation. CCA also displays, using infographics, completion statistics, particularly on-time graduation rates that as presented, would appear strikingly low. As an example, a Texas-specific report illustrates in a pie chart that 2% of all students complete their one- to two-year certificates on time. One CCA official reflected on this blunt-messaging approach, noting, I think that in our advocacy mission . . . I think that we are overzealous or over simplifying sometimes. The official continued by speculating that if they were less inflammatory and more nuanced, they would not get the same attraction from people.
These uses of data on graduation rates constitute examples of political uses of information. When asked what he liked about CCAs data, the TABs president responded, Because it validates what we all know. . . . [Institutions] are doing a lousy job. As such, he was confirming his preference for data that bolsters his preconceived notions for policy foci.
In addition to the famous TAB billboards and to CCA, a prominent and potentially impactful source of information relating to college completion is the THECB, and, in particular, its yearly Almanacs (glossy 8 ½ in.-by-14 in. publications containing data for each postsecondary institution in the state). These reports are distributed to members of the legislature, trustees, businesspersons, executive branch officials, and others. The tone of these documents is more neutral. The message is not as punchy as that provided by the billboards or by CCA, and we found less evidence of shaming in the documents themselves. However, in an interview, a coordinating board official noted that the Almanac puts pressure on institutions to do better. This comment is consistent with other impressions of the role of data as a mechanism for shedding light on what some observers perceive as poor outcomes. Thus, the analysis revealed that intermediaries and stakeholders in Texas have employed a distinctively blunt method of disseminating information coupled with a shaming tactic to highlight poor college performance. In doing so, they have aimed to raise public awareness, motivate institutions to pursue improvements, and encourage policymakers to pressure institutions to do better.
The analysis also revealed a potential drawback to the punchy messaging technique coupled with a shaming strategyhigher education institution officials hesitation to provide information or data on their institutions. Some campus officials expressed reservations with sharing information on their institutions performance, given the potential uses of this information to attract criticism. One higher education official at a four-year regional university noted that, despite her support for data analysis and for efforts to improve her institution (and others), she has worried about what the press or state leaders might do with the data. Triangulating this finding, the leader of a state-level intermediary acknowledged some higher education leaders hesitation to provide data. Reflecting on a recent experience, he recalled:
I just was at a thing in the [Rio Grande] Valley and . . . Ive never seen this before[a university official] voluntarily said . . . weve worked really hard in the last six years and our graduation rate has gone, you know, from 35 to 46. I mean, it still wasnt kicking it . . . but they were honest, and I actually called [the colleges president] out, because . . . Ive never been at a meeting where . . . voluntarily . . . a college or university president has shared [graduation rates].
DISCUSSION AND IMPLICATIONS
At a time when college-completion policies continue to be ubiquitous on state-level higher education policy agendas, this in-depth case study shed light on the policy activity surrounding this issue in one state. In particular, this study examined Texas, which has a highly professional legislature and a long history with college-completion policy, both features that enhanced opportunities for examining use of evidence in college-completion policymaking.
Many of our findings are unique to Texas, given its distinct demographic, college-completion policy, and structural characteristics. The plethora of data collected by the THECB is rare across the states, but results in significant reliance on the coordinating board for information related to college completion. Indeed, overwhelmingly higher education policy actors tend to prefer Texas-based data, particularly that provided by the coordinating board. Respondents cited three major reasons for this preference: the high quality of the THECBs data, Texass unique demographics, and the accessibility of the THECBs data.
Also noteworthy is the perception among policy actors of Texass uniqueness. Like Texas, other states are becoming increasingly aware of their unique state characteristics, leading to a type of policy filtering (Rubin & Hearn, 2018). That is, Texas may not be alone in its scrutiny of cross-state comparisons. The widespread belief of the incomparability of demographics in Texas to those in other states, coupled with the critical role of the THECB in supplying college-completion information, appear to have limited external (out-of-state) influence over college-completion policymaking at the state level, especially in agenda setting. Indeed, external intermediaries played a largely supportive role, and a rather localized one, providing technical assistance as these initiatives move forward.
Turning to the information supplied by these external groups, policy actors in Texas used it but emphasized the limited value of cross-state comparisons given Texass unique demographics. These findings point to the importance of taking into account the context of the decision-making level (in this case, states) in examining the role of intermediary organizations and new governance arrangements, which recent studies have found to be significant and increasing (Ball, 2008). This finding also highlights the mediating role that is played not only by state structural characteristics, but also by culture. Perceptions of Texass distinctive inward-looking nature permeated our interviews and set the stage for the role that intermediaries play in the state and the preferences for information.
In addition to the policy-filtering phenomenon (Rubin & Hearn, 2018), several findings related to the use of information in the college-completion policymaking process have significance beyond this state. First, the role of the business community in college-completion policy in Texas is consistent with previous findings of the role of business in higher education policymaking, particularly around performance funding (Dougherty, Natow, Bork, Jones, & Vega, 2013). Given their prominence in supplying information for college-completion policy in Texas, future research should explore the unique role played by these organizations in higher education policy activity across states.
Second, consistent with previous literature (Lubienski, Scott, & Debray, 2011; Scott et al., 2014), systematic research evidence is far from the forefront of policymaking in Texas. It is important to note that relative to other education policy areas, such as charter schools or voucher programs, the literature base on college-completion policies is nascent. However, increasingly, higher education scholars, including ones working for research-based intermediaries (e.g., the Community College Research Center) and scholars who are not affiliated with research institutes, have begun to evaluate the effectiveness of certain college-completion policies, including remedial education approaches (e.g., Logue & Watanabe-Rose, 2014).
The work produced by the Community College Research Center and Georgetowns Center on Education and the Workforce were the only research reports referenced in Texas in our data. Indeed, legislators noted their preference for concise and relevant information, which these organizations provide. In addition, these groups make themselves accessible to policymakers, which, as previous research has shown, matters greatly in enhancing the use of evidence in policymaking (Nutley, Walter, & Davies, 2007). Those intending to influence policymaking should consider making information, especially research evidence, concise and easily accessible and establish relationships with policymakers and their staff members.
Although research evidence was present and utilized, local information and localized initiatives seem to be most welcomed and effective. Intermediaries wishing to inform college-completion policy activity at the state level should consider the uniqueness of the state context in supplying information. For states that are more insular, like Texas, working through in-state intermediaries, such as the TAB, may be an effective strategy.
Our analysis also revealed that legislative staff members and coordinating board officials most commonly used information conceptually, which aligns with what previous research has found (Amara, Ouimet, & Landry, 2004; Coburn, 2005; Weiss, 1977). While researchers view instrumental use of information as somewhat of a gold standard, as Nutley and colleagues (2007) have noted, political use of evidence, particularly high-quality evidence, can promote sound policymaking. For instance, policymakers may reference data on workforce needs or low completion rates to support their argument for focusing on college completion; by visiting these sources, they become exposed to other important information (e.g., the need to focus on remedial education and the gap in workforce outcomes by racial/ethnic and socioeconomic groups) that can point to specific solutions.
Finally, we found that some intermediaries use a shaming tone with the intent to pressure institutions. The TAB and CCAs blunt messaging approach appeared to be effective at garnering attention from policymakers and the public. Indeed, the TABs billboards featured prominently in our interviews, and many respondents were familiar with CCAs reports. Despite this success at bringing attention to this issue, one possible unintended consequence of punchy messaging is campus representatives hesitation to share data, incited by a fear of becoming targets of shaming. In addition, punchy messaging runs the risk of omitting important nuances in evidence. For example, the graduation rates presented on the billboards, while striking to those who are unfamiliar with graduation rates across states, masks the challenges faced by institutions, particularly those that serve the students with highest needs, including community colleges. In addition, placing the onus entirely on institutions by shaming them can lead to policy initiatives that punish institutions that serve students with lower likelihoods of completing their degrees.
In conclusion, an overt recognition by policy actors of the preference for brief and simple information suggests that higher education stakeholders wishing to bring attention to this issue might consider simple messaging as a first step. Caution should be taken, however, to avoid shaming institutions and to forego nuances excessively, in light of potential unintended consequences.
Amara, N., Ouimet, M., & Landry, R. (2004). New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Science Communication, 26(1), 75106.
Ball, S. (2008). New philanthropy, new networks and new governance in education. Political Studies, 56, 747765.
Ball, S., & Junemann, C. (2012). Networks, new governance, and education. Chicago, IL: The Policy Press.
Birnbaum, R. (2000). Policy scholars are from Venus; policy makers are from Mars. Review of Higher Education, 23(2), 119132.
Caplan, N. (1979). The two-communities theory and knowledge utilization. American Behavioral Scientist, 22(3), 459470.
Coburn, C. E. (2005). Shaping teacher sensemaking: School leaders and the enactment of reading policy. Educational Policy, 19, 476509.
Coburn, C. E., Toure, J., & Yamashita, M. (2009). Evidence, interpretation, and persuasion: Instructional decision making at the district central office. Teachers College Record, 111(4), 11151161.
College Board. (2010). The college completion agenda state policy guide. New York City, NY: The College Board. Retrieved from http://www.ncsl.org/documents/educ/policyguide_062810sm.pdf
Conaway, C., Keesler, V., & Schwartz, N. (2015). What research do state education agencies really need? The promise and limitations of state longitudinal data systems. Educational Evaluation & Policy Analysis, 37(1), 16S28S.
DeBray, E., Scott, J., Lubienski, C., & Jabbar, H. (2014). Intermediary organizations in charter school policy coalitions: Evidence from New Orleans. Educational Policy, 28(2), 175206.
Dougherty, K. J., Natow, R. S., Bork, R. H., Jones, S. M., & Vega, B. E. (2013). Accounting for higher education accountability: Political origins of state performance funding for higher education. Teachers College Record, 115(1), 150.
Dougherty, K. J., & Reddy, V. (2013). Performance funding for higher education: What are the mechanisms? What are the impacts? ASHE Higher Education Report, 39(2).
Dunn, W. N. (1980). The two-communities metaphor and models of knowledge use. Knowledge: Creation, Diffusion, Utilization, 1, 515536.
Finnigan, K. S., & Daly, A. J. (2014). Using research evidence in education: From the Schoolhouse Door to Capitol Hill. New York, NY: Springer International Publishing.
Flores, S. M., & Shepherd, J. C. (2014). Pricing out the disadvantaged? The effect of tuition deregulation in Texas public four-year institutions. Annals of the American Academy of Political and Social Science, 655, 99122.
Gándara, D., Rippner, J. A., & Ness, E. C. (2017). Exploring the how in policy diffusion: National intermediary organizations roles in facilitating the spread of performance-based funding policies in the states. The Journal of Higher Education, 88(5), 701725.
Gray, V., & Lowery, D. (2000). Where do policy ideas come from? A study of Minnesota legislators and staffers. Journal of Public Administration Research and Theory, 10, 573597.
Guston, D. H., Jones, M., & Branscomb, L. M. (1997). The demand for and supply of technical information and analysis in state legislatures. Policy Studies Journal, 25(3), 451469.
Hamann, E. T., & Lane, B. (2004). The roles of state departments of education as policy intermediaries: Two cases. Educational Policy, 18(3), 426455.
Hamilton, R., & Ramshaw, E. (2013, May 16). Coordinating board nears end of tough Sunset process. Texas Tribune. Retrieved from https://www.texastribune.org/2013/05/16/coordinating-board-nears-end-tough-sunset-process/
Hearn, J. C. (1997). Research on higher education in a mass and diversified system: The case of the United States. In J. Sadlak & P. Altbach (Eds.), Higher education research at the turn of the century: Structures, issues, and trends (pp. 271319). New York, NY: Garland Publishing, Inc.
Hird, J. A. (2006). Power, knowledge, and politics: Policy analysis in the states. Washington, DC: Georgetown University Press.
Honig, M. I. (2004). The new middle management: Intermediary organizations in education policy implementation. Educational Evaluation and Policy Analysis, 26(1), 6587.
Jabbar, H., Lalonde, P., DeBray, E., Scott, J., & Lubienski, C. (2014). How policymakers define evidence: The politics of research use in New Orleans. Policy Futures in Education 12(8), 10131027.
Jillson, C. (2014). Texas politics: Governing the Lone Star State. New York, NY: Routledge.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: SAGE.
Logue, A. W., & Watanabe-Rose, M. (2014). Mainstreaming remedial mathematics students in introductory statistics: Results using a randomized controlled trial, presented at the Society for Research on Educational Effectiveness Fall 2014 Conference, Washington, DC, 2014.
Lubienski, C., Scott, J., & DeBray, E. (2011). The rise of IOs in knowledge production, advocacy, and educational policy [Commentary]. Teachers College Record.
Lumina Foundation. (2013). Strategic plan 2013 to 2016. Indianapolis, IN: Author. Retrieved from https://www.luminafoundation.org/files/resources/2013-lumina-strategic-plan.pdf
Malandra, G. H. (2012). Efforts to improve productivity: The impact of higher education reform in Texas. In A. P. Kelly & M. Schneider (Eds.), Getting to graduation: The completion agenda in higher education (pp. 246268). Baltimore, MD: Johns Hopkins University Press.
McKeown‐Moak, M. P. (1999). Higher education funding formulas. New Directions for Higher Education, 1999(107), 99–107.
Merriam, S. B. (2009). Qualitative research: A guide to design and interpretation. San Francisco, CA: Jossey-Bass.
Mooney, C. Z. (1991). Information sources in state legislative decision making. Legislative Studies Quarterly, 16, 445455.
Murdock, S. H., Cline, M. E., Zey, M., Jeanty, P. W., & Perez, D. (2014). Changing Texas: Implications of addressing or ignoring the Texas challenge. College Station, TX: Texas A&M University Press.
National Center for Education Statistics. (2014). Digest of Education Statistics. Washington, DC: U.S. Department of Education, Institute of Education Sciences.
National Governors Association. (2010). Complete to compete. Washington, DC: National Governors Association Center for Best Practices. Retrieved from https://www.nga.org/files/live/sites/NGA/files/pdf/1011COMMONTECHGUIDE.PDF
Ness, E. C. (2010). The role of information in the policy process: Implications for the examination of research utilization in higher education policy. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. XXV; pp. 149). New York, NY: Springer.
Ness, E. C., Tandberg, D. A., & McLendon, M. K. (2015). Interest groups and state policy for higher education: New conceptual understandings and future research directions. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (pp. 151186). New York, NY: Springer.
Nutley, S. M., Walter, I., & Davies, H. T. (2007). Using evidence: How research can inform public services. Bristol, UK: Policy Press.
Patton, M. Q., & Patton, M. Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, CA: SAGE.
Perna, L. W., Finney, J. E., & Callan, P. M. (2014). Hard choices ahead: The performance and state policies of higher education in Texas. In L. W. Perna & J. E. Finney (Eds.), The attainment agenda (pp. 137167). Baltimore, MD: The Johns Hopkins University Press.
Reckhow, S. (2012). Follow the money: How foundation dollars change public school politics. Oxford University Press.
Reckhow, S., & Snyder, J. W. (2014). The expanding role of philanthropy in education politics. Educational Researcher, 43(4), 186195.
Rubin, P. G., & Hearn, J. C. (2016, November). The policy filtering process: Understanding distinctive state responses to the national college completion agenda. Paper presented at the annual meeting of the Association for the Study of Higher Education, Columbus, OH.
Rubin, H. J., & Rubin, I. S. (1995). Qualitative interviewing: The art of hearing data. Thousand Oaks, CA: SAGE.
Rutherford, A., & Rabovsky, T. (2014). Evaluating impacts of performance funding policies on student outcomes in higher education. Annals of the American Academy of Political and Social Science, 655, 185208.
Scott, J., Jabbar, H., LaLonde, P., DeBray, E., & Lubienski, C. (2015). Evidence use and advocacy coalitions: Intermediary organizations and philanthropies in Denver, Colorado. Education Policy Analysis Archives, 23(124), n124.
Scott, J., Lubienski, C., DeBray, E., & Jabbar, H. (2014). The intermediary function in evidence production, promotion, and utilization: The case of educational incentives. In K. Finnigan & A. Daly (Eds.), Using research evidence in education: From the schoolhouse door to Capitol Hill (pp. 6989). New York, NY: Springer.
Shakespeare, C. (2008). Uncovering informations role in the state higher education policymaking process. Educational Policy, 22(6), 875899.
Snow, C. P. (1961). Science and government. Cambridge, MA: Harvard University Press.
Snyder, M. (2015, February 12). Driving better outcomes: Typology and principles to inform outcomes-based funding models. Retrieved on August 19, 2015, from http://hcmstrategists.com/drivingoutcomes/wp-content/themes/hcm/pdf/Driving Outcomes.pdf
Squire, P. (2007). Measuring state legislative professionalism: The Squire index revisited. State Politics & Policy Quarterly, 7(2), 211227.
Strauss, A. L., & Corbin (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory. Los Angeles, CA: SAGE.
Tandberg, D., & Hillman, N. (2014). State higher education performance funding: Data, outcomes and policy implications. Journal of Education Finance, 39(1), 222243.
Texas Higher Education Coordinating Board. (2015, April 20). 60x30TX: 60 percent of Generation Texas will have a postsecondary credential or degree by 2030. Retrieved on May 2, 2014, from http://content.govdelivery.com/attachments/TXHECB/2015/04/30/file_attachments/386255/TX_HESP_Draft_APR_2015v4.pdf
Webber, D. J. (1987). Legislators use of policy information. American Behavioral Scientist, 30, 612631.
Weiss, C. (1977). Research for policys sake: The enlightenment function of social research. Policy Analysis, 3, 531546.
Weiss, C. (1979). The many meanings of research utilization. Public Administration Review, 39, 426431.
Yin, R. K. (2009). Case study research: Design and methods. Los Angeles, CA: SAGE.
Sample Interview Protocols
Interview Protocol Intermediary Officials
To start off, could you please provide an overview of [organizations] major college completion initiatives?
What are some specific policy proposals you pursued?
What strategies did you employ to reach your college completion-policy related goals and do you feel like these were effective?
From your perspective, what role, if any, does [organization] play in influencing college completion policy discussions and decisions in Texas?
Networking (connecting to other states or organizations)
Legitimization (especially with governor and legislature)
Expertise (providing information)
Can you think of any ways in which [organization] influenced college completion policies in Texas?
Who were the key people you targeted to inform or influence college completion policies (these could be elected officials, their aides, higher education agency officials, governors staff members)?
When [specific state actors] reached out to you regarding college completion- related information, what was typically the nature of the discussions and/or requests?
In general, to what extent would you say research evidence that might be used to inform college completion policy is produced in your office?
What type of information that might inform college completion do you produce?
In what format (e.g., white papers, talking points, or research articles)?
What type (e.g., best practices or local descriptive information like district-level data or data specific to a college or university)?
How do you distribute or promote the evidence that you produce?
By whom is it used?
Do you ever distribute or promote information from other sources?
What type of information do you distribute?
How do you distribute or promote it?
Of the information you distribute, how much of it would you say is produced by your organization?
Do you ever receive requests for evidence that might be used to inform college completion policy?
How is the evidence you produce used?
To inform a specific policy decision? (Instrumental)
To provide broad support for a policy approach? (Conceptual)
To support already determined policy preferences? (Political/Tactical)
Would you say that [organization] is an important resource for information or evidence for Texas?
How would you characterize your organization as a source of college-completion information relative to other organizations or potential sources of information?
In general, to what extent would you say evidence is used in your office to inform college completion policy recommendations?
What type of information do you use?
In what format (e.g., white papers, talking points, or research articles)?
What type (e.g., best practices or local descriptive information like district-level data or data specific to a college or university)?
From what source (e.g., your office, state agency, regional organization, national organization, university, trade journal)?
Can you think of any other individuals or organizations that might provide information relating to college completion?
Do you have any additional thoughts on the role of intermediary organizations in the use of evidence to inform college completion policies in Texas?
Individuals to interview
Events to observe
Interview Protocol Texas Elected Officials and Staff
To start off, could you please provide a brief overview of college completion-related activity in Texas?
In your view, what are the major influences on college-completion policy activity in Texas?
In general, to what extent would you say research evidence, broadly defined, or other forms of information are used in your office to inform college-completion policy discussions, decisions, or recommendations?
What type of information do you typically use (e.g., best practices evidence, legislative district-level data, system- or campus-level data, or others)?
What are your most common sources for information?
Do you prefer information in a certain format? How and to what ends do you use research evidence or other types of information?
Can you think of any examples of ways in which youve used research evidence?
In your view, do certain state actors, including elected officials, influence the type of college completion information that you acquire or use?
Do you think the professional staffs capacity plays a role in the acquisition or use of college completion-related information?
Can you think of any other state agencies or organizations (e.g., the university system offices) that play a role in the type of college-completion information that you acquire or use?
From your perspective, what role, if any, do [intermediaries] play in influencing college-completion policy discussions and decisions in Texas?
Would you say that [intermediaries] are important resources for information or research evidence?
Do [intermediaries] ever provide information to you?
How would you characterize the information that is most often provided to you by [intermediaries] (e.g., in terms of length, target audience, whether it is evidence-based)?
When research evidence or information is provided to you, what do you do with it?
What type of information do you prefer or are you more likely to use to inform college-completion policies?
At what stage in the policy process do you use information provided by [intermediaries]?
Do you ever request information from [intermediaries]?
What kind of information have you requested most frequently in the past?
Typically, how would you go about requesting information (e.g., phone call, e-mail, in person, Internet search)?
How do you use the research evidence or other forms of information?
Can you think of any examples of ways in which youve used evidence?
Can you think of any other individuals or organizations that might provide information relating to college completion?
Do you have any additional thoughts on the role of intermediary organizations in the use of research evidence to inform college-completion policies in Texas?
Individuals to interview
Events to observe
Note. Codes are as follows: Demand = D [preferred sources = Dps; preferred types = Dpt (academic research = Dptar; policy report = Dptpr); use = Du (non-use = Dunu)]; Other States = OS; State Characteristics = SC (state demographics = SCsd; state uniqueness = SCsu); State/Governmental Key Players = G (legislative = Gl)
Example 1 (Intermediary Official)
Interviewer: I wanted to ask about to what extent you view Texas as being different or not from some of the other states in terms of their use of information to inform college-completion policies, or their reliance on research, or even their data capacity within the state relative to other states.
Participant: We dont really look at other states and try to copy them OS/Dunu, and I dont think thats really much of a pride issue, I think its because Texas is so diverse SCsd/SCsu. I mean, you haveyou have urban, you have, you know, growing populations in certain cities, the valley where you have a huge region that is 80, 90% Latino . . . and so thats really likethats what the nations going to look like, and so were kind of like, on the cusp of what will be the reality as a nation, years to come SCsd, and so who can we look to? We look to ourselves and look to the people within those regions that are smart, that havehave a sensitivity to the population and know it best Dps. That being said, yes, we can learn a lot from similar states like Florida and California, you know, perhaps New Mexico and maybe Colorado Dps/OS in terms of its Hispanic population SCsd, but we arewe are such a large state that, I mean, we can go from El Paso, and thats very different from whats going on in Houston, and then the valley SCsd/SCsu.
Example 2 (State Legislator)
Interviewer: Could you say a word about ourour theme of research evidence or data and what role those have played in driving the completion agenda and policymaking here?
Participant: I think they play a role Dptar, but you have to careful about burying people like me in data and research. Thats not my job Gl/Dpt. I need to know what the question is andand what the problem is its meant to address Dpt. You know, itsmymine aremine are policy initiatives, not research Gl. Its nice to know what it is as a technicality, or something. I just need to know empirically and very clearly and succinctly what the research is, what its meant to show Dptar, and then if anybody has any insights on paths of success oror achievement, great Dptpr.