Home Articles Reader Opinion Editorial Book Reviews Discussion Writers Guide About TCRecord
transparent 13
Topics
Discussion
Announcements
 

The Philosophical Underpinnings and Key Features of the Dynamic Learning Maps Alternate Assessment


by Neal M. Kingston, Meagan Karvonen, Sue Bechard & Karen A. Erickson - 2016

The Dynamic Learning Maps™ Alternate Assessment is based on a different set of guiding principles than other assessments. In this article we describe its characteristics and look at the history of alternate assessment and the problems in implementing useful assessment programs for students with significant cognitive disabilities.

Large-scale, statewide assessment systems have been implemented at least since the 1980s to assess student achievement of state-adopted curriculum standards. When used for accountability purposes, the information yielded by these assessments typically was used to make school- or program-level decisions and to help decision makers allocate resources to improve education for children. However, prior to the reauthorization of the Individuals with Disabilities Education Act (IDEA) in 1997, students with disabilities were unevenly included in these efforts. Some were tested in their enrolled grade level (with or without accommodations), and many were tested “off-grade level,” whereby an eighth-grade student might be given a test of his or her achievement in third-grade academic content (Thurlow, Elliott, & Ysseldyke, 1999). During this time, students with significant cognitive disabilities were excluded from large-scale assessments. Instead, they were taught a separate, primarily functional curriculum (e.g., safety, hygiene and personal living, community participation), often in separate settings.

The landscape changed dramatically with the passage of IDEA in 1997. IDEA required that all students with disabilities have access to the general curriculum and provided an option for students with significant cognitive disabilities to participate in large-scale assessment systems through “alternate assessments” (AAs). By law, AAs under these qualifications had to be developed and fully implemented by 2000. Still, AAs varied significantly from state to state until 2001 and the passage of the No Child Left Behind (NCLB) act that mandated every state to align its AAs to grade-level academic content with alternate achievement standards (AAS). In other words, students would be taught content related to what their age-appropriate peers were taught, but at reduced depth, breadth, or complexity, and with different expectations for what they should achieve in order to have demonstrated “proficiency” in the grade level. Under this new policy era, students with significant cognitive disabilities (SWSCDs) throughout the nation were mandated to understand standardized achievement tests that were aligned with grade-level expectations, regardless of the manner or significance of the student’s disability.

WHO ARE STUDENTS WITH SIGNIFICANT COGNITIVE DISABILITIES?

SWSCDs represent a small but highly diverse group of students that comprise approximately 10% of all students with disabilities, and about 1% of all students (U.S. Department of Education, 2005). In a 2013 census survey of more than 40,000 SWSCDs (Clark & Karvonen, 2015), 44% were eligible for special education under the category of Intellectual Disability, 23% as Autism, and 14% were eligible for special education because of Multiple Disabilities, defined as concomitant impairments, such as an intellectual disability and hearing impairment (U.S. Department of Education, 2004). In some states multiple disabilities are reported separately, thus the true rate of students with multiple disabilities was not clear in the census.

Most SWSCDs have significant challenges with communication, adding another layer of complexity to how we might assess what students know. For example, of the 40,000 SWSCDs who participated in a 2013 census survey (Clark & Karvonen, 2015), approximately 79% used some form of speech to communicate and 21% did not. Of those who could communicate, 71% could regularly combine three or more spoken words (according to grammatical rules to accomplish a variety of communicative purposes), 20% could only put two words together to convey meaning, and 9% could only speak one word at a time. Of those who could not talk, 71% used augmentative or alternative communication systems to communicate (e.g., printed symbols on Velcro board, eye gaze devices or tablets), and 29% used sign language (typically an idiosyncratic system rather than American Sign Language or Signed Exact English) to augment or replace speech. A smaller percentage had no means of expressive communication. The variety of communication impairments demonstrated by this population of SWSCDs provide significant challenges for developing assessments that can accurately measure students’ knowledge, skills, and abilities.

HISTORY OF ALTERNATE ASSESSMENTS

The purposes of AAs have shifted over time. Prior to NCLB, most AAs were used to measure SWSCDs’ performance on functional skills including some functional academic skills (making change, recognizing symbols). But, in response to NCLB, states created alternate assessment systems to gauge students’ performance against alternate academic achievement standards (AA-AAS). The new assessments were aligned to content standards for the students’ chronologically age-appropriate grade level. Some states relied on the general academic content standards, while other states developed extended content standards that linked to the general content standards but reflected reduced depth, breadth, or complexity to provide an appropriate level of access for SWSCDs. The requirement to assess on grade level-aligned academic content was a significant shift in the educational programs for SWSCDs (Browder et al., 2007), who also have educational programs that include therapeutic goals, functional skills, and goals related to post-high school transition. Although AA-AASs were redesigned to align to grade-level content standards, some states had difficulty reaching the goal of assessing content aligned to grade level standards (Karvonen, Almond, Wakeman, & Bechard, 2010).

Over the past 15 years, AA-AAS formats have included portfolios, standardized performance tasks, rater checklists, and systems that combined these types of evidence (Altman et al., 2010; Thompson & Thurlow, 2003). Each format has strengths and drawbacks. For example, portfolios allow for a more flexible evaluation of students’ academic skills and more opportunity to choose academic content that fits best with the student’s curricular priorities. However, there have been concerns that portfolio-based AA-AAS scores provide more evidence of the teacher’s skill in assembling a portfolio than the student’s achievement (Flowers, Browder, & Ahlgrim-Delzell, 2006). Performance assessments tended to have better alignment with academic content standards (Flowers, Browder, Wakeman, & Karvonen, 2007) but also introduced accessibility barriers when standardized administration prevented the use of accommodations.

Advances in the field have led to improvements in AA-AAS systems, such as better measurement and scoring models, improved content alignment, expanded options for student demonstration of knowledge and skills, and improved teacher training on assessment administration. These types of improvements lend more support to the inferences that may be made about AA-AAS scores.

Although AA-AAS systems have improved over time, they perpetuate many of the same problems seen in large-scale general assessments for students without significant cognitive disabilities. In many states, AA-AASs still yield one measure of the student’s performance at the end of the year (Cameto et al., 2009). Scores provide limited information about what students know and can do, and when teachers perceive the results as irrelevant, the results have limited use for instructional planning. For example, in a survey of 983 teachers across five states, Flowers, Ahlgrim-Delzell, Browder, and Spooner (2005) found that only about one quarter of the teachers reported students participating in the alternate assessment had greater access to the general curriculum, improved quality of Individualized Education Program (IEP) objectives, or overall better quality of education, but 82% of the teachers reported increased paperwork.

Standardized AA-AASs are based on assumptions that knowledge is unidimensional and that it is fair to test all students the same way, even when they cannot demonstrate their knowledge in the same way. Testing models focus on student similarities, with an assumption of a single path of learning and a consistent, sequential development of knowledge and skills. Finally, when embedded in high-stakes accountability systems and used to evaluate programs and educator effectiveness, high-stakes assessments, including alternate assessments, risk unintended consequences such as narrowing the curriculum and using tests as models for instruction (Au, 2007; Crocco & Costigan, 2007; Jones, Jones, & Hargrove, 2003; Linn, Grau, & Sanders, 1990; Popham, 2001, pp. 15–25; Shepard, 1990). With every negative unintended consequence, like teaching to the test, narrowed curriculum, and teacher and administrator cheating, we ratchet up the stakes and act as if we expect those consequences to disappear.

Despite shifts in legislation and improvements in AA-AAS systems, there are still limitations in current AA-AAS design and practice, such as (1) limited ability to measure student growth; (2) difficulty assessing the technical quality of assessment systems due to the limited sampling of content and small, heterogeneous populations of students who take AA-AAS; and (3) perceived irrelevance of AA-AAS results, which limits their utility to inform instructional planning. With a shared belief that small modifications to existing approaches have been insufficient to support the needed improvements to learning for students with significant cognitive disabilities, a consortium of states developed the Dynamic Learning Maps (DLM®) Alternate Assessment System® that provides an alternative framework for assessing SWSCDs’ mastery of academic content.

DYNAMIC LEARNING MAPS ALTERNATE ASSESSMENT SYSTEM

For the world of educational assessment to better serve students with significant cognitive disabilities, we must begin with a goal for large-scale assessment that helps students learn. It is with that goal in mind that the Center for Educational Testing and Evaluation at the University of Kansas, in partnership with the Center for Literacy and Disability Studies at the University of North Carolina at Chapel Hill and a consortium of 18 states, developed a dramatically different type of academic assessment tool for students with significant cognitive disabilities—the Dynamic Learning Maps Alternate Assessment. The DLM theory of action (see Appendix A) is that high expectations for students with significant cognitive disabilities combined with appropriate educational supports for teachers result in improved academic experiences and outcomes for students.

There are six high-level features that guide the design of the DLM system: (1) fine-grained learning maps that guide instruction and assessment, (2) a subset of particularly important nodes that serve as content standards in order to provide an organizational structure for teachers, (3) instructionally embedded assessments that reinforce the primacy of instruction, (4) instructionally relevant testlets that model good instruction and reinforce learning, (5) accessibility by design (vs. accommodation) and alternate testlets, and (6) status and growth reporting that is readily actionable.

In this article we briefly describe each of these features and the rationale for their incorporation into the DLM. We also describe tools to support implementation of the DLM alternate assessment system, including professional development to support instruction and a technology platform to manage and deliver assessments. We conclude with a discussion of potential implications of DLM implementation for policy and practice, including improved outcomes for students with significant cognitive disabilities.

FINE-GRAINED LEARNING MAPS

Since the early 1990s large-scale assessments used by state departments of education for accountability purposes have increasingly been based on academic content standards that specify what children should know and be able to do by the end of each school year or grade span. This has been uniformly so in the United States since the passage of the No Child Left Behind Act of 2001. However, the structure of learning is more complex than a simple list of standards. Cameto, Bechard, and Almond (2012) described various types of cognitive learning models, which include fine-grained learning maps:

Cognitive learning models are distinguished from a scope and sequence, pacing guide, or a curricular progression based on end-of-year standards in that they are developed based on research syntheses and conceptual analyses (Smith, Wiser, Anderson, & Krajcik, 2006). Learning models (such as learning progressions) articulate a central theory of learning, visually and verbally representing hypothesized pathways to increased understanding of the learning targets (Hess, Kurizaki, & Holt, 2009), articulating successively more sophisticated ways of thinking (Wilson & Bertenthal, 2005). Unlike standards, learning models reflect the systematic consideration of interactions among the learner, the content, and the context for learning (e.g., situational, sociocultural, nature of support/scaffolding), as well as the dynamic, cumulative outcomes of these interactions. (p. 6)

Learning maps can embed these critical learning targets and additionally reflect the systematic consideration of interactions among the learner, the content, and the context for learning, as well as the dynamic, cumulative outcomes of these interactions. Currently, there are few references to the use of fine-grained learning maps for the development of assessments employed for accountability purposes, but this concept is central to the evidence-centered design approach used by the DLM Alternate Assessment Consortium.

Learning map descriptors can be specified at various grain sizes that range from discrete skills and knowledge typically developed in a linear manner (e.g., recognizing letters and sounds before decoding words) to broader networks of related concepts and skills, developed in tandem en route to mastery. Fine-grained learning maps identify small steps in the learning pathways that allow for observations of progress. For example, the description of a pathway leading to learning how to differentiate a fractional part from the whole could include partitioning. A fine-grained map would specifically identify the various models that can be partitioned, such as length, sets, and area, rather than assuming it is a singular skill. Grain size can also be a function of content area, as mathematics is generally seen as having descriptors of a smaller grain size while English language arts tends to be more coarsely grained (Wiliam, 2011).

For reasons beyond the scope of this article, we know that statistical modeling of learning, such as Bayesian networks, benefits from the use of acyclic directed graphs—maps where the pathways do not loop back on themselves (Almond, DiBello, Moulder, & Zapata-Rivera, 2007). That is, one skill supports the learning of a second skill, but both skills are defined at a sufficiently small grain size so that the second skill will not also support learning of the first skill. As illustrated in the upcoming discussion of nodes, competence can develop along multiple pathways to reach the same understandings, and we are learning that some pathways will be followed more often than others; these typical pathways evidenced by research provide the basis for developing learning maps. Use of a small grain size helps support this desired statistical feature.

Learning maps not only specify content, but because they reflect a synthesis of existing research on the relationships and learning pathways among different concepts, knowledge, and cognition, learning maps also provide a framework that better supports inferences about student learning needs (Bechard, Hess, Camacho, Russell, & Thomas, 2012). For a traditional, standards-based assessment, the key objective is to examine whether or not students are proficient relative to agreed-on expectations at each grade level. A standards-based learning map assessment seeks to identify where a student falls along a learning pathway while also maintaining the ability to determine whether a student is proficient on a given grade-level standard. Therefore, learning maps represent a framework for developing meaningful assessments, allowing both large-scale and classroom-based assessments to be grounded in models of how understanding and learning develop in a given domain (Alonzo & Steedle, 2008). Finally, learning maps provide understandable points of reference for designing assessments for summative, interim, and formative uses that can report where students are in terms of those steps, rather than reporting only in terms of where students stand relative to their peers (Daro, Mosher, & Corcoran, 2011).

There is an approach to statistical inference that is very flexible in interpreting data from learning maps—variously known as Bayesian network analysis (Almond, Mislevy, Steinberg, Yan, & Williamson, 2015) or Diagnostic classification modeling (Rupp, Templin, & Henson, 2010). By tagging items within testlets to the appropriate nodes on the map, those items become part of the network. The details of this approach are beyond the scope of this article, but using the aforementioned method, responses to items allow inferences to be made about mastery of individual nodes and networks of nodes within the map.

Figure 1 shows a portion (less than one third) of the pre-K–12 mathematics learning map developed at the Center for Educational Testing and Evaluation. The mathematics map currently has 2,395 nodes and 5,147 connections, which reflect knowledge, skills, and aspects of cognition related to mathematics and the multiple pathways that connect the nodes. The English language arts map has 1,526 nodes and 3,524 connections. The cognitive foundations portion of the map has 140 nodes and 254 connections. These nodes specify the cognitive and communicative foundations required for students to engage in instruction in grade-level-aligned academic content. The assumptions underlying these learning map pathways are hypotheses about learning to be tested and refined with data.

Figure 1. A portion of the DLM alternate assessment mathematics map.

[39_21546.htm_g/00002.jpg]

A fine-grained learning map provides a great advantage in measuring growth, especially growth within short periods of time or for students who learn more slowly or idiosyncratically than the typical learner. A learning-map framework holds promise for overcoming a longstanding problem in the measurement of growth—low reliability of difference scores for individuals. And when considered as the foundation for assessments that require data aggregation above the classroom level and for high-stakes decisions, researchers acknowledge that this is a new frontier (Alonzo, 2007). However, the use of learning maps helps to realize a vision of a cohesive, comprehensive system of assessment.

A SUBSET OF PARTICULARLY IMPORTANT NODES THAT SERVE AS CONTENT STANDARDS AND STRUCTURES TO SUPPORT THEM

Crucial to the success of fine-grained learning maps is the selection of important nodes that serve as content standards accompanied by the selection of nodes that build the knowledge, skills, and abilities (KSA) required to achieve the content standards. This process results in a neighborhood of nodes that forms a local learning progression toward a specific learning target (see Figure 2).

Figure 2. Nodes related to identifying the feelings of the characters in a story.

[39_21546.htm_g/00004.jpg]

Without these important neighborhoods of nodes, learning maps would leave the door open to the possibility that teachers would approach instruction at the molecular level and focus only on one small node until it was mastered before moving on to the next node. A node-by-node approach may impede student ability to integrate and apply KSA as they are acquired. For example, teachers taking a molecular approach to beginning reading instruction might teach each letter of the alphabet to mastery before moving on to the next, or focus on mastery of individual words before focusing on comprehension and providing opportunities to interact with connected text.  


In medicine it is important to understand disease at the cellular or even molecular level, but doctors do not try to cure a disease one cell at a time until all cells are healthy. In learning and medicine, it may be easier to teach a single skill or cure a single cell, but doing so may come at the expense of achieving the desired long-term outcome. Focusing on a neighborhood of particularly important nodes in instruction allows students to make connections to previously integrated ideas and eventually form a coherent set of relationships that support generalization to variant and new scenarios.

The neighborhood approach takes into account the fact that not all nodes are created equal. For example, the nodes at the center of each neighborhood that reflect the content standards, called Essential Elements in DLM, are clearly more important than some of the supporting nodes in the neighborhood. As such, these more important nodes should be the primary targets of instruction, while other nodes should be addressed to the extent that they help students master the target nodes that serve as content standards.

The neighborhood approach also takes into account the fact that not all students make progress toward content standards in the same way. Mapping out a neighborhood of nodes that lead to and extend from content standards allows for the identification of the multiple and alternate pathways some students may follow. Figure 3 shows a section of the English language arts learning map that deals with constructing understandings of text. The nodes with red outlines are those that reflect the DLM Essential Elements or content standards. Take note of the multiple pathways available to move from one content standard node to another.

Figure 3. Section of the DLM ELA Learning Map for the conceptual area, construct understandings of text.

[39_21546.htm_g/00006.jpg]

Identifying the connections or pathways that lead from one node to the next is a unique feature of learning maps. Traditional learning progressions have an inherent sequence along which students move incrementally toward the intended target (Heritage, 2008). Traditional learning progressions can be extracted from a learning map, but the map goes beyond the progression by also depicting the multiple and alternate ways that students might progress toward the intended target. For example, the traditional, research-based learning progression that most students follow on the way to decoding single-syllable words with a consonant-vowel-consonant (CVC) structure includes the development of an awareness of words within spoken sentences, syllables within spoken words, and sounds within spoken syllables and words. That is followed by the development of the ability to segment and blend sounds and identify sound–symbol relationships until the student finally reaches the target of decoding written CVC words. While this is the most common progression, students can take other pathways. For example, some students first learn to identify whole printed words by sight before learning to identify the individual sounds in those words to decode new words by analogy. Other students, such as those with hearing loss or complex communication needs, achieve the target of decoding CVC words through a pathway that includes recognizing patterns in their environment, identifying letters, and recognizing spelling patterns in words. This visual, as opposed to sound-based recognition of spelling patterns, eventually leads students to the target of decoding unfamiliar CVC words. In a learning map, each of these possibilities is mapped through multiple and alternate pathways that link one node to another.

These multiple and alternate pathways can be found within learning map neighborhoods that are small, reflecting only single content standards and the nodes supporting and extending from them (as depicted in Figure 2), or collections of related content standards and the nodes that support and extend from them (as depicted in Figure 3). In DLM, the neighborhoods that reflect single content standards can be displayed in mini-maps and described in detail at the level of the nodes that comprise them. In contrast, the larger neighborhoods are too complex to visually depict in a manageable map view or describe on a node-by-node basis. Instead, the larger neighborhoods are defined based upon a series of claims and conceptual areas. The claims are overt statements of what students are intended to learn as a result of mastering skills within a very large neighborhood of the map. Conceptual areas are nested within claims and are comprised of multiple conceptually related content standards and nodes that support and extend beyond them. This system of claims and conceptual areas serves to organize a map that is otherwise too complex to understand.

While serving to organize the learning maps, the claims and conceptual areas are important to modern test development approaches, such as evidence-centered design. These approaches agree that test design must start with the claims you want to be able to make and then the evidence you will need to support such claims. In DLM, we created four broad claims in both ELA and mathematics and subdivided each set of four claims into nine conceptual areas. Table 1 presents these claims and conceptual areas.

Table 1. DLM alternate assessment claims and conceptual areas.

Subject

Major Claims

Conceptual Areas

Mathematics

Students demonstrate increasingly complex understanding of number sense.

Understand number structures (counting, place value, fraction)

Compare, compose, and decompose numbers and sets

Calculate accurately and efficiently using simple arithmetic operations

Students demonstrate increasingly complex spatial reasoning and understanding of geometric principles.

Understand and use geometric properties of two- and three-dimensional shapes

Solve problems involving area, perimeter, and volume

Students demonstrate increasingly complex understanding of measurement, data, and analytic procedures.

Understand and use measurement principles and units of measure

Represent and interpret data displays

Students solve increasingly complex mathematical problems, making productive use of algebra and functions.

Use operations and models to solve problems

Understand patterns and functional thinking

English Language Arts

Students can comprehend text in increasingly complex ways.

Determine critical elements of text

Construct understandings of text

Integrate ideas and information from text

Students can produce writing for a range of purposes and audiences.

Use writing to communicate

Integrate ideas and information in writing

Students can communicate for a range of purposes and audiences.

Use language to communicate with others

Clarify and contribute to discussion

Students can investigate topics and present information.

Use sources and information

Collaborate and present ideas


Multiple levels of organizational structure connect a learning map to different intended uses. In addition to developing claims and conceptual areas as one important means of organizing the learning maps in DLM, the nodes within the neighborhoods that represent single achievement standards are organized into linkage levels. Earlier, Figure 2 showed a neighborhood of nodes that are associated with an achievement standard, identifying the feelings of characters in a story. The nodes in Figure 2 are from the conceptual area depicted in Figure 3, Construct understandings of text. The nodes within Figure 3 are organized into linkage levels. A linkage level contains one or more nodes that represent critical points in the development of the KSA represented in the achievement standard, or Essential Element in DLM. For test development purposes, in DLM we refer to these levels as follows: initial precursor, distal precursor, proximal precursor, target, and successor.

The target linkage level is the node or collection of nodes associated with the Essential Element. In Figure 2, the fourth box represents the Essential Element Can identify the feelings of characters in a story. Each of the other nodes is linked to that Essential Element, but represents a different level of KSA related to the standard. The earliest linkage level, the initial precursor level, extends back to the portion of the map that represents the foundational cognitive skills aligned to the grade-level Essential Element. The distal and proximal precursor linkage levels represent critical waypoints that lead toward the target Essential Element, and the successor node extends beyond the Essential Element toward the general education grade-level standard. In DLM, each testlet is comprised of items that align with nodes at a single linkage level, and students are dynamically routed to testlets at higher or lower linkage levels based upon their performance.

INSTRUCTIONALLY EMBEDDED ASSESSMENTS

Summative year-end assessment can be useful, but if our goal is to maximize student learning, it is not sufficient (or even particularly useful) to gather information one time at the end of the year. Year-end assessment occurs too late and is too disconnected from instruction for results to inform teaching and learning throughout the academic year. At best, year-end assessment can provide general feedback about curricular and instructional efficacy. Moreover, research on instructional sensitivity strongly suggests that most items on end-of-year assessments are not impacted by the quality of instruction, or even whether instruction has occurred at all (Chen & Kingston, 2013; Longabach, Chen, & Kingston, 2013).

Hattie and Timperley (2007) synthesized thousands of effect sizes about the efficacy of feedback used in classrooms and developed a four-level model to explain their findings. They showed that effective use of feedback is among the most powerful techniques for improving student learning. Unfortunately, research on formative assessment has not shown results as positive as those of Hattie and Timperley (Kingston & Nash, 2012). This is likely due to both a lack of good studies and many low-quality, unfocused assessment and feedback processes masquerading as formative assessment. Anyone can call anything a formative assessment, and most commercial test publishers do, such as pre-packaged off-the-shelf tests that are created in the absence of curriculum and instruction (Popham, 2006).

Montessori provides an example of an approach to the ongoing use of feedback that improves student learning (Lillard, 2012); however, Montessori education does not include standardized or formal approaches to assessment. Instead, Montessori employs a stockpile of standardized instructional activities from which the most appropriate ones are chosen by the student or the teacher. Each activity provides ongoing feedback opportunities. Assessment in Montessori is low key, constant, and most importantly, integrated with instruction. It occurs far more than two to three times a week, and when it is implemented with high levels of fidelity, the outcomes for children are excellent (Lillard, 2012).

As in a Montessori classroom, in an assessment system based on dynamic learning maps, assessment must occur as a regular part of instruction. While best practice as to the frequency of instructionally embedded assessment is not yet known (see Kingston & Nash, 2011 and 2012, regarding the inadequacies of the empirical research base on formative assessment), we suggest that a short, focused, structured assessment activity one to three times a week might be optimal. There has been public concern about the frequency and amount of testing in the schools (e.g., Rose & Gallup, 2007), but we would argue that the appropriateness of the frequency depends on the nature of the assessment activity and whether it intrudes on, interferes with, or supports instruction. While currently there is an insufficient research base to identify an optimal frequency, this question is in our research agenda for the next several years.

The DLM Alternate Assessment offers an instructionally embedded approach to assessment that is intended to be every bit as low key, constant, and integrated into instruction as the assessments found in good Montessori classrooms. It also offers an end-of-year assessment that, either separately or in combination with the instructionally embedded assessment, will be used to meet the requirements of accountability systems. The consortium will do research on the quality and comparability of the two approaches, and if successful, states will have a choice of using an instructionally embedded assessment instead of a year-end assessment for their accountability systems.

INSTRUCTIONALLY RELEVANT TESTLETS

Instructionally embedded assessment should also be instructionally relevant—as seamlessly connected with instruction and instructional goals as possible. A less comprehensive version of instructional relevance was espoused by Lindquist (1958, as cited in Schmeiser & Welch, 2006), who said the following:

While they have not generally done so in the past, wide-scale scholarship and college entrance testing programs can make a significant contribution to these basic educational needs. By providing appropriate types of examinations the programs can give the students a concrete and immediately effective incentive to work harder at the job of getting ready for college. To serve this purpose, the examinations must measure directly the student’s readiness for college experience. That is, they must measure as directly as possible his ability to perform exactly the same kinds of complex tasks that he will have the occasion to perform in college and in his later intellectual activities in general. The examination should therefore consist in large part of requiring the students to interpret and evaluate critically the same kinds of reading materials that he will have occasion to read and study in college, and particularly, that will require him to do the same kinds of complex reasoning and problem solving. (p. 308)

Lindquist talked about cognitive relevance, but there are other aspects of instructional relevance. Instructionally relevant items should model good instructional activities to the extent that teachers should want to use them even if no formal assessment were going on. The major difference between instructionally relevant assessment and normal instructional activities should be the systematic collection and computer-assisted analysis of data.

Instructionally relevant assessment takes different forms depending on the population of students and concepts being taught. However, an instructionally relevant assessment begins by linking related items together into meaningful groups called testlets. To serve the three following purposes each instructionally relevant testlet begins with a short unscored engagement activity:  

Foster student motivation;

Form a base with which the student can reinforce recent learning anchored to pre-existing knowledge, skills, or understandings; and

Provide a context that will be used for the subsequent tasks in the testlet.


Directions will first guide the test administrator through the completion of the engagement activity and then the rest of the testlet. In the English language arts testlets that address reading and language, the engagement activity is a shared reading of a text written for the assessment. In the English language arts testlets that address writing, the engagement activity involves choosing a topic to write about. In math, the engagement activity requires the test administrator to provide the students with an opportunity to explore the objects that will be used in the testlet itself.

Grouping items into a testlet that follows the engagement activity is important to support the use of engagement activities, but also on its own accord. Engagement activities are too time inefficient to have a separate engagement activity for every single item. Moreover, using single items does not sufficiently reinforce the learned concepts. A single item is like a disconnected factoid—by itself it cannot be instructionally relevant. It takes at least a small set of items or tasks to be instructionally relevant. The items in a set reinforce one another as they reinforce recent learning.

In summary, instructionally relevant assessment should consist of activities a teacher would have wanted to do for purely instructional purposes, combined with the systematic gathering and analysis of data.

ACCESSIBILITY BY DESIGN (VS. ACCOMMODATION)

Accessibility is a desired characteristic of testing by which students with various physical, cognitive, sensory, or linguistic barriers are provided the opportunity to demonstrate the KSA intended for measurement (Almond et al., 2010). As such, accessibility is a prerequisite to validity, the degree to which a test score interpretation is justifiable for a particular purpose and supported by evidence and theory (AERA, APA, & NCME, 2014).

From the beginning, the DLM system was designed with accessibility in mind to support both learning and assessment. Students must understand what is being asked in an item or task and have the tools to respond in order to demonstrate what they know and can do (Karvonen, Bechard, & Wells-Moreaux, 2015). There are three aspects of accessibility addressed in DLM design: (a) accessible content, (b) accessible delivery via technology, and (c) dynamic routing. While these aspects are important for all children, they are especially important for students with significant cognitive disabilities (SCDs) because of the extreme educational challenges they are already facing.

Accessible Content

The first aspect of DLM assessment design is accessible content. Accessible content is essential to the success of the student with SCDs. As described above, DLM has used the fine-grained learning maps to differentiate content complexity by developing testlets at various linkage levels. In addition, DLM item writing guidelines are based on universal design for learning—a well-researched framework for guiding education practice that provides flexibility in the ways students respond or demonstrate knowledge and skills, as well as in the ways students are engaged (Dolan & Hall, 2007). Using these guidelines, items are constructed to allow students to access the content and response options using a variety of modalities.

Accessible delivery via technology. Second, personalized access based on the needs and preferences of each student with SCDs within the DLM-AA is an intentional and integral component of the DLM-AA system’s technology design and specifications. Since all students taking DLM assessments have disabilities, all are eligible for the system’s built-in accessibility features. Everything listed on the student’s IEP as a testing accommodation should be implemented as part of the Personal Needs and Preferences (PNP) profile; however, the PNP settings encompass more than just IEP accommodations. The settings also include preferred adjustments (for example). Although there may be rare circumstances when a testing accommodation that is not already integrated into the DLM-AA system is needed, the DLM-AA system has been designed and developed to eliminate the need for these testing accommodations. Consider the general statewide assessment administered via paper and pencil. Previously, a student with low vision was provided with a large-print version of the paper test. This accommodation had to be previously arranged so that the large-print version of the test could be printed, shipped, and available for the student with low vision at the time of testing. In today’s online, computer-based testing environment, the text on the screen can be magnified automatically based on the information recorded in the student’s PNP. Virtual tools and features embedded in online assessments become part of the environment rather than post-hoc accommodations. It is important to acknowledge that although digital tools enhance accessibility so that assessments can provide more valid evidence of the knowledge, skills, and abilities of students with SCDs, the DLM design includes systematic practices, research plans, and quality-assurance processes to ensure that accessibility features and supports do not threaten the construct of the test or provide differential advantages within the test.

Dynamic Routing

Finally, technology enriches the interaction between the student and the content by delivering a special user interface to allow for dynamic routing. After the educator completes the Access Profile (PNP plus First Contact information regarding the student’s communication and academic abilities), the system uses that information to route the student to a first testlet that provides an appropriate balance of accessibility and challenge for that student. Continued student performance in the system helps refine the system’s recommendations (Wells-Moreaux, Bechard, & Karvonen, 2014). Most models of adaptive testing assume that the order of item difficulties is the same for all students. Dynamic routing uses (1) the structure of the learning map, (2) conditional node-mastery probabilities (the probability of a correct response to an item that relates to a new node, given the nodes on which a student has previously demonstrated mastery) based on the set of participating students, and (3) the correctness of a student’s responses to previously administered testlets to determine which testlet to administer next.

The power of dynamic routing greatly increases immediate instructional use for an instructionally embedded assessment in a learning map environment, as discussed below. Dynamic routing also allows some testlets to be avoided and alternate testlets to be delivered to students with specific needs. For example, a student who is blind would not receive a mathematics testlet that requires vision to analyze a graph on the screen. Instead, that student would receive a testlet that provides an offline task using manipulatives set up and delivered by the test administrator, with a script and scoring rubric.

In conclusion, what is necessary for both learning and assessment (and, of course, instructionally embedded assessment) is accessibility. Students must have the tools necessary to understand what is asked in the item or task and to demonstrate what they know and can do. Students must have these tools in all aspects of their lives, and that includes when they prepare to take a test. Additionally, if the assessment is based on a learning map and items are carefully crafted to avoid construct-irrelevant aspects, then accessibility features will not alter the construct.

REPORTING THAT IS READILY ACTIONABLE

In order for educators to leverage the fine-grained learning map to maximize student growth, they must have access to information about student learning relative to the learning map. In the majority of large-scale assessment programs, results are based on scaled scores that represent a student’s performance in the subject as a whole. A student’s status might also be expressed in subdomains of the larger subject (e.g., numbers and operations within mathematics). In the DLM alternate assessment system, results are calculated by starting with the smallest grain size and building up. This scoring model supports the design of reports that can be immediately used to guide instruction. As the system calculates a student’s probability of mastering every node in the learning map, higher probabilities (e.g., .80 or higher) indicate a greater likelihood of mastery. In the DLM system, a threshold is applied to identify a probability that is high enough to be considered “mastery.” That information is then combined across nodes within a linkage level to determine whether a student has mastered the linkage level.

Linkage-level mastery is the unit of information that is conveyed in DLM progress reports. In the progress report, Essential Elements are grouped together by conceptual area (see Figure 4). The contents of nodes at each linkage level are synthesized into a short description. Levels that a student has mastered are shaded in green and the most recent assessment date is provided so the reader knows when the student mastered that level. Yellow shading is used to indicate an instructional goal the teacher has recorded in the KITE system. Descriptions are provided for all five linkage levels so the reader can see the pathway toward the target. Teachers may generate a progress report on demand and use it to discuss student progress and instructional goals with parents or other educators.

Figure 4. Sample of a portion of a progress report.

[39_21546.htm_g/00008.jpg]

Because large-scale alternate assessments are also intended to meet accountability needs, DLM summative score reports are designed to convey the required classification (i.e., proficient/non-proficient) to help guide annual instructional planning. The summative report includes two components. The learning profile shows all linkage levels mastered for all Essential Elements assessed that year. It looks much like the progress report in Figure 4, but without mastery dates and without instructional goals. The performance profile aggregates information about linkage-level mastery within the conceptual area and the overall subject. To meet accountability requirements, overall performance is categorized as emerging, approaching the target, at target, or advanced; at target is considered proficient for accountability purposes.

Unlike typical large-scale assessment score reports, the DLM performance profile also includes brief narrative statements about the student’s mastery in each conceptual area. For example, here is a description of a student’s performance in the conceptual area Determining critical elements of text:

Susie is interested in shared reading. Susie understands actions that are part of routines familiar to her. Susie understands that words have meanings that relate to people and objects around her. Susie can identify characters’ feelings and illustrations in familiar texts.

Educators and parents who participated in focus groups about the prototype DLM score reports appreciated the focus on strengths rather than deficits and saw potential for these statements to inform the IEP-development process (Clark, Karvonen, Kingston, Anderson, & Wells-Moreaux, 2015).

Currently, DLM reports are static views. Eventually, the goal is to move to more dynamic online displays that show segments of the map and more fine-grained information about node mastery. In these displays, nodes are color coded to indicate mastery with hue (red through green indicates non-mastery through mastery), value (lighter shades indicate non-mastery and darker shades indicate mastery), and probabilities of mastery value for people who prefer quantitative information. Figure 5 presents a screen shot of a prototype dynamic report. The actual dynamic report would allow a user (teacher, parent, or student) to search the map as well as condense and expand it.

Figure 5. Prototype dynamic report showing mastery probability of nodes in a section of a learning map.

[39_21546.htm_g/00010.jpg]

FINE-GRAINED LEARNING MAPS IN PRACTICE

Assessment and instruction based on a learning map is very different from what many students with SCDs receive today. Instead of assessment focused on examining whether or not students are proficient relative to agreed-on expectations at each grade level, and instruction focused on mastery of skills identified through a task analysis of those expectations (Collins, 2012), learning-map-based assessment and instruction focus on integrated, conceptual development that builds over time. Because the learning map is multifaceted and complex, technology is required to realize its potential in assessment and instruction.

DLM uses a proprietary technology platform, KITE®, to support teachers using the learning map to guide assessment and instruction. KITE has two main components: Educator Portal and the Test Delivery Engine. Educator Portal allows teachers to register students, provide critical information regarding students’ access needs, present levels of academic and communication success, as well as monitor student progress. KITE’s Test Delivery Engine allows students to perform instructionally embedded tasks throughout the year and take a summative test at the end of the year. Most students enrolled in the alternate assessment interact directly with the computer or through assistive technologies that they already use. Others need support from teachers or paraprofessionals to use the computer. An even smaller group of students enrolled in the alternate assessment are still developing intentional and/or symbolic communication and do not use the computer directly. Instead of these students interacting with KITE’s Test Development Engine, educators who are familiar with the students follow instructions on the screen to administer the testlet, observe the students’ responses, and record the responses in the system.

Whether a student interacts with the computer directly or interacts with a teacher who then enters the student’s responses into the system, the system recommends next steps each time a testlet is completed in the instructionally embedded assessment. Depending on the student’s performance, the computer recommends a next Essential Element or linkage level for instruction. The teacher may accept that recommendation or choose a different Essential Element or linkage level for instruction. In either case, the teacher then initiates instruction and, when the teacher feels the student is ready to be assessed, the appropriate next testlet will be ready for administration.

Helping teachers plan instruction that addresses the many available Essential Elements and linkage levels that might be assessed at each grade level requires a comprehensive system of training and professional development. Teachers must have access to training that teaches them how to use Educator Portal to accurately record student access needs and present levels of performance in academics and communication. Training must also teach educators to think about accessibility rather than accommodations, and how to select appropriate Essential Elements and linkage levels for instruction. In addition, professional development must support teachers in learning about the organization of the learning maps into claims and conceptual areas, the domains of ELA and mathematics, and specific teaching strategies that support the integrated, applied learning required by the DLM Essential Elements and college and career readiness standards in general.

As a primary mechanism to address the needs of teachers, DLM provides on-demand, online, interactive professional development. These self-directed modules focus on a broad range of topics including understanding learning maps, claims and conceptual areas, universal design for learning, symbols and communication, and specifics regarding instruction, administration of the DLM assessment, and use of the DLM technology. These self-directed modules are also available in facilitated versions that support face-to-face professional development or small-group study. The facilitated modules combine video-delivered content with prepared interactive activities that allow groups of educators to work together in making the shifts required by DLM. Between a series of required training modules focused on test administration and a collection of professional development modules focused on instruction, there are approximately 55 modules available for teachers.

In addition to providing important information through the modules, DLM has created a virtual community of practice to support educators in connecting with others from across the consortium. Educators and therapists who are deeply involved in the development of the DLM system and have years of experience working with students with SCDs facilitate this virtual community of practice. Furthermore, these educators and therapists create instructional materials, communication supports, and other resources to help educators as they work to teach and assess students with SCDs.

Consider what this might look like in practice. Imagine a middle school special education teacher and his student with SCDs. The student spends a portion of his day in the general education classroom learning academic content alongside his peers. The special education teacher works with the general education teacher to identify Essential Elements that align most closely with the instruction planned for the general education classroom. The special education teacher then uses Educator Portal to select and assign those Essential Elements to the student and identify the appropriate linkage levels. Together, the teachers complete a facilitated module from the professional development system that addresses instruction targeting the Essential Elements they selected. Of course, the general education teacher knows how to address the needs of the diverse students in her classroom, but the module helps her extend that knowledge to include the needs of the student with SCDs at the appropriate linkage levels. As the general education teacher wraps up several days of instruction focused on the selected area, the special education teacher uses KITE to administer a testlet to the student with SCDs. When the student completes the testlet, KITE suggests another Essential Element as a next step in instruction. The special education teacher makes note of the suggestion and brings it to the general education teacher to see how well the suggestion aligns with the planned next steps in the general education classroom. If it aligns, then the special education teacher confirms the Essential Element, accepts the suggested linkage level or selects an alternative, and refocuses instruction accordingly.  

ENVOI

This article started by describing the history of alternate assessment and the serious problems facing all large-scale assessments, especially those embedded in accountability systems. We then proposed six features that could be used to create a better assessment system—one that will support improved student learning. The DLM alternate assessment system is implementing a program based on these tactics. We hope that our early successes will influence the design of other assessment systems, but of course more research is necessary. Over the next several years we will conduct studies of the technical characteristics and instructional consequences of this system.

The DLM theory of action (see Appendix) presents nine short-term outcomes, five intermediate-term outcomes, and four long-term outcomes. Success of the first short-term outcome, “Instructionally embedded and year-end versions of the DLM assessment are implemented in member states,” has been achieved. For the other eight short-term outcomes, data will be gathered and level of success will be documented throughout the next year. The intermediate and long-term outcomes will take longer.

Admittedly there is a subversive (in a good sense) aspect to our work, not fully illuminated in the theory of action. Beyond our hope that by providing useful tools for educators, we will help teachers improve the learning of students with significant cognitive disabilities, we also hope that teachers will recognize that these tools make it easier to provide personalized instruction to students by assisting teachers in determining and supporting the different learning pathways that are optimal for each and every student. We hope the use of these tools will eventually encourage greater inclusivity in all classrooms. Ultimately, we hope this one small step for educators will become a giant leap for all students. Early informal feedback from educators is positive, but time will tell.

Note

This paper was developed as part of the Dynamic Learning Maps™ Alternate Assessment Project under grant 84.373X100001 from the U.S. Department of Education, Office of Special Education Programs. The views expressed herein are solely those of the authors and no official endorsement by the U.S. Department of Education should be inferred.

References

Almond, P., Winter, P., Cameto, R., Russell, M., Sato, E., Clarke, J., . . . Lazarus, S. (2010). Technology-enabled and universally designed assessment: Considering access in measuring the achievement of students with disabilities—A foundation for research. Dover, NH: Measured Progress, & Menlo Park, CA: SRI International.

Almond, R. G., DiBello, L. V., Moulder, B., & Zapata-Rivera, J.-D. (2007). Modeling diagnostic assessment with Bayesian networks. Journal of Educational Measurement, 44(4), 341–359.

Almond, R. G., Mislevy, R. J., Steinberg, L., Yan, D., & Williamson, D. (2015). Bayesian Networks in Educational Assessment. New York, NY: Springer. 

 Alonzo, A. C. (2007). Challenges of simultaneously defining and measuring knowledge for teaching. Measurement: Interdisciplinary Research and Perspectives, 5, 131–137.

Alonzo, A. C., & Steedle, J. T. (2008). Developing and assessing a force and motion learning progression. Science Education, 93(3), 389–421.

Altman, J. R., Lazarus, S. S., Quenemoen, R. F., Kearns, J., Quenemoen, M., & Thurlow, M. L. (2010). 2009 survey of states: Accomplishments and new issues at the end of a decade of change. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved from http://www.cehd.umn.edu/NCEO/OnlinePubs/StateReports/2009_survey_of_states.htm.

American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

Au, W. (2007). High-stakes testing and curricular control: A qualitative metasynthesis. Educational Researcher, 36, 258–267. doi:10.3102/0013189X07306523 

Bechard, S., Hess, K., Camacho, C., Russell, M., & Thomas, K. (2012). Why should cognitive learning models be used as the foundation for designing next generation assessment systems? (Symposium 2011 White Paper No. 1.1). Menlo Park, CA: SRI International.

Browder, D. M., Wakeman, S. Y., Flowers, C., Rickelman, R. J., Pugalee, D., & Karvonen, M. (2007). Creating access to the general curriculum with links to grade level content for students with significant cognitive disabilities: An explication of the concept. Journal of Special Education, 41(1), 2–16. doi:10.1177/00224669070410010101

Cameto, R., Knokey, A.-M., Nagle, K., Sanford, C., Blackorby, J., Sinclair, B., & Riley, D. (2009). State profiles on alternate assessments based on alternate achievement standards. A report from the National Study on Alternate Assessments (NCSER 2009-3013). Menlo Park, CA: SRI International. Retrieved from http://ies.ed.gov/ncser/pdf/20093013.pdf.

Chen, J., & Kingston, N. M. (2013, November). A comparison of empirical and judgmental approaches for detecting instructionally sensitive items. Paper presented at the Conference on Instructional Sensitivity, Lawrence, KS.

Clark, A. K., & Karvonen, M. (2015). First contact: A census report on the characteristics of students eligible to take alternate assessments. Lawrence, KS: University of Kansas, Center for Educational Testing and Evaluation.

Clark, A. K., Karvonen, M., Kingston, N., Anderson, G., & Wells-Moreaux, S. (2015, April). Designing alternate assessment score reports that maximize instructional impact. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.

Collins, B. V. (2012). Systematic instruction for students with moderate and severe disabilities. Baltimore, MD: Paul H. Brookes Publishing.

Crocco, M. S., & Costigan, A. T. (2007). The narrowing of curriculum and pedagogy in the age of accountability: Urban educators speak out. Urban Education, 42(6), 512–535.

Daro, P., Mosher, F., & Corcoran, T. (2011). Learning trajectories in mathematics: A foundation for standards, curriculum, assessment, and instruction (CPRE Research Report #RR-68). Philadelphia, PA: Consortium for Policy Research in Education.

Dolan, R. P., & Hall, T. E. (2007). Developing accessible tests with universal design and digital technologies: Ensuring we standardize the right things. In L. L. Cook & C. C. Cahalan (Eds.), Large-scale assessment and accommodations: What works (pp. 95–111). Arlington, VA: Council for Exceptional Children.

Flowers, C., Ahlgrim-Delzell, L., Browder, D., & Spooner, F. (2005). Teachers’ perceptions of alternate assessments. Research and Practice for Persons with Severe Disabilities, 30(2), 81–92.

Flowers, C., Browder, D. M., & Ahlgrim-Delzell, L. (2006). An analysis of three states’ alignment between language arts and mathematics standards and alternate assessments. Exceptional Children, 72, 201–215.

Flowers, C., Browder, D., Wakeman, S., & Karvonen, M. (2007, April). Comparison of alignment indices across two alternate assessments based on alternate achievement standards: Performance and portfolio formats. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.

Heritage, M. (2008). Learning progressions: Supporting instruction and formative instruction. Washington, DC: Council of Chief State School Officers. Retrieved from http://www.ccsso.org/Documents/2008/Learning_Progressions_Supporting_2008.pdf.

Individuals with Disabilities Education Act of 1997, 120 U.S.C. •1400 et seq.

Jones, M. H., Jones, B., & Hargrove, T. (2003). The unintended consequences of high-stakes testing. Lanham, MD: Rowman & Littlefield.

Karvonen, M., Almond, P., Wakeman, S., & Bechard, S. (2010, April). A synthesis of ten alignment studies of alternate assessment systems based on alternate achievement standards (1%): Implications for assessment system design and policy. Paper presented at the annual meeting of the National Council on Measurement in Education, Denver, CO.

Karvonen, M., Bechard, S., & Wells-Moreaux, S. (2015). Accessibility considerations for students with significant cognitive disabilities who take computer-based alternate assessments. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.

Kingston, N. M., & Nash, B. (2011). Formative assessment: A meta-analysis and a call for research. Educational Measurement: Issues and Practice, 30(4), 28–37.

Kingston, N. M., & Nash, B. (2012). How many formative assessment angels can dance on the head of a meta-analytic pin: .2. Educational Measurement: Issues and Practice, 31(4), 18–19.

Lillard, A. S. (2012). Preschool children’s development in classic Montessori, supplemented Montessori, and conventional programs. Journal of School Psychology, 50(3), 379–401.

Linn, R. L., Grau, M. E., & Sanders, N. M. (1990). Comparing state and district norms to national norms: The validity of the claim that “everyone is above average.” Educational Measurement: Issues and Practice, 9(3), 5–14.

Longabach, T., Chen, J., & Kingston, N. M. (2013, November). Relationship between test item cognitive complexity, content, and instructional sensitivity. Paper presented at the Conference on Instructional Sensitivity, Lawrence, KS.

No Child Left Behind Act of 2001, Public Law No. 107–110, 115 Stat. 1425 (2002).

Popham, W. J. (2001). The truth about testing: An educator’s call to action. ASCD, Alexandria, VA.

Popham, W. J. (2006). All about accountability/phony formative assessments: Buyer beware. Educational Leadership, 64(3), 86–87.

Quenemoen, R., Thompson, S., & Thurlow, M. (2003). Measuring academic achievement of students with significant cognitive disabilities: Building understanding of alternate assessment scoring criteria (Synthesis Report 50). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved from http://education.umn.edu/NCEO/OnlinePubs/Synthesis50.html.

Rose, L. C., & Gallup, A. M. (2007). The 39th annual Phi Delta Kappa/Gallup Poll of the public’s attitudes toward the public schools. Phi Delta Kappan, 89(1), 33–48.

Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford Press.

Schmeiser, C. B., & Welch, C. J. (2006). Test development. In R. L. Brennan (Ed.), Educational measurement (pp. 307–354). Westport, CT: Praeger Publishers.

Shepard, L. A. (1990). Inflated test score gains: Is the problem old norms or teaching the test? Educational Measurement: Issues and Practice, 9(3), 15–22.

Thompson, S., & Thurlow, M. (2003). 2003 state special education outcomes: Marching on. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved from http://education.umn.edu/NCEO/OnlinePubs/2003StateReport.htm.

Thurlow, M., Elliott, J., & Ysseldyke, J. (1999, April). Out-of-level testing: Pros and cons (Policy Directions No. 9). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved from http://www.cehd.umn.edu/NCEO/onlinepubs/Policy9.htm.

U.S. Department of Education. (2004). Sec. 300.8 Child with a disability. Retrieved from http://idea.ed.gov/explore/view/p/,root,regs,300,A,300%252E8.

U.S. Department of Education. (2005). Alternate achievement standards for students with the most significant cognitive disabilities: Non-regulatory guidance. Retrieved from https://www2.ed.gov/policy/elsec/guid/altguidance.pdf.

Wells-Moreaux, S., Bechard, S., & Karvonen, M. (2014). Accessibility manual for the Dynamic Learning Maps™ Alternate Assessment 2014-15. Lawrence, KS: University of Kansas, Center for Educational Testing and Evaluation.

Wiliam, D. (2011, April). Trends in formative assessment theory and practice in the United States. Paper presented at the Annual Meeting of the American Education Research Association, New Orleans, LA.


Appendix  


[39_21546.htm_g/00012.jpg]




Cite This Article as: Teachers College Record Volume 118 Number 14, 2016, p. 1-30
https://www.tcrecord.org ID Number: 21546, Date Accessed: 1/22/2022 9:44:21 PM

Purchase Reprint Rights for this article or review
 
Article Tools
Related Articles

Related Discussion
 
Post a Comment | Read All

About the Author
  • Neal Kingston
    University of Kansas
    E-mail Author
    NEAL KINGSTON is a Professor in the Educational Psychology Department at the University of Kansas and serves as Director of the Achievement and Assessment Institute. His research focuses on improving large-scale assessments so they better support student learning, especially by using fine-grained learning maps as an organizing structure for formative assessment. Before coming to the University of Kansas in 2006, Kingston worked at several educational testing companies and was Associate Commissioner for Curriculum and Assessment at the Kentucky Department of Education during the early years of the Kentucky Educational Reform Act.
  • Meagan Karvonen
    Center for Educational Testing and Evaluation
    E-mail Author
    MEAGAN KARVONEN is the Director of the Dynamic Learning Maps project and also Associate Director of the Center for Educational Testing and Evaluation. Most of her research focuses on validity, with emphases on issues of fairness, opportunity to learn, and the implications of assessment and accountability policy for students with disabilities and their teachers. Recent publications examined factors associated with students’ access to the general curriculum and with student outcomes on alternate assessments.
  • Sue Bechard
    Dynamic Learning Maps™
    E-mail Author
    SUE BECHARD, PhD, is currently serving as Senior Advisor for the Dynamic Learning Maps™. Previously, she worked at Measured Progress where she directed the Office of Inclusive Educational Assessment and led research and development grants to investigate issues related to large-scale assessment and students in special populations. Prior experience includes state supervisor of special education, professor, and classroom teacher, focusing on standards and assessment for students with disabilities. 
  • Karen Erickson
    University of North Carolina at Chapel Hill
    E-mail Author
    KAREN ERICKSON is Yoder Distinguished Professor and Director, Center for Literacy and Disability Studies at the University of North Carolina at Chapel Hill. A former teacher of children with significant disabilities, her research addresses literacy and communication assessment and intervention for all students including students with complex communication needs and significant cognitive disabilities. 
 
Member Center
In Print
This Month's Issue

Submit
EMAIL

Twitter

RSS