Home Articles Reader Opinion Editorial Book Reviews Discussion Writers Guide About TCRecord
transparent 13
Topics
Discussion
Announcements
 

Accommodation Practices for English Language Learners in States’ Mathematics Assessments


by Mikyung Kim Wolf, Jenny C. Kao, Nichole M. Rivera & Sandy M. Chang - 2012

Background/Context: Testing accommodations have been widely utilized as a way of increasing the validity of content assessments for English language learner (ELL) students. However, concerns have also arisen regarding the appropriateness of accommodation use, including the accessibility and fairness of accommodations. While many states have developed ELL-specific accommodation policies and guidelines, little research has been available on how the accommodation policies are carried out in practice.

Purpose/Objective/Research Question/Focus of Study: The present study investigated two states’ accommodation policies, specifically for the states’ respective large-scale Grade 8 math assessments, and conducted a case study to examine teachers’ understanding of the policies and uses. The study aimed to identify issues to consider for an appropriate use of accommodations and provide useful information for policymakers to improve their accommodation policies.

Research Design: The study utilized a qualitative method employing teacher surveys and interviews. The survey and interview instruments were developed based on previous literature and experts’ feedback. The interview transcripts were coded by two researchers using a systematic coding scheme. Descriptive statistics of the responses were computed to observe trends across and within districts and schools. The results were also compared between states or between ELL and math teachers when applicable.

Conclusions/Recommendations: Despite the limitation of the small sample in this study, the findings of the study offer practical implications for policymakers and educators in the use of accommodations for ELL students. The study found considerable variation reported by teachers with regard to the perception of accommodation decision makers, selection criteria, and the types of accommodations allowed in each state’s math assessment. This variation raised serious concerns regarding the adequacy of the accommodation uses and the comparability of accommodated test results across schools. In this paper, we discussed a number of possible reasons for teachers’ reported difficulty in keeping up with state policies: (a) lack of clear guidelines in making accommodation decisions and implementing accommodations in a standardized way, (b) lack of or limited opportunities in receiving information and communicating about accommodations among decision makers and teachers, and (c) limited resources and logistical difficulties. To support an appropriate use of accommodations for teachers, we recommend that states make efforts to provide comprehensive, operationalized guidelines for ELL accommodations, monitor the use of the guidelines, and hold regular professional meetings for ELL and content teachers.

Policies pertaining to the assessment of English language learner (ELL) students have gained increasing attention as a result of federal mandates to include ELL students into states’ large-scale, standards-based content assessments for accountability purposes. One critical issue in assessing ELL students relates to the validity of inferences made about the students’ knowledge and skills based on content assessments. That is, students’ lack of English proficiency may interfere with demonstrating what the students know and can do in a content assessment. To make assessments valid and accessible to ELL students, a growing number of states have established or modified testing accommodation policies specific to ELL students, distinguishing from ones made for students with disabilities (Rivera & Collum, 2006; Shafer Willner, Rivera, & Acosta, 2008; Wolf, Kao, Griffin, et al., 2008; Wolf, Kao, Herman, et al., 2008).


Accommodations, which are changes to a test or testing environment, are intended to help ELL students reduce any language barriers the students may have when taking a test. However, a widespread concern has developed regarding the appropriate use of accommodations, which may affect the validity of accommodated test results. Recognizing the important role that accommodations play in determining test validity, researchers have examined the state of current accommodation policies and practices and identified common validity issues and challenges in the use of accommodations for ELL students. For example, Rivera, Collum, Shafer Willner, and Sia (2006) examined accommodation policies for all 50 states and the District of Columbia in 2000–2001 and found substantial variations in accommodation policies and practices. The authors’ analysis revealed that most states did not focus on the unique linguistic needs of ELL students, making little distinction between ELL students and students with disabilities regarding test accommodations. This implies a potential validity threat if accommodations were not properly used for ELL students in practice. Further, one plausible reason for the previously mixed results on the effects of accommodations might be related to the policy and practice of accommodation usage. Thus, examining the surrounding policies and practices of ELL students is essential in the validation process for ELL assessment.


A recent study of the accommodation assignment criteria conducted by Kopriva, Emick, Hipolito-Delgado, and Cameron (2007) provided evidence about the importance of examining accommodation practices to inform the validity of accommodations. The researchers discussed the lack of existing guidance for decision makers in assigning specific accommodations for ELL students with different characteristics. Through a series of experimental studies, the researchers found that ELL students who were assigned accommodations based on specific criteria to meet their individual needs performed better compared to ELL students who were assigned accommodations not guided by any consistent decision rules. There was no significant difference between the group with no accommodations and the group with accommodations with inconsistent rules. This study underscores the importance of identifying and utilizing appropriate accommodation selection criteria.


Research shows that teachers may be unaware of the best way to meet the needs of ELLs in the classroom (Yoon, 2008). This lack of knowledge may adversely impact teachers’ attitude and beliefs, with negative outcomes, especially for minority students such as ELLs (Flores & Howard, 2008). Despite the importance, research examining teacher knowledge and the actual practice of ELL accommodations for states’ accountability assessments is limited. The latest policy review studies (Shafer Willner et al., 2008; Wolf, Kao, Griffin, et al., 2008) noted that states have made progress in establishing accommodation policies for ELL students. In a 2006–2007 policy review, Wolf, Kao, Griffin, et al. found that most states listed allowable accommodations for ELL students separately from students with disabilities. However, a gap often exists between policies created at the state level and the practices performed at the local district and school levels. Investigating the actual use of accommodations will offer insight into the validity issues to consider when interpreting accommodated test results. It will also provide valuable information for policymakers and practitioners to consider in developing and implementing accommodation guidelines for ELL students.


PURPOSES OF THE STUDY


The present study examined two states’ accommodation policies for the states’ large-scale, standards-based mathematics assessments in Grade 8 and how the policies were carried out in practice. The study also investigated teachers’ perspectives on the usage and helpfulness of accommodations for ELL assessment through surveys and interviews. The study aimed to identify issues to consider for an appropriate use of accommodations and provide useful information for policymakers to improve their accommodation policies. Specifically, we addressed the following research questions: (a) How varied are the state and school ELL accommodation policies within a given state for the state’s large-scale, standards-based mathematics assessment? (b) How do teachers use the accommodations for ELL students in their state’s standards-based mathematics assessment? (c) What are teachers’ perceptions of the helpfulness of accommodations for ELL students?


To investigate our research questions, we explored the following topics related to the two states’ policies and practices of accommodations for ELL students: accommodation decision makers, accommodation selection criteria, permitted and used accommodations for the state’s large-scale mathematics assessment, accommodations during instruction, policy communication channel, recording practice of the accommodation data, and teachers’ perceptions of the helpfulness of specific accommodations. Each topic is more fully described in its respective section.


ELL ACCOMMODATION POLICIES IN TWO STATES OF THE STUDY


The two states that participated in this study are henceforth referred to as State X and State Y to preserve anonymity. Table 1 illustrates states’ policies related to ELL students’ accommodations, based on information gathered from the states’ public websites, including the states’ ELL accommodation guideline documents. Note that both states allowed local districts to consider their specific needs to make their own decisions in the use of accommodations.


Table 1. Policies on Accommodations for ELL Students in State Standards-Based Assessments in Mathematics for the 2007–2008 School Year by State


Policy

State X

State Y

Decision makers

Team consisting of students, parents, teachers, and school administrators most familiar with the student’s English language acquisition

Team consisting of students, parents, and math teacher

Records/documentation

Form provided by state

Document before administration date (no date specified)

List of acceptable education plans where accommodations can be documented

Document before administration date (date specified)

Allowable indirect linguistic support accommodations

Extra time

Extra time

Breaks during test sessions

Shorter sessions with breaks in between

Test individually administered

Test administered in small groups

Test administered in alternative setting

Test administered in study carrel

specific individual administers the test (e.g., ELL teacher)

Test individually administered

Test administered in small groups

Test taker provided preferential seating

Allowable direct linguistic support accommodations

Bilingual, word-to-word Dictionary or electronic translator

Read and reread aloud directions in English

Read and reread aloud directions in native language

Read aloud entire test (text only, no numerals or symbols) in English

Word-to-word dictionaries

Read aloud directions in English

Read aloud directions in native language

Read aloud entire test in English

Read aloud entire test in native language

Student highlights or underlines Key words or phrases in the directions

Student highlights or underlines key words or phrases in the assessment

Prohibited accommodations

Read aloud entire test in a language other than English  

 

Note. The following practices are considered best practices for administering a test for all students in State Y, and are not considered to be accommodations and do not require documentation: teacher faces test taker, test administered with minimal distractions, person familiar to test taker (e.g., ELL teacher) administers the test, and test administered in familiar room.


METHOD


PARTICIPANTS


The targeted sample included ELL specialists or teachers as well as math teachers who taught Grade 8 ELL students during the 2007–2008 school year. A total of 165 middle school teachers from two states voluntarily participated in this study. In State X, a total of 64 teachers from 19 schools in two large urban districts participated in the study. Thirty-four teachers completed the survey, and 42 teachers participated in the interview. Some of the teachers completed the interview and survey, while others completed only one component. In State Y, a total of 101 teachers from 54 schools in 18 mid-size urban and suburban districts participated in the study. All 101 teachers completed the survey. Of those, 35 teachers also participated in the interview. Table 2 summarizes the number of teachers who participated in the two components of this study by state and by teacher assignment or type (i.e., math teacher vs. ELL teacher).


Although the two participating states do not have the largest proportion of ELL students in the country, the states’ students’ public school enrollment consisted of between 10% and 20% of their total student populations, consistent with average ELL population distributions nationwide. The proportion of Grade 8 ELL students of the participating schools ranged from 7.6% to 41.0% for State X and from 1% to 53.1% for State Y.


Table 2. Number of Participants by State, Teacher, and Participation Type


 

Survey

Interview

State X

(18 schools)*

(18 schools)*

ELL

5

22

Math

29

20

State X Total

34

42

State Y

(54 schools)

(25 schools)

ELL

27

16

Math

74

19

State Y Total

101

35

 Note. *Not the same 18 schools. A total of 19 schools from State X participated. One school did not participate in interviews, and one did not participate in surveys.


INSTRUMENTS


Teacher Survey


The teacher survey scales (i.e., constructs and item scales) were constructed based upon previous research and surveys regarding the use of accommodations and instructional strategies for ELL students (see Abedi, Courtney, Leon, Kao, & Azzam, 2006; Martinez, Bailey, Kerr, Huang, & Beauregard, 2009). The survey included five subscales: instructional strategies, classroom assessment practices, accommodation practices during instruction, experience with accommodations in state testing, and teacher perception of the helpfulness of accommodations. Background questions (e.g., license status, teaching experience, degree) were also included. Some items were asked on a nominal scale; others were asked on a Likert-type scale. The majority of the items were previously examined to ensure the reliability and validity of the scales. The draft of the survey was also reviewed by three former and current teachers as well as two external researchers for the appropriateness of the survey content and the clarity of the items.


Teacher Interview Protocol


The teacher interview protocol was developed to gather in-depth information about schools’ accommodation practices and teachers’ perceptions of the use of accommodations. The questions primarily focused on four topics: knowledge of accommodation policies, mechanism to record accommodation data, previous experience with accommodation usage for state testing administration, as well as accommodation usage during instruction. The interview was designed to be completed in approximately 30 to 45 minutes. (See Appendix for sample survey and interview questions.)


PROCEDURE AND ANALYSIS


Surveys were collected via mail, fax, e-mail, or online. Teacher interviews were conducted one-on-one, either face-to-face or via telephone. All interviews were audio recorded, and then later transcribed.


For reliability of the survey instrument, internal consistency using Cronbach’s alpha was examined for the items on Likert-type scales. Acceptable levels of reliability were obtained; .86 for the items on instructional strategies, .59 for the items on classroom assessment practices, .76 for the items on accommodation practices during instruction, and .81 for the items on teacher perception on the helpfulness of accommodations. Descriptive statistics including the frequencies and means of the responses were computed to observe trends across and within districts and schools. The results were also compared between states or between ELL and math teachers when applicable.


For interview data, a coding scheme was developed and applied by trained researchers to transcripts using Atlas.ti.1 More than 28% of the transcripts were coded by two researchers with 80.1% of exact agreement on average. Where there were disagreements between the two raters, consensus was reached through discussion. Once acceptable reliability was obtained, the remaining transcripts were coded by one researcher each.


RESULTS


Survey results for the first two research questions are presented according to subtopics, across and within states as well as by teacher type (i.e., math vs. ELL teacher). Interview results are also reported where applicable to provide more in-depth information.


ACCOMMODATION DECISION MAKERS


The policies for States X and Y were similar with respect to accommodation decision makers. Both stated that accommodation decisions should be made by a team of people, including teachers most familiar with students’ English proficiency (in State X) or content area teachers (in State Y). To investigate the personnel involved in making accommodation decisions for each individual ELL student in schools, teachers were asked who determined the selection of accommodations for ELL students at their school for the Grade 8 state math assessment. Teachers were given the following options and asked to choose all that applied or specify other personnel or source, if applicable: ELL teacher, math teacher, principal, parents, students, don’t know, and other.


Table 3. Decision Makers for Selecting Accommodations by State: Frequency (Percentage)


Decision Maker

State X

State Y

Principal only

1 (7.4)

1 (1.0)

ELL teacher only

7 (25.9)

9 (9.1)

Math teacher only

0 (0.0)

7 (7.1)

ELL & Math team

4 (14.8)

53 (53.5)

ELL & other team

2 (7.4)

2 (2.0)

Math & other team

1 (3.7)

6 (6.1)

Other team

1 (3.7)

1 (1.0)

Other single source

0 (0.0)

3 (3.0)

Don’t know

10 (37.0)

17 (17.2)

Total

27(100.0)

99 (100.0)

Note. ELL & Math team included both math and ELL teachers. ELL & other team and Math & other team included some personnel and ELL teacher, some personnel and math teacher, respectively. Other included special education teacher, testing coordinator, assistant principal, district personnel, parent, and student.


As shown in Table 3, many teachers in both states responded that the decision was made by a team, typically including an ELL teacher. In State X, however, an ELL teacher was often reported as the sole decision maker. In State Y, a math teacher was frequently reported as being involved in the accommodation decision. A considerable percentage of the respondents reported don’t know to the question about decision makers. The majority of teachers who reported don’t know were math teachers, across both states. While a few of these math teachers reported that less than 5% of the students in their math classes were ELLs, most math teachers in the sample reported that 15% to 95% of their students were ELLs.


ACCOMMODATION SELECTION CRITERIA


Both states’ guidelines suggested that accommodations should meet individual students’ needs. To examine the specific criteria or rules on which schools relied in selecting accommodations, teachers were asked about the sources or criteria used to make this decision for the Grade 8 state math assessment. Teachers were given the following options and asked to choose all that applied or to specify other criteria, if applicable: English language proficiency (ELP) level, student IEP (Individualized Education Plan), state standardized test scores, no specific criteria (e.g., blanket accommodation in which all ELL students receive the same accommodations), don’t know, and other. Table 4 presents the frequency and percentage of the criteria chosen by the teachers. Results indicate a variety of options were utilized, ranging from a single source to varied combinations of multiple sources.  Table 4 also shows that 31.3% of the sample math teachers selected don’t know compared to 9.4% of the sample ELL teachers.


Table 4. Accommodation Selection Criteria by State and by Teacher Type: Frequency (Percentage)


Criteria

State X

State Y

Math

ELL

ELP Level only

4 (14.3)

11 (11.0)

12 (12.5)

3 (9.4)

Individualized Education Plan (IEP) only

5 (17.9)

7 (7.0)

12 (12.5)

0 (0.0)

ELP Level & IEP

4 (14.3)

20 (20.0)

17 (17.7)

7 (21.9)

ELP Level, IEP, & State Test Score

2 (7.1)

3 (3.0)

3 (3.1)

2 (6.2)

ELP Level & Other

0 (0.0)

5 (5.0)

0 (0.0)

5 (15.6)

ELP Level, IEP, & Other

0 (0.0)

8 (8.0)

5 (5.2)

3 (9.4)

ELP Level, IEP, State Test Score, & Other

0 (0.0)

2 (2.0)

1 (1.0)

1 (3.1)

IEP & Other

1 (3.6)

5 (5.0)

5 (5.2)

1 (3.1)

Blanket accommodation

4 (14.3)

5 (5.0)

4 (4.2)

5 (15.6)

Other only

0 (0.0)

9 (9.0)

7 (7.3)

2 (6.2)

Don’t know

8 (28.6)

25 (25.0)

30 (31.3)

3 (9.4)

Total

28 (100.0)

100 (100.0)

96 (100.0)

32 (100.0)

Note. Other criteria included instructional uses, teacher recommendations, teacher observations, performance on other assessments, other types of educational plans, or based on what students received the prior year.


During the interview, teachers mentioned logistic difficulties and lack of resources as the rationale for providing blanket accommodations, as in the following excerpts:


State X ELL teacher: They all get the same accommodations because it’s just easier logistically. Because say, if I gave to all my students the same accommodations, they can all test with me. Whereas if I gave half my students [a] bilingual dictionary to use, and the other half no dictionary to use, or no re-reading the directions, they couldn’t—we’d have to find another testing room for them . . . So my main criteria is, well, if they have me for English, they get to test with me, and they get all the same accommodations.


State Y ELL teacher: I think it’s frustrating because we’re such a highly impacted district. In our situation, [ELL students are] the majority of our school, and we don’t have a lot of extra places to take kids for smaller groupings for different accommodations. And, in addition, the staff is limited for breaking kids up into groups and giving them their accommodations.


PERMITTED AND USED ACCOMMODATIONS


As summarized earlier, both states specified a list of allowable accommodations for their state math assessments. To examine the extent to which this state policy was enacted at the school level, teachers were given a list of accommodations (see Appendix) and asked to identify the types of accommodations permitted as well as the types of accommodations actually used for the state math assessment at Grade 8. The results are summarized for State X and State Y in Table 5 and Table 6, respectively. For both states, results indicate differing knowledge about permitted accommodations among the teachers. For instance, although State X’s policy restricted the use of reading aloud test items in students’ native language for the state math assessment, a sizeable number of State X sample teachers reported that reading aloud test items in the students’ native language was permitted and even used in practice. Similarly, nearly a quarter of State Y sample teachers reported that reading aloud test items in students’ native language was not permitted, while it was permitted according to the state’s policy documents. The results also indicate that this different knowledge may stem from district-level policies compared to the state policy. More than 40% of the sample teachers in both states chose not sure about whether a dictionary was allowed or not for the math assessment.


Table 5. State X Permitted and Used Accommodations for the State Math Assessment: Frequency (Percentage)


Accommodation

Permitted

 

Used

Yes

No

Not Sure

Total

 

Yes

No

Not Sure

Total

Extended time

27

(79.4)

2

(5.9)

5

(14.7)

34

(100.0)

 

22

(66.7)

6

(18.2)

5

(15.2)

33

(100.0)

 

Individually administered

12

(35.3)

2

(5.9)

14

(41.1)

34

(100.0)

 

5

(16.1)

14

(45.1)

12

(38.7)

31

(100.0)

 

Small group

16

(47.1)

7

(20.6)

11

(32.4)

34

(100.0)

 

13

(41.9)

8

(25.8)

10

(32.6)

31

(100.0)

 

Separate location

18

(52.9)

8

(23.5)

8

(23.5)

34

(100.0)

 

14

(46.7)

10

(33.3)

6

(20.0)

30

(100.0)

 

Administered by ELL teacher

22

(78.6)

0

(0.0)

6

(21.4)

28

(100.0)

 

18

(69.2)

1

(3.8)

7

(26.9)

26

(100.0)

 

Directions read aloud

24

(70.6)

6

(17.6)

4

(11.8)

34

(100.0)

 

18

(58.1)

7

(22.6)

6

(19.4)

31

(100.0)

 

Items read aloud

17

(50.0)

7

(20.6)

10

(29.4)

34

(100.0)

 

14

(45.2)

8

(26.7)

9

(30.0)

31

(100.0)

 

Items read aloud in native language

4

(11.8)

17

(50.0)

13

(38.2)

34

(100.0)

 

3

(9.7)

16

(51.6)

12

(38.7)

31

(100.0)

 

Glossary

9

(26.5)

13

(38.2)

12

(35.3)

34

(100.0)

 

4

(13.3)

15

(50.0)

12

(40.0)

31

(100.0)

Dictionary

10

(35.7)

6

(21.4)

12

(42.9)

28

(100.0)

 

9

(36.0)

4

(16.0)

12

(48.0)

25

(100.0)

Electronic translator

2

(5.9)

16

(47.1)

16

(47.1)

34

(100.0)

 

1

(3.2)

16

(51.6)

14

(45.2)

31

(100.0)

Note. In State X, the language of dictionary (i.e., bilingual, English) was not specified in the survey.


Table 6. State Y Permitted and Used Accommodations for the State Math Assessment: Frequency (Percentage)


Accommodation

Permitted

 

Used

Yes

No

Not Sure

Total

 

Yes

No

Not Sure

Total

Extended time

89

(90.8)

1

(1.0)

8

(8.2)

98

(100.0)

 

77

(78.6)

4

(4.1)

17

(17.3)

98

(100.0)

 

Individually administered

58

(61.1)

11

(11.6)

26

(27.4)

95 (100.0)

 

39

( 42.9)

21

(23.1)

31

(34.1)

91

(100.0)

 

Small group

81

(82.7)

5

(5.1)

12

(12.2 )

98

(100.0)

 

73

(74.5)

9

(9.2)

16

(16.3)

98

(100.0)

 

Separate location

72

(75.0)

5

(5.2)

19

(19.8)

96

(100.0)

 

66

(69.5)

7

(7.4)

22

(23.2)

95

(100.0)

 

Administered by ELL teacher

72

(74.2)

4

( 4.1)

21

(21.6)

97

(100.0)

 

60

(63.2)

12

(12.6)

23

( 4.2)

95

(100.0)

 

Directions read aloud

81

(82.7)

1

(1.0)

16

(16.3)

98

(100.0)

 

70

(72.9)

3

(3.1)

23

(24.0)

96

(100.0)

 

Items read aloud

66

(68.8)

9

(9.4)

21

(21.9)

96 (100.0)

 

52

(55.3)

12

(12.8)

30

(31.9)

94

(100.0)

 

Items read aloud in native language

27

(28.1)

22

(22.9)

47

(49.0)

96

(100.0)

 

16

(17.0)

33

(35.1)

45

(47.9 )

94

(100.0)

 

Glossary

8

(8.4)

38

(40.0)

49

(51.6)

95

(100.0)

 

6

(6.5)

47

(51.1)

39

(42.4)

92

(100.0)

 

Bilingual dictionary

30

(31.6)

22

(23.2)

43

(45.3)

95

(100.0)

 

23

(25.0)

30

(32.6)

39

(42.4)

92

(100.0)

 

Dictionary

8

(8.4)

45

( 47.4)

42

(44.2)

95

(100.0)

 

5

(5.4)

50

(53.8)

38

(40.9)

93 (100.0)

Electronic translator

9

(4.5)

42

(44.2)

44

(46.3)

95

(100.0)

 

4

(4.3)

53

(57.0)

36

(38.7)

93

(100.0)


Regarding accommodations that were actually used in practice, extended time and directions being read aloud in English were frequently reported in both states. Test administration by an ELL teacher was the most frequently used accommodation as reported by the majority of State X sample teachers, while test administration in a small group was the most frequently used accommodation reported by State Y sample teachers. Again, the types of accommodations reported used varied depending on the teachers and schools within each state. Teacher interview data revealed that this variation occurred across districts and even within the same district. For example, about half of the sample teachers in one district of State X reported that they read aloud the test in English while other teachers in the same district reported not using this accommodation. This within-district variation was also found in State Y. In one district of State Y, a little more than 40% of the sample teachers reported that they read aloud the test in English.


ACCOMMODATIONS DURING INSTRUCTION


Both previous research and the states’ policies emphasized that the accommodations selected for state testing should be ones in which students are familiar with through instructional practice. To examine the instructional uses of accommodations, teachers were asked to indicate how frequently a specific accommodation was provided to ELL students during classroom tests and instructional tasks, on a 6-point scale with 1 meaning never used and 6 meaning used every day.


Table 7. Accommodations Provided to ELL Students for Grade 8 Math Classroom Tests by State: Mean and Standard Deviation (SD)


Accommodation

State X

State Y

M

SD

M

SD

Extended time

4.37

1.18

3.51

1.14

Individual testing

1.96

1.19

2.01

1.03

Small groups

2.00

1.27

2.26

1.15

Directions read aloud

3.37

1.67

3.37

1.53

Items read aloud

3.00

1.66

2.79

1.47

Items read aloud in native language

1.33

0.88

1.23

0.62

Glossary

2.59

1.76

1.78

1.51

Bilingual dictionary

-

-

1.51

1.23

Dictionary

2.22

2.00

1.65

1.41

Electronic translator

1.30

1.03

1.14

0.69

Note. Rating scale, 1 = never to 6 = every day. Language of dictionary was not specified for State X.


Table 8. Accommodations Provided to ELL Students to Complete Tasks During Grade 8 Math Class by State: Mean and Standard Deviation (SD)


Accommodation

State X

State Y

M

SD

M

SD

Extended time

5.33

0.91

4.74

1.16

Work one-on-one w/teacher

3.78

1.50

4.12

1.39

Small groups

4.07

1.30

5.24

0.70

Problems read aloud

4.85

1.31

5.42

0.91

Problems read aloud in native language

2.10

1.34

1.42

1.03

Glossary

3.11

1.78

3.22

1.72

Bilingual dictionary

-

-

2.22

1.68

Dictionary

2.29

1.65

2.46

1.61

Electronic translator

1.15

0.49

1.36

1.20

Note. Rating scale, 1 = never to 6 = every day. Language of dictionary was unspecified for State X.


Table 7 shows the means and standard deviations of the scale of how often teachers provided accommodations to ELL students during math classroom assessments. Teachers in both states reported extended time most frequently, followed by read aloud of directions. Similarly, results for accommodations provided to ELL students during math class to complete tasks or problems, as shown in Table 8, indicated extended time followed by read aloud of problems as the most frequently reported in both states.


Figure 1. Percentage of teachers reporting never to a given accommodation for ELL students during Grade 8 math classroom assessments.

[39_16302.htm_g/00002.jpg]

Note. Language of dictionary was unspecified for State X.


Teachers often mentioned that they never gave accommodations during classroom assessments. Figure 1 demonstrates accommodations reported as never having been provided by teachers during classroom assessments. Higher percentages of never were reported for some accommodation types, such as use of dictionary and glossary. Roughly three quarters of both State X and State Y reported never providing a dictionary for classroom math assessments.


RECORDING OF ACCOMMODATION USE FOR STATE TESTING


A series of questions regarding the recording practices included teachers’ familiarity with accommodation data records, recording procedures, personnel in charge of recording and maintaining the accommodation, and data accessibility to teachers. Overall, ELL teachers in both states reported being familiar with accommodation recording practices, with 73.7% and 93.8% of ELL teachers in State X and Y, respectively, reporting being at least somewhat familiar with the recording policies. More State Y teachers overall reported familiarity (74.3%), while just a little over half of State X teachers reported familiarity (51.7%).


One of the recording procedure questions was related to the personnel responsible for keeping records of the accommodations given to individual ELL students during the state math assessment. While an ELL teacher or the school’s ELL department was predominantly reported as responsible for the accommodation records in State X, many math teachers in State Y reported that they were involved in the record-keeping procedures. As described in the State Y policy, math teachers were required to document an individualized accommodation plan for ELL students for their math instruction, by a specific date before the state math assessment. Subsequently, math teachers in State Y may have had a tendency to consider themselves the record-keepers for the state math assessment. However, when prompted, they were often not aware of what happened with the documentation plans after the plans left their hands.


Both states’ sample teachers were unclear about procedures regarding the maintenance and accessibility of accommodation data. While about half of the ELL teachers in State X referred to their state’s official accommodations record form, some were uncertain whether those paper forms were later transferred to any electronic database. The teachers’ responses can imply either a nonsystemic method of maintaining accommodation records or limited knowledge about the process.


COMMUNICATION CHANNEL


The interview also asked teachers how they were informed about state and school accommodation policies with the assumption that the means and quality of communication could have an influence on the use of accommodations. The interview questions specifically focused on the clarity with which information on accommodation policies was distributed to teachers, and the ways in which accommodation policy information was delivered to the teachers. Many teachers expressed reservations about whether information on accommodation policies was made clear to teachers. A math teacher in State Y commented on the lack of clarity. A State Y Math teacher observed, “Last year, I feel like it was a hodgepodge. I don’t think it was really clear . . . I think it wasn’t clear what options were possible, how to make sure those people got them.”


Regarding the ways teachers received accommodation information, many teachers in both states mentioned that they had meetings before testing with a school-based or itinerant district testing coordinator and typically received a testing manual including accommodation information. However, whether a discussion specifically focused on accommodation policies for ELL students occurred was unclear. An ELL teacher in State Y remarked that the 2007–2008 school year was the first year the state’s Department of Education distributed a separate accommodations manual specific to ELL students, separate from students with disabilities:


State Y ELL teacher: They [The State Department of Education] put out an accommodations manual for ELL students. So in the past it had kind of been part of the regular accommodations manual. And last year was the first year that they did it separately. And it’s extensive documents. It’s sixty something pages, and there’s charts for every conceivable situation of an ELL student in their previous instruction, whether it was in English or native language and whether they were literate in their native language or not.


State Y math teachers also consistently commented that they received the accommodation information in the beginning of the school year to begin planning and using the accommodations. The teachers attributed this practice to the State Y policy requirement of documenting the accommodation plan by a specific date before the testing as well as for content instruction.


PERCEPTIONS OF THE HELPFULNESS OF ACCOMMODATIONS


The third research question explored teachers’ perceptions of the helpfulness or usefulness of accommodations. In the survey, teachers were given a list of accommodations and asked to rank each one on a scale of 1 to 4, with 1 meaning not very helpful and 4 meaning extremely helpful. To examine whether ELL and math teachers possessed different perceptions of the helpfulness of accommodations, the responses were grouped by teacher type, as shown in Table 9.


Table 9. Teachers’ Perception on the Helpfulness of Accommodations for ELL Students by Teacher Type: Mean (SD)


Accommodation

ELL

Math

Total

n

M
(SD)

n

M
(SD)

N

M
(SD)

Extended time

32

2.91 (0.96)

90

2.96 (0.85)

122

2.94 (0.88)

Test individually administered

21

2.38 (0.81)

82

2.79 (0.90)

103

2.71 (0.89)

Test administered in a small group

29

3.17 (0.89)

89

2.85 (0.82)

118

2.93 (0.85)

Test administered by an ELL teacher

30

3.43 (0.73)

87

2.87 (0.93)

117

3.02 (0.91)

Directions read aloud in English

32

3.22 (0.91)

86

2.86 (0.81)

118

2.96 (0.85)

Test items read aloud in English

29

3.10 (0.90)

81

2.67 (0.95)

110

2.78 (0.95)

Test items read aloud in native language

19

2.89 (1.10)

68

2.91 (1.03)

87

2.91 (1.04)

Glossary

12

1.92 (1.00)

69

2.22 (0.92)

81

2.17 (0.93)

Dictionary (State X only)

5

2.60 (0.89)

23

2.61 (0.99)

28

2.61 (0.96)

Dictionary (State Y only)

8

1.75 (0.89)

47

1.87 (0.88)

55

1.85 (0.87)

Bilingual dictionary (State Y only)

18

2.39 (0.70)

46

2.35 (0.95)

64

2.36 (0.88)

Electronic translator

13

2.00 (1.08)

66

2.27 (1.12)

79

2.23 (1.11)

Note. Dictionary cannot be compared across states, since State X teachers did not have the option of a bilingual dictionary. Score scale of 1 = not very helpful to 4 = extremely helpful.


Results by teacher type indicated that math teachers reported extended time as the most helpful (M = 2.96) and that ELL teachers reported test administered by an ELL teacher as the most helpful (M = 3.43). Math teachers and ELL teachers both reported English dictionary, glossary, and electronic translator accommodations as relatively low in helpfulness.


Teacher responses from interview data regarding accommodation helpfulness were categorized as “positive,” “negative,” or “mixed” in which common themes emerged within the categories. While positive opinions about accommodation helpfulness were related to reducing ELL students’ language barriers, negative opinions were concerned with logistical issues and students’ lack of familiarity. Teachers with mixed opinions considered ELL students’ background when discussing the helpfulness of accommodations. For instance, an ELL teacher mentioned students’ language proficiency levels as factors that affect the helpfulness of a dictionary accommodation for students, as indicated in the following excerpt:


State Y ELL teacher: It depends on their proficiency level because if they feel that they don’t understand anything, then they’re going to look up every word, and [using a dictionary accommodation is] a daunting task. I think most students will just start guessing . . . It’s pretty overwhelming, so I think, their background understanding of the dictionary, practice with the dictionary, and then if it’s appropriate for their fluency level.


To illustrate the common themes on teachers’ perceptions about accommodation helpfulness, a summary of teachers’ general opinions on the helpfulness of read aloud and dictionary or glossary, as examples, is presented in Figure 2.


Figure 2. Summary of teacher perceptions of the helpfulness of read aloud and dictionary/glossary accommodations.


[39_16302.htm_g/00004.jpg]

DISCUSSION


Before the findings of the study are discussed, the nature of this study and its limitations should be noted. Although the study was concerned with statewide policies and their implementation in practice, as a case study, the sample size is limited, and thus, generalization of the findings should be made with caution.


This study investigated how state accommodation policies were communicated to schools and implemented in practice, to identify important validity issues to consider in the use of accommodations for ELL students. The states in our study have made great, initial efforts to provide a systematic, principled way of using accommodations for ELL students, separate from students with disabilities. However, the wide range of variation found between the policies and practices within this modest sample suggests that, while progress has been made in policies for ELL accommodations, additional work must be done in ensuring these policies are appropriately communicated and implemented. We discuss some key findings and their implications in relation to the three research questions (RQs) posited in this study.

 

RQ1. HOW VARIED ARE THE STATE AND SCHOOL POLICIES WITHIN A GIVEN STATE ON THE USE OF ACCOMMODATIONS FOR ELL STUDENTS IN THE STATE’S LARGE-SCALE, STANDARDS-BASED MATHEMATICS ASSESSMENT?


We found that teachers reported considerable variation about perception of accommodation decision makers, selection criteria, and the types of accommodations allowed in each state’s math assessment. This variation raised a question about how adequately accommodations were provided for ELL students across schools. This finding also raised a question about the comparability of accommodated test results across schools.


The interview data suggested that the variation in the use of accommodations may be due to limited guidelines, limited communication of guidelines, or limited resources. For instance, while both states strongly recommended that accommodations should be selected based upon students’ needs, what sources teachers should utilize to identify the students’ different needs were not specified in the state guidelines. Previous studies reviewing the states’ policies indicated that various criteria were used in assigning accommodations across states, including students’ language proficiency, academic performance, instructional service, parental input, and teacher observation, to name a few (Rivera et al., 2006; Wolf, Kao, Griffin, et al., 2008). The present study found that this variation also occurred within the state, and often within the same district. This result was partly due to local districts and schools determining their own accommodation rules for their own needs. Yet the lack of detailed, operationalized guidelines raises a concern as some teachers, even those who were involved as decision makers, reported they had little knowledge of any systematic criteria used in making accommodation decisions.


The state recommendation of selecting accommodations on an individual-need basis was not followed in some schools. Some teachers reported that a blanket accommodation administration, or an accommodation given to all qualifying ELL students regardless of actual need, was inevitable regardless of the status of ELL students, citing logistical challenges and strained resources as reasons for the difficulty of providing tailored accommodations for each student. For instance, several teachers commented on the difficulty of acquiring bilingual dictionaries in every language for their diverse students. Thus, while states listed a number of allowable accommodations, at the school level, a smaller set of accommodations were permitted and feasible.


While the results suggested that the accommodation policies and practices varied among the schools in this study, states’ policies still had a clear impact on the accommodation uses in schools, in general. For example, one of the accommodation policy stipulations in State Y included a strict requirement for documenting individual accommodation plans for instruction and assessment with specific dates before the state testing. As a result, a recurring trend was observed: The sample math teachers in State Y tended to report more familiarity with accommodation policies for ELL students and more involvement in accommodation uses and data recording procedures, compared to the sample State X math teachers. The absence of this requirement in State X seemed to lead to a greater communication gap between ELL and math teachers on the accommodation uses for the state math assessment, which can subsequently result in a gap between the instructional uses and testing uses of accommodations.


Another example was found in the different patterns of accommodation decision makers. State X documents stated that decision makers should include the “teacher or school administrator most familiar with students’ English language acquisition” while State Y documents stated that decision makers should include the “teacher primarily responsible for delivery of instruction in the content area being assessed.” Our finding demonstrated that, in State X sample schools, ELL teachers were often reported as the sole decision maker whereas in State Y sample schools, ELL and math teachers were reported as decision makers.


RQ2. HOW DO TEACHERS USE THE ACCOMMODATIONS FOR ELL STUDENTS IN THEIR STATE’S STANDARDS-BASED MATHEMATICS ASSESSMENT?


While some general patterns emerged regarding the most frequently used accommodations, wide variation was also noted across schools in terms of the actual provision of accommodations. The most frequently used accommodations reported by the teachers of this study in both states were extended time and directions read aloud. Extended time and read-aloud directions seemed to be relatively easily provided to all students without physically separating any specific ELL student or requiring additional resources. This result suggests that the ease of implementation, that is, feasibility and practicality, likely played a key role in selecting accommodations.


The use of other types of accommodations varied widely depending on schools and teachers. For example, results indicated that read aloud of the test was reportedly used by half of the schools in one district, posing the question why other schools in the same district did not provide the accommodation. On a related note, we found that teachers reported different permitted accommodations for the state math assessment even within the same district and within the same school. Surprisingly, a number of teachers reported that they were not sure whether certain types of accommodations were allowable for the state math assessment such as reading aloud the test in students’ native language, glossary, and dictionary. Teachers’ varied knowledge and familiarity with the policies seemed to have an effect on limiting the types of accommodations that the teachers’ actually provided during the state math assessment. This alarming finding emphasizes the need to exercise caution when examining accommodated test scores and aggregating test data.


The varied practice of the accommodation uses across schools is, to some extent, attributed to a lack of regular, systematic communication channels and training meetings for the use of accommodations. While some variation is inevitable due to local decisions, the decisions should be made in a systematic way. To prevent variation resulting from an unsystematic, random use of accommodations, ELL and content teachers must be provided opportunities to engage in discussing the policies and practices of accommodations for ELL students.


RQ3. WHAT ARE TEACHERS’ PERCEPTIONS OF THE HELPFULNESS OF ACCOMMODATIONS FOR ELL STUDENTS?


Teachers’ perceptions of the helpfulness of accommodations for ELL students were mixed. For example, ELL teachers generally rated test administered by an ELL teacher most helpful, while math teachers rated extended time most helpful. Many teachers recognized that different individual students had different needs for accommodations, and thus, evaluating the general helpfulness of accommodations was different. Teachers also pointed out the students’ different language proficiency levels and skills were factors in the accommodations’ effectiveness. These opinions raise an important issue—the backgrounds of students and the contexts should be taken into consideration when evaluating effective types of accommodations.


RECOMMENDATIONS AND CONCLUSIONS


We discussed a number of possible reasons for teachers’ reported difficulty in keeping up with state policies: (a) lack of clear guidelines in making accommodation decisions and implementing accommodations in a standardized way, (b) lack of or limited opportunities to receive information and communicate on accommodations among decision makers and teachers, and (c) limited resources and logistical difficulties. Despite the study’s limitations, we believe that the findings suggest some initial, relatively simple steps that state policymakers and practitioners could implement to support the valid use of accommodations in their schools: (a) States should provide clear, operationalized guidelines (such as in a comprehensive manual) on the use of accommodations, including the definition, selection criteria, allowable or prohibited accommodations, and details of the implementation procedures. (b) States or districts should monitor the use of guidelines regularly to ensure the adequate application of accommodation policies in practice. (c) Regular professional development meetings should be held to communicate the policies and intended use of accommodations across the state, districts, and schools, as well as between ELL and content teachers. (d) The types of accommodations provided to each individual student during the state’s large-scale assessment should be accurately recorded and accessible to teachers as well as administrators. Establishing systematic recording practices for accommodations is essential for accurate reporting of results. Analyses of such data also can be an important source for evaluating the validity of accommodations as well as for monitoring, communicating, and improving accommodation policies and practices.


The intent of the study was to learn from the existing policies and practices to help policymakers in improving accommodation guidelines and to inform practitioners of the current status of accommodation uses to consider important accommodation issues for ELL students. The study findings highlight the significance of examining practices when evaluating the validity of accommodation uses. Implications for these findings draw attention to the comparability issue of accommodated and nonaccommodated test scores among ELL students, within and across districts and states. Further, this finding brings up equity and accessibility issues in that some students were excluded from receiving accommodations for reasons unrelated to the students’ needs. We hope the present paper can be used as a resource to initiate dialogue among policymakers and practitioners.


Author Note


The study was conducted while Mikyung Kim Wolf and Jenny C. Kao were at CRESST/UCLA.


This study was supported by the research grant (R305A050004) awarded by the U.S. Department of Education’s Institute of Education Sciences (IES). The findings and opinions expressed herein are those of the authors and do not necessarily reflect the positions or policies of the IES. We would like to thank Joan Herman and Noelle Griffin for their invaluable feedback. We also thank Patina Bachman, Julie Nollner, Alice Hsu, Yuichiro Otani, Jean Jho, and Stella Tsang Li for their helpful research assistance. Finally, our sincere thanks go to all the teachers who participated in this study, and all the state and district representatives who provided enormous support and help with our study.


Note


1. ATLAS.ti Scientific Software Development GmbH, Nassauische Str. 58, D-10717 Berlin Germany.


References


Abedi, J., Courtney, M., Leon, S., Kao, J., & Azzam, T. (2006). English language learners and math achievement: A study of opportunity to learn and language accommodation (CSE Tech. Rep. No. 702). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.


Flores, B. B., & Smith, H. L. (2008). Teachers' characteristics and attitudinal beliefs about linguistic and cultural diversity. Bilingual Research Journal, 31(1), 323–358.


Kopriva, R. J., Emick, J. E., Hipolito-Delgado, C. P., & Cameron, C. A. (2007). Do proper accommodation assignments make a difference? Examining the impact of improved decision making on scores for English language learners. Educational Measurement: Issues and Practice, 26(3), 11–20.


Martinez, J. F., Bailey, A. L., Kerr, D., Huang, B. H., & Beauregard, S. (2009, April). Measuring opportunity to learn and academic language exposure for English language learners in elementary science classrooms. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.


Rivera, C., & Collum, E. (Eds.). (2006). State assessment policy and practice for English language learners: A national perspective. Mahwah, NJ: Erlbaum.


Rivera, C., Collum, E., Shafer Willner, L., & Sia, J. K., Jr. (2006). An analysis of state assessment policies regarding the accommodation of English language learners. In C. Rivera & E. Collum (Eds.), State assessment policy and practice for English language learners: A national perspective (pp. 1–173). Mahwah, NJ: Erlbaum.


Shafer Willner, L., Rivera, C., & Acosta, B. (2008). Descriptive study: State assessment policies for accommodating English language learners. Arlington, VA: The George Washington University Center for Equity and Excellence in Education.


Wolf, M. K., Kao, J., Griffin, N., Herman, J. L., Bachman, P., Chang, S. M., & Farnsworth, T. (2008). Issues in assessing English language learners: English language proficiency measures and accommodation uses--Practice review (CRESST Rep. No. 732). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.


Wolf, M. K., Kao, J., Herman, J., Bachman, L. F., Bailey, A., Bachman, P. L., . . . Chang, S. M. (2008). Issues in assessing English language learners: English language proficiency measures and accommodation uses--Literature review (CRESST Rep. No. 731). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.


Yoon, B. (2008). Uninvited guests: The influence of teachers’ roles and pedagogies on the positioning of English language learners in the regular classroom. American Educational Research Journal, 45(2), 495–522.


APPENDIX


SAMPLE TEACHER SURVEY AND INTERVIEW QUESTIONS


[39_16302.htm_g/00006.jpg]

[39_16302.htm_g/00008.jpg]


[39_16302.htm_g/00010.jpg]


Policy Communication Channel

Do you feel that state and district policies on accommodations for ELLs are made clear to you?

How do you get information on accommodation policies?


Recording Practice of the Accommodation Data

Are you familiar with how students’ accommodations are kept on file?

How are students’ accommodations recorded?

Who records them? Or fills out the forms?

How often are these forms accessed or updated?




Cite This Article as: Teachers College Record Volume 114 Number 3, 2012, p. 1-26
https://www.tcrecord.org ID Number: 16302, Date Accessed: 10/21/2021 7:55:08 PM

Purchase Reprint Rights for this article or review
 
Article Tools

Related Media


Related Articles

Related Discussion
 
Post a Comment | Read All

About the Author
  • Mikyung Kim Wolf
    Educational Testing Service
    E-mail Author
    MIKYUNG KIM WOLF is currently a research scientist at Educational Testing Service (ETS). Her research interests include English language learner assessment and instruction, English language proficiency assessments, and second language acquisition. She recently published a CRESST policy brief, “Improving the Validity of English Language Learner Assessment Systems,” and articles related to the language demands of content assessments for English language learners in the Educational Assessment journal and the NCELA newsletter AccELLerate!
  • Jenny Kao
    Teachers College, Columbia University
    E-mail Author
    JENNY C. KAO was formerly a research analyst at the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) at UCLA and is currently a doctoral student at Teachers College, Columbia University. Her research interests are in language acquisition and development, specifically for students of diverse language backgrounds. She has coauthored several technical reports focusing on English language learners and students with disabilities and recently coauthored an article in the Applied Measurement in Education journal.
  • Nichole Rivera
    University of California, Los Angeles
    E-mail Author
    NICHOLE M. RIVERA is a research analyst at the National Center for Research on Evaluation, Standards, and Student Testing (CRESST), and is currently working on projects related to quality teacher practice and military assessment. Her research interests include educational assessment and social psychology, particularly as they apply to disadvantaged groups.
  • Sandy Chang
    University of California, Los Angeles
    E-mail Author
    SANDY M. CHANG is a doctoral student in psychological studies in education in the Graduate School of Education and Information Sciences at the University of California, Los Angeles. Her research interests include reading comprehension, English language learners, and academic language.
 
Member Center
In Print
This Month's Issue

Submit
EMAIL

Twitter

RSS