The purpose and primary objective of Idol’s (2006) program evaluation was to examine and describe the provision of special education programs in a metropolitan school district using four elementary schools and four secondary schools. The second objective was to determine how each school provided a least restrictive environment for students with special education needs. This review will discuss whether the model, design, and data collection strategy were consistent with the aims of the program evaluation, and it will also identify the stakeholders, outcomes, and limitations of the evaluation.
Evaluation Model
The evaluation model used by Idol (2006) is the advocacy model. According to Stufflebeam (2001), the purpose of the advocacy or social agenda model is to use the evaluation for making a difference in society. An evaluation conducted using the advocacy model seeks to “ensure that all segments of society have equal access to educational and social opportunities and services” (Stufflebeam, 2001, p. 62). While reporting the results of the evaluation, Idol (2006) focuses on pointing out the favorable results of the interview with educators and concludes that the interviews “strongly support the practice of including students with special education challenges in general education programs” (p. 94). The only merits of the program investigated were the effects of inclusion on other students and statewide test scores, but the effect of inclusion was not quantified, which is consistent with the advocacy model of the evaluation. Rather than assessing the merits of the program, the primary purpose of the evaluation is to encourage the development and improvement of special education programs.
Evaluation Design
The evaluation was designed as a case study. Even though the design of the evaluation is not explicitly stated, there are three characteristics of the program evaluation that are consistent with the case study design. First, a case study is defined as an in-depth description and analysis of a specific program (Stufflebeam, 2001), and Idol (2006) describes the implementation of special education programs in a school district and analyzes them based on the qualitative and quantitative data collected. Second, the main purpose of a case study “is to delineate and illuminate a program, not necessarily to guide its development or to assess and judge its merit and worth” (Stufflebeam, 2001, p. 34). Although this evaluation does provide some insight into possible directions for future development, it is mostly a descriptive study that illustrates the current state of special education in a school district. Third, Stufflebeam (2001) emphasizes the necessity of using multiple sources of information and the use of both qualitative and quantitative data to conduct an effective case study evaluation. The data for this evaluation was collected from educators in various roles (i.e., principals, assistant principals, special education teachers, special area teachers, classroom teachers, support staff, administrators, and instructional assistants), and the evaluation used historical statewide test data as well.
The main strength of case study evaluations is the ability to investigate a specific program in depth and examine it holistically (Stufflebeam, 2001). Idol (2006) takes advantage of that strength and investigates multiple aspects of the special education programs. Although the educators’ perceptions are the main source of information, the evaluation presents their attitudes toward inclusion, the effects of inclusion on other students, perceptions of their teaching skills for delivering the program benefits, and various other aspects of the programs. However, Stufflebeam (2001) also warns evaluators that the case study has several limitations. Specifically, the broad focus, descriptive nature of case studies, and the lack of control groups often lead to the lack of rigor and fail to produce an assessment of a program’s merits. In this evaluation, those limitations can be observed because the evidence is not sufficient to improve the understanding of the relationship between the educators’ attitudes toward inclusion and special education programs. It is also not possible to measure the effects of inclusion on other students and statewide test data using the data from this case study. However, the primary purpose of the case study is to illuminate and delineate a program rather than judge its effectiveness, so it is possible to argue that those limitations are not critical for this particular evaluation.
Data Collection Strategy
The eight schools selected for participation were all from a single participating school district, and the inclusion criteria for selection were: (a) the schools have a well-developed program for special education and (b) the staff at each school considered their approach to special education appropriate. Because the criteria were defined to select a specific type of school without randomization, it is possible to conclude that purposeful sampling was used. Although purposeful sampling may be feasible for a case study, this program evaluation included only schools in which the staff considers the programs appropriate, so it is obvious that the data would reveal positive attitudes about the inclusion of students with disabilities in general education. The evaluation does not reveal why some schools may have inappropriate special education programs and what the effects of those programs on educator attitudes are.
The program evaluation used both qualitative and quantitative data. Qualitative data was obtained with structured interviews. The entire staff of each school participated because the researchers did not want to limit the data collection to educators with a specific role or to a random sample, so it is possible to state that the sample of educators used was representative for all types of educators. The statewide test data for all students was used to determine how the testing of students with disabilities affects the average test scores of the entire school. Although the effects of inclusion on statewide test scores is important for schools, it is possible to argue that various more important, holistic outcomes were neglected, such as the effects of inclusion on social development of all students.
The researcher’s interpretation of the results was consistent with the strengths and limitations of the data collection and analysis. Causal inferences were not made, and the data was used only to describe the trends observed in teacher attitudes toward the effects of inclusion, attitudes toward their own professional skills for working with special needs students, perceptions of administrative support quality, and the impact of inclusion on statewide testing and other students. Although some recommendations for improvement were made in the evaluation, they were not based on the data analysis, but on the recommendations explicitly identified by the educators participating in the data collection.
Description of Clients and Stakeholders
Primary stakeholders are those stakeholders who are continuously involved in organizational processes, and the organization needs those groups of people to survive (Clarkson, 1995). For the special education programs in this case, the primary stakeholders are those individuals who are involved in the creation of program policies because the evaluation results will inform their future decisions concerning the program. For each school, the primary stakeholders are the administration staff (i.e., advisory council and principals). However, Idol (2006) is also targeting local public education policy makers for the metropolitan school district because multiple schools were selected for the study, but they were all from the same school district. Therefore, the description of the special education programs, as well as the educators’ general recommendations for improving special education service delivery, could inform the creation of district-level special education policies for all local schools.
Secondary stakeholders are groups that are not involved in an organization’s functions, but they can affect the organization’s well-being, and the organization can affect them as well (Clarkson, 1995). The program staff and the program’s beneficiaries can be considered the secondary stakeholders in this case because any changes in special education programs inspired by the evaluation will affect those groups. The effects of the evaluation on those groups can be positive if the primary stakeholders implement the general recommendations suggested. For example, visiting other schools with inclusion programs can inform the development of program policies and teaching skills, whereas supporting teachers in their professional development will improve their skills to modify instructions and curricula for special meeds students effectively. Consequently, the special needs students as the program’s beneficiaries will receive a better education than before.
Outcomes of Evaluation
The findings of the program evaluation are extensive because of the in depth review of special education programs, attitudes toward inclusion, educators’ attitudes toward their level of professional skills for working with students with disabilities, and various other aspects of the special education programs. However, Idol (2006) defined two intents at the beginning of the evaluation, so it is expected that an interested reader would want to know the degree of inclusion of students with disabilities, as well as the findings related to the development of a least restrictive environment for students with disabilities.
In elementary schools, special needs students participated in general education classes between 75% and 99% of the time, and only one school reported that all of their students with disabilities spent 100% of their time in general education programs. In secondary education, two schools reported that all students with disabilities attend separate programs; one school reported that all students attend general education classes; one school reported that all students participated in general education classes, but the school’s principal commented that the number of students with disabilities in some classes is too high. The teacher attitudes toward inclusion were mostly favorable, but Idol (2006) points out that most teachers expressed the desire for further professional development so that they can better work with students with special education needs.
Regarding the least restrictive environment, most teachers believe that students with disabilities should be integrated with the rest of the class, and only two elementary school educators (1.7%) and two secondary school educators (1.2%) believed that students with disabilities should be educated in separate classrooms. Most educators also considered the administration to be supportive and primary used various strategies to accommodate the inclusion of students with disabilities in regular classrooms, including instruction modification and curricular modifications. Some schools also reported using electronic beepers so that classroom teachers can call for assistance when the behaviors of their special needs students became disruptive. Therefore, it is possible to suggest that the schools demonstrated willingness and invested the effort to develop a least restrictive environment for students with disabilities.
Although the impact of special education programs was not mentioned as one of the objectives, it is important to list those outcomes because they determine if the special education programs are beneficial. According to the perceptions of educators in elementary schools, the presence of special needs students in the general program had a positive effect or no effect on other students (68%), and only a few educators mentioned observing a negative effect of the inclusive program on other students (6%). The inclusion of students with special needs in statewide testing also had no effect on the average student scores in elementary schools. However, educators in secondary schools reported no significant effects of the inclusions of students with disabilities on other students, and the inclusion of test data obtained from students with disabilities lowered the average results. According to Idol (2006), the inclusion of students with disabilities in statewide testing is feasible for disabilities that do not impair intellectual functions, whereas students with intellectual disabilities should participate in regular education only to experience the social integration benefits of education.
Limitations of Evaluation
Although Idol (2006) does not discuss the limitations of the program evaluation, two limitations have to be noted. First, the data obtained from the program evaluation is cross-sectional, which means that it is collected at one point in time, and a control group was not used. That means inferring conclusions about causal relationships between special education programs and the educators’ inclusion attitudes or other outcomes. Second, the transferability of the findings is low because purposeful sampling was used. The schools participating in the evaluation were all from a single district, which means that the students and teachers in that school also probably share similar characteristics (e.g., socioeconomic status). The characteristics of the district in which the evaluation took place make it impossible to generalize the findings to other districts. Furthermore, one inclusion criterion stated that the staff had to consider the special education program appropriate, so the evaluation included only educators who already had a positive attitude about the programs in place. Therefore, it is not possible to generalize the findings because educators who do not consider their special education programs appropriate may have different attitudes about the inclusion of students with special education needs in general education classrooms.
References
Clarkson, M. E. (1995). A stakeholder framework for analyzing and evaluating corporate social performance. Academy of Management Review, 20(1), 92-117.
Idol, L. (2006). Toward inclusion of special education students in general education: A program evaluation of eight schools. Remedial and Special Education, 27(2), 77-94.
Stufflebeam, D. (2001). Evaluation models. New Directions for Evaluation, 2001(89), 7-98.
