Impact Measures (CAEP Standard 4)
1. Impact on P-12 learning and development (Component 4.1)
Like most state departments of education, the Alabama State Department of Education does not provide Alabama EPPs with access to student performance data associated with individual completers, making this component of standard 4 difficult to address. The EPP is finalizing and will implement in 2019 a comprehensive plan for capturing P-12 student impact data through the EPP’s new Induction and Mentoring Program. In order to gather rich data, the EPP will request the completer provide the following: 1) written lesson plan, 2) video or in-person observation teaching the lesson, 3) pre- and posttest assessment of student learning, and 4) reflection on effectiveness of lesson. The rubrics developed for the EPP’s Signature Assessment-Planning and Assessment will be used to evaluate the lesson plan, data analysis and reflection, and planning commentary. Data from the pre- and posttest student assessments will be compiled and evaluated for impact on P-12 learning and development. The EPP will use the Danielson Framework to assess the instructional video or observation as a measure of teacher effectiveness. By using instruments where data was collected from candidates during their final year of preparation, the EPP will have a benchmark to measure growth as completers enter their teaching fields.
Finally, the EPP will analyze the P-12 student performance data (both from the classroom lesson and from an appropriate state or national achievement test) as an additional measure of completer impact. Currently, the collection of that data is ongoing and is expected to be completed at the end of spring 2019 for this academic year. Data collected by the EPP will continue to be de-identified and presented to the faculty at Assessment day and to the Program Advisory Councils consisting of P-12 practitioners, candidates, EPP faculty, and completers. These stakeholders have the opportunity to review the data and provide feedback and suggestions to the EPP. In turn, the EPP uses this input to support programmatic changes for the betterment of its program offerings.
2. Indicators of teaching effectiveness (Component 4.2)
The Alabama State Department of Education does not provide Alabama EPPs with access to employer evaluation data associated with individual completers, making this component of standard 4 difficult to address. All EPPs in the state hold membership in the Alabama Association of Colleges for Teacher Education (ALACTE). Due to the absence of state reported information about employer satisfaction or new teacher effectiveness the membership, in collaboration with the Alabama State Department of Education (ALSDE), developed, validated, and piloted an employer satisfaction survey for new teachers. This instrument reveals not only employer satisfaction, but also elucidates teacher effectiveness. The State Department administers the survey through the New Teacher Mentoring program and began in spring 2018. For the 2017-2018 reporting period selected items from the Alabama employer satisfaction survey addressing teacher effectiveness were used to address this component. A compilation of the data on teacher effectiveness data can be found here.
The EPP is currently finalizing and will implement in 2019 a plan to collect additional teacher effectiveness data using a case study of the EPP’s new Induction and Mentoring Program involving two samples of initial program completers; 1) completers of the Black Belt Teaching Corps Program, and 2) completers who were not members of the BBTC. Contact will be made with these completers at the beginning of the school year and a request to participate in the case study will be made. In order to gather rich data, the EPP will request the completer provide the following: 1) written lesson plan, 2) video or in-person observation teaching the lesson, 3) pre- and posttest assessment of student learning, and 4) reflection on effectiveness of lesson. The EPP will use the Danielson Framework to assess the instructional video or in-person observation as a measure of teacher effectiveness. It will use the rubrics developed for the EPP’s Signature Assessment-Planning and Assessment to evaluate the lesson plan, data analysis and reflection, and planning commentary. In doing this, the EPP will have a growth measure comparing the candidate’s performance while in the preparation program to the completer’s performance. Finally, the EPP will request the completer share his/her employer evaluation to provide a comparison of EPP faculty evaluations using the Danielson Framework to the district’s teacher evaluation method. This comparison will provide insight into the EPP’s standards of teacher effectiveness compared to the employers’. The goal for the EPP is to fully implement the case study by fall 2019; however, this plan is still contingent on the program completers’ willingness to share their student impact data and their employer evaluations which may be daunting for some novice teachers.
3. Satisfaction of employers and employment milestones (Component 4.3 | A.4.1)
The unit, like all other EPPs in the state of Alabama, has a difficulty in the collection of employer satisfaction and employment milestone data from the Alabama State Department of Education (ALSDE). Currently the ALSDE does not report out specific student impact and/or teacher effectiveness data on its teachers. All EPPs in the state hold membership in the Alabama Association of Colleges for Teacher Education (ALACTE). Due to the absence of state reported information about employer satisfaction or new teacher effectiveness the membership, in collaboration with the Alabama State Department of Education (ALSDE), developed, validated, and piloted an employer satisfaction survey for new teachers. The State Department administers the survey through their New Teacher Mentoring program and began in spring 2018.
Results of the employer satisfaction survey can be found here. Data from employers’ responses (N=15) indicated that the first year program completers were rated as “teacher leader” or “effective” at an approximately equal percentage to those rated as “emerging”. This is seen as a positive indicator of preparation. One would expect new teachers being assessed after completing only one semester of full-time teaching to be rated “emerging” as opposed to the two higher categories.
Currently the Alabama State Department of Education nor the EPP is not administering and collecting employer satisfaction data on advanced program completers. Thus far this component poses unique challenges in that our advanced program completers are spread across the United States and are often in locations that they have not disclosed to the EPP. In previous years, the EPP has attempted to gather this information, but low response rates and lack of employer contact information has led the EPP to the conclusion that it must enhance its data gathering techniques for this component. The EPP has devised a new plan and will begin collecting this data for the upcoming year. First the EPP has begun building a database of advanced program completers and their personal email addresses. This data is being collected upon graduation. The data will assist in building a database of employer information which is connected directly to the program completer allowing for an employer survey to be administered. Survey data collection will be supplemented through interviews of a sample of school administrators who supervise our advanced completers. With both qualitative and quantitative data, the EPP will be able to have a more complete picture of advanced program completer performance.
4. Satisfaction of completers (Component 4.4 | A.4.2)
Beginning in spring 2018 a revised completer satisfaction survey was administered by the Alabama State Department of Education (ALSDE). This survey was developed by an Alabama Association of Colleges for Teacher Education collaborative committee in response to CAEP standard 4. This survey has been validated and is also being utilized to collect data from new in-service teachers. Initial program completers (N=23) with 17 being first year teachers responded to the survey. It is unclear how many surveys reached the EPP’s completers since the ALSDE administers this survey with no data provided on completers teaching out of state or in private schools. This survey is only sent to Alabama public school teachers. Results from the completer survey indicated that the new teachers predominantly strongly agreed or agreed to items written in a positive form regarding their preparation by the EPP.
Advanced program completers were surveyed via the School of Graduate Studies. Data revealed that 29.5 % of respondents strongly agreed; 50% agreed; and only 4.5% disagreed that they were prepared for a career in their chosen field (N= 37).
5. Graduation Rates (initial & advanced levels)
The graduation rates for both initial and advanced programs were calculated based on 2013 cohorts. For initial programs, any candidate accepted to the Educator Preparation Program was included. For advanced programs, any candidate accepted to and enrolled in an advanced program for the first time was included. For initial programs the EPP had a total of 113 potential graduates. Of those 27 never completed a program, and 5 more of those completed a program, but not a program supported by the unit, and thus they cannot be included in the unit’s graduation rate. The unit’s calculated graduation rate for the 2013 initial programs’ cohort is 71.2%. For advanced programs the unit had a total of 301 potential graduates (first time candidate in a master’s or specialist degree program). Of those 121 never completed a program. The unit’s calculated graduation rate for the 2013 advanced programs’ cohort is 59.8%. The EPP has addressed the retention rate through the use of an “attention alert” system in recent years. Any candidate appearing to struggle with attendance, timely work submission, or class participation was flagged and the appropriate personnel reached out to provide support in an effort to help the candidate maintain an active program status.
6. Ability of completers to meet licensing (certification) and any additional state requirements; Title II (initial & advanced levels)
In order for a candidate to complete any program supported by the unit they must meet all licensing requirements outlined by the Alabama State Department of Education. As a result, 100% of the unit's completers have the ability to meet licensing requirements. The following chart outlines the unit completer numbers for the 2016-17 academic year.
7. Ability of completers to be hired in education positions for which they have prepared (initial & advanced levels)
The University’s Counseling and Career Services Department collected data from graduates using the National Association of Colleges and Employers’ Final Destination Survey. Among other data, the survey collected employment information, which is reported above. The data does not provide specific information related to employment field, but it does provide information about the employment status of some graduates.
The reported graduate numbers do not match the EPP’s completer numbers, and there are gaps in reporting. Locating graduates and successfully collecting survey data after matriculation is challenging. Although a large number of graduates reported current employment, many graduates did not respond to the survey and the list of graduates contacted is incomplete. It is also not possible to state whether unit completers are employed in their field. This data further supports the need for the unit to develop a graduate tracking system so that employment data, in addition to satisfaction data, can be collected.
8. Student loan default rates and other consumer information (initial & advanced levels)
The University's Official Default Notification Letter was provided by the financial aid office. The data is based on the cohort of loans that began repayment in 2015 and defaulted by September 30, 2017, but the data is not disaggregated by college or programs within a college. The current UWA loan default rate is 7.8 %.