Sunday, December 11, 2016

EVALUATING THE MCREL 
TEACHER EVALUATION AUDIT

McREL International, a Denver-based non-profit which has developed its own teacher evaluation system, was hired by the Idaho Department of Education to review Idaho's progress in implementing its evaluation system, which is based on the Charlotte Danielson evaluation model.

McREL was asked to audit 225 randomly selected teacher evaluations from across the state (36 of the 225 came from Boise, which has about 12% of the state's teachers). A State Department of Education committee, the Principal Evaluation Review Committee, provided the questions that McREL was to use in preparing its report.

Idaho Education News greeted the publication of the audit data with the headline "Audit Finds 99% of Teacher Evaluations were Inaccurate or Incomplete". Idaho Board of Education President Emma Atchley said soon after the publication of the article, “The audit raises serious concerns regarding the teacher evaluation process conducted during the 2014-2015 school year.” 

But did it? Let's take a look at the questions provided by the committee and used by the auditors and how they relate to Idaho Code and practice.

Question 1 - What are the components that were on the Individual Professional Learning Plan (IPLP)?

The audited evaluations were from the 2014-15 school year. The IPLP was not required that year; it came into effect in 2015-16. No wonder a large number of evaluations missed the mark in this area.

Question 2 - Does the professional practice portion include all 22 components of the Charlotte Danielson Framework (2nd ed.)?

State Board  rule requires that “each district evaluation model shall be aligned to state minimum standards that are based on Charlotte Danielson’s Framework for Teaching Second Edition domains and components of instruction. The rule, by design, does not require districts to adopt verbatim the Danielson Framework domains and components as was implied in the questions given to McREL for the evaluation.

For example, the Bonneville District uses a different evaluation system which is aligned the the Danielson system, but did not satisfy the criteria in the question, and is not required to. The Bonneville system was approved by the Idaho Department of Education.

Question 3 - Record the levels of performance for each component (1, 2, 3 or 4)

Again, State Board rule requires districts to establish a “rating system with a minimum of three (3) rankings (not 4) used to differentiate performance of teachers and pupil personnel certificate holders including: Unsatisfactory being equal to “1”; Basic being equal to “2”; and Proficient being equal to “3”. For example, Boise uses 3 categories, as is required. Even though every one of Boise's evals met State Board rule, they would all have missed the mark on this question.

Question 4 -What are the dates of the two documented observations?

Idaho teacher evaluations must include a minimum of two documented observations, one of which shall be completed prior to January 1 of each year. In the Boise School District, we require our principals to complete, at a minimum, one formal observation prior to January 1st in compliance with the law as well as a second formal observation that becomes incorporated into the final formal evaluation. The dates may not have been noted on the evaluation but in every case two observations or more were made.

Further, in two cases, the audit identified employees who did not have an evaluation because they were on leave during the second semester of the school year, one for family leave and the other for medical reasons.  However, the auditors did not ask for the reasons for missing evaluations.

Question 5 - Which additional measure(s) was included to inform professional
practice?


  • Student Input
  • Parent Input
  • Portfolio
  • None

The Boise District uses parent input to satisfy this State Board required element.


Question 6 - Which measures were used for student achievement?


The data elements provided to McREL for this review come from the options to measure
student achievement under the Career Ladder which was not part of the state law and was not required until 2015­-2016 with the introduction of the Career Ladder Legislation that was established during the 2015 Legislative Session. In 2014­-2015, districts had to include ISAT data in the evaluation as well as one or more of multiple objective measures of growth in student achievement, which could be determined by the board of trustees. Districts could choose for the ISAT to count for 1% up to 32% of the 33% of the evaluation that was to be
based on student achievement.

Additionally, the rule states that “growth in student achievement may be considered as an optional measure for all other school based and district based staff, as determined by the local board of trustees.”

As a result of this provision, any evaluations that were reviewed by McREL that were not instructional staff, may not have had student achievement data included because it did not have to be included in accordance with State Board rule. This was the case for two evaluations that were randomly selected as part of the Boise School District sampling.

Question 7 - What is the summative rating?

Question 8 - Does the summative rating include combining professional practice (67%) and student achievement (33%) 

The Boise School District requires a separate rating for the 33% of the
evaluation that is based on growth in student achievement and an overall
summative rating that includes combining the 67% and 33%.

Question 9 - What is the date of the summative evaluation?

Question 10 - Was it completed by May 1st?

Three (3) 2014-2015 Boise District audited evaluations were submitted between May and June 1. For 2015-16, that submittal date was changed by the legislature to June 1 because May 1 was deemed too early.

Question 11 - Is there a written evaluation policy?

The State Department of Education did not ask for the District's policy, or we would have provided it.

In summary, McREL was asked in the 2014-15 audit to:

  • conduct analyses of elements not required by law or under Idaho Code
  • analyze adherence to deadlines that had already been determined unreasonable by the legislature
  • summarize without asking for explanation:
    • observations required by law and by local policy
    • missing evaluations that may have been due to unforeseen circumstances
In order for legislators to have an understanding of the errors inherent to the audit process as they make decisions about the Career Ladder, it makes sense that the State Department of Education confer with districts involved in the audit as soon as possible. We believe we have done a thorough job of outlining the flaws of the audit. Now is the time to set it right.