Goodbye, Old ISAT... what were the results?

In spring 2014-15, the state of Idaho will officially move from the NCLB-era Idaho Standards Achievement Tests (called herein the “old ISAT”), to the Smarter Balanced Assessments (also to be dubbed the “ISAT”).

Beginning in spring 2007, a new vendor, Data Recognition Corporation, was selected to administer the old ISAT, and a new test was delivered to Idaho students. That first year, 79.9% of Idaho students in grades 3-8 and 10 received a mark of “proficient” or “advanced” on the test. In math, the percentage was 75.8%. In 2012-13, in the final statewide administration of the old ISAT, the percentages were 89.1% in Reading and 80.9% in Math. So, then, the improvement in proficient/advanced percentages over the six years of old ISAT administration was 9.2% in Reading and 5.1% Math.

However, this comparison of proficiency rates conceals something much more important that was happening in Idaho’s public schools. If we examine just the percentages of advanced students in each year, we can see that important shift.

In old ISAT Reading, only a third of Idaho's 3-8 and 10 students scored in the “Advanced” category in 2007. By 2012, half of Idaho’s kids had reached that performance level. And, though the ISAT “advanced” level is not quite so high as that of the SAT, it’s clear that many more Idaho students reached a higher level of rigor in the time of the ISAT. In fact, almost 30,000 more Idaho students were “advanced” in 2012 than were in 2007.

In Math, though the growth was somewhat less than that made in Reading, it was still impressive. Again, only a third of Idaho students were advanced in 2007. By 2012, 45% were in that category. That’s an improvement of more than 17,000advanced students from 2007 to 2012.

The measurement criteria used under the NCLB law, which labeled schools “failing” if they did not meet “Adequate Yearly Progress” with as few as 34 students among their total populations, obscured the tremendous gains made by Idaho schools and districts during the administration of the old ISAT. Those gains were truly impressive.

Century Scholars - From Here They Can (and DO) Go Everywhere

For the past nine years, the Boise District has honored top students at Boise high schools with the Century Scholars award. These students, who have the highest weighted grade point averages (gpa) at their respective high schools, receive a medal and a certificate and are asked to choose teachers who have had a significant impact on their lives. These “distinguished educators” are honored along with the students. The ceremony typically takes place in April of each year and is sponsored by the Rotary Clubs of Boise.

It’s interesting to look at the characteristics of these top students in the Boise District. What follows is an analysis of the 2012-13 class of Century Scholars. This post focuses on the students from the four traditional Boise high schools.

Grades and Advanced Coursework

The eighty-four (84) students (representing the top 5% of the class of 2013) sported an average weighted gpa of 4.27. The Boise District gives credit for Advanced Placement coursework on a 5-point scale, while credit for all other classes is on a 4-point scale. The Scholars took an average of 10 Advanced Placement courses. Two students took 18 AP classes, 1 took 17, and 3 took 16. 40 of the students took Calculus AB and BC, and another 32 took Calculus AB as their capstone math course.

College Entrance Examinations and Activities

The average ACT composite score from this group was 31 (the ACT is scored on a 36 point scale) which was at the 97th percentile on the ACT scoring scale. There were three perfect ACT scores from this group, and 22 Scholars scored 33 or higher, at the 99th percentile nationwide.

On the SAT Critical Reading subtest, the average score was 645, or about the 90thpercentile nationwide. There were 5 perfect Critical Reading scores from the Scholars. In math, the average of 687 is at about the 92nd percentile, and there were 6 perfect math scores. The writing average was at 639, at the 89thpercentile. The average composite score for Century Scholars was 1971, or about the 92nd percentile. The maximum composite score on the SAT is 2400. Twelve scholars scored above 2220, at the 99th percentile.

Century Scholars participated in all sorts of school activities and classes outside of Advanced Placement – from leadership to varsity athletics, chamber and jazz orchestra to ceramics, from debate to journalism, taking full advantage of the offerings of an elite public school system.

College Destinations

In terms of post-secondary choices, Class of 2013 Century Scholars enrolled in college in 21 states and the District of Columbia. Some stayed close to home – 17 (20%) stayed in Idaho; 7 at BSU, 3 each at the University of Idaho, the College of Idaho, and Idaho State University, and one at BYU-Idaho .

The majority of Scholars chose the northwest and the western United States for their pursuits – 5 attended in Oregon, 5 in Washington, 5 in Montana, 2 in Colorado, 11 in California, 6 in Utah, and 4 in Arizona. Others traveled across the country – 2 to the District of Columbia, 4 to Massachusetts, 2 to Maine, 4 to New York, and 2 to Pennsylvania, while two left the mainland for Hawaii.

A number of students are attending the most elite schools in the country. Georgetown, the University of Chicago, Brown, Harvard, MIT, Tufts, Duke, Columbia, Penn, Cornell, Carnegie Mellon, Santa Clara, and Stanford are among the destinations of 2013 Century Scholars.

Other Scholars are enrolled at some of the most prestigious liberal arts colleges in the U.S. – Grinnell (IA), Bowdoin and Colby (ME), Carleton (MN), Oberlin (OH), Lewis and Clark, Reed and Pacific (OR), and Whitman (WA) are included.

In 2013, two Scholars received appointments at military academies, one to The United States Military Academy at West Point, and the other at the United States Air Force Academy in Colorado Springs.

Wherever our Century Scholars choose to enroll, they bring an excellent academic record and a well-rounded background to their college and university experiences. They are truly an impressive group.

What the Star Rating System Tells Us, Part 2

The Star Rating System provides a limited portrayal of the quality of a school, mostly because it relies only on growth and achievement in standardized test scores (elementary and junior high), or on those measures and college-ready courses and test scores (high schools). However, given its limitations, the system does provide comparative achievement and growth information about schools across the state.

As indicated in part 1 of this post, elementary and junior high Star Ratings will likely see some revision of growth/achievement weighting by the Star Rating Committee, but will retain the same basic structure.

However, the high school ratings are problematic, in that they tell us little about the achievement outcomes of schools across the state, and instead rely too heavily on growth. Chester Finn, Jr. made an argument for using achievement as the primary measure in rating high schools.

So, if we could adjust the weighting for high schools, could we more accurately reflect outcomes and preparation for post-secondary? Yes, if we used a formula something like this:

• Achievement – 45%
• Growth – 20%
• Graduation Rate – 10%
• Advanced Opportunities – 25%

In this proposal, fully 70% of the points in the rating system would be earned through college-readiness achievement and course-taking patterns. The effect? High schools with the most students taking Dual Credit or Advanced Placement coursework and showing evidence of college readiness on the SAT or the SBAC would have the highest ratings. For example, using growth points, high school graduation, and advanced opportunities from the old system and reweighting them with 2013 SAT performance, the highest ranked schools would be:



Note that every high school received the maximum possible points for high school graduation and for advanced opportunities. Idaho's high school graduation rate is among the best in the nation, so that makes some sense. But the bar for advanced opportunities is set at a very low level. In fact, only 25% of juniors in the high school had to take an advanced opportunity course (AP, Dual Credit, Tech-Prep, International Baccalaureate), as long as 90% of the students enrolled received a grade of "C" or better, for the high school to receive the maximum number of points.

Comparing to the chart we used in Part 1, we see that the proposed weighting system better reflects achievement outcomes in Idaho high schools.



So what happens to high schools that rated 5 stars on the 2013 Star System, but had lower preparation indicators on the SAT?



Even though the Star system is an imperfect measure of school quality, this proposal provides for consumers a better indicator of college and career readiness among Idaho high schools.

Teacher Evaluations and Standardized Test Scores

The Idaho Department of Education recently released its newest application for a waiver from provisions of the No Child Left Behind law for public comment. The waiver application includes a timeline for the implementation of a statewide teacher evaluation system (pages 290-299).

Idaho’s proposed teacher evaluation system is based upon a controversial plan from the state of New Mexico, which recently provided teachers with their evaluation rankings, sparking some reaction from teachers across the state. 50% of a teacher’s rating in the New Mexico system is based on student growth and achievement on the New Mexico Standards-Based Assessment (to be replaced by PARCC assessment in 2015) and/or other assessments (such as end of course and reading assessments), 25% on teacher observations, and 25% on multiple measures, such as teacher attendance, preparation and planning, and professionalism. Teacher evaluation ratings were not made public in New Mexico in 2014. In Idaho’s waiver application, the first statewide teacher evaluation ratings would occur in Summer 2015.

Idaho’s system currently calls for 33% of the evaluation to be based on achievement and growth on standardized tests, though legislation in 2012 that increased the percentage to 50% was rendered null and void by the overturn of Propositions 1, 2, and 3, and House Bill 557, held in committee in 2014, would have increased the percentage to 50% over a period of years, comporting with the new NCLB waiver application.

The Technical Advisory Committee (TAC), a subcommittee of the Tiered Licensure Committee reviewing Idaho’s licensure requirements as part of the planning for Idaho’s Career Ladder, recommended last week that the SBAC (or another statewide assessment) be a mandatory part of teacher evaluation at applicable grades (the SBAC is currently planned for administration in Idaho in grades 3-11 in spring 2015).

These test-based teacher effectiveness measures are called Value Added Measures (VAM) and purportedly measure individual teacher contribution to student achievement by comparing the expected growth of the individual student against the actual individual student growth while in the teachers' classroom (source: New Mexico Department of Education).

However, two recent studies from reputable statistical organizations question the relationship of student growth on standardized test measures to teacher quality.

First, the American Statistical Association produced a statement regarding Value Added measures and teacher evaluation, making these points:

• VAMs are generally based on standardized test scores, and do not directly measure potential teacher contributions toward other student outcomes.
  
• VAMs typically measure correlation, not causation: Effects – positive or negative - attributed to a teacher may actually be caused by other factors that are not captured in the model.
  
• VAMs should be viewed within the context of quality improvement, which distinguishes aspects of quality that can be attributed to the system from those that can be attributed to individual teachers, teacher preparation programs, or schools. Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality.

Here is an article from Valerie Strauss of the Washington Post detailing the ASA findings.

In early May, Morgan Polikoff of USC and Andrew Porter of the University of Pennsylvania published another study, "Instructional Alignment as a Measure of Teacher Quality” in the peer-reviewed journal Educational Evaluation and Policy Analysis. Using results of the Gates Foundation Measures In Effectiveness of Teaching study, they found “no association between value-added results and other widely-accepted measures of teaching quality.” (see article in Education Week).

The seeming disconnect between the work of the Technical Advisory Committee and the research on value-added measures and teacher evaluation hearkens back to the introduction of Proposition 2, Pay for Performance, in 2011. Research showing the ineffectiveness of Pay for Performance implementation in other states was plentiful, as is the emerging research on use of value-added measurement for teacher evaluation. The first distribution of funds highlighted the same inequities that an evaluation system based on standardized test scores will show.

The work of teachers with their students during the school year can easily be taken into account at the local level in evaluations done by principals with teachers without using standardized test scores and value-added measurement. Progress of orchestra students on musical pieces, Pre-Algebra students on a statistics unit, U.S. History students in understanding the causes of the Civil War – all should be handled at the local level through judicious use of content-specific assessment data, observations of instruction, and analysis of classroom management and preparation, as well as other variables from effective teaching frameworks such as the Charlotte Danielson model.

College Majors of Boise District Graduates

Data about everything from fees to college grad rates are available at an excellent website hosted by the National Center for Educational Statistics and called the College Navigator.

Colleges and universities report their data based on "first-time full-time" enrollees, which means that students in a particular high school class who enter college as full-time students for the first time in the semester after graduation are tracked. The NCES provides four and six-year college grad rates for every college that takes federal funds.

We've used the same statistical calculations to track Boise District students for the past six years, and we now have complete six-year data for the 2007 Boise District high school graduating class. It's interesting that the highest percentage of Boise District college grads major in Science, Technology, Engineering, and Mathematics.

Here's the full report of their academic progress, and their majors at graduation.

Classes of 2007 and 2008 - Degrees Obtained

We now have data further illustrating the degrees obtained and major areas for members of the high school graduating classes of 2007 and 2008. Instead of just “first-time full-time” grads, these data are for all students in those classes who have obtained certificates, Associate degrees, and Baccalaureate degrees.

For the class of 2007, we have data for 6 ½ years after high school graduation. 43 students have obtained Associate’s Degrees, 25 have Certificates, and 471 have four-year degrees. For the class of 2008, (5 ½ years out from high school graduation), 40 have Associate degrees, 31 have Certificates, and 498 have four-year degrees.

Of the 56 certificates reported by colleges for the classes of 2007 and 2008, 39 came from Carrington College of Boise (College of Western Idaho came into existence after these classes began their college journey). 36 of 56 certificates were in areas of medicine (Medical Assisting and Dental Assisting were the most popular certificates).

Of the 83 Associate’s Degrees earned by the classes of 2007 and 2008, 24 came from Boise State University and the College of Western Idaho; the remainder were from colleges scattered around the region and the country.

969 four-year degrees have been earned by the classes of 2007 ad 2008. Here is the distribution of four-year degree major areas for those graduates:



Of the BSD 4-year college grads, 284 (29%) graduated from Boise State University. Here is the distribution of majors among the BSU grads:



167 (17%) of the BSD grads matriculated from the University of Idaho. Here's a chart showing the major areas for those students:



In November, we'll have 5 years of data from the high school class of 2009, and additional information about 2007 and 2008. We'll add those data into the mix, in order to continue tracking college destinations and majors.

Update
To date, 32.3% of the Class of 2007 have received a 2 or 4 year degree or a certificate from a college or university. For the class of 2008, the figure is 29%. College matriculation rates for every state are available at the National Council for Higher Education Management Systems site. These graduation rates are about three times those claimed in the Idaho Business for Education Field Guide to Education in Idaho on the Don't Fail Idaho website.

What the Star Rating System Tells Us, Part 1

In August of 2013, Michael Petrilli posted an article in the online magazine Education Next,entitled “The Problem with Proficiency”. In the post, he argued that proficiency levels are determined largely by demographics, and advocated for a system in which schools are judged largely on student academic growth. Petrilli wrote: “To be sure, proficiency rates should be reported publicly, and parents should be told whether their children are on track for college or a well-paying career. (That’s one of the great benefits of a high standard like the Common Core.) But using these rates to evaluate schools will end up mislabeling many as failures that might in fact be doing incredible work at helping their students make progress over time.”

In the same month, Chester E. Finn, Jr. responded with “Let’s Hear it for Proficiency”, in which he argued that proficiency has its rightful place as an indicator of readiness, especially for college, and should not be dropped: “Not, at least, so long as it matters greatly in the real world. Do you want the pilot of your plane to be proficient at take-offs and landings or simply to demonstrate improvement in those skills? (Do you want to fly on an airline that uses only “growth measures” when hiring pilots?) How about dining in restaurants that use only growth measures when selecting chefs? Having your chest cut open by thoracic surgeons who showed “gains” on their surgical boards but didn’t actually “pass” them?”

The Idaho Star Rating system was initiated by the Idaho State Department of Education in 2012. The first school ratings were published in the 2012-13 school year, combining growth and proficiency measures. Though many states have opted for an “A-F” rating system, Idaho’s SDE decided to use a rating system similar to that used for hotels – 5 is the highest rating a school can receive, and 1 is the lowest. The Star Rating system was implemented as part of Idaho’s NCLB waiver request , submitted in September of 2012.

Though the Star Rating system itself is complicated, suffice it to say that the points earned by schools in the system are based primarily on school performance on the ISAT; in fact, 100% of the ratings for elementary and middle/junior high schools are based on test scores. In 2014-15, the system will replace the ISAT with scores from the Smarter Balanced assessments which are being piloted statewide this year.

At elementary and middle/junior high levels, the distribution of Star rating points is as follows:

• ISAT Achievement – 25%
• ISAT Growth – 75%

This distribution allows schools that are achieving at a relatively low level to gain Star Rating points by demonstrating that students are “growing” from grade to grade at a high level. In other words, if the schools in a system are showing excellent growth, outcomes should be better over time for that system.

Members of the Star Rating Advisory Committee (established this year by the State Department of Education) recommended that a more balanced distribution of points between achievement and growth be implemented beginning in 2014-15, in order that schools with high achievement receive more credit for their performance. However, the basic structure of the elementary/junior high system will remain intact.

At the senior high school level, the system contains a few more variables. Here is the basic structure:

• ISAT Achievement – 20%
• ISAT Growth – 50%
• High School Graduation Rate – 15%
• SAT College Readiness – 7.5%
• Advanced Coursework – 7.5%

The ISAT was taken by 9th and 10th graders across Idaho, which meant it counted in the Star Rating system for freshmen and sophomores at 4-year high schools, and for just sophomores at 3-year high schools. In the current state testing plan, the SBAC will count for 9th, 10th, and 11th graders in 4-year high schools, for sophomores and juniors in 3-year high schools

The Star system at the high school level relies on growth for half of the rating. The result is that it provides a misleading view of high school performance when considered against measurement of progress on the Idaho State Board of Education goal:

“60% of young Idahoans, ages 25-34, will have a post-secondary degree or
certificate of value by 2020.”

In light of this goal, It would be helpful to know how Idaho schools are doing in preparing students for college. The high school Star Ratings don’t tell us that. If we use the SAT performance of Idaho juniors and cross-reference Star Ratings for Idaho high schools, this is what we find:



So, while Timberlake High School (Lakeland District) and Moscow High School are among the top high schools in the state at preparing students for college math, they are “4-star schools”, and American Falls High School and Marsing High School, with far fewer students prepared, are “5-star schools”.

This is caused primarily by the weighting of the factors in the Star Rating system. Grade-to-grade student growth counts for half of the Star Ratings for high schools. If the system were more outcome-based, that is, if it put more weight on preparation for college and career in Idaho high schools (as Finn recommends), we’d have a better picture of how we measure up in preparing students. In Part 2 of this post, we’ll examine an idea for a different weighting system that would be more reflective of outcomes and preparation.