Monday, June 9, 2014

Century Scholars - From Here They Can (and DO) Go Everywhere

For the past nine years, the Boise District has honored top students at Boise high schools with the Century Scholars award. These students, who have the highest weighted grade point averages (gpa) at their respective high schools, receive a medal and a certificate and are asked to choose teachers who have had a significant impact on their lives. These “distinguished educators” are honored along with the students. The ceremony typically takes place in April of each year and is sponsored by the Rotary Clubs of Boise.

It’s interesting to look at the characteristics of these top students in the Boise District. What follows is an analysis of the 2012-13 class of Century Scholars. This post focuses on the students from the four traditional Boise high schools.

Grades and Advanced Coursework

The eighty-four (84) students (representing the top 5% of the class of 2013) sported an average weighted gpa of 4.27. The Boise District gives credit for Advanced Placement coursework on a 5-point scale, while credit for all other classes is on a 4-point scale. The Scholars took an average of 10 Advanced Placement courses. Two students took 18 AP classes, 1 took 17, and 3 took 16. 40 of the students took Calculus AB and BC, and another 32 took Calculus AB as their capstone math course.

College Entrance Examinations and Activities

The average ACT composite score from this group was 31 (the ACT is scored on a 36 point scale) which was at the 97th percentile on the ACT scoring scale. There were three perfect ACT scores from this group, and 22 Scholars scored 33 or higher, at the 99th percentile nationwide.

On the SAT Critical Reading subtest, the average score was 645, or about the 90thpercentile nationwide. There were 5 perfect Critical Reading scores from the Scholars. In math, the average of 687 is at about the 92nd percentile, and there were 6 perfect math scores. The writing average was at 639, at the 89thpercentile. The average composite score for Century Scholars was 1971, or about the 92nd percentile. The maximum composite score on the SAT is 2400. Twelve scholars scored above 2220, at the 99th percentile.

Century Scholars participated in all sorts of school activities and classes outside of Advanced Placement – from leadership to varsity athletics, chamber and jazz orchestra to ceramics, from debate to journalism, taking full advantage of the offerings of an elite public school system.

College Destinations

In terms of post-secondary choices, Class of 2013 Century Scholars enrolled in college in 21 states and the District of Columbia. Some stayed close to home – 17 (20%) stayed in Idaho; 7 at BSU, 3 each at the University of Idaho, the College of Idaho, and Idaho State University, and one at BYU-Idaho .

The majority of Scholars chose the northwest and the western United States for their pursuits – 5 attended in Oregon, 5 in Washington, 5 in Montana, 2 in Colorado, 11 in California, 6 in Utah, and 4 in Arizona. Others traveled across the country – 2 to the District of Columbia, 4 to Massachusetts, 2 to Maine, 4 to New York, and 2 to Pennsylvania, while two left the mainland for Hawaii.

A number of students are attending the most elite schools in the country. Georgetown, the University of Chicago, Brown, Harvard, MIT, Tufts, Duke, Columbia, Penn, Cornell, Carnegie Mellon, Santa Clara, and Stanford are among the destinations of 2013 Century Scholars.

Other Scholars are enrolled at some of the most prestigious liberal arts colleges in the U.S. – Grinnell (IA), Bowdoin and Colby (ME), Carleton (MN), Oberlin (OH), Lewis and Clark, Reed and Pacific (OR), and Whitman (WA) are included.

In 2013, two Scholars received appointments at military academies, one to The United States Military Academy at West Point, and the other at the United States Air Force Academy in Colorado Springs.

Wherever our Century Scholars choose to enroll, they bring an excellent academic record and a well-rounded background to their college and university experiences. They are truly an impressive group.

Tuesday, June 3, 2014

What the Star Rating System Tells Us, Part 2

The Star Rating System provides a limited portrayal of the quality of a school, mostly because it relies only on growth and achievement in standardized test scores (elementary and junior high), or on those measures and college-ready courses and test scores (high schools). However, given its limitations, the system does provide comparative achievement and growth information about schools across the state.

As indicated in part 1 of this post, elementary and junior high Star Ratings will likely see some revision of growth/achievement weighting by the Star Rating Committee, but will retain the same basic structure.

However, the high school ratings are problematic, in that they tell us little about the achievement outcomes of schools across the state, and instead rely too heavily on growth. Chester Finn, Jr. made an argument for using achievement as the primary measure in rating high schools.

So, if we could adjust the weighting for high schools, could we more accurately reflect outcomes and preparation for post-secondary? Yes, if we used a formula something like this:
  • Achievement – 45%
  • Growth – 20%
  • Graduation Rate – 10%
  • Advanced Opportunities – 25%
In this proposal, fully 70% of the points in the rating system would be earned through college-readiness achievement and course-taking patterns. The effect? High schools with the most students taking Dual Credit or Advanced Placement coursework and showing evidence of college readiness on the SAT or the SBAC would have the highest ratings. For example, using growth points, high school graduation, and advanced opportunities from the old system and reweighting them with 2013 SAT performance, the highest ranked schools would be:

Note that every high school received the maximum possible points for high school graduation and for advanced opportunities. Idaho's high school graduation rate is among the best in the nation, so that makes some sense. But the bar for advanced opportunities is set at a very low level. In fact, only 25% of juniors in the high school had to take an advanced opportunity course (AP, Dual Credit, Tech-Prep, International Baccalaureate), as long as 90% of the students enrolled received a grade of "C" or better, for the high school to receive the maximum number of points.

Comparing to the chart we used in Part 1, we see that the proposed weighting system better reflects achievement outcomes in Idaho high schools.

So what happens to high schools that rated 5 stars on the 2013 Star System, but had lower preparation indicators on the SAT?

Even though the Star system is an imperfect measure of school quality, this proposal provides for consumers a better indicator of college and career readiness among Idaho high schools.

Monday, June 2, 2014

Teacher Evaluations and Standardized Test Scores

The Idaho Department of Education recently released its newest application for a waiver from provisions of the No Child Left Behind law for public comment. The waiver application includes a timeline for the implementation of a statewide teacher evaluation system (pages 290-299).

Idaho’s proposed teacher evaluation system is based upon a controversial plan from the state of New Mexico, which recently provided teachers with their evaluation rankings, sparking some reaction from teachers across the state. 50% of a teacher’s rating in the New Mexico system is based on student growth and achievement on the New Mexico Standards-Based Assessment (to be replaced by PARCC assessment in 2015) and/or other assessments (such as end of course and reading assessments), 25% on teacher observations, and 25% on multiple measures, such as teacher attendance, preparation and planning, and professionalism. Teacher evaluation ratings were not made public in New Mexico in 2014. In Idaho’s waiver application, the first statewide teacher evaluation ratings would occur in Summer 2015.

Idaho’s system currently calls for 33% of the evaluation to be based on achievement and growth on standardized tests, though legislation in 2012 that increased the percentage to 50% was rendered null and void by the overturn of Propositions 1, 2, and 3, and House Bill 557, held in committee in 2014, would have increased the percentage to 50% over a period of years, comporting with the new NCLB waiver application.

The Technical Advisory Committee (TAC), a subcommittee of the Tiered Licensure Committee reviewing Idaho’s licensure requirements as part of the planning for Idaho’s Career Ladder, recommended last week that the SBAC (or another statewide assessment) be a mandatory part of teacher evaluation at applicable grades (the SBAC is currently planned for administration in Idaho in grades 3-11 in spring 2015).

These test-based teacher effectiveness measures are called Value Added Measures (VAM) and purportedly measure individual teacher contribution to student achievement by comparing the expected growth of the individual student against the actual individual student growth while in the teachers' classroom (source: New Mexico Department of Education).

However, two recent studies from reputable statistical organizations question the relationship of student growth on standardized test measures to teacher quality.

First, the American Statistical Association produced a statement regarding Value Added measures and teacher evaluation, making these points:

  • VAMs are generally based on standardized test scores, and do not directly measure potential teacher contributions toward other student outcomes.
  • VAMs typically measure correlation, not causation: Effects – positive or negative - attributed to a teacher may actually be caused by other factors that are not captured in the model.
  • VAMs should be viewed within the context of quality improvement, which distinguishes aspects of quality that can be attributed to the system from those that can be attributed to individual teachers, teacher preparation programs, or schools. Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality.
Here is an article from Valerie Strauss of the Washington Post detailing the ASA findings.

In early May, Morgan Polikoff of USC and Andrew Porter of the University of Pennsylvania published another study, "Instructional Alignment as a Measure of Teacher Quality” in the peer-reviewed journal Educational Evaluation and Policy Analysis. Using results of the Gates Foundation Measures In Effectiveness of Teaching study, they found “no association between value-added results and other widely-accepted measures of teaching quality.” (see article in Education Week).

The seeming disconnect between the work of the Technical Advisory Committee and the research on value-added measures and teacher evaluation hearkens back to the introduction of Proposition 2, Pay for Performance, in 2011. Research showing the ineffectiveness of Pay for Performance implementation in other states was plentiful, as is the emerging research on use of value-added measurement for teacher evaluation. The first distribution of funds highlighted the same inequities that an evaluation system based on standardized test scores will show.

The work of teachers with their students during the school year can easily be taken into account at the local level in evaluations done by principals with teachers without using standardized test scores and value-added measurement. Progress of orchestra students on musical pieces, Pre-Algebra students on a statistics unit, U.S. History students in understanding the causes of the Civil War – all should be handled at the local level through judicious use of content-specific assessment data, observations of instruction, and analysis of classroom management and preparation, as well as other variables from effective teaching frameworks such as the Charlotte Danielson model.

College Majors of Boise District Graduates

Data about everything from fees to college grad rates are available at an excellent website hosted by the National Center for Educational Statistics and called the College Navigator.

Colleges and universities report their data based on "first-time full-time" enrollees, which means that students in a particular high school class who enter college as full-time students for the first time in the semester after graduation are tracked. The NCES provides four and six-year college grad rates for every college that takes federal funds.

We've used the same statistical calculations to track Boise District students for the past six years, and we now have complete six-year data for the 2007 Boise District high school graduating class. It's interesting that the highest percentage of Boise District college grads major in Science, Technology, Engineering, and Mathematics.