JUDGING FROM AFAR

There are a number of school ranking systems out there these days. Each has criteria upon which its rankings are based. Most school rating systems examine a small slice of a school's characteristics, namely standardized test scores. For example, Great Schools uses standardized test scores as their primary rating tool, as does Schooldigger, which also includes information about a school's free/reduced lunch percentage.

Two of the nation's most prominent school ranking systems focus on high schools, those of Newsweek magazine and U.S. News and World Reports. These two ranking systems use fairly complex systems which consider standardized test scores and a few other variables, including how well the schools serves its underprivileged population. A third, the Washington Post's "Most Challenging High Schools" list, relies on a much simpler formula for its rankings.

U.S. News High School Rankings

Kevin Welner, the Director of the National Education Policy Center at the University of Colorado-Boulder, wrote a critique of the U.S. News high school rankings which was published in the Answer Sheet, Valerie Strauss' blog for the Washington Post newspaper. The critique, which spanned two posts, can be found here (Part 1) and here (Part 2). In Part 1, Welner looks primarily at the methodology of the U.S. News rankings, and in Part 2 he examines the characteristics of some of the top-ranked schools.

From Welner's critique:

"There is an important lesson here about rankings in general—not just the U.S. News rankings. A disproportionate focus on outcomes will always reward those schools that excel at enrolling high-achievers. Perhaps this is more obvious when we look outside the school realm. For instance, University of Kentucky basketball coach John Calipari is such a fantastic recruiter that when looking at his success it’s almost impossible to disentangle his recruiting and coaching talents."

and:

"This year, the nation’s top-ranked high schools are: (1) Dallas’ “School for the Talented and Gifted,” a magnet school; and (2) BASIS Scottsdale (AZ), a charter school. The Dallas school has an application process that begins with a GPA screen and a test score screen, then the applicant student must “design and carry out a school-related or other project that demonstrates extended effort and creativity,” and then the applicant student must attend an “application session.” At this session, the applicant student must complete a timed (1 ½ hours) hand-written, prompted essay; must complete a “30-minute timed logic and reasoning activity;” and finally is given a 15-minute scored interview. This highly competitive process yields an entering class of 65 students.

The Scottsdale charter school is not selective in the same way; admission is by lottery. But the selectivity is arguably just as potent as with the Dallas school. Parents are warned in no uncertain terms that only the most gifted and committed students will survive. Just a handful of students are admitted after the early grades (e.g., a parent was told that there would be a total of four slots open for entering eighth graders). Moreover, as described by the mother of a student in another BASIS charter, the warnings appear to understate the extremity of the BASIS approach."

U.S. News responded with a written statement about Welner's critique, which read in part:

"Mr. Welner’s overall argument is incorrect. Schools that do better and are awarded medals in the U.S. News rankings are the ones that are doing more (better than the average in the state or way better than the average in the state) than what is expected given their level of poverty or proportion of economic disadvantaged."

Newsweek Rankings

Newsweek ranks the top 500 high schools in the country, based on its own analysis. Westat, a Maryland research company, provided an accompanying document to the rankings, entitled "Identification of Newsweek's 2015 Public High School Rankings", which explained the methodology of the rankings. In the Section called "Limitations", the authors make the following remarks:

"There are limitations associated with conducting this analysis, many of which stem from the availability of data and their suitability for comparing schools in different states. We have no information about a range of school factors that may influence school performance, such as fiscal resources, teacher quality and effectiveness, school leadership, and school climate. These school factors could potentially contribute to student achievement and college readiness, but data are unavailable for a variety of reasons.

Additionally, the rankings are still dependent on self-reported data, which has implications for our sample and data collection standards. There are circumstances in which the variables may not have been reported consistently, and for 2015, this issue may be particularly problematic for the new dual enrollment items."

and this:

"We did not control for any other factor besides student socioeconomic status. Many of the top schools on the relative list are magnet and charter schools and may have an application process that allows them to select high achieving students. These types of schools would have an advantage in our ranking methodology over schools that do not have an application process."

So, Newsweek's rankings suffer from some of the flaws noted by Mr. Welner on the U.S. News rankings. Further, some of the magnet and charter schools he referenced are among the highest ranked schools on both lists.

Washington Post Most Challenging Schools Index

The Newsweek rankings were formerly the "Most Challenging High School Rankings" until the Washington Post Company sold Newsweek in 2010. Since then, Post education columnist Jay Mathews has continued with his rankings, using two simple factors for the evaluation of "Challenging High Schools" : If the number of Advanced Placement or International Baccalaureate Exams given in a particular year at the school divided by the number of school's graduating seniors that year is greater than 1.0, the school makes the list.

To be sure, the Post rankings suffer from some of the same shortcomings as the other ranking systems. The top-ranked schools are typically magnets or charter high schools. And, on the Post rankings, the number of exams and graduates are self-reported.

In Idaho last year, 10 high schools applied for and made the Washington Post list. Here they are, with their ratio (AP or IB/Grads) and free/reduced lunch percentages:

Especially impressive on the Post list are the achievements of Borah, Capital, Century, and Vallivue, which have successfully emphasized rigorous coursework and achievement with high numbers of disadvantaged students.

On balance, we prefer the Washington Post list to the others, since it is simple and understandable. It was a District goal in the early 2000's to have all 4 comprehensive high schools on the Post list, simply to prove to ourselves that we could do it. Now that all four have made the list for 6 straight years, our challenge is to expand opportunities for all students to access rigorous coursework and preparatory opportunities.

Clearly, it's extremely difficult to put together a ranking system that reflects all of the characteristics of a successful school.Witness the difficulty Idaho has had with the establishment of the "Five Star System", which relied only on achievement and produced school  rankings which were unstable and unreliable.


The combined value of school culture, achievement and depth of offerings such as music, art and athletics may be harder to quantify, but is perhaps more important to student success than any annual school ranking list can adequately portray.