Allegheny College: President's Office

Macalester College's president, Michael McPherson, calls it "silliness." Bard's president, Leon Botstein, is more stinging in his assessment: "It is the most successful journalistic scam I have seen in my entire adult lifetime -- corrupt, intellectually bankrupt and revolting." A national survey indicates that 92 percent of college and university presidents believe that the results are inaccurate.

"It" is the U.S. News and World Report's annual "America's Best Colleges" issue, which purports to rank some 1,400 colleges and universities across the country. Newsstand sales of this issue run 40 percent higher than typical issues, and a highly profitable industry has grown up around the idea of measuring the quality of America's wide range of institutions of higher education. Although Allegheny College is grouped among other highly ranked national liberal arts colleges, we recognize that the ratings fail to capture our historic values, unique characteristics, and educational effectiveness.

Criticism of the rankings is widespread, including presidents of some of the most highly ranked institutions. Even the former director of data collection and analysis for the study says that the U.S. News and World Report (USNWR) ranking system "defies common sense and doesn't measure anything that actually occurred while a student was attending the school or any learning that takes place there."

Is there anything wrong with the rankings, and does it matter? The answer to both questions is a definite yes. Through its use of survey data and a complex formula of calculation, the rankings imply qualitative measures and quantitative precision that simply do not exist.

Reputation remains the single most important variable in the USNWR methodology. Each year, college presidents, chief academic officers, and admissions deans are asked to place each of as many as several hundred of their peer institutions into one of four "quality" categories. It is unrealistic to think that even experienced college administrators can know much about the real quality of more than a handful of institutions. One of the most fascinating indicators of this fact is the surprisingly high reputational rank of one tiny, struggling college that happens to bear a variation of the name of a prominent and well-respected college in the same geographical region. Hardly the makings of a reliable study.

Few of the other measures have any direct correlation with the effectiveness of the student experience, and none seeks information from students themselves. Wealthy institutions fare well because colleges are directly rewarded in the formula for program expenditures, faculty compensation, and alumni giving. Indirectly, wealthy schools benefit in the rankings from name recognition, reputation, and high SAT scores of incoming students, influenced in part by the level of merit- and need-based aid a college awards.

The sensitivity of the rankings to this "wealth effect" is dramatically illustrated by specific examples of institutions that have experienced significant changes in their rankings in recent years. One college leapfrogged into the top echelon when it received an endowment bequest totaling several hundred million dollars. A major research university jumped to the prized Number 1 spot when ranking methodology was changed to standardize how research expenditures were counted. Does anyone imagine that these kinds of ranking artifacts have much to do with institutional quality?

Rank does seem to matter, however, both in student behavior and institutional reaction. A recent study at Cornell found that improvement in college rankings resulted in more applications, increased SAT scores among applicants, and better yield of matriculants.

To improve performance in the rankings, some institutions emphasize early-decision programs in admissions to improve yield or have changed their admissions decisions and financial aid policies to improve the most easily quantifiable aspects of incoming classes' academic profiles. Pouring time and resources into improving reported data or reputation in an attempt to improve rank has resulted in an unfortunate arms race that has little to do with improving actual educational quality.

Not only do rankings run the risk of distorting institutional mission and integrity, they may mislead students into selecting the wrong institution. But it's not enough to simply complain about the rankings. The continuing popularity of the USNWR rankings since their introduction in 1983 indicates that students and parents want accessible, comparative information about colleges and universities. Educational researchers have not provided useful comparisons of institutional characteristics and effectiveness, so we should not be surprised that a news magazine is perceived by many as the most authoritative source of information and judgments on college quality.

Fortunately, researchers and colleges recently have taken a giant leap forward to close the information gap. Sponsored by the Pew Charitable Trusts and the Carnegie Foundation for the Advancement of Teaching, the National Study of Student Engagement (NSSE) measures actual student-reported experiences that are known to be correlated with learning effectiveness. NSSE director George Kuh is fond of saying, "It isn't what a school has, it's what a school does that counts." Allegheny was selected to participate in the second round of this study and has recently received the results. I look forward to sharing them with you in the next issue of Allegheny.