Criticism of college and university rankings (2007 United States)
Criticism of college and university rankings has been voiced by a 2007 movement which developed among faculty and administrators in American Institutions of Higher Education. It follows previous movements in the U.S. and Canada which have criticized the practice of college rankings. The arguments of those who criticize the ranking are that it is not possible to come with a single number that characterizes university performance. Ratings, as argued by academic institutions and their leaders, can be easily manipulated and include such subjective characteristics as the “reputation” determined by surveying university administrators such as chancellors or deans. Methodology of many rankings emphasizes research expenditures as the only measure of scientific accomplishments despite the concern that measuring science by the amount of money spent rather than by the importance and impact of scientific discoveries or the depth of the ideas could encourage costly projects that are not necessary scientifically sound.
Sarah Lawrence College
In 2007, educators in the United States began to question the impact of rankings on the college admissions process, coinciding with the newly released Washington Post article "The Cost of Bucking College Rankings" by Dr. Michele Tolela Myers. As Sarah Lawrence College dropped its SAT test score submission requirement for its undergraduate applicants in 2003, SLC does not have SAT data to send to U.S. News for its national survey. Of this decision, Myers states, "We are a writing-intensive school, and the information produced by SAT scores added little to our ability to predict how a student would do at our college; it did, however, do much to bias admission in favor of those who could afford expensive coaching sessions.
As a result of this policy, in the same Washington Post article, Dr. Myers stated that: "I was recently informed by the director of data research at U.S. News, the person at the magazine who has a lot to say about how the rankings are computed, that absent students' SAT scores, the magazine will calculate the college's ranking by assuming an arbitrary average SAT score of one standard deviation below the average score of our peer group. In other words, in the absence of real data, they will make up a number. He made clear to me that he believes that schools that do not use SAT scores in their admission process are admitting less capable students and therefore should lose points on their selectivity index."
Myers further stated that "several faculty members and deans suggested that perhaps it was time to stop playing ranking roulette and opt out of the survey." Myers next argued that at the NEAIR 33rd Annual Conference in 2006, a talk given by U.S. News, "indicated that if a school stops sending data, the default assumption will be that it performs one standard deviation below the mean on numerous factors for which U.S. News can't find published data. Again, making up the numbers it can't get. The message is clear. Unless we are willing to be badly misrepresented, we had better send the information the magazine wants."
U.S. News and World Report official response
U.S. News and World Report issued a response to this article on 12 March 2007 which stated: "Sarah Lawrence's decision is unique, and the magazine's handling of it is still under consideration. Some colleges have made SAT or ACT scores optional in the admissions process, but to our knowledge, no other major college has decided to disregard them completely. Our rankings are painstakingly tabulated, using the best data available. U.S. News data researchers regularly participate in briefings and conferences where the most complicated nuances of the process are discussed with the ranked institutions. We regularly adjust to changes in the educational environment, and we plan to address this circumstance in a similar manner."Presidents Letter
The Presidents Letter, developed by Lloyd Thacker of the Education Conservancy, was sent to college and university presidents in the United States in May 2007, concerning the U.S. News and World Report college rankings. The letter does not ask for a full boycott but rather states that:Instead, it asks presidents not to participate in the "reputational survey" portion of the overall survey. The letter also asks presidents not to use the rankings as a form of publicity:
List of colleges and universities
Twelve college and university presidents originally signed the letter in early May. The letter currently has sixty-one signatures, though others may be added at a later date.- Bethany College
- Dickinson College
- Drew University
- Earlham College
- Heritage University
- Lafayette College
- Marlboro College
- Southwestern University
- St. John's College
- St. John's College
- Trinity Washington University
- Wheelock College
Current list of additional schools
Annapolis Group meeting
On 19 June 2007, during the annual meeting of the Annapolis Group, which represents over 100 liberal arts colleges, members discussed the letter to college presidents. As a result, "a majority of the approximately 80 presidents at the meeting said that they did not intend to participate in the U.S. News reputational rankings in the future." However, the decision to fill out the reputational survey or not will be left up to each individual college as: "the Annapolis Group is not a legislative body and any decision about participating in the US News rankings rests with the individual institutions."The statement also said that its members "have agreed to participate in the development of an alternative common format that presents information about their colleges for students and their families to use in the college search process." This database will be web based and developed in conjunction with higher education organizations including the National Association of Independent Colleges and Universities and the Council of Independent Colleges.
The new database was described in TIME magazine as "a web-based alternative to the rankings that is being spearheaded by the 900-member National Association of Independent Colleges and Universities. NAICU's easy-to-read template, which is expected to be rolled out by hundreds of schools in September, allows students and their families to pull up extensive information organized in an objective format that includes such data as what percentage of students graduate in four years compared to those who graduate in five or six years."
U.S. News and World Report official response and debate
On 22 June 2007, U.S. News and World Report editor Robert Morse issued a response in which he argued, "in terms of the peer assessment survey, we at U.S. News firmly believe the survey has significant value because it allows us to measure the "intangibles" of a college that we can't measure through statistical data. Plus, the reputation of a school can help get that all-important first job and plays a key part in which grad school someone will be able to get into. The peer survey is by nature subjective, but the technique of asking industry leaders to rate their competitors is a commonly accepted practice. The results from the peer survey also can act to level the playing field between private and public colleges."In reference to the alternative database discussed by the Annapolis Group, Morse also argued, "it's important to point out that the Annapolis Group's stated goal of presenting college data in a common format has been tried before U.S. News has been supplying this exact college information for many years already. And it appears that NAICU will be doing it with significantly less comparability and functionality. U.S. News first collects all these data. Then we post the data on our website in easily accessible, comparable tables. In other words, the Annapolis Group and the others in the NAICU initiative actually are following the lead of U.S. News."
A debate on this issue was published as a podcast in the 25 June 2007 issue of Inside Higher Ed. The debate was between Lloyd Thacker, director of the Education Conservancy, who is a well known critic of the U.S. News rankings, and U.S. News editor Brian Kelly. The debate was moderated by Inside Higher Ed reporter, Scott Jaschik.
Statement from Annapolis Group chair
Chair of the Annapolis Group, and president of Gettysburg College, Katherine Haley Will, discussed this decision further in a 9 July 2007 article for The Washington Post. In this article, Hill states that this decision was not based upon "a lack of concern about providing accurate, comprehensive information to help students and their families make decisions about college." Rather, she argued against the methodology of the U.S. News rankings. In particular, she argues against "the largest single factor in the U.S. News rating formula" which is the reputational survey as, "it is unrealistic to expect academic officials to know enough about hundreds of institutions to fairly evaluate the quality of their programs." Hill then argues that, "by contrast, 1 percent of the U.S. News ratings formula is assigned to student-to-faculty ratios, which many faculty members and students consider the most important factor in educational experience." Hill states that the members of the Annapolis Group will offer the same information in an alternative, free, format which will not rank schools, as, "an educational experience can't be reduced to one number, a school's so-called rank. The simplicity of a rank is understandably more appealing than spending hours poring over college catalogues and visiting campuses, but myriad complex variables can't be reduced to a single number." Instead, Hill asks students and parents to "compare schools on a variety of factors they should visit campuses and go on what feels like a good match rather than relying on filtered or secondhand information. We must encourage students to look inside their hearts and trust their instincts when it comes to choosing a college, not whether parents or friends think a university is cool or prestigious."College presidents: responses
A number of presidents have issued responses to these events. One of them, Presbyterian College president John Griffith, compared this movement to a form of revolution: "I have lived long enough to come to the conclusion that major shifts occur every quarter century or so in the way American culture approaches matters of importance. We often call those shifts revolutions because people revolt against old and outmoded ways of doing things in favor of new approaches, new technologies and new ideas that better meet the needs of the time. We have experienced revolutions in information technology, travel and communication. There is one going on now that is symbolized by the introduction of iPhones this past week; we know what this one is about. But there is another revolution going on related to choosing a college -- and the role that public rankings play in that choice -- that may be less clear."Economics and endowment
Presidents have also discussed the role of endowment, correlating a high ranking on the survey with institutional wealth. President of Muhlenberg College, Peyton Helm, argued that "most of the other factors weighted by U.S. News in their rankings are based, ultimately, on institutional wealth A trustee once asked me what it would take for Muhlenberg to be ranked in the top five by U.S. News. My answer was simple: A check for $800 million placed directly in the endowment would do it -- even if we never changed another thing we were doing." Helms also noted that, "what you won't read in U.S. News is that most of the data they use is public information, readily available on the Web sites of most colleges and universities, as well as on the U.S. Department of Education Web site. There is no single formula for weighting these factors -- they will have different significance for different students and families. So, next year I and many other leaders of our nation's best colleges and universities will be working on a new and better Web-based tool for families engaged in the college search." Millsaps College president, Frances Lucas, further noted that, "she previously had paid little attention to the rankings debate because her own institution was rated highly in U.S. News. But after learning more about the magazine's methodology and discussing the issue with colleagues at this week's meeting, she concluded that the rankings were based too heavily on measurements determined by institutional wealth."Historically Black Colleges and Universities
President Walter Kimbrough of the historically black college, Philander Smith College, argued that U.S. News, "focuses on institutional resources, student selectivity and graduation rates to select the top institutions. But since many HBCUs struggle with these issues, he says the rankings in effect discourage students from going to those schools If there are people looking at the rankings as a measurement of the quality of an institution, they think do not have any type of qualities do not tell you who the best schools are, just the most privileged."Reputational survey
Former president of Sarah Lawrence College, Michele Tolela Myers, in discussing her decision to no longer submit information to U.S. News, stated, "they will do what they will do, we will do what we will do. And we want to do it in a principled way." Myers also indicated in a press release for the college magazine, Sarah Lawrence, that the college will be involved in developing the new database of colleges discussed in the Annapolis Group statement as they "believe in accountability and openness, and that the public has a right to solid and reliable information about the important decisions involved in choosing a college." The press release also indicated that Sarah Lawrence "plans not to participate in the peer reputational survey or data collection for U.S. News and World Report's rankings" as, according to Myers, "by submitting data and the peer reputation survey we have tacitly been endorsing these rankings all the information we have provided to U.S. News in the past will be available to the public through other channels.”Other presidents have also commented on the reputational survey. Former president of Scripps College, Nancy Y. Bekavac also stated in a press release on the college website that Scripps will also no longer submit the Reputational Survey to U.S. News as "for years we have known of flaws in the methodology; many of us have spoken with editors at U.S. News in an attempt to improve its approach but nothing can really improve a system that seeks to reduce 3,300 educational programs in American higher education to one set of numbers, and then rank them. College presidents, academic deans and deans of admission do not know enough about other institutions to make meaningful comparisons. This gives a false sense of reliability to what is a ranking system without any real validity." Sweet Briar College president, Elizabeth S. Muhlenfeld stated that, "one of our colleagues likened it to trying to rank composers. It's a great analogy. How can you say that Beethoven and Brahms are better than Mahler or Mozart?" Trinity Washington University president Patricia McGuire argued that, "the survey asks me to 'rate the academic quality of undergraduate programs,' assigning each school a single score using a 1-to-5 scale from 'marginal' to 'distinguished.' That I have little real information about these 181 institutions does not seem to matter to the U.S. News editors Some of the actual best colleges in this nation do not fare well in the U.S. News survey because they do not have the wealth, big-time sports notoriety or public relations clout to influence the peer voting system." Finally, DePauw University president, Robert G. Bottoms, argued that, "I, in fact, did not fill out the reputational survey for this past year. I came to the conclusion that I am not in a position to make judgments on other schools, many of which I have little or no familiarity with. The fact that one quarter of a college's ranking is based upon what is, in essence, its popularity, is very disturbing and we choose not to be a part of the process."
Third-party involvement
, president of Vassar College argued that, "many of us in higher education dislike popular college rankings such as the annual academic beauty pageant from US News & World Report. But expecting them to go away is naive, and attempting to undermine them is unwise since students and families could perceive that as petulant and paternalistic. Worse, it could seem as if we have something to hide." Rather than not sending in the reputation survey, she argued, it would be of value to focus on, "a third-party non-profit or foundation", sending them "the same data that we already submit to US News and other rating organizations." On this point, she argues, "a one-size ranking does not fit all, because students and families care about different things What if a school doesn't use the SAT in making admissions decisions and therefore doesn't collect or report these data? In a new system, that school couldn't be ranked if a student chose a positive weight for the SATs. Students would know that the school doesn't value that piece of information. They could then run the rankings with other information, excluding the SAT, and see what those rankings look like. Alternatively, they could decide they actually do care about the average SATs of the student body and decide to look at other schools. Fair enough."Responses
Economics and endowment
Other academic administrators have discussed the correlation between economics, college choice, and the rankings. David McGrath, emeritus professor of English, College of DuPage, discussed his own decision to attend Chicago State University in the 24 July 2007 Chicago Tribune article, Ode to a fourth-tier college. Of this decision, he noted that, "I qualified for admission elsewhere, but CSU was close to my part-time job, and it was cheap I never required a student loan since I earned enough as a grocery bagger to pay tuition and fees in 1970 that totaled $300 per year. All told, a pretty good value, even for a fourth-tier school." McGrath considered it a "good value," because, "CSU eventually led to a teaching career, and my working alongside professors from Princeton, Northwestern, and the University of Chicago. I earned the same ample salary and benefits as they, and, more important, was privileged to engage in the same kind of fulfilling work." He also referenced the 2000 Krueger-Dale study and noted that, "too often, it seems, students choose colleges the way they choose jeans or athletic shoes. They would rather bust the family budget than be caught dead in sweats bearing an unrecognizable school logo. But it's their ability, work ethic and dedication that determine the height of their achievement."Author and journalist, Peter Sacks, narrows the argument by suggesting a direct correlation between the wealth of school and its rank. He suggests that "the ranking amounts to little more than a pseudo-scientific and yet popularly legitimate tool for perpetuating inequality between educational haves and have nots -- the rich families from the poor ones, and the well-endowed schools from the poorly endowed ones. Toss in the most heavily weighted factor in the U.S. News survey, the assessment of deans, college presidents, admissions officials and others regarding their peer institutions, and you get the perfect recipe for a self-perpetuating, class-based rankings system driven by brand names, marketing hype, and prestige."
University recruitment
In the summer of 1996, Marshall University sociology professor Dr. William Westbrook was having a conversation with a recent Master's graduate. The graduate inquired about Marshall's traditional admissions policy, which had a lower SAT/ACT apparent requirement than his undergraduate school, Shepherd University, with a "selective" admissions policy and higher SAT/ACT apparent requirement. The graduate made an incorrect inference from the averages and thought Shepherd was arbitrarily turning new students away below a higher fixed ACT or SAT score. Dr. Westbrook clarified the situation first by explaining that every university or college is going to meet its enrollment quota with the best students available, and asked where both Marshall University and Shepherd University got their students. Marshall typically recruited from Cabell, Logan, Wayne, and Putnam Counties, WV, and Lawrence County, OH. Shepherd typically recruited both from West Virginia, both locally in the Eastern Panhandle and in the interior of and elsewhere in the state, and from the comparatively wealthy Maryland and Virginia suburbs surrounding Washington D.C. Dr. Westbrook made the point that SAT and ACT scores are an indirect measure of socioeconomic status. County kindergarten to twelfth-grade school boards are funded by property taxes assessed on home values. At nearly the end of grade school, a college-bound student takes the SAT or ACT. The difference between Shepherd University and Marshall University is that Shepherd recruited more from areas whose home values were higher, whose homeowners paid more in property taxes, whose school systems were better funded, and whose students benefitted as a result, took the SAT or ACT, some of whom applied to Shepherd University and were accepted, and attended. Colleges and universities, then, have less control over setting their ACT and SAT requirements than the graduate assumed and than most parents might imagine. Parents need to understand that large scale political and economic forces such as the level of Federal employment and commensurate income around Washington, or other regional differences in living standards, have more to do setting ACT and SAT requirements than the decisions of college committees trying to enroll the best students they can under the circumstances over which they have no control.Gaming and methodology
Assistant to the Dean for the University of North Carolina School of Law, Sarah E. Wald, noted, "the rankings purport to give an overall order to colleges and graduate schools to help students make the best decisions about where to attend school. But universities all know how misleading and even destructive these rankings can be. It's common knowledge how the statistics can be 'gamed.' Colleges can solicit applications from students with little chance of acceptance to boost how selective they appear. Schools can adjust when they allow faculty to take leave in order to raise the faculty/student ratio. And admitting more "risky" students on transfer rather than in the initial class results in a higher freshman SAT average."Professor Marty Kaplan of the USC Annenberg School for Communication further argued that, "the problem with U.S. News' college rankings isn't that institutions of higher education shouldn't be held accountable for the quality of services they provide The problem is that the fierce competition among colleges to raise their rankings torques the priorities of colleges toward the criteria that U.S. News uses So this week, when an association of 80 liberal arts college presidents, including Barnard, Sarah Lawrence and Kenyon, announced that a majority of them would no longer participate in the U.S. News annual survey, and that they would fashion their own way to collect and report common data, it was bad news for the magazine, but good news for families. It's also good news for American higher education, some of whose institutions may now become less timid about accepting the quirky applicant, less nuts about generating journalistic puff pieces, and more bold about declaring unique educational missions that don't derive from focus groups.
Senior scholar at The Carnegie Foundation for the Advancement of Teaching, Alexander C. McCormick, adds to the above discussion by arguing against the way in which the Carnegie Classification of Institutions of Higher Education is used in the creation of U.S. News rankings. The problem, he argues, with this use is that there is "no basis for inferring national versus regional focus, because it's not a factor in the classification criteria. So it should come as no surprise that the national and regional lists contain a great many inconsistencies and bizarre placements By continuing to rely on the Carnegie Classification, they avoid the tough job of defining their terms."