Bob Morse, Champion for Change

The most passive journalist on the planet is at it again today, where he discusses the latest ABA jobs data changes.

U.S. News had been among those urging the ABA to take action to improve the credibility of the important consumer information.

Yes, "urging." See, kids, this is how reform works. Figureheads like Bob Morse are a position to change the system immediately, and yet they do nothing. Then, when the people who actually care about reform show up and make a self-evident point, the coat-riders like Morse join the chorus. Then, when they write the history, they were a full-fledged member of the justified opposition, even though they refused to enact any reform steps for years.

But that's not all the magic from Bob's hand.

If the new ABA rules are implemented, U.S. News will use our own law school statistical surveys in fall 2011 to collect and eventually publish the entire new richer and more detailed set of employment and jobs data from each law school for 2010 J.D. graduates. When we gather this richer data set, we will be able to make a more exact determination of how our ranking methodology will change.

Think about this for a moment. The new ABA guidelines already make clear what information will be collected. Granted, Morse and Co. don't know what the response rates will be or how reliable the state-wide surveys will be, but you'd think they could be working with the methodology prior to getting the data itself. After all, statistical methodology should be theoretically-sound independent of the data.

Is Morse being ambiguous and as commitment-free as possible so he can make methadological adjustments to keep the T-14 the T-14, or the T-6 the T-6? I'll leave you to speculate, but it's hard for a cynic like me to not make that assumption given the nature of the propaganda business.

Bob Morse Responds, Tries to Push Blame on Law Schools

Well, after last week's New York Times story on how law schools use merit scholarships to boost their USNWR ranking, which get pulled away after the first year. As I noted, this is the expected response to such a system where schools are incentivized to act in a way that doesn't always neatly conform to standards of ethics or what an objectively-good law school would do. All of this is the natural consequence of the USNWR rankings having almost no merit in actually evaluating law schools.

Naturally, Bob Morse has decided to respond to USNWR role in the scholarship game and, as seemingly usual, wants to pretend it's someone else's issue. After a "no duh" paragraph about students understanding risks and the need for disclosure, Bob lays out this gem of a passage (numbers mine for purposes of commentary below):

It's clear that the U.S. News law school rankings have a (1) large impact on law schools and prospective law school students. (2) However, the U.S. News Best Law School rankings are not why students lose their scholarships. In addition, the article (3) implies that the U.S. News rankings are the key factor behind why law schools are offering more merit-based aid and less need-based aid in order to enroll students with higher LSATs and GPAs and, as a result, improve in the rankings. (4) Law schools need to take far more direct responsibility for their policies instead of citing the oft-repeated claims that they are forced into these actions solely because U.S. News exerts so much power over law school behavior.

(1) With the power of national journalism (or are we going to admit that USNWR is no longer "journalism?") comes a grave responsibility in understanding the effects of one's words. That's why it's essential for the journalist to report accurately what happens, not base his conclusions on small sample sizes or unexamined self-reporting. And the journalist has a responsibility not to have a conflict of interest in the resolution of what happens. Bob's admission that his rankings have great effect is an admission that he either has a great amount of responsibility to get it right or that he's not a journalist at all.

(2) Technically, no, Bob, but your rankings are essential in establishing a system where parties act on reliance on insincere (as in, they're getting pulled after one year) scholarship offers - often advertised as three-year deals - that only exist because your magazine irrationally believes the LSAT to be relevant to how good a law school is; and many of those students actually do lose their scholarships. It also causes the rankings to favor richer schools, possibly at the expense of more efficient ones, and it causes schools to shift more money to their merit scholarship balances. Your magazine is a but-for cause of this mess, and disclaiming responsibility because you had nothing to do with Johnny taking the bait and getting a C+ in torts is absurd.

(3) "Implies?" No, Bob. The article "implied" nothing, and to say otherwise is disingenuous weasel-wording.

(4) Why don't you take responsibility as a "journalist" for the power your words have over people's actions, especially those who need metrics to boast of their accomplishments? There are countless steps you could take to prevent potentially-deceptive and abusive behavior from occurring. You could factor scholarship retention rates into your rankings. You could factor need-based scholarships into your rankings. You could collect and disclose scholarship offering and retention rates. You could do surveys comparing students' expected debt totals to their actual debt totals and penalize schools for the difference.

There are all sorts of steps to take to aid USNWR be a detached voice that helps the consumers understand the truth and have the facts necessary to make a decision about what law schools are good. To date, USNWR has done none of them and has persisted in using a formula that has nothing to do with actual law school merit. And yet, when they're called on it, they throw up their hands and say "that's the law schools' fault!"

Well, no, it's not. As a news institution, USNWR has a responsibility not to be a party in things like this. As long as they have the word "news" in the masthead, people like Bob Morse can't abnegate that responsibility by some twisted logic. So Bob, don't throw up your hands. Do your damned job.

Either that, or change the name and admit you're just a puppet for the elite educational lobby.

What's Up with the Virginia Bar Exam?

We all know that schools manipulate employment information because it's self-reported to the ABA and the rankings guru, but I wouldn't think a school could have any way of manipulating its bar exam passage rates. But then I saw this article today and I wondered if the wizards behind the curtain at Liberty had pulled it off:

Liberty University School of Law students achieved a 100 percent pass rate on the Virginia Bar Examination in February 2011.
...
Liberty is the only Virginia school to achieve the 100 percent rate in the state this year. Washington & Lee was at 90 percent; George Mason, 84.8 percent; University of Virginia, 75 percent; and the College of William & Mary, 73.3 percent.

I might actually believe the 100 percent claim with minimal skepticism if all the schools were above 90 percent. But with UVa and William and Mary at 75 and 73.3, I'm not sure I can buy it. I'm not an elitist by any stretch, but come on. Liberty is bottom-tier. The median LSAT in 2009 was a 150. 1L attrition in the Class of 2008 was 27.1%.

Liberty has in the past overstated its bar performance, but as far as I know there's no accounting metric that allows you to say student x passed they bar when they clearly failed. The official page says they hit 100%. Did only 5 students sit for it? The percentages look as though it may have been a low turnout.

In any event, it looks like the University of Virginia is taking measures to boost its pass rate (and claim it's doing something for unemployed graduates), but in the grand scheme of things, isn't this report - and others where third-tier toilets occasionally have higher pass rates than better-ranked schools - either an indictment of the bar exam process OR of the rankings system and prestige-based law?

The bar exam is either a good metric of competency or it isn't. If it is, the fact that lower-ranked schools can "do more with less" by taking 153 LSAT kids and getting them licensed while a school with 165 LSAT kids cannot get as many to pass signifies that the former institution is actually a become a competent lawyer. If it's not, then we should just scrap the thing and find a different way to certify people.

See also: http://www.ilrg.com/rankings/law/index.php/1/desc/Bar, where it shows in the 2009 data where you can see many lower-ranked schools outperforming higher ranked peers.

On Malcom Gladwell's Law School Rankings

Smug intellectual Malcom Gladwell has come with his own law school rankings:

Gladwell, relying partly on a ranking tool devised by Indiana Law professor Jeffrey Stake, takes a stab at ranking law schools based on 3 factors: the value for the dollar (40%); LSAT scores (40%); and faculty publishing (20%).

As I've previously pointed out, such an approach is 60% hopelessly dim-witted. But he gets kudos for at least attempted to factor in investment principles, although I have no idea how he computed "value" and he's hardly the first person to suggest we should be looking at tuition amounts.

He's still wrong (Faculty publishing? Really?), he's just not as wrong as the US News and World Report.

Propagandist of the Week: Bob Morse

This is from Bob Morse's latest defense of the USNWR rankings, titled "Law Schools Report Accurate Data to U.S. News":

Just how honest were law schools when they reported their data to U.S. News for our 2011 Best Law Schools rankings? Each year, we ask law schools to report the same statistical information to us that they report on the American Bar Association's (ABA) annual accreditation questionnaire. Despite some notable exceptions and data errors over the years, it turns out the schools are pretty reliable in their data reporting.

To support this absurd confusion of reliability with honesty (you can reliably lie, can't you?), he points to a correlative study done by Chapman's Tom Bell, which takes the data provided to the ABA and uses the USNWR's methodology to reach the same conclusions that USNWR did. Then Morse drops this absolute gem of propaganda bullshit:

The fact that Professor Bell was able to duplicate our methodology by using law school data he obtained directly from the ABA proves that the U.S. News rankings process is very transparent and can be duplicated using publicly available data. This exercise also establishes that U.S. News is calculating and weighting the ranking variables as stated in the posted methodology. Users of our law school rankings can be confident that the results are correct given the weights and rankings variables that U.S. News has chosen.

Finally, does Bell's study also prove that law schools are being accurate in how they report their statistical profile data to the ABA, the general public, and U.S. News? No [ed.: yet your opening speaks of "honest" schools because _______?], this only proves that law schools are being very careful to report the same data to U.S. News that they report to the ABA for accrediting purposes.

Has anyone, in the history of the USNWR rankings, doubted that USNWR was using data that wasn't identical to that given to the ABA, especially since the ABA information is public and USNWR, being excellent journalists, would surely cross-check their data? Has anyone ever suggested that law schools are not "being very careful to report the same data to U.S. News that they report to the ABA for accrediting purposes?"

As far as I know, they haven't. Instead, all Morse did here was publish a blog entry to say that some professor took the listed methodology applied it to the exact same data set, and came up with scientifically-similar results. What the hell is that supposed to show, again?

Ah, yes, that "the results are correct given the weights and rankings variables that U.S. News has chosen."

For most people, the word "correct" would imply that the substance of the rankings is correct, e.g., that Harvard is a better law school than Georgetown, rather than the process used to form the rankings was scientifically sound. Take a look at certain fragments of this piece in order:

• ...schools are pretty reliable in their data reporting.
• ...the U.S. News rankings process is very transparent...
• ...U.S. News is calculating and weighting the ranking variables as stated...
• ...the results are correct...

The way this was composed is deliberately designed to defend the whole system, including the substantive results. The fact that in the next paragraph he disclaims "but the data may not be good" doesn't change the fact that he spent the rest of the piece defending the results derived from that data.

The first issue is, and always has been, that their methodology is fatally flawed. The second issue is, and always has been, that the numbers presented to them are inaccurate.

The fact that Morse was able to take a complete non-story not even worthy of a blog entry and turn it into a rhetorically-sound defense of the rankings' value is admirable. I'm baffled that a professional writer and statistician can conflate the concepts of reliability, honesty, accuracy and precision all in one post. Of course, he did almost the exact same thing a year and a half ago.

The fact that Professor Bell was able to duplicate our methodology and outcomes using data collected from a different source proves that the U.S. News rankings process is very transparent and accurate.

Professor Bell concludes:

"For now, I'll just offer this happy observation: The close fit between USN&WR's scores and the model's scores suggests that law schools did not try game the rankings by telling USN&WR one thing and the ABA (the source of much of the data used in my model) another. Even a skeptic of law school rankings can find something to like in that."

Again, a complete non-story that does not justify the claim made ("the U.S. News rankings process is very . . . accurate") as read by lay people. It's not a "happy" observation and there is no hint that a "skeptic . . . can find something to like" since no ground of skepticism is weakened. The law schools' gaming of the rankings does not take place in presenting different data to the USNWR and the ABA. No one has ever thought that. What law school in its right mind would lie to a journalist when there's contradictory public information available? Do you think law school administrators are dopes?

On the bright side, at least this year's article had a one-line disclaimer that was somehow supposed to alleviate a plausible interpretation of the other four paragraphs. That still doesn't change the fact that it's shameless propaganda calculated to make the rankings seem "accurate," "honest," and "transparent," which seems odd given that just last week Bob was telling us not to take them so seriously.

On the USNWR Rankings

Kimber at Shilling Me Softly is reporting that the USNWR will adjust the information presented about its law school rankings to give a better portrait of the information colleges submit to them (i.e. by providing more information to put employment numbers in context, not any change to the methodology) based on articulated criticism from Law School Transparency.

Even though I don't like the concept of the rankings generally, I realize they do have an effect. I obviously think this is good news; isn't anything that presents a more honest picture of the legal profession?

But I wanted to talk more about USNRW in general. I found this article here, about a recent study of which USNWR was a partner that looked at the nature of college rankings (the article is part 2 of a 3-part series:

3. Do the U.S. News rankings "put pressure on institutions to invest in strategies and practices primarily for the purpose of maintaining or strengthening position in the rankings," either consistently or occasionally? An overwhelming majority (95.1 percent) of NACAC's members at both colleges and high schools believe that "Yes, colleges either occasionally or consistently invest in strategies and policies to improve in the rankings."
...
4. Does your school or institution make programmatic changes because of rankings? The report concluded that 54.1 percent of NACAC's members representing colleges reported that their particular institutions do not make any programmatic changes based on the ranking. However, it said that 7.6 percent say that their school consistently makes changes and 38.4 percent say that their schools make changes occasionally because of the rankings.

Although this is for the undergraduate sector, I strongly suspect that it applies equally (or greater) to law schools, and I find the timing of these revelations significant.

Notice anything curious about questions 3 and 4 when put together? Question 3 asked if schools in the generic sense put in policies to improve their USNWR rank. Question 4 asked members if their institution in particular did it (although I'm not sure what they mean by "programmatic" change and that seems like a question flaw; under a liberal definition of the word, it would mean anything, but people might interpret it narrowly).

In any event, I find the disparity between 3 and 4 interesting. 95% of respondents believe schools make changes, but only 46% of those reporting say their own school does it. Assuming no one thought they were the only school in the country that did it, that's a hypocrisy/sanctimonious rate of around 49%, i.e., those who thought other schools put in programs with the ratings in mind, but their school did not.

That's really high and USNWR isn't dumb. Although normally I'm cynical about such things, there's a part of me that honestly believes USNWR is getting the results back from this survey (and maybe others) and is realizing how widespread and destructive to academia their ratings have become. In the past few decades, USNWR has gone from being a news magazine to being a journalistic outpost whose sole practical purpose to most people anymore is to put out these silly rankings. And when someone's bread and butter may be tainted - and the above discrepancies show the college administrators have no qualms lying about their actions or their motives - it's usually wise to do something about it and find ways to correct the problem.

In the past few years, they've gone entirely online and faced major criticism from college presidents for their rankings. Given the results of both those developments, it wouldn't surprise me in the least if they were actually trying to take care of their rankings and nip accurate criticism - such as LST's that the information presented was not giving a complete picture - in the bud to maintain relevant.