(Most) Law Firm Rankings Make No Sense To Me

I’ve spent over 20 years working in the legal industry now, including 14 years at law firms — the last dozen or so I’ve specialized in working with legal data.

For years, I watched with interest at how much time and money firms and lawyers sink into their efforts to rank highly in various legal directories and rankings (Chambers, Legal500, IFLR1000 etc.).

These directory rankings seemed great to me… until I actually read the fine print.

The Problem with “Reputation” Rankings Is in the Fine Print

If you haven’t read the fine print in these rankings lists, you should. If you do, you'll see that, in general, they all follow a similar approach:

1. They ask for deal highlights for the past 12-18 months.
2. They make a bunch of [mostly] subjective judgements about these deals re: their complexity, size, impact, innovation, etc.
3. They gather client feedback from people hand selected by the firm or the lawyer being rated.

Chambers, for example, says this in its methodology: "Our rankings are based on real interviews with your selected references."

Maybe I’ve spent too many days looking at data tables, but that approach makes no sense to me.

I had to check with my lawyer friends to ensure I wasn't missing something.

Is This Kind of Research Meaningful?

Why would I care about your reputation rank when those rankings are based largely on what junior researchers hear on phone calls held with people referred to them by the people and firms being ranked? Who are discussing the matters those same people selected?

Does anyone believe that a lawyer would give a rankings agency a reference who wouldn’t provide a sparkling recommendation?

As you’ll see below, in my current role here at PERSUIT, we've used our data to provide a much more objective (and trustworthy) set of rankings for law firms.

Rankings that are based on repeated buying behavior, not just someone’s opinion. So we can get data that looks like this, tracking trust over time:

More on PERSUIT’s data in a bit…

A Broken Process

This week, Silvia Van den Bruel, Marketing and BD Director at Hausfeld, wrote an article commenting on these directories — and on the Chambers business model in particular.

And she totally nailed it.

“The line between research and payment has faded. Today, if you do not pay, you can't play … Dealing with Chambers and The Legal 500 directories has become an all-consuming task between November and March and, being part of a small team, takes me away from doing ‘real’ marketing and business development.”

Yet, as Van den Bruel also wrote, “not taking part as a ranked firm is totally unthinkable.” More on her article and the discussion here.

I wholeheartedly agree with Van den Bruel’s perspective (well, not the last part; it’s very ‘thinkable’ to me), but I also think we can and need to say it more strongly.

Most law firm and lawyer rankings are just not valuable.

And I'm pretty sure everyone knows it.

Here’s why…

The Difference Between Subjective Rankings and Objective Rankings

There are some rankings that I find useful — like NHL power rankings or best-selling items on Amazon.

Or take the Am Law 100 and the Global 200, for example. These rankings are based on objective metrics such as total revenue or profit per equity partner.

Those metrics — while not perfect — let you know what you’re getting, and they’re based on actual data. It may not be data you attribute much value to, but at least they avoid rankings based on recommendations from someone’s golf partner or college fraternity member.

The other directories?

The ones based on lengthy submissions, subjective interviews, self-selected referrals and “pay-to-play” options?

Those I don’t find any value in.

Why Are Subjective Rankings a Problem?

Let me count the ways…

1. The Research Process Is Almost Entirely Subjective

If you’re reading this, you almost certainly know what I’m talking about.

Firms spend months putting together submissions they send to these directories. Those submissions are prepared by teams of professional marketers and include their very best matters and references hand-picked by firms. These are people who can “speak to” their experience of working with the firm on those matters.

In Big Law, these references are clients of the firm or the lawyer and they can speak to their relationship with the firm or the lawyer. They are also often golf buddies or former colleagues.

The ratings organizations review submissions, check references, and conduct interviews with the firms, usually with several partners on a conference call.

The interview is part of the “in-depth research” process they promote. But in reality, it’s often a call made by a junior-level attorney reading a list of standard questions.

That gives us three main inputs for these rankings:

1. Submissions by the firms.
2. Reference checks of people hand selected by the firms.
3. Research interviews (often) conducted by very junior level associates.

All of this data is subjective, and then the ranking organization make their own subjective decisions about the findings.

2. Huge Opportunities to Game the System

So many of these lists claim that their rankings reflect “reputation” or “ability.”

If I had a professional marketing team (as so many firms do) preparing a profile of my professional reputation and ability, I could make myself sound incredibly awesome too.

Even more so if the references you used were my hand-picked close friends and former colleagues — people I can easily ask to say specific things about me when asked.

It’s not like Amazon or Yelp, places where customers can report negative experiences as well as positive ones. Those systems have their own problems, but at least there’s an opportunity for customers with negative experiences to impact the ratings.

With legal directories, the only customers with a voice are the ones I hand pick to participate.

Who wouldn't game a system like this by stacking the referrals with their very best clients?

3. Little Connection to Actual Performance or Results

What does it mean to be a “good” lawyer or law firm anyway?

I would argue that it means you’re able to produce results or desirable outcomes for your clients.

Winning cases. Negotiating settlements. Closing acquisitions. Managing risk. And doing these things efficiently and effectively, without unnecessary delay or fees.

I don’t see a connection between these things and the rankings so popular in the industry.

I admit this is — at least in part — due to the nature of the industry. When clients work with one firm, there’s no direct comparison between that experience and the experience they’d have with a different firm on the same matter.

It’s not like trying on two pairs of shoes and then comparing how they fit.

4. No Measure of Real Buying Behavior (by Clients)

“Actions speak louder than words.”

It’s a cliche but it’s also true.

Imagine if these directories added objective measures of trust to their rankings — something like “repeat business generated,” for example. Doing so would include an objective measure showing which firms and lawyers are earning the trust of their clients again and again.

Some of them do talk about trying to judge how much market share they have in a certain area. I don’t know quite how they measure this (they don’t explain how), but I do appreciate that there are a few attempts to be at least a little bit objective.

As we see it at PERSUIT, the more you return to buy something, the more you trust it. And that’s true whether we’re talking about a preferred clothing brand or a multi-million-dollar relationship with a law firm.

Buying behavior is a data point that looks at actions, not words.

It’s why, in our firm rankings at PERSUIT, we lean heavily on objective measures such as “total amount awarded” and “total amount awarded from repeat clients” and we enforce minimum thresholds such as number of matters won and number of clients.

An Alternative (and Objective) Way to Rank Law Firms

Last year, our data team at PERSUIT put together a first-of-its-kind law firm rankings report.

And last week, we released the second annual edition.

It’s a rankings report that doesn’t rely on subjective interviews or references. And we didn’t accept any form of sponsorship or payment.

It’s called The PERSUIT 50: The Most Trusted Firms on PERSUIT. You can see the full report here.