International In-house Counsel Journal
Vol. 8, No. 30, Winter 2015, 1

TOBY UNWIN
Chief Information Officer, Premonition LLC, USA

Moneyball for Law?

The Brad Pitt film, Moneyball, portrays Billy Beane baseball team manager of the Oakland A’s struggling against accepted dogma in player selection. Beane’s insight was to use already available player data to pick athletes. This flew in the face of accepted conventional wisdom in the game. Talent scouts prided themselves on their “eye” for talent, choosing drafts by gut feel, pitching and batting style, even factors such as their demeanor off the field and the attractiveness of their wives and girlfriends. Despite a rocky start, strongly opposed by the A’s talent scouts, Beane put together a team that went down in baseball history. Derided, The Oakland A’s put together a roster of players believed to be lacking in talent, past their prime or unusable. Not only did the team prove itself by results, they were assembled with one of the lowest budgets in the sport. Win rates were not correlated with salary rates.

Lawyer selection is much like the bad old days of baseball player selection, contends Premonition L.L.C., a Florida based Artificial Intelligence company targeting the legal industry. Premonition claims to know which attorneys usually win before which judges. People pick attorneys based on recommendations from friends, online reviews, because they’re friends, friends of friends, went to a particular law school, have nice offices, work for a well known firm, saw an advertisement, their name was first in the phone book, etc. Many publications run “Top Lawyer” lists, people who are recognized by their peers as being “the best”. Premonition analyzed the win rates of these attorneys, it turned out most were average. The only way that they stood out was a disproportionate number of appealed and re-opened cases, i.e. they were good at dragging out litigation. They discovered that even the law firms themselves were poor at picking litigators. In a study of the United Kingdom Court of Appeals, it found a slight negative correlation of -0.1 between win rates and re-hiring rates, i.e. a barrister 20% better than their peers was actually 2% less likely to be re-hired.

If generally accepted methods for choosing a lawyer don’t work, what does? “Win rate”, says Toby Unwin, inventor of the Premonition system. “The only item that affects the likely outcome of a case is the attorney’s prior win rate, preferably for that case type before that judge”. Wins and losses have traditionally not been tracked in law, perhaps being seen by the profession as a bit tawdry. “Attorneys themselves don’t know their own statistics, nor do the firms they work for. The only thing that gets tracked in law is hours and fees.” Unwin explains. Premonition was formed in March 2014 and expected to find a fertile market for their services amongst the big law firms. They found little appetite and much opposition. “In retrospect it was a stupid idea.” confides Premonition C.E.O and Co-Founder, Guy Kurlandski. “It was like interviewing cows, asking them what they thought of McDonalds. You don’t ask the candlemakers what they think about light bulbs.”

Despite being public record, court data is surprisingly inaccessible in bulk, nor is there a unified system to access it, outside of the Federal Courts. Clerks of courts refused Premonition requests for case data. Resolved to go about it the hard way, Unwin, who was suspended from school for computer hacking at the tender age of 13, wrote a web crawler to mine courthouse web sites for the data, read it, then analyze it in a database. Unwin founded NetSearch, one of the most profitable internet businesses in Europe in 1998, selling it a year later at a $160M valuation. Additional courts would be painstakingly re-coded by hand, each new source taking a week or more. “It was the wrong approach. We had thousands of court to cover and it just wouldn’t scale. We scrapped it and I developed an Artificial Intelligence based system that could actually learn how to crawl a court house. Now each new one takes us an hour.” he says.

Win Rates

Having downloaded hundreds of thousands of cases, Unwin set about the painstaking task of teaching the Artificial Intelligence system to read the cases. Like a child, the system learned slowly at first, but rapidly gained in speed and accuracy. The system downloads new cases in real time, as they are filed. It analyzes them, then can create complex tables, mining overall win rates, differences from judge to judge looking for specialists in varying case types. Most of all, it looks for outliers, litigators with long strings of unbroken wins before particular judges. “Every judge has their favorites.” confides Kurlandski with a twinkle in his eye.

Premonition in action

Kurlandski recounted with glee the story of a paralysis case on the verge of trial for a very nervous insurance client. Premonition downloaded 15,000 cases, 3 years worth of civil actions in the Reno, Nevada Courthouse. The system found an attorney with 22 straight wins before the judge – the next person down was 7. A bit of checking revealed the lawyer was actually a criminal defense specialist who operated out of a strip mall and devoted most of his Web site to his “advocacy for the lesbian, gay, bi-sexual and transgender community”. The lawyer had little reason to be before a civil Judge, let alone so often and with such a stratospheric win rate. Premonition advised he be hired as co- counsel. The case settled within weeks.

The firm claims such outliers are far from rare. Their web site, www.Premonition- Analytics.com, shows an example of an attorney with 32 straight wins before a judge in Orange County, Florida. The attorney, an Associate at a major national law firm, has a win rate outstripping any of the Senior Partners at his office.

Fees

“In law you don’t always get what you pay for”, Kurlandski claims. The firm found no correlation between win rate and billing rate. While there are undoubtedly highly paid top performers amongst the big firms, the majority of the most successful Counsel in each court is found amongst the small firms and solo practitioners that Premonition refers to as “Strip mall superstars. The legal marketplace is broken. It has significant opportunities for “Perception Reality Arbitrage” and Premonition is the only company to know each litigator’s true worth.” He continues «Law has long been seen as a Geffen Good, an economics term whereby the more a consumer pays for something, the better it is perceived to be”.

Case duration

Win rates aren’t the only metric Premonition tracks. “Many attorneys are notorious clock runners. They’ve got away with it for a long time, because there’s been no transparency.” Unwin says. “Clients will often short-sightedly focus on an attorneys’ hourly rate. You need to look at your total spend: hourly rate x length of case losses”. Over a portfolio of cases, these savings can be significant. Premonition can fine tune individual litigators cherry-picked for each case, or pick a better panel for the firms not willing to take the plunge into selecting new litigators every time. The firm claims attorney selection has a 30.70% effect on case success. “There’s the facts, the law and the people.” says Kurlandski.

Analyzing the big firms

Premonition recently completed a study of the major law firms. Targeting the Miami 11th Circuit Court, one of the busiest in the nation, they looked at data from 2010-October 2014, over 100,000 civil cases. Restricting the survey to just the 20 largest law firms by Attorney count in Florida then were able to extrapolate the first ever ranking of Big Law by their results. The insights were quite startling:

Firm choice matters

Firm success rates ranged from 93.51% for a specialist insurance defense firm to 0% for a firm reputed to be “legal surgeons”. They had only 4 cases in the system for that court during the period analyzed, only 2 had been concluded and they lost them both. A couple of Statewide firms posted respectable 64.67% and 64.38% win rates, but 3 National firms scraped in at 37.93%,, 33.93% and 27.78%. Why do General Counsel hire White Shoe firms that lose 3 out of 4 times they go to court? “They don’t know. No one keeps score” explains Kurlandski.

Individual Attorney choice matters more

Success rates are not consistent within firms, they vary from office to office. Unwin showed a Statewide firm that did a lot of litigation in Miami with a good 64.67% win rate but did relatively little in Orlando at a win rate that averaged 50%. Premonition showed the stats for the busiest big firm in Miami. It had a 58.31% average win rate for that office. The bulk of the data came from just one litigator who was responsible for 35.42% of their cases, handling 243 for that period. His win rate was 62.14%, and it strongly affected the average for the firm. There were 2 100% win rate attorneys with 4 and 5 straight wins, but worryingly also one with 0% and 17 straight losses whose chart mark nestled next to a 3.45% win rate.

One lawyer had 28 losses out of 29 appearances. “It’s often a lottery who you get assigned to in these firms”, opines Kurlandski. “If a General Counsel feels they have such a strong relationship with their existing firm that they could never leave it, they should, at least, pick a results-based panel from within that firm, not just accept the first person offered, or insist blindly on a Senior Partner in the hope they are hiring quality.”

There are few Top 20 performers among the Big Firms

There is usually one representative from the top firms before each judge, occasionally two. However many judges have no Big Law top performers before them. It became very clear that the majority of top performers were from small firms or solo practitioners. Should companies just hire from the small firms? “No. You see that just because there are more solo and small firm attorneys than those employed at the big firms. Win rate isn’t correlated with firm size”, Unwin says.

Cover Your Back

Why do General Counsel like Big Law firms so much? “It’s Cover Your Back, Kurlandski believes. “There hasn’t been any reliable data before to track what you’re actually getting. So you hire a big firm and you hope. There’s nothing wrong with hiring a big firm, as long as you have the data to back up your attorney choice within it. If the 33rd case is a loss after 32 wins in a row before that judge, clearly you picked the very best person possible. Having data to back up your hires is the ultimate career defence. At the end of the day, the numbers don’t lie and the attorneys that win more, win more.” Is Premonition at the vanguard of a new era of accountability in law? “Every General Counsel in America will eventually be our client”, Unwin boasts, “The only question is when.”

An unfair advantage?

On March 1st 2014, Alvin Benton Esq. won Case #2013-CA-008850-O in Orange County, Florida before Hon. Margaret Schreiber. The analysts at legal analytics firm, Premonition L.L.C.(www.Premonition-Analytics.com), weren’t surprised. “It’s the 32nd time in a row that he’s won in front of her.” CIO Toby Unwin explained. Can the results of litigation be predicted in advance, based solely on the attorney and who they are appearing in front of? “Not entirely” Unwin explains, “but it has approximately a 30% influence. Over the course of several cases it adds up to significant savings for clients.”

Premonition has a unique database that pulls records directly from individual Courthouses and pools them where they can be analyzed. “That in itself is a breakthrough”, CEO Guy Kurlandski explains. “Until now, unless you had a Federal, or appealed case, there was no central system for searching cases, so you had to find 99% of them by hand at individual court houses around the country. Premonition can search many courts simultaneously.” Data like this can produce interesting results. Using advanced algorithms to spot trends, Kurlandski claims the system usually finds at least one outlier, like Benton, before each judge. “We had a client in Reno, Nevada. After crunching the data on over 15,000 cases we found an attorney with 22 straight wins before that judge. The next lawyer down was at 7. On closer examination we found a lot of anomalies with that attorney/ judge pairing… these things are more common than you might think.”

How good are attorneys? “It’s a complicated question. They themselves don’t know and it differs per judge.” says Unwin. “We’ve found that peer recognition and billing rate aren’t correlated with courtroom success. The top UK law firms consistently re-hire proven losers because, until now, nobody’s been keeping score.”

“Alvin Benton is only an Associate at Holland & Knight, yet he’s the best lawyer there for certain judges. However, if you called the firm and asked for their best attorney for Judge Schreiber, they’d probably give you a Senior Partner at vastly higher rates who isn’t even top 100 before that Judge. It sounds crazy to say, but law firms (that aren’t clients) don’t even know how good their own people are, let alone the attorneys and judges they’re up against.” Unwin says.

Former Insurance defense attorney, Brett Schlacter, agrees “Facts and law have lost too many times. As objective as judges try to be, at the end of the day they’re human and the quality of the advocate definitely makes a difference.”

How are insurers using Premonition? “We believedata-driven litigation is agame changer”, Guy Carpenter’s UK Managing Director, Donald McDonald, explains “It’s amazing that law as an industry hasn’t had these kind of metrics applied to it before. Better litigators means fewer losses, means lower premiums. It’s an exciting area”. He continues “We’re also exploring using Premonition’s general court data in areas like medical malpractice. 35% of malpractice claims come from repeat litigants, so why not screen them out at the beginning? We’re exploring programs with drastically reduced premiums for clients that litigate intelligently.”

“You have to hire an attorney anyway, why not hire a good one?” Kurlandski asks. “Many insurers track their internal Counsel results, but have no idea when it comes to outside Counsel, or new geographic areas. Premonition offers what some are calling a very, very unfair advantage in litigation, i.e. putting the right attorney before the right judge”.

Can Law Firms Recognize Great Barristers?

The Court of Appeals in the United Kingdom has arguably the finest legal minds in the World. Queens Counsel, the elite of their profession, debate points of law on the country’s most difficult cases. How good are they, and how are they chosen? Solicitors advise their clients on the choice of Barrister to represent them in Court largely on the basis of peer recognition, essentially how good their colleagues perceive them to be.

“Peer recognition is nonsense, it has no correlation whatsoever with win rate” claims Toby Unwin. The company developed an Artificial Intelligence system to mine Big Data for trends and outliers in litigation. Premonition recently completed a study on the performance of barristers in the Court of Appeals over 3 years with some quite surprising findings. The firm’s Web site, www.Premonition-Analytics.com shows a chart of barristers’ win rates plotted against the number of cases they were hired for. The company actually found a slight negative correlation of -0.1 between win rate and hiring popularity, e.g. a Barrister 20% better than a peer is actually 2% less likely to be hired.

“The market for legal talent is broken”, Unwin continues. “Basically no one has ever kept track of who wins and loses. There’s no transparency, no accountability.” The company developed a computer system that tracks and analyzes cases in courts around the World for clients seeking an edge in litigation. The UK Court of Appeals attracted the firm due to the high level of law practiced, the specialization in pure advocacy and that the English legal system’s “cab rank rule” prevents barristers from cherry-picking easy cases. “It’s a very pure test of the system in one of the most expensive legal environments in the world. There are huge opportunities in more intelligent attorney selection.” he believes.

“Picking Lawyers by win rate is the future of law.”, says Ben Wolkov, C.E.O. of Litigas, a U.S. firm that promises guaranteed top 20 lawyers by their win rate for that particular Judge. “The profession has been able to escape accountability, but big data is bringing transparency. Picking attorneys by anything other than their win rate no longer makes sense” he added.

The study found that the most popular barrister, with 41 appearances over 3 years had a 60% win rate, far short of the 77-100% range a number of colleagues had. More worryingly, the 2nd, 3rd and 5th most popular hires were amongst the worst performers, all having win rates in the 20% range. “Firms are failing clients by picking barristers using peer recognition. Win rates are the only true indication of ability. It’s not unreasonable to ask your lawyer how much they win and ask to see some numbers to back up their claims.” says Unwin. Premonition sees big changes in the litigation market as clients increasingly demand performance metrics.

Associates vs. Partners – Litigation Win Rates compared

Alvin Benton, an attorney at Holland & Knight in Orlando, Florida has a 97% win rate. He’s 100% in front of some Judges and has had a run of 32 consecutive wins before one of them according to legal analytics firm, Premonition L.L.C.(www.Premonition- Analytics.com). They claim he is one of the best lawyers in Orange County and that his win rate is better than any other litigator in that office. “They’re mainly timeshare cases and fairly easy to win, but he’s still head and shoulders above the industry average win rate of 75% for that case type. He doesn’t handle the type of cases our clients have, but he’s clearly very talented. Our system is designed to spot outliers like him” Premonition’s Toby Unwin said.

Benton is only an Associate at Holland & Knight despite a stellar litigation track record. “Partners at Big Law firms are usually chosen based on the number of hours they bill and the value of clients they bring in. Courtroom prowess is a minor factor.” says Richard Waserstein, a prominent Miami Attorney and Premonition Board member. “It’s a widely held believe that Partners are better than Associates, but we often come across Associates that have higher track records.” claims Premonition CEO and Co-Founder, Guy Kurlandski. “We’re very disciplined here about not holding opinions unless we have the data to back them up, but it occurred to us that no one had ever done a study on this. If Partners aren’t selected based on win rate, then maybe there was no difference between them? We decided to find out.” he continued.

Premonition’s recent Miami 11th Circuit Court study covered over 100,000 civil cases. Restricting the survey to just the 20 largest law firms by attorney headcount in Florida they were able to extrapolate the first ever ranking of Big Law by their results. From this they looked at the titles of the litigators identifying 79 attorneys who handled 1,648 completed cases. There were 52 Partners and 27 Associates, i.e. the Partners handled nearly twice as many cases.

Premonition found a 58.1% win rate for the Associates. The Partners? 59.5%, just 1.4% more. “We often find that in law, you don’t necessarily get what you pay for. While there are some Partners at the Big Law firms with phenomenal track records, on average you’re paying around 50% more for only a minor increase in performance” Unwin said.

“It could be argued that Partners try the harder cases, but we found there was a lot of cross-over on case type within firms. Many Partners will oversee Associates’ work and many Associates will work on Partners’ cases, so a blending of results is inevitable, but even we were surprised at how close it was.” says Kurlandski. Are all attorneys pretty much the same, does it make no difference who you choose? “Absolutely not. There are vast differences in attorney performance. Their title and how much you pay them doesn’t matter, but if you want to win, you need to select a top performer by case type for your Judge” said Unwin.

How long does a Lawsuit last?

The answer, according to legal analytics firm, Premonition LLC (www.premonition- analytics.com), very much depends on your choice of Attorney. Premonition recently completed a study of 24,507 divorce cases from 2011-2014 at the Orange County Courthouse in Orlando, Florida. “Divorce cases are useful to compare. While some are undoubtedly more complicated than others, or have more intransigent parties, at the end of the day, they all boil down to filling out 2 forms at the courthouse. Pretty much everything else is at the discretion of the Attorneys involved. Over thousands of cases these differences get smoothed out and clear patterns emerge.” says Toby Unwin.

Unwin invented an Artificial Intelligence system that mines Court data for unusual trends and outliers. The system is typically used by General Counsel at big companies seeking an edge in large litigation portfolios, or high stakes trials. The human element, Premonition claims, is worth an average 30.70% increase in win rate, which translates to substantial savings for clients.

Premonition’s database of lawsuits is one of the largest in private hands. Its sheer volume of information has enabled them to write database algorithms to solve questions such as:

• Which Attorneys usually win before which judges? Premonition claims most judges have favorite attorneys, the average Top 20 lawyer before each judge has an 80.70% likelihood of winning.
• Are big law firms better than small firms? Yes, but only by 6.98% and at a substantially higher average billing rate.
• How does billing rate correlate to win rate? Apparently it doesn’t, at least not by a significant amount.

“We had a client ask us for a family law attorney”, Guy Kurlandski explained. “It’s traditionally not an area we spend a lot of time on because companies don’t get divorced” he quipped. “Family law is the Wild West of the Legal profession”, Unwin claims, “It has a remarkable lack of precedents, the clients are usually legally unsophisticated and determining win rates is highly problematic. Our Artificial Intelligence system finds that there really aren’t many clear “winners” in domestic proceedings.”

They had a client ask for a “fast lawyer”, an unusual request for a company that specializes in finding “Winners”. The analysis showed that it was indeed possible to process divorces quickly, and that even contested cases could be handled in a matter of months, and often were. However there was a cluster of attorneys that had consistently high case durations. “Even though it’s the Public Record, there’s no sign outside law offices saying “this is my win rate and this is how long my cases usually take”, says Kurlandski, “Even major corporations are usually litigating blindly, relying on big brand law firms that are, on average, only 3.49% better than chance.” he says. Running attorney analytics is the only way for a consumer to know.

While there were some cases lasting up to 1,800 days, the lawyers handling them usually did few divorce cases. The firm found an outlier with 5 cases averaging 1,016. The last one had been going for over four and a half years and was set for trial. “It’s disappointing that attorneys like this are allowed to continue practicing” Unwin said. Premonition declined to name the attorney, but said they would provide the Florida Bar with the information, if requested.

How does the average consumer avoid hiring a clock running lawyer? “If you can’t afford to select an attorney with analytics, then at lease check your short list at the local courthouse website, see how long their cases usually run for.” Kurlandski recommends.

Published in the International In-house Counsel Journal (Vol. 8, No. 30, Winter 2015, 1)