Accuracy

“History is the fiction we invent to persuade ourselves that events are knowable and that life has order and direction.” – Calvin, from Calvin & Hobbes

Law School Predictor is of far greater use to applicants when applicants can gauge the accuracy of LSP predictions.  These results are based off of more than 33,500 Law School Numbers-listed law school admission decisions from the 2008-09 cycle for all ABA schools listed on both LSN and LSP. The version of LSP tested was Version 2.4. A big thanks goes out to TLS forum member CyLaw for compiling the data from LSN; I couldn’t have done it without CyLaw.

The admit rate was calculated by taking the number of admitted applicants divided by total applicants for each respective prediciton category (total applicants includes admitted, waitlisted, and rejected applicants, but does not include those who status was listed as “pending”). The prediction categories used in testing accuracy are the same as LSP’s prediction engine with the exception of early prediction (which was not tested); predictions are adjusted to account for splitter-ness, weak GPAs, and URM status.

All Applicants

Target admit rates, based on how LSP renders predictions:
When LSP said Admit: >= 87%
When LSP said Strong Consider: ~ 69%
When LSP said Consider: ~ 50%
When LSP said Weak Consider: ~ 31%
When LSP said Deny: <= 13%

Based on the 33,500+ LSN decisions from the 2008-09 admission cycle:
When LSP said Admit, 8402 instances: 87.0%, +0.0%
When LSP said Strong Consider, 4842 instances: 78.0%, +9.0%
When LSP said Consider, 10066 instances: 50.6%, +0.6%
When LSP said Weak Consider, 3646 instances: 22.8%, -8.2%
When LSP said Deny, 6754 instances: 10.0%, -3.0%

Difference between actual admit rate and predicted admit rate: (prediction categories weighted evenly)
Average rate: -0.3%
Median rate: -3.0%

Summary:
Prediction categories that were right on the money: Admit, Consider
Prediction categories that were reasonably close: Deny
Prediction categories that were somewhat off: Strong Consider and Weak Consider

A closer look
A possible explanation for the Strong/Weak Consider results is that the chance of being admitted (based on admission index scores and LSP adjustments) should look like a parabolic curve (since it should be a normal distribution), and, based on these accuracy results, a curve with a high and narrow peak in the center (leptokurtic distribution).

Underrepresented Minorities (URMs)

Target admit rates for URMs, based on how LSP renders predictions: (same method as non-URM applicants after URM boost is applied)
When LSP said Admit: >= 87%
When LSP said Strong Consider: ~ 69%
When LSP said Consider: ~ 50%
When LSP said Weak Consider: ~ 31%
When LSP said Deny: <= 13%

Based on the 4,300+ LSN 2008-09 decisions for self-identified URMs and with LSP URM feature enabled:
When LSP said Admit, 963 instances: 87.1%, +0.1%
When LSP said Strong Consider, 418 instances: 70.8%, +1.8%
When LSP said Consider, 1034 instances: 55.3%, +5.3%
When LSP said Weak Consider, 436 instances: 33.2%, +2.3%
When LSP said Deny, 1468 instances: 14.9%, +1.9%

Difference between actual URM admit rate and predicted URM admit rate: (prediction categories weighted evenly)
Average rate: +2.3%
Median rate: +1.9%

Summary:
Prediction categories that were right on the money: Admit
Prediction categories that were reasonably close: Strong Consider, Weak Consider, Deny
Prediction categories that were somewhat off: Consider

A closer look
While the URM boost was not devised by using a particularly statistically rigorous technique (the URM feature debuted in Version 1.5, and has been tweaked a couple times since), it turns out that LSP is just about as good (or arguably better) at making predictions for URM applicants on LSN when compared to the average applicant on LSN. For those who sometimes suggest that decisions for URM applicants are largely unpredictable based on an applicant’s numbers, these results would shed some doubt on that assertion. This is not to say that URM applicant cycles are entirely predictable, but if a URM applicant plugs his/her numbers into LSP, s/he should get a decent of his/her chances.

It’s worth noting that URM applications (where a decision was noted) made up less than 13% of LSN application decisions. For URM applications where a decision was noted, the average applicant had an LSAT of 157 and a GPA of 3.31.