Friday, November 22, 2013

Comment on early EDI - Oracle Study results

The Electronic Discovery Institute yesterday released via Law Technology News some preliminary results of its study on a dataset provided by Oracle related to its acquisition of Sun Microsystems. The study involved multiple providers of technology-assisted review, including Backstop, which categorized documents for three tags: responsive, privilege, and hot. EDI's release is the first step towards what should eventuate in ground-breaking raw data and analysis. While skeletal (we only have ordinal F1 rankings thus far), it affords the basis for some thoughts, including very imperfect cost-adjusted performance measures.  Interestingly, the results show no correlation between cost and accuracy ranking. Backstop and the other study participants are forbidden to identify their own entries, and EDI can only tell a vendor which results are the vendor's own. So, while we are very pleased with our results, we cannot identify them or those of any other participant.  In this post I will share some thoughts on difficulties with F1 as a benchmark for accuracy, then delve into a first attempt at a cost-adjusted performance spreadsheet, which you can sort, view, edit and download.

Wednesday, November 20, 2013

EDI - Oracle Study Preliminary Results Released

Preliminary results have been released from the Electronic Discovery Institute - Oracle study, in which Backstop participated.  See an article at Law Technology News and the chart below.  We are very pleased to see these results and will soon share a preliminary analysis on this blog.  We also look forward to seeing more granular detail (viz., recall and precision figures) in the near future.