Grossman et al.: Overview of the TREC 2011 Legal Track

Maura R. Grossman, Esq., of Wachtell, Lipton, Rosen & Katz, and colleagues, have published Overview of the TREC 2011 Legal Track, in The Twentieth Text REtrieval Conference (TREC 2011) Proceedings.

Here is the abstract:

The TREC 2011 Legal Track consisted of a single task: the learning task, which captured elements of both the TREC 2010 learning and interactive tasks. Participants were required to rank the entire corpus of 685,592 documents by their estimate of the probability of responsiveness to each of three topics, and also to provide a quantitative estimate of that probability. Participants were permitted to request up to 1,000 responsiveness determinations from a Topic Authority for each topic. Participants elected either to use only these responsiveness determinations in preparing automatic submissions, or to augment these determinations with their own manual review in preparing technology-assisted submissions. We provide an overview of the task and a summary of the results. More detailed results are available in the Appendix to the TREC 2011 Proceedings.

HT Gordon V. Cormack.

Cormack et al., Overview of the TREC 2010 Legal Track

Professor Dr. Gordon V. Cormack of the University of Waterloo; Maura R. Grossman, Esq., of Wachtell, Lipton; Bruce Hedin of H5, and Professor Dr. Douglas W. Oard of the University of Maryland, have posted Overview of the TREC 2010 Legal Track. Here is the abstract:

TREC 2010 was the fifth year of the Legal Track, which focuses on evaluation of search technology for discovery of electronically stored information in litigation and regulatory settings. The TREC 2010 Legal Track consisted of two distinct tasks: the Learning task, in which participants were required to estimate the probability of relevance for each document in a large collection, given a seed set of documents, each coded as responsive or non-responsive; and the Interactive task, in which participants were required to identify all relevant documents using a human-in-the-loop process.

Hedin et al., Overview of the TREC 2009 Legal Track

Dr. Bruce Hedin of H5, and colleagues, have posted Overview of the TREC 2009 Legal Track, in The Eighteenth Text REtrieval Conference (TREC 2009) Proceedings. Here is the abstract:

TREC 2009 was the fourth year of the Legal Track, which focuses on evaluation of search technology for “discovery” (i.e., responsive review) of electronically stored information in litigation and regulatory settings. The track included two tasks: an Interactive task (in which real users could iteratively refine their queries and/or engage in multi-pass relevance feedback) and a Batch task (two-pass search in a controlled setting with some relevant and nonrelevant documents manually marked after the first pass). This paper describes the design of the two tasks and presents the results.

Oard et al., Some Lessons Learned To Date from the TREC Legal Track (2006-2009)

Associate Dean Douglas W. Oard of the University of Maryland College of Information Studies, and colleagues, have posted Some Lessons Learned To Date from the TREC Legal Track (2006-2009) (Feb. 2010). Here is the abstract:

For four years now, the Text Retrieval Conference (TREC) Legal Track administered by the US National Institute of Standards (NIST) has undertaken yearly studies evaluating the application of Information Retrieval (IR) methods to e-discovery in the context of U.S. civil litigation. In this short paper, we distill some of what has been learned. As we write this, analysis is not yet complete for some parts of the 2009 Legal Track [click here for proceedings of the 2009 Legal Track (scroll down)], while the 2010 Legal Track is only now beginning. “Lessons learned” are, therefore, a work in progress. In an effort to be concise, we stick to bullet points – much more could be said about any of this. We are indebted to our colleagues for much of what we have learned, but we emphasize that these are our personal observations and that others might see things differently. Other perspectives on the Legal Track can be found in TREC proceedings papers by organizers and participants at http://trec.nist.gov/.