Slides: Legal Informatics Research Today: Implications for Legal Prediction, 3D Printing, and eDiscovery

I’ve posted slides of my presentation entitled Legal Informatics Research Today: Implications for Legal Prediction, 3D Printing, and eDiscovery, given 16 May 2013 at CICL 2013: The Fifth Conference on Innovation and Communications Law, 16 May 2013, Glen Arbor, Michigan, sponsored by Michigan State University College of Law.

Here is the abstract:

This presentation describes methodologies and results of recent legal informatics research on eDiscovery and legal prediction, and describes two possible scenarios for the application of legal technology to 3D printing. In addition, the presentation describes a four-level framework that enables comparison of legal informatics research studies in different areas.

I thank Professor Adam Candeub of Michigan State University College of Law for inviting me to give this presentation.

Call for Papers: AICOL 2013: Workshop on AI Approaches to the Complexity of Legal Systems

A call for papers — with abstract submission deadline of 28 February 2013 and full paper submission deadline of 15 May 2013 — has been issued for AICOL 2013: Workshop on AI Approaches to the Complexity of Legal Systems, to be held at a date to be determined, between 21 and 27 July 2013, in Belo Horizonte, Brazil.

The workshop is being collocated with XXVI. World Congress of Philosophy of Law and Social Philosophy.

Papers for AICOL 2013 are invited on the following topics:

• Law and Science
• Knowledge Management
• Law and Cognitive Science
• Cognitive schemas
• Law and Complexity Theory
• Law and Robotics
• Complex Systems
• Law and Mathematics
• Legal Theory
• Legal Graphic Representation
• Legal Culture
• Game Theory
• Computer Ethics
• Formalization of Legal Systems and Norms
• Artificial Societies
• Rules and Standards
• Argumentative Frameworks
• Agreement technologies
• Legal Ontologies
• Electronic Institutions
• Governance
• Legal Concepts
• Legal Information Retrieval
• Legal Thesauri
• Online Dispute Resolution
• Taxonomies
• Trends in e-Discovery, e-Courts, e-Administration
• Natural Language Processing (NLP)
• Legal Knowledge Acquisition
• Users’ studies
• Legal Knowledge Representation

For more details, please see the call.

HT Professor Dr. Monica Palmirani

Katz and Bommarito: Three Thoughts on E-Discovery in 2015 and Beyond

Professor Dr. Daniel Martin Katz of Michigan State University College of Law and Michael Bommarito of Computational Legal Studies have posted slides of their 2013 Legal Tech New York presentation entitled 3 Thoughts on E-Discovery in 2015 and Beyond.

The slides address topics including predictive coding, machine learning, information visualization, pattern recognition, and “scaling relationships,” including scaling involving costs of computer processing and rates of growth of electronically stored information.

Call for Papers: DESI V: Workshop on Standards for Using Predictive Coding and Other Machine Learning Algorithms

A call for papers — with submission deadline of 1 May 2013 — has been issued for DESI V: Workshop on Standards for Using Predictive Coding and Other Machine Learning Algorithms, to be held 14 June 2013 in Rome, Italy, following ICAIL 2013: International Conference on Artificial Intelligence and Law.

Papers addressing the following questions are invited for DESI V:

1) How transparent can and should the process be in sharing seed sets or training sets of documents with opposing parties, including the sharing of privileged documents?

2) What differences if any exist between seed sets developed through random sampling versus other forms of judgmental sampling (including picking seed documents using keywords)?

3) How are non-relevant documents used to optimize machine learning algorithms and should they be subject to similar disclosure?

4) Are there ways in which predictive coding and machine learning methods can be tuned to find highly relevant (“hot”) documents in large collections?

5) To what extent is metadata important in tuning predictive coding software to find similarity in documents?

6) In light of past research at the TREC Legal Track and elsewhere, are there absolute targets for metrics in recall and precision that could serve as standards in every case, or are achieving certain metrics dependent on the relevant data set and legal context?

7) What kinds of best practice standards are needed to help improve mutual understanding of what was actually done, and to improve overall “search quality”?

8) How should predictive coding techniques be audited in connection with an entity submitting itself to an ISO 9001 quality measurement process?

9) To what extent can and should machine learning approaches be used in other phases of the litigation process, to assist in aspects of the process such as identification, preservation, and collection?

10)What are the applications of predictive coding and other forms of machine learning in related “compliance” areas, including regulatory, enforcement, and investigations?

The workshop discussion will be grounded in the results of the recently completed TREC Legal Track, especially where supervised learning methods have shown promising results in terms of being able to more cost-effectively demonstrate rates of recall and precision that approximate the best that could be obtained through other methods, including exhaustive manual review.

For more details, please see the complete call.

HT Jurix.

TREC Legal Track 2013 Reported Canceled

The 2013 TREC Legal Track has been cancelled “because of ongoing unspecified problems with the data set,” Evan Koblentz reports TREC Legal Track co-organizer Professor Dr. Gordon Cormack as having said after a 29 January 2013 panel at the Legal Tech New York conference, according to Koblentz’s article, “LegalTech Day One: Relativity 8, Statistical Sampling, Law Firm Security,” Law Technology News, 30 January 2013.

This would be the second consecutive year in which TREC Legal Track has been cancelled.

The TREC Legal Track is a major legal information retrieval conference. Its findings have contributed substantially to research respecting electronic discovery (ediscovery).

Click here for earlier posts about the TREC Legal Track.

Click here for proceedings of previous years of the TREC Legal Track, contained in the general TREC proceedings.

Click here for published papers that report results of research involving data used in connection with TREC Legal Track.

HT @LawTechEvan

Yablon and Landsman-Roos: Sampling Practice and the Resolution of Discovery Disputes

Professor Charles M. Yablon of Yeshiva University Cardozo School of Law, and Nick Landsman-Roos, have published Discovery about Discovery: Sampling Practice and the Resolution of Discovery Disputes in an Age of Ever-Increasing Information, Cardozo Law Review, 34(2), 101-161 (2012).

Here is the abstract:

This Article provides the first extended academic consideration of a new practice adopted by an increasing number of courts to resolve e-discovery disputes — the sampling of a small portion of the information sought in backup or other relatively inaccessible files. We provides a comprehensive overview and statistical analysis of contemporary sampling techniques, identifying issues where sampling practice is inconsistent or where additional guidance appears to be required. Our aim is to provide a coherent theoretical approach to the use of sampling, suggesting “best practices” for many unresolved issues, and locating sampling practice within broader contemporary debates about discovery.

Call for Papers: ICAIL 2013: International Conference on Artificial Intelligence and Law

A call for papers — with paper submission deadline of 18 January 2013 — has been issued for ICAIL 2013: 14th International Conference on Artificial Intelligence and Law, to be held 10-14 June 2013 in Rome, Italy.

The Twitter account for the conference is @ICAIL2013 . The Twitter hashtag for the conference is #ICAIL2013. The conference organizers invite those interested to follow the Twitter account and hashtag and to comment and contribute with the latest news.

The conference features two tracks: one for “regular papers” and one for “innovative applications papers.”

Here is the complete list of deadlines:

• Mentoring program request deadline: November 9, 2012
• Mentoring program paper deadline: November 16, 2012
• Submission of workshop and tutorial proposals: December 7, 2012
• Submission of abstracts (optional): January 11, 2013
• Submission of papers deadline: January 18, 2013
• Notification of acceptance: March 20, 2013
• Final revised and formatted papers due: April 19, 2013
• Conference: June 10 – June 14, 2013

Papers are invited on the following topics:

• Formal and computational models of legal reasoning
• Knowledge acquisition techniques for the legal domain, including natural language processing and data mining
• Computational models of argumentation and decision making
• Legal knowledge representation including legal ontologies and common sense knowledge
• Automatic legal text classification and summarization
• Automated information extraction from legal databases and texts
• Machine learning and data mining applied to legal databases
• Conceptual or model-based legal information retrieval
• E-discovery and e-disclosure
• E-government and e-justice
• Computational models of evidential reasoning
• Modeling norms for multi-agent systems
• Modeling negotiation and contract formation
• Computational models of case-based legal reasoning
• Online dispute resolution
• Intelligent legal tutoring systems
• Intelligent support systems for the legal domain
• Interdisciplinary applications of legal informatics methods and systems

For more information, please see the call for papers.

HT Anne Gardner

[NOTE: Updated 23 November 2012 to add the Twitter account and hashtag. HT Enrico Francesconi]

Ghosh et al. on Cluster-based Relevance Feedback in Legal Information Retrieval

Kripabandhu Ghosh of the Indian Statistical Institute Information Retrieval Lab, and colleagues, have published Cluster-based Relevance Feedback: TREC Legal Track 2011, in The Twentieth Text REtrieval Conference (TREC 2011) Proceedings.

Here is the abstract:

This is our second participation in the TREC Legal Track. The TREC Legal Track 2011 featured only the Learning Task. We participated in Topics 401 and 403. We used Lemur 4.11 for Boolean retrieval and followed it with a clustering technique, where we chose members from each cluster (which we called seeds) for relevance judgement by the TA and assumed all other members of the cluster whose seeds are assessed as relevant to be relevant. Based on the relevance information from seeds and their clusters, we applied Rocchio relevance feedback technique implemented in Terrier 3.0. Then, we used the feedback terms for the expansion of both the text queries and the Boolean queries. Finally, we used Z-fusion, a data fusion technique, on two of our runs.

Click here for reports of the authors’ results (scroll down to INDIAN STATISTICAL INSTITUTE, KOLKATA).

Click here for the overview paper on the 2011 TREC Legal Track.

Zhang et al. on Legal Information Discovery Based on Relevant Feedback

Jiayue Zhang and colleagues of the Beijing University of Posts and Telecommunications School of Information and Communication Engineering have published PRIS at TREC 2011 Legal Track: Discovery Based on Relevant Feedback, in The Twentieth Text REtrieval Conference (TREC 2011) Proceedings.

Here is the abstract:

In order to finish the task of TREC 2011 Legal Track, this paper puts forward an experiment method, which combines indri and relevant feedback to evaluate the probability of relevance of every document in a collection.

For reports of the authors’ results, please see:

• priindA1
• priindA2

Click here for the overview paper on the 2011 TREC Legal Track.

Grossman et al.: Overview of the TREC 2011 Legal Track

Maura R. Grossman, Esq., of Wachtell, Lipton, Rosen & Katz, and colleagues, have published Overview of the TREC 2011 Legal Track, in The Twentieth Text REtrieval Conference (TREC 2011) Proceedings.

Here is the abstract:

The TREC 2011 Legal Track consisted of a single task: the learning task, which captured elements of both the TREC 2010 learning and interactive tasks. Participants were required to rank the entire corpus of 685,592 documents by their estimate of the probability of responsiveness to each of three topics, and also to provide a quantitative estimate of that probability. Participants were permitted to request up to 1,000 responsiveness determinations from a Topic Authority for each topic. Participants elected either to use only these responsiveness determinations in preparing automatic submissions, or to augment these determinations with their own manual review in preparing technology-assisted submissions. We provide an overview of the task and a summary of the results. More detailed results are available in the Appendix to the TREC 2011 Proceedings.

HT Gordon V. Cormack.