TREC Legal Track
About the Legal Track
The goal of the Legal Track at the Text
Retrieval Conference (TREC) is to assess the ability of
information retrieval techniques to meet the needs of the legal
profession for tools and methods capable of helping with the retrieval
of electronic business records, principally for use as evidence in
civil litigation. In the USA, this problem is referred to as
"e-discovery." Like all TREC tracks, the Legal Track seeks to foster
the development of a research community by providing a venue for
shared development of evaluation resources ("test collections") and
baseline results to which future results can be compared.
The Legal Track operates on an annual cycle that moves through three
phases: task planning, experimentation, and reporting. Task planning
occurs in the winter and spring, experimentation in the spring and
summer, and reporting in the fall and winter. Registration is open to
all interested research teams. Participants must register each year
with NIST using a link that is typically available on the TREC Web site from mid-January to
mid-April. Registration with NIST provides access to
password-protected parts of NIST's TREC Web site and membership for
one email address on low-volume mailing list that is used by NIST for
announcements that are of interest to all tracks (e.g., TREC
conference submission deadlines and registration procedures).
The Legal Track maintains a separate mailing list (https://mailman.umiacs.umd.edu/mailman/listinfo/trec-legal)
as the principal online venue for announcements regarding the track
and for discussion of issues related to the design of the Legal Track.
The volume of messages on this mailing list varies throughout the
year, but is generally fairly light. Anyone may join at any time,
regardless of whether they intend to participate in the track in a
specific year, and membership in the mailing list typically does not
need to be renewed each year. Members have access to an archive of
prior posts to the mailing list.
TREC is a research community in which the organizers and participants
are volunteers. Participants agree in writing to abide by the agreement
concerning dissemination of TREC results. Neither TREC nor the
Legal Track exercises any other editorial control over oral or written
statements made by track organizers, track participants or future
users of the test collection. Moreover, a stated affiliation by an
individual with an organization should not normally be interpreted to
imply that organization's endorsement of statements made by that
individual in the course of their academic research.
Participating research teams in any track of each year's TREC are
invited to participate in an invitation-only conference for
participants at NIST in Gaithersburg, MD at which research results
from that year will be discussed. Working notes papers written by
each research team in October are made available only to participating
research teams at that time. The research results from each year's
TREC tracks are then disseminated more broadly in writing on the TREC
Web site in February of the following year.
2012 Legal Track
The TREC Legal Track will not run in 2012. A new data set will be
made available shortly; however, it will not be possible to develop
the necessary topics and gold standard, and to conduct the proposed
experiment within the time constraints of TREC 2012.
The data set, consisting of approximately one million email messages
and attachments from a liquidated business enterprise, will be made
available by the University of Maryland, subject to a usage agreement.
The availability of this data set, as well as future plans to conduct
experiments similar to those proposed for TREC 2012, will be announced
at this location at a future date.
2012 Track Coordinators
- Gordon V. Cormack
- Maura R. Grossman
- Bruce Hedin
- David D. Lewis
- Douglas W. Oard
2011 Legal Track (Reporting Phase)
In 2011, the Legal Track has a single task, referred to as the
learning task, in which participating teams can use either an
interactive or a fully automated process to perform review for
responsiveness.
2011 Learning Task Test Collection
The goal of the 2011 learning task is to determine which documents
(email messages or attachments, treated separately) should be produced
in response to a production request for which a set of "training"
relevance judgments are available. In 2011, participating teams can
request training judgments on specific documents.
2011 Track Coordinators
- Gordon V. Cormack
- Maura R. Grossman
- Bruce Hedin
- Douglas W. Oard
2010 Legal Track (Completed)
In 2010, the Legal Track included two tasks, an interactive task
focused on end-to-end evaluation of an interactive process of review
for responsiveness or privilege and a learning task focused on
technology evaluation.
2010 Interactive Task Test Collection
The goal of the 2010 interactive task was to determine which document
sets (where a set was defined to be an email messages with its
attachments) should be produced in response to a production request
for which a "topic authority" was available to answer specific
questions posed by a participating team.
2010 Learning Task Test Collection
The goal of the 2010 learning task was to determine which documents
(email messages or attachments, treated separately) should be produced
in response to a production request for which a set of "training"
relevance judgments are available. In 2010, all participating teams
used the same training judgments.
2010 Legal Track Results
2010 Track Coordinators
- Gordon V. Cormack
- Maura R. Grossman
- Bruce Hedin
- Douglas W. Oard
2009 Legal Track (Completed)
In 2009, the Legal Track included two tasks, an interactive task
focused on end-to-end evaluation of an interactive process of review
for responsiveness or privilege and a batch task focused on
technology evaluation.
2009 Interactive Task Test Collection
The goal of the 2009 interactive task was to determine which document
sets (where a set was defined to be an email messages with its
attachments) should be produced in response to a production request
for which a "topic authority" was available to answer specific
questions posed by a participating team.
2009 Batch Task Test Collection
The goal of the 2009 batch task was to determine which documents
(scanned business records) should be produced in response to a
production request. Teams could optionally use a predefined set of
"training" relevance judgments that were available.
2009 Legal Track Results
2009 Track Coordinators
- Jason R. Baron
- Bruce Hedin
- Douglas W. Oard
- Stephen Tomlinson
2008 Legal Track (Completed)
In 2008, the Legal Track included three tasks, an interactive task
focused on end-to-end evaluation of an interactive process of review
for responsiveness, a relevance feedback task focused on technology
evaluation when some "training" relevance judgments are available, and
an ad hoc task focused on technology evaluation when no training
relevance judgments are available.
2008 Interactive Task Test Collection
The goal of the 2008 interactive task was to determine which documents
(scanned business records) should be produced in response to a
production request for which a "topic authority" was available to
answer specific questions posed by a participating team.
2008 Relevance Feedback Task Test Collection
The goal of the 2008 relevance feedback task was to determine which
documents (scanned business records) should be produced in response to
a production request. Teams could optionally use a predefined set of
"training" relevance judgments that were available.
2008 Ad Hoc Task Test Collection
The goal of the 2008 ad hoc task was to determine which documents
(scanned business records) should be produced in response to a
production request based on the complaint, the production request, and
a Boolean query negotiation history.
2008 Legal Track Results
2008 Track Coordinators
- Jason R. Baron
- Bruce Hedin
- Douglas W. Oard
- Stephen Tomlinson
2007 Legal Track (Completed)
In 2007, the Legal Track included three tasks, a "main" task focused
on technology evaluation when no training relevance judgments are
available, a relevance feedback task in which some "training"
relevance judgments are also available, and an interactive challenge
task in which the goal is to balance underproduction and
overproduction.
2007 Main Task Test Collection
The goal of the 2007 "main" task was to determine which documents
(scanned business records) should be produced in response to a
production request based on the complaint, the production request, and
a Boolean query negotiation history. The 2007 main task is referred
to in the Track Overview paper as the "ad hoc" task.
2007 Relevance Feedback Task Test Collection
The goal of the 2007 relevance feedback task was to determine which
documents (scanned business records) should be produced in response to
a production request based on the complaint, the production request, a
Boolean query negotiation history, and some "training" documents for
which relevance judgments are available.
2007 Interactive Task Challenge Test Collection
The goal of the 2007 interactive challenge task was to perform
machine-assisted human review to balance the cost of type 1 (false
positive) and type 2 (false negative) errors when determining which
documents (scanned business records) should be produced in response to
a production request for which only the complaint and the production
request are available.
2007 Legal Track Results
2007 Track Coordinators
- Jason R. Baron
- Douglas W. Oard
- Paul Thompson
- Stephen Tomlinson
2006 Legal Track (Completed)
In 2006, the Legal Track included one task, which was focused on
technology evaluation when no training relevance judgments are
available.
2006 Legal Track Test Collection
The goal of the 2006 Legal Track was to determine which documents
(scanned business records) should be produced in response to a
production request based on the complaint, the production request, and
a Boolean query negotiation history.
2006 Legal Track Results
2006 Track Coordinators
- Jason R. Baron
- David D. Lewis
- Douglas W. Oard
Published Research using Legal Track Test Collections
The test collections produced in the Legal Track are freely available
for research and commercial use under the conditions indicated in the
distribution package for each test collection. Research publications
based at least in part on use of TREC Legal Track test collections
that we are aware of (other than those in the TREC
Proceedings) are listed here.
- A. Arampatzis, J. Kamps and S. Robertson, Where to Stop
Reading in a Ranked List?: Threshold Optimization Using Truncated
Score Distributions, Proceedings of the 32nd International
ACM SIGIR Conference on Research and Development in Information
Retrieval (SIGIR), Boston, MA, July (2009)
- R. Bauer, D. Brassil, C. Hogan, G. Taranto and J.S. Brown, Impedance
Matching of Humans and Machine in High-Q Information Retrieval
Systems, IEEE International Conference on Systems, Man and
Cybernetics (SMC), San Antonio, TX, USA, October (2009)
- D. Brassil, C. Hogan and S. Attfield, The
Centrality of User Modeling to High Recall with High Precision
Search, IEEE International Conference on Systems, Man and
Cybernetics (SMC), San Antonio, TX, USA, October (2009)
- H. Chu, Factors
affecting relevance judgment: A report from TREC Legal Track,
Journal of Documentation, 67(2)264-278 (2011)
- M. Grossman and G. Cormack, Technology-Assisted
Review in E-Discovery Can Be More Effective and More Efficient
Than Exhaustive Manual Review, Richmond Journal of Law and
Technology, 17(3), Spring (2011)
- M. Grossman and G. Cormack, Inconsistent
Assessment of
Responsiveness in E-Discovery: Difference of Opinion or Human
Error?, ICAIL 2011 Workshop on Setting Standards for Searching
Electronically Stored Information in Discovery Proceedings (DESI
IV), Pittsburgh, PA, USA, June (2011)
- B. Hedin and D. Oard, Replication
and Automation of Expert Judgments: Information Engineering in
Legal E-Discovery, IEEE International Conference on Systems,
Man and Cybernetics (SMC), San Antonio, TX, USA, October (2009)
- C. Hogan, R. Bauer, and D. Brassil, Automation of Legal
Sensemaking in E-Discovery, Artificial Intelligence and Law
18(4)431-457 (2011).
- C. Hogan, D. Brassil and M. Marcus, Human Aided
Computer Assessment for Exhaustive Search, IEEE International
Conference on Systems, Man and Cybernetics (SMC), San Antonio,
TX, USA, October (2009)
- A. Kontostathis and S. Kulp, The
Effect of Normalization when Recall Really Matters,
Proceedings of the International Conference on Information and
Knowledge Engineering (IKE), Las Vegas, NV, USA, July (2008)
- D. Oard, J. Baron, B. Hedin, D. Lewis and S. Tomlinson,
Evaluation
of Information Retrieval for E-Discovery, Artificial
Intelligence and Law 18(4)347-386 (2011).
- J. Parapar, A. Freire and A. Barreiro, Revisiting
N-Gram Based Models for Retrieval in Degraded Large
Collections, Proceedings of the 31st European Conference on
IR Research (ECIR), Toulouse, France, April (2009)
- V. Rangan, Discovery
of Related Terms in a Corpus using Reflective Random Indexing,
ICAIL 2011 Workshop on Setting Standards for Searching
Electronically Stored Information in Discovery Proceedings (DESI
IV), Pittsburgh, PA, USA, June (2011)
- S. Tomlinson and B. Hedin, Measuring
Effectiveness in the TREC Legal Track, in M. Lapu, K. Mayer,
J. Tait and A. Trippe (eds.), Current Challenges in Patent
Information Retrieval, Springer (2011)
- J. Wang, Accuracy,
Agreement, Speech and Perceived Difficulty of Users' Relevance
Judgments for E-Discovery, SIGIR 2011 Information Retrieval
for E-Discovery (SIRE) Workshop, Beijing, China, July (2011)
- J. Wang and D. Soergel, A
User Study of Relevance Judgments for E-Discovery,
Proceedings of the Annual Meeting for the American Society for
Information Science and Technology (ASIST), Pittsburgh, PA, USA,
October(2010).
- W. Webber, D. Oard, F. Scholer and Bruce Hedin, Assessor error
in stratified evaluation, Proceedings of the 19th ACM
Conference on Information and Knowledge Management (CIKM),
Toronto, ON, Canada, pp. 539-548, October (2010)
- W. Webber, Re-examining
the Effectiveness of Manual Review, SIGIR 2011 Information
Retrieval for E-Discovery (SIRE) Workshop, Beijing, China, July
(2011)
- F. Zhao, D. Oard and J. Baron, Improving
Search Effectiveness in the Legal E-Discovery Process Using Relevance
Feedback, ICAIL DESI 2009 III Global E-Discovery/E-Disclosure
Workshop, Barcelona, Spain, June (2009)
Other Materials
Additional materials related to the Legal Track are listed here in
most-recent-first order. Some of these materials are outdated, and
are included here only for archival purposes.
- Reflections from the 2009 Topic
Authorities (July 18, 2010)
- Sedona Conference
2010 open letter on the TREC Legal Track (June 30, 2010)
- Some Lessons Learned To Date from
the TREC Legal Track (2006-2009) (February 24, 2010)
- Legal Track FAQ (June 24, 2009)
- Sedona Conference 2009 open letter on the
TREC Legal Track (April 22, 2009)
- Reflections from the 2008
Topic Authorities (January 6, 2009)
- Summary of
TREC 2006, 2007, and 2008 Complaints and Production Requests,
noting some cases in which different complaints with the same
name (i.e., A, B, C, ...) were used in different years. (August
17, 2008)
- Call for Participation by Relevance
Assessors (August 11, 2008)
- Example of a Boolean
Negotiation History (May 4, 2007)
- Legacy Tobacco
Documents Library, an interactive search engine at the
University of California San Francisco that searches the document
collection from which the CDIP test collection used in the
2006-2009 Legal Tracks was derived.
- Tobacco Documents
Online, a second interactive search engine for the materials
contained in the CDIP test collection used in the 2006-2009 Legal
Tracks.
The old Legal Track Web page is still available, but no
longer being updated.
Last modified: Thu May 10 23:11:35 2012
Doug Oard
oard@umd.edu