"Bash shell script that calculates system effectiveness metrics for TREC 2010 and TREC 2011 Legal Learning systems", "calc1.c" =>"C program used in dolegal10eval", "calc2.c" => "C program used in dolegal10eval", "qrels.t10legallearn.gz" => "Official qrels for the TREC 2010 Legal Learning task", "qrels.t11legallearn.gz" => "Official qrels for the TREC 2011 Legal Learning task", "t10legallearn-results.zip" => "Official results for TREC 2010 Legal Learning participant systems", ) ?> TREC 2010 Legal Track Learning Task Results and Toolkit

TREC 2010 Legal Track Learning Task Results and Toolkit

$desc) { printf("\n"); printf(" \n", $file, $file); printf(" \n", date('Y-m-d\&\n\b\s\p\;H:i:s', filemtime($file))); $fsz = filesize($file); $gb = 1024 * 1024 * 1024; $mb = 1024 * 1024; $kb = 1024; if ($fsz > $gb) { $fsz_s = sprintf("%dG", $fsz/$gb); } else if ($fsz > $mb) { $fsz_s = sprintf("%dM", $fsz/$mb); } else if ($fsz > $kb) { $fsz_s = sprintf("%dK", $fsz/$kb); } else { $fsz_s = sprintf("%dB", $fsz); } printf(" \n", $fsz_s); printf(" \n", $desc); } ?>

Name	Last Modified (UTC)	Size	Description
%s	%s	%s	%s

TREC 2010 Legal Track Learning Task Results and Toolkit

Results:

contained in t10legallearn-results.zip
files of the form RUNNAME.sum give summary statistics
files of the form RUNNAME.table give progressive statistics

Relevance assessments:

the file qrels.t10legallearn.gz contains the relevance assessments for 2010
the file qrels.t11legallearn.gz contains PRELIMINARY assessments for 2011

Toolkit:

download the files:

uncompress the qrels file:

       gunzip qrels.t10legallearn.gz

on a *nix system, type:
```
       ./dolegal10 qrels.t10legallearn RUNNAME
     
```
where qrels.t10legallearn is the uncompressed version of the relevance assessments and RUNNAME is a learning task run, in the format given in the guidelines
the result will be a RUNNAME.sum and RUNNAME.table files
You may use Linux or MacOS or Cygwin on Windows to run the toolkit. It has been tested on Fedora and Ubuntu Linux.