COLIEE 2026
Pilot Task Evaluation (Archived)

Pilot Task Evaluation (Archived)

⚠️

This is an archived version of the Pilot Task section of the Evaluation page. For the latest information, please visit the current version.

Legal Judgment Prediction for Japanese Tort cases (Pilot Task)

Tort Prediction

For Tort Prediction of Pilot Task, the evaluation measure will be accuracy, with respect to whether the True/False label for court_decision was correctly predicted:

Accuracy:

the number of instances which were correctly predicted as true or falsethe number of all instances\frac{\text{the number of instances which were correctly predicted as true or false}}{\text{the number of all instances}}

Rationale Extraction

For Rationale Extraction of Pilot Task, the evaluation metrics will be F1-measure concerning True label for is_acceptedand it is:

Precision:

the number of claims correctly predicted as Truethe number of claims predicted as True\frac{\text{the number of claims correctly predicted as True}}{\text{the number of claims predicted as True}}

Recall:

the number of claims correctly predicted as Truethe number of claims whose gold labels are True\frac{\text{the number of claims correctly predicted as True}}{\text{the number of claims whose gold labels are True}}

F-measure:

2×Precision×RecallPrecision+Recall\frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}

We first calculate the F-measure for each tort instance, then average it over all tort instances. (A tort instance corresponds to a line in a submission JSON file.)