COLIEE 2026
Evaluation

Measuring the Competition Results

Case Law Competition Results (Tasks 1 and 2)

For Tasks 1 and 2, the evaluation metrics will be precision, recall, and F-measure:

Precision:

the number of correctly retrieved cases(paragraphs) for all queriesthe number of retrieved cases(paragraphs) for all queries\frac{\text{the number of correctly retrieved cases(paragraphs) for all queries}}{\text{the number of retrieved cases(paragraphs) for all queries}}

Recall:

the number of correctly retrieved cases(paragraphs) for all queriesthe number of relevant cases(paragraphs) for all queries\frac{\text{the number of correctly retrieved cases(paragraphs) for all queries}}{\text{the number of relevant cases(paragraphs) for all queries}}

F-measure:

2×Precision×RecallPrecision+Recall\frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}

In the evaluation of Task 1 and Task 2, we simply use micro-average (evaluation measure is calculated using the results of all queries) rather than macro-average (evaluation measure is calculated for each query and then take average).

Statute Law Competition Results (Tasks 3 and 4)

For Task 3, the primary evaluation metric is accuracy, defined as the proportion of correct entailment results accompanied by sufficient relevant articles (Recall = 1). In the case of a tie in accuracy between two or more teams, the F2-measure will be used as the secondary evaluation metric.

Accuracy:

the number of queries which were correctly predicted as true or false​ using sufficient relevant articles​the number of all queries\frac{\text{the number of queries which were correctly predicted as true or false​ using sufficient relevant articles​}}{\text{the number of all queries}}

Precision:

average of (the number of correctly retrieved articles for each query)the number of retrieved articles for each query\frac{\text{average of (the number of correctly retrieved articles for each query)}}{\text{the number of retrieved articles for each query}}

Recall:

average of (the number of correctly retrieved articles for each query)the number of relevant articles for each query\frac{\text{average of (the number of correctly retrieved articles for each query)}}{\text{the number of relevant articles for each query}}

F2-measure:

5×Precision×Recall4×Precision+Recall\frac{5 \times \text{Precision} \times \text{Recall}}{4 \times \text{Precision} + \text{Recall}}

For Task 4, the evaluation measure will be accuracy, with respect to whether the yes/no question was correctly confirmed:

Accuracy:

the number of queries which were correctly confirmed as true or falsethe number of all queries\frac{\text{the number of queries which were correctly confirmed as true or false}}{\text{the number of all queries}}

Legal Judgment Prediction for Japanese Tort cases (Pilot Task)

Updated (October 2026): This section previously described only macro F1, which was the metric in our earliest plan. It has been revised to match the evaluation actually used in the competition. The scores shown on the leaderboard during the competition are not affected.

Tort Prediction

For Tort Prediction of Pilot Task, the evaluation measure was accuracy, with respect to whether the True/False label for court_decision was correctly predicted:

Accuracy:

the number of instances which were correctly predicted as true or falsethe number of all instances\frac{\text{the number of instances which were correctly predicted as true or false}}{\text{the number of all instances}}

Rationale Extraction

For Rationale Extraction of Pilot Task, the evaluation metrics were F1-measure concerning True label for is_accepted and it is:

Precision:

the number of claims correctly predicted as Truethe number of claims predicted as True\frac{\text{the number of claims correctly predicted as True}}{\text{the number of claims predicted as True}}

Recall:

the number of claims correctly predicted as Truethe number of claims whose gold labels are True\frac{\text{the number of claims correctly predicted as True}}{\text{the number of claims whose gold labels are True}}

F-measure:

2×Precision×RecallPrecision+Recall\frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}

Participants were required not to submit null predictions. A submission with any null court_decision was disqualified from the Tort Prediction ranking, and one with any null is_accepted was disqualified from the Rationale Extraction ranking.

Official metrics in the Leaderboard system

During the competition, evaluations were automatically performed on the online leaderboard system (the system is offline after the competition period). The system returned the following metrics for each submission. In the metric names, all, p and d mean that the target claims are all claims, the plaintiff's claims and the defendant's claims, respectively.

Tort Prediction metrics

MetricDescription
tort_prediction_task.accuracyAccuracy of court_decision. The metric used for the final ranking in Tort Prediction

Rationale Extraction metrics

MetricDescription
binary_all_f1 / binary_p_f1 / binary_d_f1F1 computed over all target claims pooled together, treating is_accepted = true as the positive class. They were shown as F1 (All) / F1 (Plaintiff) / F1 (Defendant) on the leaderboard. binary_all_f1 is the metric used for the final ranking in Rationale Extraction
binary_*_precision / binary_*_recallThe precision and recall that correspond to the F1 above
macro_*The mean of the values computed for each tort (torts with no target claims are excluded). A tort with no true among the gold labels of its target claims counts as 0 in the mean, regardless of the predictions
micro_*Micro level calculation.
accuracy_all / accuracy_plaintiff_claims / accuracy_defendant_claimsAccuracy of is_accepted
Pilot Task (LJPJT26)Submission