Pilot Task Results

This is the final results of pilot tasks.

Tort Prediction

There are 11 valid official submissions out of 14 leaderboard submissions.

Rank	Model	Team	Affiliation	Accuracy
1	JAIST-LJPJT25	CAPTAIN	Japan Advanced Institute of Science and Technology	76.5%
2	KIS5	KIS	Shizuoka University	71.3%
2	KIS6	KIS	Shizuoka University	71.3%
4	KIS4	KIS	Shizuoka University	69.7%
5	system2	NOWJ	VNU University of Engineering and Technology	67.1%
6	modernbert	omega	freelance	66.6%
7	system1	NOWJ	VNU University of Engineering and Technology	63.8%
8	system3	NOWJ	VNU University of Engineering and Technology	59.7%
9	OVGU2	OVGU	Otto von Guericke University Magdeburg	55.3%
10	OVGU3	OVGU	Otto von Guericke University Magdeburg	53.2%
11	OVGU1	OVGU	Otto von Guericke University Magdeburg	51.5%

There are 11 valid official submissions out of 15 leaderboard submissions.

Rank	Model	Team	Affiliation	F1 (All)
1	KIS5	KIS	Shizuoka University	71.2%
2	JAIST-LJPJT25	CAPTAIN	Japan Advanced Institute of Science and Technology	70.6%
3	system2	NOWJ	VNU University of Engineering and Technology	69.2%
4	modernbert	omega	freelance	69.1%
5	KIS4	KIS	Shizuoka University	68.2%
6	system1	NOWJ	VNU University of Engineering and Technology	68.1%
7	KIS6	KIS	Shizuoka University	67.3%
8	OVGU1	OVGU	Otto von Guericke University Magdeburg	65.7%
9	system3	NOWJ	VNU University of Engineering and Technology	55.9%
10	OVGU2	OVGU	Otto von Guericke University Magdeburg	48.6%
11	OVGU3	OVGU	Otto von Guericke University Magdeburg	31.6%

Online Leaderboards (including unofficial submissions): https://ljpjt25.coliee.org/ (opens in a new tab)

Last updated on March 25, 2025