Rules

Usage of LLM

Based on the discussion at COLIEE 2025, we would like to change the rules for using LLMs. You can use LLMs whose training data and/or model are publicly available (e.g. Huggingface or similar sites), but you cannot use LLMs whose models are closed (e.g. GPT-4o, Gemini,...). For reproducibility purposes, please include information on how to obtain the model in the paper.

For tasks 3 and 4, to avoid contamination of the test data, you can only use LLMs released before July 9, 2024 (JST), the last day before the Japanese bar exam.

Usage of bilingual data

In the previous COLIEE, there were systems that used both English and Japanese question texts to generate the final results. Since this setting is not realistic, we would like to prevent the use of English and Japanese question texts at the same time. However, you can use both Japanese and English civil law texts at the same time because manually translated English civil law texts are available to the public. Therefore, if you want to use the bilingual system, you should use the machine translation system to generate Japanese/English question texts from English/Japanese ones to retrieve results from Japanese civil law texts and English civil law texts.