Submission Guidelines

Paper Submissions Format

Participants are required to submit a paper on their method and experimental results.

TBA

Results Submission Format

All the run results should be submitted following the instructions specified for each task in the sections below.

In addition, run results should be submitted with a text file briefly explaining the system used in the run. The explanation should comply with the following format. In the description, please specify the task ID and run tag. "Run tag" is defined in the following sections (Tasks 1, 2, 3, 4, Pilot).

Task: [1,2,3,4,pilot]
# Choose one task your submitting

Run tag: [Your run tag here]
  
- Machine learning models:
# If you use any machine learning models such as pretrained LLM,
# please provide name(s) and URL(s) for the model. For the Task 3
# and Task 4 participants, please provide the model update date that
# can be identfied from URL(s). It should be before July 15, 2025 (JST).
  
- External resources:
# If you use any external resources such as Wikipedia, legal
# documents, please provide name(s) and URL(s) for the resources.
  
- Explaination:
# Please provide brief explanation of the system that will be
# considered for making the explanation of your submission information
# in the overview paper. (1-3 lines)


# If you have more than one run, repeat the above as many times as necessary.

Submit runs (tasks 1-4) and system descriptions (tasks 1-4, and pilot) to: coliee_participation@nii.ac.jp

Upon submission, the subject of the mail should be like "[submision] YOUR_GROUP_ID".

Task 1

For Task 1, a submission should consist of a single ASCII text file. Use a single space to separate columns, with three columns per line as follows:

000001 000018 univABC  
000001 000045 univABC  
000001 000130 univABC  
000002 000433 univABC  
...

where:

1. The first column is the query file name.
2. The second column is the official case number of the retrieved case.
3. The third column is called the "run tag" and should be a unique identifier for the submitting group, i.e., each run should have a different tag that identifies the group. Please restrict run tags to 12 or fewer letters and numbers, with no punctuation.

At most three runs for each group should be assessed.

Task 2

For Task 2, a submission should consist of a single ASCII text file. Use a single space to separate columns, with three columns per line as follows:

001 013 univABC  
002 037 univABC  
002 002 univABC  
003 008 univABC  
...  

Where:

1. The first column is the query id.
2. The second column is the paragraph number which entails the decision.
3. The third column is called the "run tag" and should be a unique identifier for the submitting group, i.e., each run should have a different tag that identifies the group. Please restrict run tags to 12 or fewer letters and numbers, with no punctuation.

At most three runs for each group should be assessed.

Task 3

We need two submission files for one submission. One is retrieval results and the other is entailment results.

Retrieval result

The retrieval results submission format is the TREC eval format used in trec_eval program. Use a single space to separate columns, with six columns per line as follows:

H21-5-3 Q0 213 1 0.8 univABC  

Where:

1. The first column is the query id.
2. The second column is "iter" for trec_eval and not used in the evaluation. Information of the column will be ignored. But please write Q0 in this column.
3. The third column is the official article number of the retrieved article.
4. The fourth column is the rank of the retrieved articles.
5. The fifth column is the similarity value (float value) of the retrieved articles.
6. The sixth column is called the "run tag" and should be a unique identifier for the submitting group, i.e., each run should have a different tag that identifies the group. Please restrict run tags to 12 or fewer letters and numbers, with no punctuation.

Entailment result

The entailment results should consist of a single ASCII text file. Use a single space to separate columns, with three columns per line as follows:

H18-1-2 Y univABC 
H18-5-A N univABC 
H19-19-I Y univABC 
H21-5-3 N univABC 
... 

where:

1. The first column is the query id.
2. "Y" or "N" indicating whether the Y/N question was confirmed to be true ("Y") by the relevant articles, or confirmed to be false ("N").
3. The third column is called the "run tag" and should be a unique identifier for the submitting group, i.e., each run should have a different tag that identifies the group. Please restrict run tags to 12 or fewer letters and numbers, with no punctuation.

At most three runs for each group should be assessed.

Task 4

Submission format of Task 4 is same as entailment results of Task 3.

H18-1-2 Y univABC  
H18-5-A N univABC  
H19-19-I Y univABC  
H21-5-3 N univABC  
...  

At most three runs for each group should be assessed.

In order to facilitate comparison with systems from previous competitions, participants are also required to submit answers and evaluation results under the formal run settings of the past three years. Specifically, each of the past three years’ datasets (H30, R01, and R02) should be used as test datasets, while datasets from earlier years (–H29, –H30, and –R01) should be used as training datasets.

In your submission, please add the dataset name as a prefix to the original file name:

R06.task4.YOURID Final submission for TestData_{jp,en}.xml

R02.task4.YOURID for riteval_R02_{jp,en}.xml
R01.task4.YOURID for riteval_R01_{jp,en}.xml
H30.task4.YOURID for riteval_H30_{jp,en}.xml

Pilot Task (LJPJT26)

To submit your results, you have to implement a solver (opens in a new tab), which will upload your results to the leaderboard and return metrics. Please read the readme file (EN (opens in a new tab)/JA (opens in a new tab)) of this template repository (opens in a new tab) for the detailed steps.

If you register to join the pilot task and submit your pilot dataset memorandum, we will send you an API key required to use the leaderboard on 15 Feb 2026 via email. Further details will be announced by 15 Feb 2026.