Submission

Submission Guidelines

Participants are required to submit a paper on their method and experimental results. At least one of the authors of an accepted paper has to present the paper at the COLIEE session of JURISIN 2024. Papers should conform to the standards set out at the JURISIN 2024 webpage:

JURISIN 2024 (opens in a new tab)
Submission page at JURISIN 2024 (opens in a new tab)

We will publish COLIEE papers in one of two categories according to the quality of the papers:

  • Proceedings in LNAI series by Springer Verlag which will (electronically) be distributed at the COLIEE workshop
  • Local Proceedings published by IsAI-JSAI (parent symposium of JURISIN 2024) for papers which are not qualified for LNAI publication, but are selected to be presented at the COLIEE workshop.

Participants should clearly mention what dataset was used (for example: pretrained by Wikipedia dump data as of 2022xxxx, fine-tuned by...) for reproducibility purposes. Participants can use any external data, but it is assumed that they do not use the test dataset and/or something which could directly contain the correct answers of the test dataset.

Results Submission Format

Task 1

For Task 1, a submission should consist of a single ASCII text file. Use a single space to separate columns, with three columns per line as follows:

000001 000018 univABC  
000001 000045 univABC  
000001 000130 univABC  
000002 000433 univABC  
...

where:

  1. The first column is the query file name.
  2. The second column is the official case number of the retrieved case.
  3. The third column is called the "run tag" and should be a unique identifier for the submitting group, i.e., each run should have a different tag that identifies the group. Please restrict run tags to 12 or fewer letters and numbers, with no punctuation.

Task 2

For Task 2, a submission should consist of a single ASCII text file. Use a single space to separate columns, with three columns per line as follows:

001 013 univABC  
002 037 univABC  
002 002 univABC  
003 008 univABC  
...

where:

  1. The first column is the query id.
  2. The second column is the paragraph number which entails the decision.
  3. The third column is called the "run tag" and should be a unique identifier for the submitting group, i.e., each run should have a different tag that identifies the group. Please restrict run tags to 12 or fewer letters and numbers, with no punctuation.

Task 3

Submission format in Task 3 is the TREC eval format used in trec_eval program. Use a single space to separate columns, with six columns per line as follows:

H21-5-3 Q0 213 1 0.8 univABC

Where

  1. The first column is the query id.
  2. The second column is "iter" for trec_eval and not used in the evaluation. Information of the column will be ignored. But please write Q0 in this column.
  3. The third column is the official article number of the retrieved article.
  4. The fourth column is the rank of the retrieved articles.
  5. The fifth column is the similarity value (float value) of the retrieved articles.
  6. The sixth column is called the "run tag" and should be a unique identifier for the submitting group, i.e., each run should have a different tag that identifies the group. Please restrict run tags to 12 or fewer letters and numbers, with no punctuation.

Task 4

For Task 4, again a submission should consist of a single ASCII text file. Use a single space to separate columns, with three columns per line as follows:

H18-1-2 Y univABC  
H18-5-A N univABC  
H19-19-I Y univABC  
H21-5-3 N univABC  
...

where:

  1. The first column is the query id.
  2. "Y" or "N" indicating whether the Y/N question was confirmed to be true ("Y") by the relevant articles, or confirmed to be false ("N").
  3. The third column is called the "run tag" and should be a unique identifier for the submitting group, i.e., each run should have a different tag that identifies the group. Please restrict run tags to 12 or fewer letters and numbers, with no punctuation.

At most three runs for each group should be assessed.

EvaluationResults