Task 3

Statute retrieval and entailment for yes/no questions (SL-QA)

Our goal is to explore and evaluate legal document retrieval and entailment technologies within an integrated framework. Since reliable legal entailment systems are expected to provide explanations supported by a set of relevant articles, this task investigates the performance of systems that determine entailment for unseen queries using a fixed collection of civil law articles.

Specifically, the task requires systems to return an entailment decision together with the relevant articles from the collection for each query. An article is considered relevant to a query if the query can be answered with Yes or No and the answer is entailed by the meaning of the article.

Japanese civil law articles will be provided in Japanese only;however, participants may use machine translation systems as part of their approach. The training data consist of sets of queries, relevant articles, and entailment labels. The entire process of executing queries over the articles and generating experimental runs must be fully automatic.

The test data will include queries only, and participants are required to submit entailment results together with the corresponding sets of relevant articles.

No human intervention is allowed at any stage, including modifications to the retrieval system based on inspection of the queries. Participants must not materially modify their retrieval systems between the time the queries are downloaded and the time the runs are submitted.